Hi folks, I'm having trouble getting my remote worker VM to pick up the code for a deployment I crea...

Jacob Bedard

03/02/2023, 12:56 AM

Hi folks, I'm having trouble getting my remote worker VM to pick up the code for a deployment I created. I created a flow, then in python, did a Deployment, shown in the second screencap here. I can run this thing on an agent on my machine, but when I try to run it from an agent on my VM, it seems unable to retrieve the file from gitlab. I've got the SSH key and everything set up. I think my problem is the storage block, but I can't quite see what I'm doing wrong because there aren't many examples in the docs about this particular thing

✅ 1

Jacob Bedard

03/02/2023, 1:00 AM

I'm finding myself focusing on that "downloading flow code from storage at none" part now... But I'm still not sure what it is I've got wrong

Ryan Peden

03/02/2023, 1:03 AM

That seems like the right place to focus. I'm going to quickly check something and get right back to you so we can get to the bottom of this 🙂

Jacob Bedard

03/02/2023, 1:03 AM

Thanks!

Jacob Bedard

03/02/2023, 1:03 AM

for reference, my gitlab repo layout has a folder called "flows", and the python files containing flows just inside it

Jacob Bedard

03/02/2023, 1:04 AM

But I think my issue is one step before that, based on the error message saying "storage at None"

Jacob Bedard

03/02/2023, 1:12 AM

And I used the SSH way to point to the repo, so I got an error when I tried using both gitlab credentials and gitlab repository, which is why that part isn't there in the screencap

Ryan Peden

03/02/2023, 1:15 AM

We might need to adjust the deployment a little

Jacob Bedard

03/02/2023, 1:15 AM

I'm ready, hit me

Ryan Peden

03/02/2023, 1:22 AM

Looking in the deployment docstring I see:

Copy code

path: The path to the working directory for the workflow, relative to remote storage or, if stored on a local filesystem, an absolute path

entrypoint: The path to the entrypoint for the workflow, always relative to the `path`

Not your fault; it's in the docstring for the

Deployment

class but not

build_from_flow

, even though

build_from_flow

forwards kwargs to

Deployment

. So what I think the deployment needs is:

Copy code

...
path="/flows",
entrypoint="jacob_simplest_test_flow_ever.py:print_a_message_flow",
...

Jacob Bedard

03/02/2023, 1:23 AM

Ahhhh I need to pass a path argument

Jacob Bedard

03/02/2023, 1:24 AM

I'm gonna try that right now. I can just alter that same code block I screencapped and run it again and that will update the same deployment, right?

Jacob Bedard

03/02/2023, 1:24 AM

I was previously in the habit of just re-registering flows on v1. I'm assuming I can simply update a pre-existing deployment this way, right?

Jacob Bedard

03/02/2023, 1:27 AM

Got a different error message at least 🙂

Jacob Bedard

03/02/2023, 1:27 AM

I wonder, shouldn't the message being returned be mentioning my gitlab repo or the block name?

Jacob Bedard

03/02/2023, 1:31 AM

The machine can definitely reach into that repo - I just cloned it in another dir to check

Jacob Bedard

03/02/2023, 1:31 AM

I've got something wrong with the block itself, maybe?

Ryan Peden

03/02/2023, 1:32 AM

It could be - I'm doing to dig through the code a bit

Jacob Bedard

03/02/2023, 1:32 AM

This doc here is set up pretty much the same as my deployment. I was going for the most vanilla flow ever

Jacob Bedard

03/02/2023, 1:44 AM

Yeah it's pretty weird

Jacob Bedard

03/02/2023, 1:44 AM

I did however just get it to work when I tried the CLI

Ryan Peden

03/02/2023, 1:45 AM

The deployment CLI?

Jacob Bedard

03/02/2023, 1:45 AM

Ryan Peden

03/02/2023, 1:46 AM

I was just about to ask for that 🙂

Jacob Bedard

03/02/2023, 1:46 AM

And there's the message I was looking for:

Jacob Bedard

03/02/2023, 1:46 AM

Ok so it's something to do with the python deployment builder

Jacob Bedard

03/02/2023, 1:46 AM

Looks like maybe I used it correctly but there's a bug? Or if I didn't, maybe some stuff needs to be added to the docs

Ryan Peden

03/02/2023, 1:47 AM

I think it's a documentation issue - but the code and CLI you shared will help us verify one way or the other

Jacob Bedard

03/02/2023, 1:48 AM

Well, you certainly helped out. Thanks for helping me get pointed in the right direction!

Jacob Bedard

03/02/2023, 1:48 AM

I'm a very happy camper right now

Ryan Peden

03/02/2023, 1:48 AM

Your feedback and perseverance make a big difference here - better documenting things like this and making deployments easier are both very much on our radar right now

Jacob Bedard

03/02/2023, 1:48 AM

one last question - how do I have it output the deployment yaml somewhere I want it to go?

Jacob Bedard

03/02/2023, 1:49 AM

I have just 2 folders in that repo: flows, and deployments

Jacob Bedard

03/02/2023, 1:49 AM

I was planning to put the ymls in the deployments folder

Jacob Bedard

03/02/2023, 1:49 AM

I can also figure that out myself, but while I've got you, I figure I'd ask

Ryan Peden

03/02/2023, 1:52 AM

The

prefect deployment build

command has a

--output

flag that lets you specify an output path/file. You can use

-o

as a shortcut

🙏 1

Jacob Bedard

03/02/2023, 1:52 AM

Thanks again!

Ryan Peden

03/02/2023, 1:54 AM

If you run into anything else that feels like it adds friction, please feel free to share. We listen, and your feedback helps us make Prefect better 🙂

Ryan Peden

03/02/2023, 1:55 AM

And you're welcome - I'm happy the deployment is working for you!

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.