https://prefect.io logo
Channels
Powered by
t

Tom Klein

12/16/2021, 12:03 AM
is there a way to add a label to a flow through the UI?
k

Kevin Kho

12/16/2021, 12:04 AM
Try clicking the plus
t

Tom Klein

12/16/2021, 12:11 AM
Thanks, found it after i asked 😆 but now i’m facing two other problems: • flows i create using the CLI (or through python) on the EC2 machine end up getting the local IP as a label (even when i didn’t select that label), and then it tells me that no matching agents have that label… when i created the agent i just gave a single label (
poc
) and i try to give the same one to the flow i register but it also gets the host label… • even after i manually remove this label manually through the GUI, and try to run the flow - i get an error i don’t understand:
the run-config is a `UniversalRun`…
this is my entire flow:
Copy code
from prefect import task, Flow
from prefect.backend.artifacts import create_markdown_artifact
import requests


@task
def fetch():
    res = requests.get('<https://xkcd.com/454/info.0.json>')
    j = res.json()
    print(j)
    mrkdwn = "# My XKCD\n\n ![" + j['alt']+ "](" + j['img'] + ")"
    create_markdown_artifact(mrkdwn)


with Flow('xkcd') as flow:
    fetch()
flow was registered with:
Copy code
prefect register --project poc -p xkcd.py -l poc
agent was started with:
Copy code
prefect agent docker  start --key "XXX" -l poc
i’m guessing i need either a remote storage for the flow or… ?
k

Kevin Kho

12/16/2021, 12:14 AM
You can turn off the default label by passing 
add_default_label=False
to your Storage. The default is local
t

Tom Klein

12/16/2021, 12:15 AM
ok that would solve the first bullet, what about the 2nd?
k

Kevin Kho

12/16/2021, 12:16 AM
The default storage is local storage so Prefect serializes your Flow and saved it under the 
.prefect
folder by default. When the agent picks up the flow run, it goes to this place to fetch the flow and run it.
So if you register with local storage on machine A and run a flow on machine B that doesn’t have the file, you will get an error like this. If you need a Flow across machines, you need a remote storage
t

Tom Klein

12/16/2021, 12:17 AM
it is in fact the same machine (same EC2), but the agent is docker
k

Kevin Kho

12/16/2021, 12:18 AM
Ah in this case, it’s being looked for inside the container, but it’s on your local.
This will work if the same file exists in the container at that specific location
t

Tom Klein

12/16/2021, 12:21 AM
so… how would i generally go about pushing flows into the agent docker?
(i’m asking more for sport/poc, obviously we’ll pick something more viable like s3 or github or whatever in production)
ah nvm i found it
k

Kevin Kho

12/16/2021, 12:23 AM
That will work, but you can also make your own container and copy files is (if you have a lot of custom modules also). You can also use Docker storage and DockerRun to package that flow in a container
t

Tom Klein

12/16/2021, 12:23 AM
isn’t 
DockerRun
the default for docker agents?
k

Kevin Kho

12/16/2021, 12:24 AM
No UniversalRun is the default which can be picked up by all agents but not really configured
t

Tom Klein

12/16/2021, 12:24 AM
ok, i meant - in practice, what will the 
DockerAgent
do when it gets a 
UniversalRun
flow to run?
wouldn’t it run it in a 
Docker
?
k

Kevin Kho

12/16/2021, 12:25 AM
It will use the base prefecthq/prefect image yep
t

Tom Klein

12/16/2021, 12:26 AM
my problem with the 
Docker Storage
is that it expects a registry (so it becomes no better than using S3 which would now involve permissions etc. which im trying to avoid for the time being) Whoops nvm i see that’s optional
can 
Docker Agents
run flows who are configured with the 
Module Storage
? not sure what the module storage means (and it didn’t work when i tried, anyway) i guess it needs to be available in the docker image
k

Kevin Kho

12/16/2021, 12:31 AM
Yes Docker Storage doesn’t need a registry. It can just be on local. Agent and RunConfigs are pairs but ALL Storage + RunConfig combinations can be used. Some just make less sense but they are all compatible. Yes the Module needs to be installed in the image.
t

Tom Klein

12/16/2021, 12:33 AM
wooo
k

Kevin Kho

12/16/2021, 12:33 AM
Nice!
t

Tom Klein

12/16/2021, 12:34 AM
ok, one thing i’m still not clear on: if i configure my flow with a docker storage, does that mean it gets its own image (built locally, in this case)?
if i had a 2nd flow would that be a 2nd image?
k

Kevin Kho

12/16/2021, 12:35 AM
Yes that’s right but if you want multiple flows in 1 image, you can follow this
t

Tom Klein

12/16/2021, 12:36 AM
ahhh i see, ya i remember running into this article before 😆 your docs are very exhaustive but there’s just so much of them
k

Kevin Kho

12/16/2021, 12:38 AM
funny you say that. more commonly we hear not enough stuff is documented 😛
t

Tom Klein

12/16/2021, 12:46 AM
dunno i think i found every answer documented so far, i just sometimes struggle to get there myself through navigation or search (also, some questions would kind of require you to know where to look) i would consider having some FAQ format page in addition - plus maybe some kind of question-bot-thingie that can find you articles based on a human-readable question….. otherwise - you become the bot yourself 😆
k

Kevin Kho

12/16/2021, 12:48 AM
I am working on our FAQ. I hope to get something out in early Jan
💯 1
t

Tom Klein

12/16/2021, 12:49 AM
the multi-flow thing worked btw
k

Kevin Kho

12/16/2021, 12:50 AM
nice!
t

Tom Klein

12/16/2021, 1:02 AM
thx for the help, much appreciated 🙏
2 Views
Powered by