Hey guys, 2 questions for today… • Do you know why I would get an output error with the following format? Please note the dodgy URL printed does not contain http, which is most likely the cause for failure but I’m not sure what it’s trying to find.. “Invalid URL ‘1.2.3.4/graphql/alpha’” • Am I right in assuming that if I want to to run a very simple flow built into Local Storage and with LocalEnvironment, I register it with my server running on amazon, the local executor agent should just pick it up? Or am I missing a magic step..?

Hi @Darragh! Could you give a bit more detail about where you're seeing the error about the invalid URL? As for running a simple local flow, you're correct. You should be able to register your flow with

flow.register()

, run a local agent, and have it pick it up. If you're seeing issues with that, the first thing I'd check is whether the labels on your flow match the labels for the agent you're running.

Labels could be it alright, I haven’t done/changed/added any. Are there recommended sets, or guidelines?

When you register your flow, it's auto-labeled with information about the computer from which it was registered. If you spun up your agent on the same computer, those labels should match, but 9 times out of 10 scheduled runs not being picked up by the local agent ends up being a label mismatch. 🙂

That’ll be the problem then - local machine calls registers it to remote machine

They'll be specific to the machine they were registered on. You should be able to navigate to the flow in your UI and see them no problem. Here's an example:

Good point, thanks! The docker ones are building and associating fine, but at runtime I get the URL error above. There’s some env variable gone bass ackwards I think

try setting the value of the graphql_url in your prefect config to the IP of your amazon server

Yeah I have that set in an env variable : PREFECT__SERVER__UI__GRAPHQL_URL=“http://${PUBLIC_IP}:4200/graphql

I don't think so, this is what my config looks like

[server]
  
    [server.ui]
        graphql_url = "http://<SERVER_IP>:4200/graphql"

[environments]

    [environments.local]
        services_host = <SERVER_IP>

this is for prefect core btw

Yeah that’s what I’m using. I don’t have the environments.local piece set, will give that a go too!

I could definitely be missing something, but it looks like you're missing the

DeepThought.local

tag on your agent.

ahhhhhh do I need ALL of the tags?

Agent labels need to be a superset of the flow's labels. This doc may be useful, in case you haven't seen it. https://docs.prefect.io/orchestration/execution/overview.html#environments

Yeah I saw that, missed the superset part 🙂 It doesn’t mention anything about removing an auto generated label, is that a thing? the machine regsitering this flows will be a temp container in my CI/CD flow, so I don’t want it added as a label

It doesn't look like we have a way to opt out of the auto-generated labels at the moment, but I've opened an issue for that! https://github.com/PrefectHQ/prefect/issues/2653

Yeah the docker storage in general is working well, but it has me beating my head against a separate problem 😂 Think I mentioned it previously…

you could just call docker login in your CI/CD scripts, i do that in mine

Yeah i do that in the CI/C D scripts and it works fine, It’s on the far side I hit the issue - the EC2 instance I register the flows to, and run the docker agent on, will get a timeout after 12 hours from ECR, so if I have a flow scheduled to run more than 12 hours after I register then I’m in trouble 🙂

won't the image the flow needs already be pulled down locally though? so no need to hit ECR again

That one might (at register time), but any subsequent ones won’t, is what my assumption is. I could be thinking about this wrong…. Let me write this out, make sure I have it right

I guess it depends on how you configure your docker storage, but you could set the local_image to True on the docker storage - https://docs.prefect.io/api/latest/environments/storage.html#docker

Pipeline is pushing the images out to AWS-ECR from GitLab, so I don’t know how that would work…

I don't know how much control you have over your CI/CD, but you could have 1 step that pushes to AWS-ECR, and then another step that runs on the agent machine that pulls that image down locally

Hmmmmmmmmmmm yeah I can see how that would work. I’d need to figure out the magic words for logging to the ec2 box - it’s created on-demand using aws-cdk, so the IP is liable to change 🙂

that could work since you could set up your fargate cluster to be able to authenticate to your ECR using an IAM_role

And ideally it would mean I’m running the tasks on Fargate env rather than on the same EC2 instance running Core

really depends on the entire use case, you'll end up saving money if you just do it on demand though

👍 Does that hold for scheduled flows as well? If I have a flow, or groups of flows, I need to run on say a weekly schedule, I’d guyess on demand is better. Flow.. (

agent = FargateAgent(...)

agent.start() flow.environment = FargateEnvironment flow.register() )

on-demand is generally going to be cheaper than having a long running instance that just sits idle, the biggest downside would be that you need to wait for your instances to spin up when running your flows, so your overall execution time will be longer

That’s fine, the jobs I’ll be running will be measured in hours anyway, so it’s not an issue.

I'm not quite sure, I'd read the docs page on the subject - https://docs.prefect.io/orchestration/execution/fargate_task_environment.html#process https://docs.prefect.io/api/latest/environments/execution.html#daskcloudproviderenvironment

The docs suggest the posaibility, but nothing explicit. Ill try it and see 😂

Most folks who use the Fargate Agent have it running on an EC2 instance or something similar and submitting jobs to Fargate. As long as Fargate's properly authed with ECR, I think that'd do the trick for you.

Thanks guys, I’ll have a crack at it and let you know how it goes. @Zachary Hughes Would you have anymore insight into my question on the on-demand agent? Is it possible to start the agent on demand within the flow?

It's a bit of a chicken and egg situation there. In order for your flow to be picked up, you'd need an agent to be running. In order for your agent to run, you'd need to have your flow picked up. So what I think you'll want to do is have your agent running constantly, but have your flows execute on Fargate. Whether the agent is running on an EC2 instance or your local machine is up to you-- it just needs proper credentials to submit things to Fargate.

Fair enough, that clears it up. Thanks guys!