i’m trying to get prefect server running on an ec2 following this tutorial: https://towardsdatascience.com/deploying-prefect-server-with-aws-ecs-fargate-and-docker-storage-36f633226c5f but I can’t seem to get it working. I’ve got server installed, the ecs agent running but when I try to run a task I get this error:

Failed to load and execute Flow's environment: ModuleNotFoundError("No module named '/home/ec2-user/'")

Hey @saml, this is because there is a concept called Storage. When you register, the Flow saves the flow in storage, and then it gets retrieved during execution. When you registered this, it used the default and put the Flow in your

/home/ec2-user/

. The ECS agent will look for that path but not find it. You need to use a Storage like Git or S3 that your ECS can find and pull the Flow from.

Thank you, I couldn’t tell which layer the error was coming from

Exactly. The same thing will happen if you register with the default local storage on one computer and try to run it with an agent on another computer (which is basically what ECS does)

ok, my mental model was flawed then. Based on what you’re saying, I think I now understand that something like the following is happening: • create a flow definition that could live anywhere • register that flow - which I think means prefect basically processes the flow itself and then decides how to call it. • As part of the registration, the agent saves it’s own version of the flow to storage that the final executor will also need to be able to access. • during execution it’s that last flow definition that gets referenced

Yes exactly except for Git where you have to upload on your own as well to your Git repo

interesting, so what happens if there is a conflict between the two? or do you mean that I need to provide a git login to my agent so that it can push it’s version

If there is a difference between the two, you can get away with it as long as the DAG structure and metadata is the same. Otherwise, Prefect will throw an error during execution. No to the agent log-in. I don’t think Git storage pushes to the repo for you. S3 storage can push to the bucket for you though

Thanks for you help, that clarifies it a lot