Hi folks! I'm trying to determine best practices/idioms for deploying and executing code shared between Flows without having everything in a single file. I'm using ECS for running my flows, and everything works great with S3/CloudPickle serialization as long as the code is dead simple (i.e. hello-world), but everything breaks down if I reference another file in my code. I have yet to find an example that utilizes more than one file to define a Flow, but surely there must be a way to structure code more sanely for large, complicated Flows. The various

Storage

classes seems to only be able to reference a single file, not entire modules/directories. Are there any examples of a somewhat complicated suite of flows with code shared between them?

Hi @Dane Gardner when you have multiple pieces of your flow in separate files then those files need to be available when your flow goes to execute. There are a couple ways you could do this depending on how you are packaging/deploying your flow. If you are running using the Local agent you can set a LocalRun on your flow with a working directory. If that working directory has all of your necessary imports on it then it will be able to import the multiple files. If you are using some containerized execution platform we recommend you package your dependencies inside of a flow’s Docker storage. This storage option lets you add files to the image at build time that can be available when your flow runs. If you don’t use docker storage and are still using containers to execute your flow then you should build a base image that contains your dependencies. Many of the container-based runconfigs allow for setting an image to run your flow in. I’m sure users in the community would have more concrete examples of how they set up their flow projects/repositories and I encourage them to jump in 🙂

Ahh, I missed that Docker allowed more than just each explicit file specified

Anything set on the RunConfig will override the defaults set on the agent. This is by design. You can set defaults on the agent but setting it on the run config will always allow you to override those defaults

@Dane Gardner I've been running into this problem the last couple days as well. I'm very happy I came across this post because it wasn't obvious to me what the issue was given the error message. I ended up slightly modifying the default task definition to

networkMode: awsvpc
cpu: 1024
memory: 2048
containerDefinitions:
  - name: flow
taskRoleArn: arn:aws:iam::12345678910:role/prefect-ecs
executionRoleArn: arn:aws:iam::12345678910:role/ecsTaskExecutionRole

and starting the agent with

agent ecs start --cluster <cluster-arn> --token <token> --label uat  --task-definition <path/to/file.yaml>

and it worked great without having to set anything on the run config of each flow.

@Carter Kwon @josh Please if you can help me on this further I am getting the error : ERROR - agent | Failed to infer default networkConfiguration, please explicitly configure using

--run-task-kwargs

ValueError: Failed to infer default networkConfiguration, please explicitly configure using

--run-task-kwargs