Amanda Wee
01/26/2021, 10:01 PMdask_cloudprovider.aws.FargateCluster
plus adaptive scaling) with the local agent. My thinking is that I could separately build a Docker image that has all the required flow dependencies (they're the same for all the flows) and upload that to ECR during CI/CD, and then the local agent can pull from ECR and initialise the Dask executor with this image, while also initialising it with the flows retrieved from S3 storage. Is this feasible?Jim Crist-Harif
01/26/2021, 10:06 PMThe catch is that because of our ECS setup, this may mean some kind of Docker-in-Docker solution, which seems to be a mess.I don't follow this. Are your tasks using docker as a service (perhaps building images?). If your flow doesn't need docker to run (you only need to run an image), you shouldn't need a Docker-in-Docker solution afaict.
My thinking is that I could separately build a Docker image that has all the required flow dependencies (they're the same for all the flows) and upload that to ECR during CI/CD, and then the local agent can pull from ECR and initialise the Dask executor with this image, while also initialising it with the flows retrieved from S3 storage. Is this feasible?Yep, that should be fine. When running with dask, the workers need all dependent libraries you use, but the source of the flow itself can live elsewhere. So provided your image has all the required dependencies, the workers themselves wouldn't need access to the flow source stored in S3.
Amanda Wee
01/26/2021, 10:18 PMI don't follow this. Are your tasks using docker as a service (perhaps building images?). If your flow doesn't need docker to run (you only need to run an image), you shouldn't need a Docker-in-Docker solution afaict.When the ECS task starts, it re-runs flow registration with the up-to-date flows, and in this process I understand that the flow will be serialised to storage. So, if I use Docker storage, wouldn't the ECS task need its own docker setup so that the flow's image can be built (and pushed to ECR)? Or can I somehow continue to use S3 storage with ECS Agent?
Jim Crist-Harif
01/26/2021, 10:24 PMOr can I somehow continue to use S3 storage with ECS Agent?All storage types should work with all agents (with the exception of
Local
storage, which only works with the local agent). S3 storage with an ECS agent is a common pattern, given both are hosted on AWS.
When the ECS task starts, it re-runs flow registration with the up-to-date flows, and in this process I understand that the flow will be serialised to storage. So, if I use Docker storage, wouldn't the ECS task need its own docker setup so that the flow's image can be built (and pushed to ECR)?Oh wait, is this an ECS task that you wrote, not an ECS task started by prefect for running a single flow run? During a flow run docker shouldn't be needed, it should only be needed during registration. So if you register your flows elsewhere (where docker is available), no docker setup should be needed during runtime.
Amanda Wee
01/26/2021, 10:31 PMOh wait, is this an ECS task that you wrote, not an ECS task started by prefect for running a single flow run?Yes. As I mentioned, my wider team's requirements is for all our services to be deployed as ECS services, hence I'm running Prefect Server, the web UI, and the Prefect agent/executor as three different ECS services, thus as the Docker entrypoint for the ECS service with Prefect agent I have a shell script that runs
prefect create project
for each project that needs to exist, runs Python scripts that declare the flows that call register_flow()
for each of them, and finally the script does prefect agent local start
.Jim Crist-Harif
01/26/2021, 10:32 PMAmanda Wee
01/26/2021, 10:37 PMLocalExecutor
, and then within the ephemeral Fargate task the executor will download the flow from S3 to run it?Jim Crist-Harif
01/26/2021, 10:37 PMAmanda Wee
01/26/2021, 10:38 PMIn order to use this agent with Prefect Server the server's GraphQL API endpoint must be accessible. This may require changes to your Prefect Server deployment and/or configuring the Prefect API address on the agent.Is this true, i.e., does it mean that in fact the flows/agents must be able to directly access the graphql endpoint, contrary to the diagram?
Jim Crist-Harif
01/26/2021, 10:46 PMlocalhost:4200
on the user's computer won't be reachable from the ECS task (since it's a different machine), which is what that comment was warning about. Sounds like you've deployed Prefect server in a different way so you're probably fine.Amanda Wee
01/26/2021, 10:48 PMJim Crist-Harif
01/26/2021, 10:49 PMAmanda Wee
01/26/2021, 11:19 PMaws_account_<http://id.dkr.ecr.region.amazonaws.com/flow-base:latest|id.dkr.ecr.region.amazonaws.com/flow-base:latest>
, but would it say, suffice to have AWS_ACCESS_KEY_ID
and AWS_SECRET_ACCESS_KEY
environment variables available, or must I do something more/else?Jim Crist-Harif
01/26/2021, 11:23 PMECSRun
run config (via execution_role_arn
), or on the agent itself (via (--execution-role-arn
). Before this had to be done through use of a custom task definition template.Amanda Wee
01/26/2021, 11:25 PMAll storage types should work with all agents (with the exception ofAnother possible docs improvement, I think. This: https://docs.prefect.io/orchestration/flow_config/storage.html#docker makes the assertion that:storage, which only works with the local agent).Local
This method of Storage has deployment compatability with the Docker Agent, Kubernetes Agent, and Fargate Agent.which to me gives the impression that Docker storage is required for those three agent types, whereas it seems more the case that Docker storage is not required, but makes it easier to setup those three agent types because the Docker image to supply is already available (e.g., via
prefect.context.image
)Sagun Garg
01/27/2021, 2:30 PMAmanda Wee
02/02/2021, 1:58 AMJim Crist-Harif
02/02/2021, 3:19 PMOh, and another question to confirm: when the flow has finished, does the ECS agent automatically stop the Fargate task?When a flow finishes, the main process will complete and the container will be cleaned up by AWS. No actions required by Prefect.
My only thing here is that how can one auto-start an ECS agent based on flows submitted and then after a timeout offset remove the ECS agent if no more flows are present...something more like a serverless setup but for fargate in ECS. Current I have to run the ECS Agent via my local machine using the CLI.A serverless model would require prefect pushes out requests to run flows - right now flows are run via agents polling prefect cloud, meaning that all traffic from inside your infrastructure is outbound and you don't need to expose any ports/auth (and all the security implications that would entail). This is part of our "hybrid model" (https://medium.com/the-prefect-blog/the-prefect-hybrid-model-1b70c7fd296) and core to how Prefect currently works. We've thought about ways to loosen this restriction, but for now an agent must be running somewhere to pick up your flow runs. Note that the agent process can be very small, since it's just IO bound and low memory - an ECS service with 0.25 cores and 512 MiB may suffice.