Greg Desmarais
07/20/2020, 3:40 PMCompleted flow run submission
. No Fargate cluster is ever created. What needs to happen on the agent (or prefect server?) to see that the next steps are takan (e.g. cluster created, work started)?executor = DaskExecutor(cluster_class='dask_cloudprovider.FargateCluster',
cluster_kwargs=cluster_kwargs)
flow.environment = FargateTaskEnvironment(
executor=executor,
region_name=DEFAULT_REGION,
**task_definition_kwargs
)
# flow.environment = LocalEnvironment(executor=executor, labels=labels)
flow_id = flow.register(labels=labels)
Kyle Moon-Wright
07/20/2020, 3:53 PM--verbose
should give you some info as the agent starts up, such as what labels it's running with and which endpoint it's querying.Greg Desmarais
07/20/2020, 3:58 PMKyle Moon-Wright
07/20/2020, 4:18 PMGreg Desmarais
07/20/2020, 4:22 PMcluster_name_template
. I'm finding two things very frustrating:
1. the serialization process when pushing a flow to the prefect server strips out a lot of parameters that are set on the environment/executor - seemingly arbitrarily.
2. the process of running the flow filters provided kwargs very aggressively, for example ignoring container definition arguments like memory or image nameChris White
07/20/2020, 4:38 PMGreg Desmarais
07/20/2020, 4:40 PMChris White
07/20/2020, 4:40 PMGreg Desmarais
07/20/2020, 4:41 PMChris White
07/20/2020, 4:46 PMGreg Desmarais
07/20/2020, 4:47 PMChris White
07/20/2020, 4:47 PMGreg Desmarais
07/20/2020, 4:48 PMChris White
07/20/2020, 4:51 PMGreg Desmarais
07/20/2020, 4:55 PMChris White
07/20/2020, 4:59 PMGreg Desmarais
07/20/2020, 5:01 PM