Darragh
06/19/2020, 6:21 PM0.12.0
release:
• Has something changed with the FlowRunTask? We were using without any issues up until we upgraded to 0.12.0 , but now our flow fails looking for a project_name. I can see in the code this is specific to backed being cloud, but we set ‘prefect backend server’ before starting the server and haven’t had issues with it before..
• I had asked about a week ago about DFE for mapped tasks on Fargate, and was told it would be in 0.12.0, but doesn’t appear to be working for us - mapped tasks are still running sequentially…nicholas
06/19/2020, 6:29 PMFlowRunTask
, can you provide some more details/a min reproducible example?
• I think there was a misunderstanding there - DFE is available with the 0.12.0
release as part of the DaskExecutor
, switching to that will give you the parallelism you're looking forDarragh
06/19/2020, 6:42 PMnicholas
06/19/2020, 7:00 PMDarragh
06/19/2020, 7:01 PM@task(trigger=triggers.all_successful, max_retries=3, retry_delay=timedelta(seconds=60), log_stdout=True, result=LocalResult())
def run_collector(partition, bucket_name, collector_name):
logger = prefect.context.get("logger")
<http://logger.info|logger.info>("Run Collector SubFlows")
FlowRun = FlowRunTask()
sub_flow_id = FlowRun.run(flow_name=collector_name,
parameters={"bucket_name": bucket_name, "input_location": partition["input"],
"output_location": partition["output"]})
if not sub_flow_id:
raise Exception("Flow run [%s] failed to start for input: %s", sub_flow_id, partition["input"])
else:
<http://logger.info|logger.info>("Started Collector Flow ID: %s", sub_flow_id)
time.sleep(partition["id"] * 2)
return sub_flow_id
And this fails with
File “/usr/local/lib/python3.6/site-packages/prefect/tasks/prefect/flow_run.py”, line 71, in run
raise ValueError(“Must provide a project name.“)
ValueError: Must provide a project name.Chris White
06/19/2020, 7:42 PMDaskExecutor
without configuration is running on the dask distributed scheduler but using multiprocessing, so no cluster required-e PREFECT__BACKEND=server
Darragh
06/19/2020, 7:54 PMprefect backend server
or setting backend = "server"
in the config.toml? Because I have both of those , but I'll test the env var nowChris White
06/19/2020, 7:55 PMDarragh
06/19/2020, 9:05 PMexport PREFECT__BACKEND=server
started my agent using:
agent = FargateAgent(ALL_THE_CONFIG)
agent.start()
But still have the same error : Must provide a project name
If I add a project name to the params I then get a different error:
400 Client Error: Bad Request for url: <http://X.X.X.X:4200/graphql/graphql/alpha>
Chris White
06/19/2020, 9:11 PMagent = FargateAgent(ALL_THE_CONFIG, env_vars={"PREFECT__BACKEND": "server"})
should do the trickDarragh
06/20/2020, 9:54 AM