Hi all, I'm hitting an issue where manually runnin...
# prefect-communityt
Tom Augspurger
09/14/2020, 8:02 PMHi all, I'm hitting an issue where manually running a flow is timing out before it starts. I'm using the prefect-cloud backend.
Copy code
$ prefect run flow --name=terraclimate --project=pangeo-forge
File "/Users/taugspurger/miniconda3/envs/pangeo-forge-37/bin/prefect", line 8, in <module>
sys.exit(cli())
File "/Users/taugspurger/miniconda3/envs/pangeo-forge-37/lib/python3.7/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/Users/taugspurger/miniconda3/envs/pangeo-forge-37/lib/python3.7/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/Users/taugspurger/miniconda3/envs/pangeo-forge-37/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/taugspurger/miniconda3/envs/pangeo-forge-37/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/taugspurger/miniconda3/envs/pangeo-forge-37/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/taugspurger/miniconda3/envs/pangeo-forge-37/lib/python3.7/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/Users/taugspurger/miniconda3/envs/pangeo-forge-37/lib/python3.7/site-packages/prefect/cli/run.py", line 125, in flow
no_url=no_url,
File "/Users/taugspurger/miniconda3/envs/pangeo-forge-37/lib/python3.7/site-packages/prefect/cli/run.py", line 377, in _run_flow
flow_id=flow_id, parameters={**file_params, **string_params}, run_name=run_name
File "/Users/taugspurger/miniconda3/envs/pangeo-forge-37/lib/python3.7/site-packages/prefect/client/client.py", line 964, in create_flow_run
res = self.graphql(create_mutation, variables=dict(input=inputs))
File "/Users/taugspurger/miniconda3/envs/pangeo-forge-37/lib/python3.7/site-packages/prefect/client/client.py", line 294, in graphql
raise ClientError(result["errors"])
prefect.utilities.exceptions.ClientError: [{'path': ['create_flow_run'], 'message': 'Operation timed out', 'extensions': {'code': 'API_ERROR'}}]👀 2
k
Kyle Moon-Wright
09/14/2020, 8:15 PMHey @Tom Augspurger,
I was just able to kick off a flow run from the cli like you have above, so it's definitely strange you're encountering this error. Just to confirm, you've set your backend with
prefect backend cloudterraclimatepangeo-forget
Tom Augspurger
09/14/2020, 8:17 PMThanks. Just re-ran
prefect backend cloud Copy code
terraclimate 13 30 minutes ago 73711833-eeca-4273-8cd3-37d787c7c75f pangeo-forgeTom Augspurger
09/14/2020, 8:17 PMI also see the "Operation timed out" error when running it through the UI
k
Kyle Moon-Wright
09/14/2020, 8:18 PMInteresting, let me get some insight for you.
Kyle Moon-Wright
09/14/2020, 8:34 PMDiving deeper: Is this still happening deterministically? Are there many flows in this project?
t
Tom Augspurger
09/14/2020, 8:39 PMIt's still happening. Just one flow with 10-20 tasks in it. The source is at https://github.com/TomAugspurger/terraclimate-feedstock/blob/debug/recipe/pipeline.py
k
Kyle Moon-Wright
09/14/2020, 8:41 PMGreat thanks, still gathering insight on this.
👍 1
Kyle Moon-Wright
09/14/2020, 9:23 PMHey there @Tom Augspurger, thanks for your patience. After looking into your flow from our end, it looks like there are a large number (thousands) of statically defined tasks that are causing a slow down when creating all the new task run states. Of course, this is not ideal and is something we definitely want to fix, however in the meantime - it may help if you could replace those static tasks with some dynamic ones with Prefect mapping.
t
Tom Augspurger
09/14/2020, 9:24 PMGreat, thanks. I'll give that a shot.
👍 1
Tom Augspurger
09/15/2020, 1:56 AMThanks Kyle, that seemed to work perfectly! https://github.com/pangeo-forge/terraclimate-feedstock/pull/4
🚀 1
k
Kyle Moon-Wright
09/15/2020, 2:03 AMNice job!