Hi all, I'm hitting an issue where manually runnin...
# prefect-community
Hi all, I'm hitting an issue where manually running a flow is timing out before it starts. I'm using the prefect-cloud backend.
Copy code
$ prefect run flow --name=terraclimate --project=pangeo-forge
  File "/Users/taugspurger/miniconda3/envs/pangeo-forge-37/bin/prefect", line 8, in <module>
  File "/Users/taugspurger/miniconda3/envs/pangeo-forge-37/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/taugspurger/miniconda3/envs/pangeo-forge-37/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/taugspurger/miniconda3/envs/pangeo-forge-37/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/taugspurger/miniconda3/envs/pangeo-forge-37/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/taugspurger/miniconda3/envs/pangeo-forge-37/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/taugspurger/miniconda3/envs/pangeo-forge-37/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/taugspurger/miniconda3/envs/pangeo-forge-37/lib/python3.7/site-packages/prefect/cli/run.py", line 125, in flow
  File "/Users/taugspurger/miniconda3/envs/pangeo-forge-37/lib/python3.7/site-packages/prefect/cli/run.py", line 377, in _run_flow
    flow_id=flow_id, parameters={**file_params, **string_params}, run_name=run_name
  File "/Users/taugspurger/miniconda3/envs/pangeo-forge-37/lib/python3.7/site-packages/prefect/client/client.py", line 964, in create_flow_run
    res = self.graphql(create_mutation, variables=dict(input=inputs))
  File "/Users/taugspurger/miniconda3/envs/pangeo-forge-37/lib/python3.7/site-packages/prefect/client/client.py", line 294, in graphql
    raise ClientError(result["errors"])
prefect.utilities.exceptions.ClientError: [{'path': ['create_flow_run'], 'message': 'Operation timed out', 'extensions': {'code': 'API_ERROR'}}]
https://prefect-community.slack.com/archives/CL09KU1K7/p1596043664390000 looks similar, and seemed to resolve itself. Any debugging suggestions?
👀 2
Hey @Tom Augspurger, I was just able to kick off a flow run from the cli like you have above, so it's definitely strange you're encountering this error. Just to confirm, you've set your backend with
prefect backend cloud
, your flow is named
, and its registered in the project
? Just want some initial information to start to debug.
Thanks. Just re-ran
prefect backend cloud
to confirm that. `prefect get flows gives
Copy code
terraclimate  13         30 minutes ago  73711833-eeca-4273-8cd3-37d787c7c75f  pangeo-forge
I also see the "Operation timed out" error when running it through the UI
Interesting, let me get some insight for you.
Diving deeper: Is this still happening deterministically? Are there many flows in this project?
It's still happening. Just one flow with 10-20 tasks in it. The source is at https://github.com/TomAugspurger/terraclimate-feedstock/blob/debug/recipe/pipeline.py
Great thanks, still gathering insight on this.
👍 1
Hey there @Tom Augspurger, thanks for your patience. After looking into your flow from our end, it looks like there are a large number (thousands) of statically defined tasks that are causing a slow down when creating all the new task run states. Of course, this is not ideal and is something we definitely want to fix, however in the meantime - it may help if you could replace those static tasks with some dynamic ones with Prefect mapping.
Great, thanks. I'll give that a shot.
👍 1
Thanks Kyle, that seemed to work perfectly! https://github.com/pangeo-forge/terraclimate-feedstock/pull/4
🚀 1
Nice job!