Hi all I m hitting an issue where manually running a flow is

Question

Hi all I m hitting an issue where manually running a flow is timing out before it starts I m using the prefect cloud backend ```$ prefect run flow name=terraclimate project=pangeo forge File Users taugspurger miniconda3 envs pangeo forge 37 bin prefect line 8 in lt module gt sys exit cli File Users taugspurger miniconda3 envs pangeo forge 37 lib python3 7 site packages click core py line 829 in call return self main args kwargs File Users taugspurger miniconda3 envs pangeo forge 37 lib python3 7 site packages click core py line 782 in main rv = self invoke ctx File Users taugspurger miniconda3 envs pangeo forge 37 lib python3 7 site packages click core py line 1259 in invoke return process result sub ctx command invoke sub ctx File Users taugspurger miniconda3 envs pangeo forge 37 lib python3 7 site packages click core py line 1259 in invoke return process result sub ctx command invoke sub ctx File Users taugspurger miniconda3 envs pangeo forge 37 lib python3 7 site packages click core py line 1066 in invoke return ctx invoke self callback ctx params File Users taugspurger miniconda3 envs pangeo forge 37 lib python3 7 site packages click core py line 610 in invoke return callback args kwargs File Users taugspurger miniconda3 envs pangeo forge 37 lib python3 7 site packages prefect cli run py line 125 in flow no url=no url File Users taugspurger miniconda3 envs pangeo forge 37 lib python3 7 site packages prefect cli run py line 377 in run flow flow id=flow id parameters= file params string params run name=run name File Users taugspurger miniconda3 envs pangeo forge 37 lib python3 7 site packages prefect client client py line 964 in create flow run res = self graphql create mutation variables=dict input=inputs File Users taugspurger miniconda3 envs pangeo forge 37 lib python3 7 site packages prefect client client py line 294 in graphql raise ClientError result errors prefect utilities exceptions ClientError path create flow run message Operation timed out extensions code API ERROR ``` <https prefect community slack com archives CL09KU1K7 p1596043664390000> looks similar and seemed to resolve itself Any debugging suggestions

Kyle Moon-Wright · Answer

Hey < Tom Augspurger> I was just able to kick off a flow run from the cli like you have above so it s definitely strange you re encountering this error Just to confirm you ve set your backend with `prefect backend cloud` your flow is named `terraclimate` and its registered in the project `pangeo forge` Just want some initial information to start to debug

Tom Augspurger · Answer

Thanks Just re ran `prefect backend cloud` to confirm that `prefect get flows gives ```terraclimate 13 30 minutes ago 73711833 eeca 4273 8cd3 37d787c7c75f pangeo forge```

Tom Augspurger · Answer

I also see the Operation timed out error when running it through the UI

Kyle Moon-Wright · Answer

Interesting let me get some insight for you

Kyle Moon-Wright · Answer

Diving deeper Is this still happening deterministically Are there many flows in this project

Tom Augspurger · Answer

It s still happening Just one flow with 10 20 tasks in it The source is at <https github com TomAugspurger terraclimate feedstock blob debug recipe pipeline py>

Kyle Moon-Wright · Answer

Great thanks still gathering insight on this

Kyle Moon-Wright · Answer

Hey there < Tom Augspurger> thanks for your patience After looking into your flow from our end it looks like there are a large number thousands of statically defined tasks that are causing a slow down when creating all the new task run states Of course this is not ideal and is something we definitely want to fix however in the meantime it may help if you could replace those static tasks with some dynamic ones with

Tom Augspurger · Answer

Great thanks I ll give that a shot

Tom Augspurger · Answer

Thanks Kyle that seemed to work perfectly <https github com pangeo forge terraclimate feedstock pull 4>

Kyle Moon-Wright · Answer

Nice job