hey all I m trying to register a flow in v1 but I keep getti

Question

hey all I m trying to register a flow in v1 but I keep getting timeouts I m trying to set my timeout to 60 per this PR <https github com PrefectHQ prefect pull 4119> using `PREFECT CLOUD REQUEST TIMEOUT=60` but don t think it s quite working does anyone have any experience with situations like this full stack trace in thread

Tomas Moreno · Answer

```Traceback most recent call last File home runner work dwh dwh deploy register flows py line 626 in lt module gt main File home runner work dwh dwh deploy register flows py line 620 in main create proj and register flows flows args File home runner work dwh dwh deploy register flows py line 283 in create proj and register flows register flow flow flow file args File home runner work dwh dwh deploy register flows py line 373 in register flow flow register File home runner work dwh dwh venv lib python3 10 site packages prefect core flow py line 1727 in register registered flow = client register File home runner work dwh dwh venv lib python3 10 site packages prefect client client py line 1176 in register res = self graphql File home runner work dwh dwh venv lib python3 10 site packages prefect client client py line 570 in graphql raise ClientError result errors prefect exceptions ClientError path create flow from compressed string message Operation timed out extensions code API ERROR ```

Alex Dancho · Answer

same here tomas anyone from prefect seeing anything on this

Bianca Hoch · Answer

Hello Thanks for reporting this We re escalating internally to see what could be happening here

Bianca Hoch · Answer

Do you have an approximate time for when this started happening

Bianca Hoch · Answer

Also how many tasks are within the flow that is being registered Is this reproducible with flows that have a smaller number of tasks

Tomas Moreno · Answer

everything worked fine on our side until this week I believe on wednesday it usually fails on one of our flows that has tons of versions tasks but sometimes it fails on one of the smaller newer flows

Bianca Hoch · Answer

Hmm how small is small thinking face

Tomas Moreno · Answer

here s an example of a flow that s not updated that failed for me locally yesterday does this help ``` 2021 01 14 15 00 00 000 DEBUG Serialized flow version 0 15 13 edges environment null name dbt debug canary parameters reference tasks run config version 0 15 13 cpu null env ENVIRONMENT user PREFECT PROJECT tmorenovasquez PREFECT CLOUD HEARTBEAT MODE thread PREFECT LOGGING EXTRA LOGGERS flows PREFECT LOGGING FORMAT levelname s name s | message s PREFECT LOGGING LEVEL DEBUG REDSHIFT DBNAME tmorenovasquez REDSHIFT HOST <http non prd data warehouse cxqs6ydij0jm us west 2 redshift amazonaws com|non prd data warehouse cxqs6ydij0jm us west 2 redshift amazonaws com> REDSHIFT PORT 5439 REDSHIFT PW NAME PREFECT non prd data warehouse tmorenovasquez REDSHIFT USERNAME tmorenovasquez execution role arn null image null labels ecs memory null run task kwargs cluster prd prefect tasks networkConfiguration awsvpcConfiguration assignPublicIp ENABLED securityGroups sg 01206324ae14d8ee5 sg 0d1550363c8c9ded6 subnets subnet 0175d939b3fb111ac subnet 049e95fa3d88ac4d6 subnet 074b1f2db66fcf8e9 subnet 0dc0ac04035637572 task definition containerDefinitions image XXX logConfiguration logDriver awslogs options awslogs group ecs prefect tasks awslogs region us west 2 awslogs stream prefix tmorenovasquez dbt debug canary name flow cpu 512 0 executionRoleArn arn aws iam 792470144447 role prefect ecs execution role family tmorenovasquez dbt debug canary memory 1024 0 networkMode awsvpc taskRoleArn XXX task definition arn null task definition path null task role arn null type ECSRun schedule version 0 15 13 adjustments clocks version 0 15 13 cron 00 10 day or true end date null labels null parameter defaults start date dt 2021 01 01T00 00 00 tz Etc UTC type CronClock filters not filters or filters storage version 0 15 13 access token secret GITHUB ACCESS TOKEN base url null flows path flows flow dbt debug canary py ref express partners hotpatch repo DispatchHealth dwh secrets type GitHub tasks version 0 15 13 auto generated false cache for null cache key null cache validator fn prefect engine cache validators never use kwargs inputs command required true type lt class str gt dbt kwargs required false type lt class dict gt env required false type lt class dict gt execute required false type lt class bool gt helper script required false type lt class str gt options required false type lt class str gt max retries 3 name DbtTask outputs typing Optional typing List str retry delay 600 skip on upstream skip true slug DbtTask 1 tags timeout null trigger fn prefect triggers all successful kwargs type flows utils dbt DbtTask type flows utils dhflow DHFlow 2021 01 14 15 00 00 000 DEBUG Hashed flow b17c314c684e792a6a86e09e634adf722cc423a208fde6e79b9e1fa0b9b7df20 2021 01 14 15 00 00 000 DEBUG Hashed register flows py adca5ee46c2931827cec7a963253763c0abfba38de9096ab6f7ade2d845586cf 2021 01 14 15 00 00 000 DEBUG Full idempotency key b17c314c684e792a6a86e09e634adf722cc423a208fde6e79b9e1fa0b9b7df20adca5ee46c2931827cec7a963253763c0abfba38de9096ab6f7ade2d845586cf```

Tomas Moreno · Answer

that flow just runs a dbt debug command so we can ensure that dbt will work successfully

Tomas Moreno · Answer

here s the github action we use to register flows as well I tried setting the env var via both the `env` section and an `export` statement just to see if it was any wonkiness with that and both time dout ``` name Register Flows feature branch if env IS PR == true amp amp env SHOULD RUN == true env PREFECT PROJECT $ github event pull request user login ENVIRONMENT user BRANCH $ github head ref run | source $VENV export PREFECT CLOUD REQUEST TIMEOUT=60 python deploy register flows py l DEBUG b $BRANCH e $ENVIRONMENT t $BRANCH no schedule```

alex · Answer

+1 I ve been seeing an increase in timeouts these past two days as well

Bianca Hoch · Answer

Hey everyone really appreciate the additional context Just wanted to give the heads up that we created an internal ticket to track this issue

Bianca Hoch · Answer

Will keep you all updated as we learn more

Bianca Hoch · Answer

Hey team just wanted to update you all that we have a maintenance period scheduled which should help address the issues you re seeing Check out our status page here for more info <https prefect status io |https prefect status io >

Tomas Moreno · Answer

Awesome thank you < Bianca Hoch> I ll check this out tomorrow when I m back in the office

Tomas Moreno · Answer

< Bianca Hoch> this is looking much better on our side thank you for the help with this