hey all, I'm trying to register a flow in v1 but I keep getting timeouts. I'm trying to set my timeout to 60 per this PR https://github.com/PrefectHQ/prefect/pull/4119 using

PREFECT__CLOUD__REQUEST_TIMEOUT=60

but don't think it's quite working. does anyone have any experience with situations like this? full stack trace in thread

same here, tomas -- anyone from prefect seeing anything on this?

Hello! Thanks for reporting this. We're escalating internally to see what could be happening here.

everything worked fine on our side until this week I believe on wednesday. it usually fails on one of our flows that has tons of versions/tasks but sometimes it fails on one of the smaller/newer flows.

Hmm..how small is small? 🤔

here's an example of a flow that's not updated that failed for me locally yesterday, does this help?

[2021-01-14 15:00:00.000] DEBUG    --- Serialized flow: {"__version__": "0.15.13", "edges": [], "environment": null, "name": "dbt_debug_canary", "parameters": [], "reference_tasks": [], "run_config": {"__version__": "0.15.13", "cpu": null, "env": {"ENVIRONMENT": "user", "PREFECT_PROJECT": "tmorenovasquez", "PREFECT__CLOUD__HEARTBEAT_MODE": "thread", "PREFECT__LOGGING__EXTRA_LOGGERS": "['flows']", "PREFECT__LOGGING__FORMAT": "%(levelname)s - %(name)s | %(message)s", "PREFECT__LOGGING__LEVEL": "DEBUG", "REDSHIFT_DBNAME": "tmorenovasquez", "REDSHIFT_HOST": "<http://non-prd-data-warehouse.cxqs6ydij0jm.us-west-2.redshift.amazonaws.com|non-prd-data-warehouse.cxqs6ydij0jm.us-west-2.redshift.amazonaws.com>", "REDSHIFT_PORT": "5439", "REDSHIFT_PW_NAME": "PREFECT-non-prd-data-warehouse-tmorenovasquez", "REDSHIFT_USERNAME": "tmorenovasquez"}, "execution_role_arn": null, "image": null, "labels": ["ecs"], "memory": null, "run_task_kwargs": {"cluster": "prd-prefect-tasks", "networkConfiguration": {"awsvpcConfiguration": {"assignPublicIp": "ENABLED", "securityGroups": ["sg-01206324ae14d8ee5", "sg-0d1550363c8c9ded6"], "subnets": ["subnet-0175d939b3fb111ac", "subnet-049e95fa3d88ac4d6", "subnet-074b1f2db66fcf8e9", "subnet-0dc0ac04035637572"]}}}, "task_definition": {"containerDefinitions": [{"image": "XXX", "logConfiguration": {"logDriver": "awslogs", "options": {"awslogs-group": "/ecs/prefect-tasks", "awslogs-region": "us-west-2", "awslogs-stream-prefix": "tmorenovasquez-dbt_debug_canary"}}, "name": "flow"}], "cpu": 512.0, "executionRoleArn": "arn:aws:iam::792470144447:role/prefect-ecs-execution-role", "family": "tmorenovasquez-dbt_debug_canary", "memory": 1024.0, "networkMode": "awsvpc", "taskRoleArn": "XXX"}, "task_definition_arn": null, "task_definition_path": null, "task_role_arn": null, "type": "ECSRun"}, "schedule": {"__version__": "0.15.13", "adjustments": [], "clocks": [{"__version__": "0.15.13", "cron": "00 10 * * *", "day_or": true, "end_date": null, "labels": null, "parameter_defaults": {}, "start_date": {"dt": "2021-01-01T00:00:00", "tz": "Etc/UTC"}, "type": "CronClock"}], "filters": [], "not_filters": [], "or_filters": []}, "storage": {"__version__": "0.15.13", "access_token_secret": "GITHUB_ACCESS_TOKEN", "base_url": null, "flows": {}, "path": "flows/flow_dbt_debug_canary.py", "ref": "express_partners_hotpatch", "repo": "DispatchHealth/dwh", "secrets": [], "type": "GitHub"}, "tasks": [{"__version__": "0.15.13", "auto_generated": false, "cache_for": null, "cache_key": null, "cache_validator": {"fn": "prefect.engine.cache_validators.never_use", "kwargs": {}}, "inputs": {"command": {"required": true, "type": "<class 'str'>"}, "dbt_kwargs": {"required": false, "type": "<class 'dict'>"}, "env": {"required": false, "type": "<class 'dict'>"}, "execute": {"required": false, "type": "<class 'bool'>"}, "helper_script": {"required": false, "type": "<class 'str'>"}, "options": {"required": false, "type": "<class 'str'>"}}, "max_retries": 3, "name": "DbtTask", "outputs": "typing.Optional[typing.List[str]]", "retry_delay": 600, "skip_on_upstream_skip": true, "slug": "DbtTask-1", "tags": [], "timeout": null, "trigger": {"fn": "prefect.triggers.all_successful", "kwargs": {}}, "type": "flows.utils.dbt.DbtTask"}], "type": "flows.utils.dhflow.DHFlow"}
[2021-01-14 15:00:00.000] DEBUG    --- Hashed flow: b17c314c684e792a6a86e09e634adf722cc423a208fde6e79b9e1fa0b9b7df20
[2021-01-14 15:00:00.000] DEBUG    --- Hashed register_flows.py adca5ee46c2931827cec7a963253763c0abfba38de9096ab6f7ade2d845586cf
[2021-01-14 15:00:00.000] DEBUG    --- Full idempotency_key b17c314c684e792a6a86e09e634adf722cc423a208fde6e79b9e1fa0b9b7df20adca5ee46c2931827cec7a963253763c0abfba38de9096ab6f7ade2d845586cf

+1 I've been seeing an increase in timeouts these past two days as well

Hey everyone, really appreciate the additional context. Just wanted to give the heads up that we created an internal ticket to track this issue.

Awesome, thank you @Bianca Hoch! I'll check this out tomorrow when I'm back in the office