https://prefect.io logo
Title
t

Tomas Moreno

02/16/2023, 9:50 PM
hey all, I'm trying to register a flow in v1 but I keep getting timeouts. I'm trying to set my timeout to 60 per this PR https://github.com/PrefectHQ/prefect/pull/4119 using
PREFECT__CLOUD__REQUEST_TIMEOUT=60
but don't think it's quite working. does anyone have any experience with situations like this? full stack trace in thread
Traceback (most recent call last):
  File "/home/runner/work/dwh/dwh/deploy/register_flows.py", line 626, in <module>
    main()
  File "/home/runner/work/dwh/dwh/deploy/register_flows.py", line 620, in main
    create_proj_and_register_flows(flows, args)
  File "/home/runner/work/dwh/dwh/deploy/register_flows.py", line 283, in create_proj_and_register_flows
    register_flow(flow, flow_file, args)
  File "/home/runner/work/dwh/dwh/deploy/register_flows.py", line 373, in register_flow
    flow.register(
  File "/home/runner/work/dwh/dwh/.venv/lib/python3.10/site-packages/prefect/core/flow.py", line 1727, in register
    registered_flow = client.register(
  File "/home/runner/work/dwh/dwh/.venv/lib/python3.10/site-packages/prefect/client/client.py", line 1176, in register
    res = self.graphql(
  File "/home/runner/work/dwh/dwh/.venv/lib/python3.10/site-packages/prefect/client/client.py", line 570, in graphql
    raise ClientError(result["errors"])
prefect.exceptions.ClientError: [{'path': ['create_flow_from_compressed_string'], 'message': 'Operation timed out', 'extensions': {'code': 'API_ERROR'}}]
a

Alex Dancho

02/17/2023, 12:25 AM
same here, tomas -- anyone from prefect seeing anything on this?
b

Bianca Hoch

02/17/2023, 2:47 PM
Hello! Thanks for reporting this. We're escalating internally to see what could be happening here.
Do you have an approximate time for when this started happening?
Also, how many tasks are within the flow that is being registered? Is this reproducible with flows that have a smaller number of tasks?
t

Tomas Moreno

02/17/2023, 2:53 PM
everything worked fine on our side until this week I believe on wednesday. it usually fails on one of our flows that has tons of versions/tasks but sometimes it fails on one of the smaller/newer flows.
b

Bianca Hoch

02/17/2023, 2:54 PM
Hmm..how small is small? 🤔
t

Tomas Moreno

02/17/2023, 2:56 PM
here's an example of a flow that's not updated that failed for me locally yesterday, does this help?
[2021-01-14 15:00:00.000] DEBUG    --- Serialized flow: {"__version__": "0.15.13", "edges": [], "environment": null, "name": "dbt_debug_canary", "parameters": [], "reference_tasks": [], "run_config": {"__version__": "0.15.13", "cpu": null, "env": {"ENVIRONMENT": "user", "PREFECT_PROJECT": "tmorenovasquez", "PREFECT__CLOUD__HEARTBEAT_MODE": "thread", "PREFECT__LOGGING__EXTRA_LOGGERS": "['flows']", "PREFECT__LOGGING__FORMAT": "%(levelname)s - %(name)s | %(message)s", "PREFECT__LOGGING__LEVEL": "DEBUG", "REDSHIFT_DBNAME": "tmorenovasquez", "REDSHIFT_HOST": "<http://non-prd-data-warehouse.cxqs6ydij0jm.us-west-2.redshift.amazonaws.com|non-prd-data-warehouse.cxqs6ydij0jm.us-west-2.redshift.amazonaws.com>", "REDSHIFT_PORT": "5439", "REDSHIFT_PW_NAME": "PREFECT-non-prd-data-warehouse-tmorenovasquez", "REDSHIFT_USERNAME": "tmorenovasquez"}, "execution_role_arn": null, "image": null, "labels": ["ecs"], "memory": null, "run_task_kwargs": {"cluster": "prd-prefect-tasks", "networkConfiguration": {"awsvpcConfiguration": {"assignPublicIp": "ENABLED", "securityGroups": ["sg-01206324ae14d8ee5", "sg-0d1550363c8c9ded6"], "subnets": ["subnet-0175d939b3fb111ac", "subnet-049e95fa3d88ac4d6", "subnet-074b1f2db66fcf8e9", "subnet-0dc0ac04035637572"]}}}, "task_definition": {"containerDefinitions": [{"image": "XXX", "logConfiguration": {"logDriver": "awslogs", "options": {"awslogs-group": "/ecs/prefect-tasks", "awslogs-region": "us-west-2", "awslogs-stream-prefix": "tmorenovasquez-dbt_debug_canary"}}, "name": "flow"}], "cpu": 512.0, "executionRoleArn": "arn:aws:iam::792470144447:role/prefect-ecs-execution-role", "family": "tmorenovasquez-dbt_debug_canary", "memory": 1024.0, "networkMode": "awsvpc", "taskRoleArn": "XXX"}, "task_definition_arn": null, "task_definition_path": null, "task_role_arn": null, "type": "ECSRun"}, "schedule": {"__version__": "0.15.13", "adjustments": [], "clocks": [{"__version__": "0.15.13", "cron": "00 10 * * *", "day_or": true, "end_date": null, "labels": null, "parameter_defaults": {}, "start_date": {"dt": "2021-01-01T00:00:00", "tz": "Etc/UTC"}, "type": "CronClock"}], "filters": [], "not_filters": [], "or_filters": []}, "storage": {"__version__": "0.15.13", "access_token_secret": "GITHUB_ACCESS_TOKEN", "base_url": null, "flows": {}, "path": "flows/flow_dbt_debug_canary.py", "ref": "express_partners_hotpatch", "repo": "DispatchHealth/dwh", "secrets": [], "type": "GitHub"}, "tasks": [{"__version__": "0.15.13", "auto_generated": false, "cache_for": null, "cache_key": null, "cache_validator": {"fn": "prefect.engine.cache_validators.never_use", "kwargs": {}}, "inputs": {"command": {"required": true, "type": "<class 'str'>"}, "dbt_kwargs": {"required": false, "type": "<class 'dict'>"}, "env": {"required": false, "type": "<class 'dict'>"}, "execute": {"required": false, "type": "<class 'bool'>"}, "helper_script": {"required": false, "type": "<class 'str'>"}, "options": {"required": false, "type": "<class 'str'>"}}, "max_retries": 3, "name": "DbtTask", "outputs": "typing.Optional[typing.List[str]]", "retry_delay": 600, "skip_on_upstream_skip": true, "slug": "DbtTask-1", "tags": [], "timeout": null, "trigger": {"fn": "prefect.triggers.all_successful", "kwargs": {}}, "type": "flows.utils.dbt.DbtTask"}], "type": "flows.utils.dhflow.DHFlow"}
[2021-01-14 15:00:00.000] DEBUG    --- Hashed flow: b17c314c684e792a6a86e09e634adf722cc423a208fde6e79b9e1fa0b9b7df20
[2021-01-14 15:00:00.000] DEBUG    --- Hashed register_flows.py adca5ee46c2931827cec7a963253763c0abfba38de9096ab6f7ade2d845586cf
[2021-01-14 15:00:00.000] DEBUG    --- Full idempotency_key b17c314c684e792a6a86e09e634adf722cc423a208fde6e79b9e1fa0b9b7df20adca5ee46c2931827cec7a963253763c0abfba38de9096ab6f7ade2d845586cf
that flow just runs a dbt debug command so we can ensure that dbt will work successfully
here's the github action we use to register flows as well, I tried setting the env var via both the
env
section and an
export
statement just to see if it was any wonkiness with that and both time dout
- name: Register Flows feature branch
        if: env.IS_PR == 'true' && env.SHOULD_RUN == 'true'
        env:
          PREFECT_PROJECT: ${{ github.event.pull_request.user.login }}
          ENVIRONMENT: user
          BRANCH: ${{ github.head_ref }}
        run: |
          source $VENV
          export PREFECT__CLOUD__REQUEST_TIMEOUT=60
          python deploy/register_flows.py -l DEBUG -b $BRANCH -e $ENVIRONMENT -t $BRANCH --no-schedule
a

alex

02/17/2023, 4:52 PM
+1 I've been seeing an increase in timeouts these past two days as well
b

Bianca Hoch

02/17/2023, 5:13 PM
Hey everyone, really appreciate the additional context. Just wanted to give the heads up that we created an internal ticket to track this issue.
❤️ 1
Will keep you all updated as we learn more.
Hey team, just wanted to update you all that we have a maintenance period scheduled which should help address the issues you're seeing. Check out our status page here for more info: https://prefect.status.io/
❤️ 2
🙌 1
t

Tomas Moreno

02/20/2023, 6:16 PM
Awesome, thank you @Bianca Hoch! I'll check this out tomorrow when I'm back in the office
@Bianca Hoch this is looking much better on our side! thank you for the help with this!
:party-parrot: 1