Hi, I have suddenly started getting these errors in prefect 2.8.4, any clue what might be wrong?

Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/prefect/engine.py", line 274, in retrieve_flow_then_begin_flow_run
    flow = await load_flow_from_flow_run(flow_run, client=client)
  File "/usr/local/lib/python3.8/dist-packages/prefect/client/utilities.py", line 47, in with_injected_client
    return await fn(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/prefect/deployments.py", line 159, in load_flow_from_flow_run
    deployment = await client.read_deployment(flow_run.deployment_id)
  File "/usr/local/lib/python3.8/dist-packages/prefect/client/orchestration.py", line 1479, in read_deployment
    return schemas.responses.DeploymentResponse.parse_obj(response.json())
  File "pydantic/main.py", line 526, in pydantic.main.BaseModel.parse_obj
  File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 8 validation errors for DeploymentResponse
schedule -> interval
  field required (type=value_error.missing)
schedule -> timezone
  argument of type 'function' is not iterable (type=type_error)
schedule -> cron
  extra fields not permitted (type=value_error.extra)
schedule -> day_or
  extra fields not permitted (type=value_error.extra)
schedule -> timezone
  argument of type 'function' is not iterable (type=type_error)
schedule -> rrule
  field required (type=value_error.missing)
schedule -> cron
  extra fields not permitted (type=value_error.extra)
schedule -> day_or
  extra fields not permitted (type=value_error.extra)

what version of pydantic do you have where you’re getting that error?

Hi @Nate I had

1.10.6

tried with

1.10.4

also

hm, would you be able to share

prefect deployment inspect <your deployment>

? that “function is not iterable” is suspicious to me 🤔

sure, here it is

{
    'id': '4dc8b1e6-8700-4cc4-a13f-669f71bcd31c',
    'created': '2023-09-04T07:50:17.706317+00:00',
    'updated': '2023-12-11T06:33:43.020877+00:00',
    'name': 'default',
    'version': '92df2354ad4fb9cc6ea61ef1264a6bdd',
    'description': '',
    'flow_id': '810df3c8-f680-419d-8f45-d64c16e5a71c',
    'schedule': {'cron': '*/30 * * * *', 'timezone': 'Asia/Calcutta', 'day_or': True},
    'is_schedule_active': True,
    'infra_overrides': {
        'customizations': [
            {'op': 'add', 'path': '/spec/template/spec/containers/0/resources', 'value': {'limits': {'cpu': 30, 'memory': '240G'}, 'requests': {'cpu': 24, 'memory': '210G'}}},
            {'op': 'add', 'path': '/spec/template/spec/nodeSelector', 'value': {'type': 'large-on-demand'}}
        ]
    },
    'parameters': {},
    'tags': [],
    'work_queue_name': 'default',
    'parameter_openapi_schema': {'type': 'object', 'title': 'Parameters', 'properties': {}},
    'path': 'test_location',
    'pull_steps': None,
    'entrypoint': 'test_location/test_flow.py:flow_function',
    'manifest_path': None,
    'storage_document_id': None,
    'infrastructure_document_id': 'a7b4912d-4b54-43f3-b675-8a996580214c',
    'created_by': None,
    'updated_by': None,
    'work_pool_name': 'default-agent-pool',
    'enforce_parameter_schema': False,
    'infrastructure': {
        'type': 'kubernetes-job',
        'env': {'EXTRA_PIP_PACKAGES': 's3fs'},
        'labels': {},
        'name': None,
        'command': None,
        'image': '<REMOVED>',
        'namespace': 'jhub',
        'service_account_name': None,
        'image_pull_policy': <KubernetesImagePullPolicy.ALWAYS: 'Always'>,
        'cluster_config': None,
        'job': {
            'apiVersion': 'batch/v1',
            'kind': 'Job',
            'metadata': {'labels': {}},
            'spec': {'template': {'spec': {'parallelism': 1, 'completions': 1, 'restartPolicy': 'Never', 'containers': [{'name': 'prefect-job', 'env': []}]}}}
        },
        'customizations': [],
        'job_watch_timeout_seconds': 100000,
        'pod_watch_timeout_seconds': 600,
        'stream_output': True,
        'finished_job_ttl': 10,
        'block_type_slug': 'kubernetes-job'
    }
}

Having the same issues starting around yesterday when trying to run the

Deployment.build_from_flow

method in Prefect

2.10.12

. It seems to be happening when are passing in a cron schedule that contains timezones into the

schedule

parameter i.e.

{"cron": "0 1 * * *", "timezone": "America/New_York"}

CC: @Saad Zaheer

Hi @Ilya Galperin, are you facing the issue when you quick run a flow?

Hi Ankit - what do you mean by quick run?

Thanks for the direction @Ilya Galperin I just had issue with my scheduled flows, I changed the timezone to browser default, everything looks fine for me

@Ankit I'd be curious if this error persists on a more recent version?

@Nate we don’t seem to be having the same issue using build_from_flow in a test environment running 2.14.10

true, just an issue with older version and it's probably the recent pydantic changes, some backward compatibility issue maybe with older version

you're both on cloud correct? off the top of my head, i'd suspect a client-server schema mismatch for

DeploymentResponse

We’re on Cloud yes

thank you! I'm looking more into this now

Hi Nate - we represent it as a Python dictionary that’s passed into Deployment.build_from_flow i.e.

Deployment.build_from_flow(**deployment_settings)

where one of the keys defines the schedule

deployment_settings["schedule"] = {"cron": "0 2 * * *", "timezone": "America/New_York"}

hmm, ok thank you. and you get the error at deployment time?

We get the error when calling build_from_flow

ok thank you - i am still trying to reproduce

Oh interesting, ya for us it does seem at least somewhat related to the schedule since if we just pass the scheduled dictionary with only the single cron key we do not get the error. We’re actually working on upgrading to 2.14.10 right now but if I get a chance I can try reverting a deployment at some point tomorrow and inspecting it

oh okay - thank you for the information!

Looks like there is another report here just FYI https://prefect-community.slack.com/archives/C048SVCEFF0/p1702933297747949

ah - one more question, whats your pydantic version?

1.10.6

hey @Ilya Galperin - could you try pinning this dependency?

pendulum >= 2.1.2, < 3.0.0

i've managed to reproduce,

pendulum

is pinned in newer versions of prefect, and pendulum 3 looks like a breaking change