https://prefect.io logo
Join Slack
Powered by
Hi <@U021J8TU753> team, did you all silently push ...
# prefect-cloud
r
Hi @Prefect team, did you all silently push a change to how Schedulers are handled recently? My flows are suddenly failing with a 
Flow could not be retrieved from deployment.
error that traces back to a 
pydantic.error_wrappers.ValidationError: 8 validation errors for DeploymentResponse
error. We have happily used Interval Schedulers for months and months without issue. Stable deployments suddenly started failing about an hour ago. This is affecting every one of our deployments with schedulers. Full stacktrace in threads
✅ 1
Copy code
Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/prefect/engine.py", line 294, in retrieve_flow_then_begin_flow_run
    flow = await load_flow_from_flow_run(flow_run, client=client)
  File "/usr/local/lib/python3.10/dist-packages/prefect/client/utilities.py", line 40, in with_injected_client
    return await fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/prefect/deployments.py", line 182, in load_flow_from_flow_run
    deployment = await client.read_deployment(flow_run.deployment_id)
  File "/usr/local/lib/python3.10/dist-packages/prefect/client/orchestration.py", line 1501, in read_deployment
    return schemas.responses.DeploymentResponse.parse_obj(response.json())
  File "pydantic/main.py", line 526, in pydantic.main.BaseModel.parse_obj
  File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 8 validation errors for DeploymentResponse
schedule -> timezone
  argument of type 'function' is not iterable (type=type_error)
schedule -> cron
  field required (type=value_error.missing)
schedule -> timezone
  argument of type 'function' is not iterable (type=type_error)
schedule -> anchor_date
  extra fields not permitted (type=value_error.extra)
schedule -> interval
  extra fields not permitted (type=value_error.extra)
schedule -> rrule
  field required (type=value_error.missing)
schedule -> anchor_date
  extra fields not permitted (type=value_error.extra)
schedule -> interval
  extra fields not permitted (type=value_error.extra)
n
hi @Robert Banick - can you try this?
you're likely using an old version of prefect before we pinned this dep - pendulum 3 just got released
r
Trying this…
I unfortunately am blocked by an unrelated issue. The old admin of our account set up 
PREFECT_EXPERIMENTAL_ENABLE_WORKSPACE_DASHBOARD
back when the dashboard was released. He’s gone but the account remains registered in his name. This is causing an error where the setting is not recognized in newer version of Prefect (I think)
Copy code
File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for Profile
settings
  Unknown setting 'PREFECT_EXPERIMENTAL_ENABLE_WORKSPACE_DASHBOARD'. (type=value_error)
@Nate could you help me remove this setting? I’m not sure how to go about doing so
n
you can open 
~/.prefect/profiles.toml
with your preferred editor and remove that setting from your active profile
r
Perfect thanks
Yep that worked
Unfortunately the original issue remains. Installing 
prefect==2.14.11
w/ the 
requirements.txt
set to install 
pendulum >= 2.1.2, < 3.0.0
and then re-deploying still results in the deployment choking on pendelum requirements when run
I’m not entirely convinced by this explanation anyways as it doesn’t make sense why Pendelum would suddenly start failing deployments in the cloud when those deployments hadn’t been re-deployed.
If I remove the Scheduler from a deployment it will run so that’s definitely the core issue here.
If I inspect the test flow I’m using I see this for the schedule:
Copy code
'schedule': {'interval': 86400.0, 'anchor_date': '2023-12-19T07:15:28.467541+00:00', 'timezone': 'UTC'}
I wasn’t thinking about this correctly — I need to change on the docker containers running our flows. Confirming now.
upvote 1
Fixed!
Thanks for the help @Nate
n
glad you got it resolved catjam
4 Views
Open in Slack
PreviousNext
Powered by