Liel Bach

05/02/2023, 7:01 AM
Hi there, Is anyone experiencing problems with submitting and running Prefect flows in the cloud? We have a GCP K8S setup (using GKE autopilot) and we started experiencing problems last week (we didn’t change anything significant on our side) The flow run successfully attaches to the code and just freezes at the second cloud API call according to the logs - From the cloud perpsective it seems like the flow is stuck at the
stage. Attaching an example flow run log from the pod itself (flow names and IDs are redacted):
22:11:09.051 | DEBUG   | prefect.profiles - Using profile 'default'
/usr/local/lib/python3.10/ RuntimeWarning: 'prefect.engine' found in sys.modules after import of package 'prefect', but prior to execution of 'prefect.engine'; this may result in unpredictable behaviour
22:11:09.084 | DEBUG   | prefect.client - Connecting to API at <><redacted>/workspaces/<redacted>/
22:11:09.916 | INFO    | Flow run 'lambda6-tarakis-instability' - Downloading flow code from storage at '/flexor'
22:11:09.918 | DEBUG   | Flow run 'lambda6-tarakis-instability' - Importing flow code from 'transformations/'
22:11:09.928 | DEBUG   | prefect.client - Connecting to API at <><redacted>/workspaces/<redacted>/
It’s worth mentioning that we don’t use any special storage block, we just use the preloaded code in the image (using the
parameter in a flow deployment)