Ilya Galperin
08/26/2022, 8:30 PM/usr/local/lib/python3.10/site-packages/prefect/deployments.py:48: UserWarning: Block document has schema checksum sha256:0ec43f8010cee4adbf73aebcc58f1e45986d765c2a224dfc9cd5428f98c516f8 which does not match the schema checksum for class 'S3'. This indicates the schema has changed and this block may not load.
storage_block = Block._from_block_document(storage_document)
Deleting and re-creating the block and deployment will sometimes cause it to work again but again, only on an intermittent basis. The flow code itself does not touch or interact with Prefect block storage. Has anyone experienced this or have any idea what might be causing the issue?Kevin Grismore
08/26/2022, 8:34 PMIlya Galperin
08/26/2022, 8:36 PMKevin Grismore
08/26/2022, 8:40 PMIlya Galperin
08/26/2022, 8:40 PMKevin Grismore
08/26/2022, 8:42 PMIlya Galperin
08/26/2022, 8:42 PMKevin Grismore
08/26/2022, 8:42 PMRyan Peden
08/26/2022, 8:49 PMIlya Galperin
08/26/2022, 8:53 PMprefecthq/prefect:2.2.0-python3.10
on the agent.Ryan Peden
08/26/2022, 9:38 PMIlya Galperin
08/26/2022, 9:39 PMAnna Geller
08/27/2022, 1:13 AMIlya Galperin
08/27/2022, 6:18 AMAnna Geller
08/27/2022, 11:01 AMIlya Galperin
08/27/2022, 5:05 PMVersion: 2.2.0
API version: 0.8.0
Python version: 3.8.9
Git commit: e3651362
Built: Tue, Aug 23, 2022 2:18 PM
OS/Arch: darwin/arm64
Profile: default
Server type: hosted
I’ve just now tried creating an entirely new flow, deployment and storage block and we continue to see the same behavior. I kick off 5 flows from the Cloud UI, 3 run successfully (pods are created for these) and 2 fail immediately.
Unfortunately there are no logs that I’m able to capture because no pods are even attempted to be created for the failed flows — they are not recognized by the agent nor do they seem to ever exist in a “scheduled” state, they just go into a “failed” state immediately so the agent does not have an opportunity grab them.
I am not sure what an MRE is if you can please explain but I have forwarded all the documentation in the support ticket and created a Github issue here. Please let me know if there are any other details i.e. flow run IDs that I can provide that could help in the investigation.
https://github.com/PrefectHQ/prefect/issues/6586#issuecomment-1229039900Anna Geller
08/27/2022, 8:52 PMprefect deployment build
CLI which is way easier to troubleshoot than doing it from Python.
Also, given that you have both a support ticket and an open GitHub issue, let's continue the discussion there and close this Slack discussion to avoid repeating ourselves across channels. I'm marking this thread as solved, and please if you have any new findings to add, add it to the GitHub issue and if the person that is handling your support ticket has access to this issue, they should have everything they need there.
Thanks a lot for the detailed write-up and for providing a thorough explanation. Someone should get back to you after the weekend.