Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
pacc-clearcover-june-12-2023
pacc-may-31-2023
ppcc-may-16-2023
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Title
a
Adam Roderick
12/12/2022, 3:53 PMHello, I have 4 tasks that run successfully when running locally using FlowRunner. When running in Prefect Cloud, I get the following results How can I troubleshoot this?
• TypeError("'generator' object is not callable")
• SystemError('unknown opcode')
• SystemError('Objects/dictobject.c:1527: bad argument to internal function')
• The 4th task hangs indefinitely
Any idea on this @Anna Geller or @Dylan?
I can pull down the docker image that is being used, manually spin up a container, and successfully run the flow using FlowRunner
d
Dylan
12/12/2022, 4:45 PMHey Adam 👋
Can you share a minimum reproducible example here?
a
Adam Roderick
12/12/2022, 4:48 PM👋 I will work on that. The problem may be in our infrastructure/devops, so I'll do my best. Any ideas on troubleshooting errors like these so I can try and narrow it down?
z
Zanie
12/12/2022, 4:50 PMI’ve never seen these
SystemErrorIt seems like something very weird must be going on. Are you mixing Python versions?
(Pickling on deploy with one version and running with another?)
a
Adam Roderick
12/12/2022, 4:52 PMWe've confirmed that the local dev environments, the build container, and the execution environment are all on python 3.8
Our searches do seem to indicate a python version mismatch, but I don't know where else to look for a version mismatch
z
Zanie
12/12/2022, 4:57 PMPerhaps it’d be helpful to run it without an agent using
prefect run -n "<flow-name>" --executeIt may also be helpful to use script-based storage instead of pickle-based (if you are).
Are you using Cython?
a
Adam Roderick
12/12/2022, 4:59 PMThanks for the thoughts Michael. We aren't using Cython but perhaps one of the dependencies is
Script-based storage is new to me, will check it out
Would
prefect run -n "<flow-name>" --executez
Zanie
12/12/2022, 5:00 PMIt’ll run it as though it is the agent — it’ll connect to Cloud.
👍 1
I think it’ll use a subprocess instead of a container though, which may be helpful.
a
Adam Roderick
12/12/2022, 8:43 PMThis can down to an issue of the snowflake connector and something about prefect's execution model. When we implement following the pattern at https://github.com/PrefectHQ/prefect/blob/1.x/src/prefect/tasks/snowflake/snowflake.py#L187 (using
fetchall()