https://prefect.io
Join Slack
Seeing this error: can't pickle CompiledFFI objects , can someone help what might be the issue?
# ask-community
n
Seeing this error: can't pickle CompiledFFI objects , can someone help what might be the issue?
✅ 1
a
you would need to provide a bit more information 🙂
is it for Prefect 1.0 or 2.0?
usually, pickle errors occur due to flow serialization issues, often happens when you serialized your flow with one Python and Prefect version during registration, but then ran it in an environment that had different versions
n
prefect 1.0
I am trying to register the flow
and seeing this error
in prefect 1.0, is there a way to call one task from another task function?
k
You need to call 
mytask.run()
but that’s calling the Python function under the hood. It’s not a task with results and retries etc
a
theoretically yes, you can invoke the .run() method but it's not recommended
n
ok
found the issue - i am using storage client for bigquery in one of the tasks and that is throwing this error. Why is it happening?
can we remove this pickling?
k
What storage are you using? You can store it as a script, but a client instantiated inside a task should be fine for registration because that code is not actually run yet
n
yes, i am not sure why i am seeing this because i have used similar logic elsewhere and it worked
GCS bucket
k
You can use the 
stored_as_script
= 
True
to not have serialization during the registration time
You can find more info the in docstrings here
n
i tried to run the script on a different prefect vm
and registration went through!
what must have changed?
this first vm is on GCP and the second is on premise - will it make a difference?
k
No that shouldn’t 😅.
n
yes! I am not understanding this! Prefect version on both is the same.
6 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by