Hi all, apologies for the vagueness of this, but I...
# prefect-community
s
Hi all, apologies for the vagueness of this, but I’m getting a bit of a heisenbug in a flow I’m running on `DaskKubernetesEnvironment`:
It’s a heisenbug to me because it’s jumping randomly around my tasks, sometimes it shows up upstream or downstream of this. Usually only one or a few mapped tasks that fail. Can someone point me in a likely direction to start reading / debugging?
Not sure if it’s related, but I’m also getting this, which is a fresh bundle of fun:
Copy code
Last State Message
[13 Nov 2020 9:22am]: Unexpected error: AttributeError("partially initialized module 'pandas' has no attribute 'core' (most likely due to a circular import)")
I’m passing pandas dataframes between tasks, but it’s not clear where exactly the failure is occuring.
This also feels like a heisenbug; in the flow I ran there, when I triggered a restart from the Cloud UI, it didn’t occur again (though the downstream task didn’t do anything, but I’m guessing that’s because I’m not persisting things).
n
Hi @Steven Hamblin - I'd say the flow not restarting has something to do with missing results, as you said. As for the error, can you share some minimum code that you're seeing the issue with for us to look at? In addition, you should take a look at the PandasSerializer, it may help with passing data between tasks (and also retries, caching, etc)
Also @Steven Hamblin - someone on the team pointed me to https://github.com/PrefectHQ/prefect/issues/3226, which might describe your issue.
s
@nicholas Yeah, I did look at that one as it looked similar; I ran the test suggested in that ticket around image versions, but mine came back all the same just as `commetta`’s did.
I’m using the same environment + storage configuration as
commetta
is, without the
worker_spec_file
and using
gcr
as the registry.
n
Hm ok in which case would you mind adding any info you have to that issue for the Core team to look into?
s
Sure, will do.
👍 1