Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

image.png

Hi all, apologies for the vagueness of this, but I’m getting a bit of a heisenbug in a flow I’m running on `DaskKubernetesEnvironment`:

It’s a heisenbug to me because it’s jumping randomly around my tasks, sometimes it shows up upstream or downstream of this.  Usually only one or a few mapped tasks that fail.

Can someone point me in a likely direction to start reading / debugging?

Not sure if it’s related, but I’m also getting this, which is a fresh bundle of fun:

```Last State Message
[13 Nov 2020 9:22am]: Unexpected error: AttributeError("partially initialized module 'pandas' has no attribute 'core' (most likely due to a circular import)")```
I’m passing pandas dataframes between tasks, but  it’s not clear where exactly the failure is occuring.

This also feels like a heisenbug;  in the flow I ran there, when I triggered a restart from the Cloud UI, it didn’t occur again (though the downstream task didn’t do anything, but I’m guessing that’s because I’m not persisting things).

Hi <@U01DYD60T17> - I'd say the flow not restarting has something to do with missing results, as you said. As for the error, can you share some minimum code that you're seeing the issue with for us to look at? In addition, you should take a look at the <https://docs.prefect.io/api/latest/engine/serializers.html#pandasserializer|PandasSerializer>, it may help with passing data between tasks (and also retries, caching, etc)

Also <@U01DYD60T17> - someone on the team pointed me to <https://github.com/PrefectHQ/prefect/issues/3226>, which might describe your issue.

<@UN6FTLFAS> Yeah, I did look at that one as it looked similar;  I ran the test suggested in that ticket  around image versions, but mine came back all the same just as `commetta`’s did.

I’m using the same environment + storage configuration as `commetta` is, without the `worker_spec_file` and using `gcr` as the registry.

Hm ok in which case would you mind adding any info you have to that issue for the Core team to look into?