Hi all, apologies for the vagueness of this, but I’m getting a bit of a heisenbug in a flow I’m runn...

Steven Hamblin

11/13/2020, 9:20 AM

Hi all, apologies for the vagueness of this, but I’m getting a bit of a heisenbug in a flow I’m running on `DaskKubernetesEnvironment`:

Steven Hamblin

11/13/2020, 9:21 AM

It’s a heisenbug to me because it’s jumping randomly around my tasks, sometimes it shows up upstream or downstream of this. Usually only one or a few mapped tasks that fail. Can someone point me in a likely direction to start reading / debugging?

Steven Hamblin

11/13/2020, 9:27 AM

Not sure if it’s related, but I’m also getting this, which is a fresh bundle of fun:

Copy code

Last State Message
[13 Nov 2020 9:22am]: Unexpected error: AttributeError("partially initialized module 'pandas' has no attribute 'core' (most likely due to a circular import)")

I’m passing pandas dataframes between tasks, but it’s not clear where exactly the failure is occuring.

Steven Hamblin

11/13/2020, 9:35 AM

This also feels like a heisenbug; in the flow I ran there, when I triggered a restart from the Cloud UI, it didn’t occur again (though the downstream task didn’t do anything, but I’m guessing that’s because I’m not persisting things).

nicholas

11/13/2020, 3:11 PM

Hi @Steven Hamblin - I'd say the flow not restarting has something to do with missing results, as you said. As for the error, can you share some minimum code that you're seeing the issue with for us to look at? In addition, you should take a look at the PandasSerializer, it may help with passing data between tasks (and also retries, caching, etc)

nicholas

11/13/2020, 4:26 PM

Also @Steven Hamblin - someone on the team pointed me to https://github.com/PrefectHQ/prefect/issues/3226, which might describe your issue.

Steven Hamblin

11/13/2020, 4:38 PM

@nicholas Yeah, I did look at that one as it looked similar; I ran the test suggested in that ticket around image versions, but mine came back all the same just as `commetta`’s did.

Steven Hamblin

11/13/2020, 4:39 PM

I’m using the same environment + storage configuration as

commetta

is, without the

worker_spec_file

and using

gcr

as the registry.

nicholas

11/13/2020, 4:40 PM

Hm ok in which case would you mind adding any info you have to that issue for the Core team to look into?

Steven Hamblin

11/13/2020, 4:42 PM

Sure, will do.

👍 1

41 Views

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.