I'm getting a weird error when trying to run a flow using the DaskExecutor:
AttributeError: 'Edge' object has no attribute 'upstream_task'
I'm getting this when dask tries to use pickle to load the first of my mapped tasks. The first two tasks of the flow are not mapped, and they don't have issues. Any thoughts?
k
Kevin Kho
05/18/2022, 6:26 PM
Are you able to share the Flow?
Kevin Kho
05/18/2022, 6:27 PM
Wait no. I think you meant to just say
upstream_tasks
?
j
Jessica Smith
05/18/2022, 6:27 PM
Definitely upstream_task, here's the screenshot:
k
Kevin Kho
05/18/2022, 6:29 PM
Ah ok i’ll need to see code I think
j
Jessica Smith
05/18/2022, 6:33 PM
Does a serialized flow work? I couldn't provide code if I wanted to, this one is generated at registration time based on specifications stored in yaml files. But I can output the serialized flow to a file if that works
k
Kevin Kho
05/18/2022, 6:36 PM
More like I need to read the code to figure out what happened so if you have a dummy example that can reproduce it, that would work
j
Jessica Smith
05/18/2022, 6:37 PM
oh gotcha. i'll have to play around and see if i can make a simple example that causes the error. since I don't know what is causing it I'm not sure how to do that. I'll follow up once I determine if I can get an example or not
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.