Channels
#prefect-community
Title
k
kevin
03/23/2022, 8:06 PMI’m like 99% certain that this is the case but I’m just looking to confirm:
when I have a task that creates a piece of mutable data, say a dictionary, and I pass it downstream to two different tasks, both tasks get a reference to the original dict right? So if one task edits it there risk for conflict.
ex:
Copy code
with Flow('foo') as f:
a = task(lambda x: {'key': x})('value')
b = task(lambda y: y.pop('key'))(a)
c = task(lambda z: z.get('key'))(a)a
Anna Geller
03/23/2022, 8:14 PM
no risk, under the hood Prefect creates a copy of a task and passes the same "a" data dependency to both b and c
you can see it well when you visualize the flow
if you want to be 100% about anything, don't take my word for it but instead test it out, run it, print the data and you'll be then 100% sure 😄
k
kevin
03/23/2022, 8:17 PMhmm yea i tried altering the flow to look like this:
Copy code
with Flow('foo') as f:
a = task(lambda x: {'key': x})('value')
b = task(lambda y: y.pop('key'))(a)
c = task(lambda z: z['key'])(a)cit seems like the reference object is shared between the two tasks
k
Kevin Kho
03/23/2022, 8:18 PM
That…is news to me but I can also believe it. you could wrap the lambda input into a deepcopy to be sure this works right?
k
kevin
03/23/2022, 8:18 PMfor task c?
k
Kevin Kho
03/23/2022, 8:19 PM
Copy code
with Flow('foo') as f:
a = task(lambda x: {'key': x})('value')
b = task(lambda y: deepcopy(y).pop('key'))(a)
c = task(lambda z: z['key'])(a)Like that? I dunno not sure
k
kevin
03/23/2022, 8:20 PMyea that gave me all tasks succeeded
we’re on a pretty old version of prefect
"==0.14.14"if that changes anything
k
Kevin Kho
03/23/2022, 8:21 PM
I dont think the prefect version will affect this. I think it’s just the execution creates a dict which is mutable and it just gets updated instead of copies created
k
kevin
03/23/2022, 8:23 PMyea it seems pythonically reasonable to have that be the case
6 Views