https://prefect.io
Join Slack
I’m like 99% certain that this is the case but I’m just looking to confirm: when I have a task that ...
# ask-community
k
I’m like 99% certain that this is the case but I’m just looking to confirm: when I have a task that creates a piece of mutable data, say a dictionary, and I pass it downstream to two different tasks, both tasks get a reference to the original dict right? So if one task edits it there risk for conflict. ex:
Copy code
with Flow('foo') as f:
    a = task(lambda x: {'key': x})('value')
    b = task(lambda y: y.pop('key'))(a)
    c = task(lambda z: z.get('key'))(a)
a
no risk, under the hood Prefect creates a copy of a task and passes the same "a" data dependency to both b and c you can see it well when you visualize the flow
if you want to be 100% about anything, don't take my word for it but instead test it out, run it, print the data and you'll be then 100% sure 😄
k
hmm yea i tried altering the flow to look like this:
Copy code
with Flow('foo') as f:
     a = task(lambda x: {'key': x})('value')
     b = task(lambda y: y.pop('key'))(a)
     c = task(lambda z: z['key'])(a)
and I get a KeyError when it hits 
c
it seems like the reference object is shared between the two tasks
k
That…is news to me but I can also believe it. you could wrap the lambda input into a deepcopy to be sure this works right?
k
for task c?
k
Copy code
with Flow('foo') as f:
     a = task(lambda x: {'key': x})('value')
     b = task(lambda y: deepcopy(y).pop('key'))(a)
     c = task(lambda z: z['key'])(a)
Like that? I dunno not sure
k
yea that gave me all tasks succeeded
we’re on a pretty old version of prefect 
"==0.14.14"
if that changes anything
k
I dont think the prefect version will affect this. I think it’s just the execution creates a dict which is mutable and it just gets updated instead of copies created
k
yea it seems pythonically reasonable to have that be the case
5 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by