https://prefect.io logo
Title
p

Parsa-SRA

06/09/2022, 7:27 PM
Hey I'm trying to run a mapped task and it looks like everything was successful but then right after reduce when I try to merge the results I'm getting this error:
1
k

Kevin Kho

06/09/2022, 8:08 PM
What is your version?
p

Parsa-SRA

06/09/2022, 8:29 PM
1.2.0
k

Kevin Kho

06/09/2022, 8:32 PM
Does this happen every time? Does a re-registration help?
p

Parsa-SRA

06/09/2022, 8:33 PM
Ya it's a recurring problem. Re-registration didn't fix it either.
k

Kevin Kho

06/09/2022, 8:47 PM
I swear we had an issue like this documented somewhere. Am trying to find it
p

Parsa-SRA

06/09/2022, 8:53 PM
Thanks Kevin
k

Kevin Kho

06/09/2022, 10:56 PM
does the last comment sound like anything you do? https://github.com/PrefectHQ/prefect/issues/5075
Are you using artifacts?
c

chicago-joe

06/10/2022, 1:36 PM
nope not using artifacts!
@Kevin Kho actually you helped me write this a long long time ago in a galaxy far far away. You may vaguely remember that script I was writing that was a multiplicative of two lists that I mapped in parallel:
daily, monthly quarterly, yearly
for each of the above, calculate with 5 different types of aggregation levels
this was working fine for several months, started experiencing issues a few weeks ago. Initially we thought it was OOM error but I hopped into the container with ctop, and also passed in custom mem settings, and don't think this is the issue that we're seeing now
k

Kevin Kho

06/10/2022, 1:38 PM
I just left a message to the team about this to get more ideas
a

Anna Geller

06/10/2022, 1:58 PM
Joe, can you share a small example we could reproduce to file a bug report? we saw similar issues in the past, but all of them were fixed when re-registering the flow
this is mainly an issue with runtime vs. build time which no longer exists in 2.0
c

chicago-joe

06/10/2022, 3:13 PM
took a while to obfuscate PIID but I've got it now, will slack you guys the zip file containing the python script and the csv datafile
thank you both so much btw, this script is massively important to us
k

Kevin Kho

06/10/2022, 3:16 PM
which task raises it?
fnCreateBpsReturnViews
? I assume it works for flow.run()?
could you verify the CodeCommit file is exactly matching the registration? Cuz this normally happens when ther’s a mismatch
c

chicago-joe

06/10/2022, 3:23 PM
will do. 99% sure it will be with the exception of the image registered. We registered an updated image tag, which is updated on the codecommit file on that branch - could this be it?
I was thinking the with Flow() parameters wouldn't need to be, as I can register a flow with one image and then choose to run the flow on another image either via python prefect Client() or the Cloud UI
k

Kevin Kho

06/10/2022, 3:26 PM
Yeah I don’t think it’s that. But worth a shot I guess
I can try running this script more after lunch.
c

chicago-joe

06/10/2022, 3:28 PM
sounds good, will double check registered vs branch flows. PyCharm's comparison feature ftw
a

Anna Geller

06/10/2022, 4:26 PM
PyCharm's comparison feature ftw
100% yes, I use it too 💯
c

chicago-joe

06/10/2022, 7:28 PM
PyCharm reports back that the contents are identical
hmm if this has to do with some black swan issue on registration, I can try deleting the flow from the cloud and re-registering?
a

Anna Geller

06/10/2022, 7:47 PM
re-registering usually fixes the issue here, so that would be great
c

chicago-joe

06/10/2022, 8:42 PM
looks like that has worked @Anna Geller @Kevin Kho @Parsa-SRA quick question, I do notice the OOM score is getting pretty high up there, I thought I had specified my host_config parameters here to allow for more memory, but it doesn't look like the mem_limit of 6g is showing in htop - what am I doing wrong here?
🙌 2
k

Kevin Kho

06/10/2022, 8:45 PM
You did it man! I don’t know on the memory. It looks pretty right. I wonder if you set a lower limit, will it enforce? That would mean it just can’t get 6g, because I think 6g is an upper bound right?