Channels
#prefect-community
Title
# prefect-community
p
Parsa-SRA
06/09/2022, 7:27 PMHey I'm trying to run a mapped task and it looks like everything was successful but then right after reduce when I try to merge the results I'm getting this error:
✅ 1
k
Kevin Kho
06/09/2022, 8:08 PM
What is your version?
p
Parsa-SRA
06/09/2022, 8:29 PM1.2.0
k
Kevin Kho
06/09/2022, 8:32 PM
Does this happen every time? Does a re-registration help?
p
Parsa-SRA
06/09/2022, 8:33 PMYa it's a recurring problem. Re-registration didn't fix it either.
k
Kevin Kho
06/09/2022, 8:47 PM
I swear we had an issue like this documented somewhere. Am trying to find it
p
Parsa-SRA
06/09/2022, 8:53 PMThanks Kevin
k
Kevin Kho
06/09/2022, 10:56 PM
does the last comment sound like anything you do? https://github.com/PrefectHQ/prefect/issues/5075
Are you using artifacts?
c
chicago-joe
06/10/2022, 1:36 PMnope not using artifacts!
@Kevin Kho actually you helped me write this a long long time ago in a galaxy far far away. You may vaguely remember that script I was writing that was a multiplicative of two lists that I mapped in parallel:
daily, monthly quarterly, yearlyfor each of the above, calculate with 5 different types of aggregation levelsthis was working fine for several months, started experiencing issues a few weeks ago. Initially we thought it was OOM error but I hopped into the container with ctop, and also passed in custom mem settings, and don't think this is the issue that we're seeing now
k
Kevin Kho
06/10/2022, 1:38 PM
I just left a message to the team about this to get more ideas
a
Anna Geller
06/10/2022, 1:58 PM
Joe, can you share a small example we could reproduce to file a bug report? we saw similar issues in the past, but all of them were fixed when re-registering the flow
this is mainly an issue with runtime vs. build time which no longer exists in 2.0
c
chicago-joe
06/10/2022, 3:13 PMtook a while to obfuscate PIID but I've got it now, will slack you guys the zip file containing the python script and the csv datafile
thank you both so much btw, this script is massively important to us
k
Kevin Kho
06/10/2022, 3:16 PM
which task raises it?
fnCreateBpsReturnViewscould you verify the CodeCommit file is exactly matching the registration? Cuz this normally happens when ther’s a mismatch
c
chicago-joe
06/10/2022, 3:23 PMwill do. 99% sure it will be with the exception of the image registered. We registered an updated image tag, which is updated on the codecommit file on that branch - could this be it?
I was thinking the with Flow() parameters wouldn't need to be, as I can register a flow with one image and then choose to run the flow on another image either via python prefect Client() or the Cloud UI
k
Kevin Kho
06/10/2022, 3:26 PM
Yeah I don’t think it’s that. But worth a shot I guess
I can try running this script more after lunch.
c
chicago-joe
06/10/2022, 3:28 PMsounds good, will double check registered vs branch flows. PyCharm's comparison feature ftw
a
Anna Geller
06/10/2022, 4:26 PM
PyCharm's comparison feature ftw100% yes, I use it too 💯
c
chicago-joe
06/10/2022, 7:28 PMPyCharm reports back that the contents are identical
hmm if this has to do with some black swan issue on registration, I can try deleting the flow from the cloud and re-registering?
a
Anna Geller
06/10/2022, 7:47 PM
re-registering usually fixes the issue here, so that would be great
c
chicago-joe
06/10/2022, 8:42 PMlooks like that has worked @Anna Geller @Kevin Kho @Parsa-SRA
quick question, I do notice the OOM score is getting pretty high up there, I thought I had specified my host_config parameters here to allow for more memory, but it doesn't look like the mem_limit of 6g is showing in htop - what am I doing wrong here?
🙌 2
k
Kevin Kho
06/10/2022, 8:45 PM
You did it man! I don’t know on the memory. It looks pretty right. I wonder if you set a lower limit, will it enforce? That would mean it just can’t get 6g, because I think 6g is an upper bound right?
3 Views