https://prefect.io
Join Slack
Hey I'm trying to run a mapped task and it looks like everything was successful but then right after...
# ask-community
p
Hey I'm trying to run a mapped task and it looks like everything was successful but then right after reduce when I try to merge the results I'm getting this error:
✅ 1
k
What is your version?
p
1.2.0
k
Does this happen every time? Does a re-registration help?
p
Ya it's a recurring problem. Re-registration didn't fix it either.
k
I swear we had an issue like this documented somewhere. Am trying to find it
p
Thanks Kevin
k
does the last comment sound like anything you do? https://github.com/PrefectHQ/prefect/issues/5075
Are you using artifacts?
c
nope not using artifacts!
@Kevin Kho actually you helped me write this a long long time ago in a galaxy far far away. You may vaguely remember that script I was writing that was a multiplicative of two lists that I mapped in parallel: 
daily, monthly quarterly, yearly
for each of the above, calculate with 5 different types of aggregation levels
this was working fine for several months, started experiencing issues a few weeks ago. Initially we thought it was OOM error but I hopped into the container with ctop, and also passed in custom mem settings, and don't think this is the issue that we're seeing now
k
I just left a message to the team about this to get more ideas
a
Joe, can you share a small example we could reproduce to file a bug report? we saw similar issues in the past, but all of them were fixed when re-registering the flow
this is mainly an issue with runtime vs. build time which no longer exists in 2.0
c
took a while to obfuscate PIID but I've got it now, will slack you guys the zip file containing the python script and the csv datafile
thank you both so much btw, this script is massively important to us
k
which task raises it? 
fnCreateBpsReturnViews
? I assume it works for flow.run()?
could you verify the CodeCommit file is exactly matching the registration? Cuz this normally happens when ther’s a mismatch
c
will do. 99% sure it will be with the exception of the image registered. We registered an updated image tag, which is updated on the codecommit file on that branch - could this be it?
I was thinking the with Flow() parameters wouldn't need to be, as I can register a flow with one image and then choose to run the flow on another image either via python prefect Client() or the Cloud UI
k
Yeah I don’t think it’s that. But worth a shot I guess
I can try running this script more after lunch.
c
sounds good, will double check registered vs branch flows. PyCharm's comparison feature ftw
a
PyCharm's comparison feature ftw
100% yes, I use it too 💯
c
PyCharm reports back that the contents are identical
hmm if this has to do with some black swan issue on registration, I can try deleting the flow from the cloud and re-registering?
a
re-registering usually fixes the issue here, so that would be great
c
looks like that has worked @Anna Geller @Kevin Kho @Parsa-SRA quick question, I do notice the OOM score is getting pretty high up there, I thought I had specified my host_config parameters here to allow for more memory, but it doesn't look like the mem_limit of 6g is showing in htop - what am I doing wrong here?
🙌 2
k
You did it man! I don’t know on the memory. It looks pretty right. I wonder if you set a lower limit, will it enforce? That would mean it just can’t get 6g, because I think 6g is an upper bound right?
4 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by