Thread
#prefect-community
    p

    parsa

    3 months ago
    Hey I'm trying to run a mapped task and it looks like everything was successful but then right after reduce when I try to merge the results I'm getting this error:
    Kevin Kho

    Kevin Kho

    3 months ago
    What is your version?
    p

    parsa

    3 months ago
    1.2.0
    Kevin Kho

    Kevin Kho

    3 months ago
    Does this happen every time? Does a re-registration help?
    p

    parsa

    3 months ago
    Ya it's a recurring problem. Re-registration didn't fix it either.
    Kevin Kho

    Kevin Kho

    3 months ago
    I swear we had an issue like this documented somewhere. Am trying to find it
    p

    parsa

    3 months ago
    Thanks Kevin
    Kevin Kho

    Kevin Kho

    3 months ago
    does the last comment sound like anything you do? https://github.com/PrefectHQ/prefect/issues/5075
    Are you using artifacts?
    chicago-joe

    chicago-joe

    3 months ago
    nope not using artifacts!
    @Kevin Kho actually you helped me write this a long long time ago in a galaxy far far away. You may vaguely remember that script I was writing that was a multiplicative of two lists that I mapped in parallel:
    daily, monthly quarterly, yearly
    for each of the above, calculate with 5 different types of aggregation levels
    this was working fine for several months, started experiencing issues a few weeks ago. Initially we thought it was OOM error but I hopped into the container with ctop, and also passed in custom mem settings, and don't think this is the issue that we're seeing now
    Kevin Kho

    Kevin Kho

    3 months ago
    I just left a message to the team about this to get more ideas
    Anna Geller

    Anna Geller

    3 months ago
    Joe, can you share a small example we could reproduce to file a bug report? we saw similar issues in the past, but all of them were fixed when re-registering the flow
    this is mainly an issue with runtime vs. build time which no longer exists in 2.0
    chicago-joe

    chicago-joe

    3 months ago
    took a while to obfuscate PIID but I've got it now, will slack you guys the zip file containing the python script and the csv datafile
    thank you both so much btw, this script is massively important to us
    Kevin Kho

    Kevin Kho

    3 months ago
    which task raises it?
    fnCreateBpsReturnViews
    ? I assume it works for flow.run()?
    could you verify the CodeCommit file is exactly matching the registration? Cuz this normally happens when ther’s a mismatch
    chicago-joe

    chicago-joe

    3 months ago
    will do. 99% sure it will be with the exception of the image registered. We registered an updated image tag, which is updated on the codecommit file on that branch - could this be it?
    I was thinking the with Flow() parameters wouldn't need to be, as I can register a flow with one image and then choose to run the flow on another image either via python prefect Client() or the Cloud UI
    Kevin Kho

    Kevin Kho

    3 months ago
    Yeah I don’t think it’s that. But worth a shot I guess
    I can try running this script more after lunch.
    chicago-joe

    chicago-joe

    3 months ago
    sounds good, will double check registered vs branch flows. PyCharm's comparison feature ftw
    Anna Geller

    Anna Geller

    3 months ago
    PyCharm's comparison feature ftw
    100% yes, I use it too 💯
    chicago-joe

    chicago-joe

    3 months ago
    PyCharm reports back that the contents are identical
    hmm if this has to do with some black swan issue on registration, I can try deleting the flow from the cloud and re-registering?
    Anna Geller

    Anna Geller

    3 months ago
    re-registering usually fixes the issue here, so that would be great
    chicago-joe

    chicago-joe

    3 months ago
    looks like that has worked @Anna Geller @Kevin Kho @parsa quick question, I do notice the OOM score is getting pretty high up there, I thought I had specified my host_config parameters here to allow for more memory, but it doesn't look like the mem_limit of 6g is showing in htop - what am I doing wrong here?
    Kevin Kho

    Kevin Kho

    3 months ago
    You did it man! I don’t know on the memory. It looks pretty right. I wonder if you set a lower limit, will it enforce? That would mean it just can’t get 6g, because I think 6g is an upper bound right?