Joseph Loss

    Joseph Loss

    1 year ago
    We have this massive script that is super important to our day to day operations, and I'm looking to convert this into a prefect-cloud oriented setup so that I can connect it with a slack webhook and also have access to logs anywhere/everywhere. But I'm having trouble setting this up correctly, for a few reasons: 1. These tasks in the graph are inter-twined, meaning that a lot of these tasks are called within a "task" itself. Example:
    def fnRunTask(parameter1,  parameter2, parameter3 .... parameter10):
        df1, df2 = fnRunSubTask(parameter3, parameter5)
        dfTmp = fnRunSubTask2(df1, parameter1, parameter7, parameter9)
    2. As shown above, each subfunction requires different parameters, and the number of parameters for each function can vary. Essentially this is all headache from converting a bunch of stuff that was defined under "if name == main:" into a prefect setup. I've played around with passing in one large dictionary at the start, but wanted to ask here before I continued debugging that (didn't work on tries #1-20 lol) Also wanted to ask about the task interconnectivity, as I was a bit confused about upstream/downstream tasks since some of them are "at the same time" within the task that calls them.
    Kevin Kho

    Kevin Kho

    1 year ago
    Hey @Joseph Loss! What error do you see when you pass a large dictionary? This person seemed to pass in a gigantic one. A bit unclear about interconnectivity, is there a problem with the DAG that was setup? Should
    print_df
    not be hooked up by multiple upstream tasks? This looks messy, but I think this can work.
    Joseph Loss

    Joseph Loss

    1 year ago
    let me grab a few snips of the functions, that will help explain I think
    These are the fn definitions, and I've put the entire "work-in-progress" script in a gist here so you can see the modifications I've made so far with the dictionary of Params, etc
    originally, a lot of these input parameters were defined above if name== main , so part of this large process is to redefine those as Parameters so that the prefect function would run correctly. On second thought, maybe I could just leave those inputs where they stand? As long the variable is defined before it is called in any function, maybe the Prefect task will still be able to run? Something I'll try tomorrow!
    Kevin Kho

    Kevin Kho

    1 year ago
    This is massive man…wow! Yes you can leave constant inputs alone. They will work. That will probably make you life a lot easier cuz you’ll need to pass in less
    Joseph Loss

    Joseph Loss

    1 year ago
    sorry for the delayed response, yes this is working perfectly dude thank you so much for the advice!
    Kevin Kho

    Kevin Kho

    1 year ago
    No worries at all about the delayed response! Good you’re making progress.