Carlos Cueto

    3 months ago
    Hi everyone. Has somebody successful ran flows that use Dask Dataframes or multiprocessing library on a Windows Local Agent run deployed on Cloud? I am finding it impossible to define a flow and wrap it inside the necessary
    if __name__ == ‘__main__’
    . It all works fine when running locally with
    but it tells me it can't find a Flow when I deploy it to Cloud and run it from there.
    Kevin Kho

    Kevin Kho

    3 months ago
    For Dask Dataframe you need to do this like this. The error of not finding a flow is because the Flow needs to be defined outside the
    if __name__ == "__main__"
    as I mentioned here. The Flow file is imported during runtime and looks for flow variables so if you define the Flow is the main guard, it won’t show up
    For multiprocessing, it will be a lot better if you use LocalDaskExecutor instead + mapping your tasks, which spins up a multiprocessing pool also