Hi everyone. Has somebody successful ran flows that use Dask Dataframes or multiprocessing library on a Windows Local Agent run deployed on Cloud?I am finding it impossible to define a flow and wrap it inside the necessary
if __name__ == ‘__main__’
. It all works fine when running locally with
but it tells me it can't find a Flow when I deploy it to Cloud and run it from there.
3 months ago
For Dask Dataframe you need to do this like this. The error of not finding a flow is because the Flow needs to be defined outside the
if __name__ == "__main__"
as I mentioned here. The Flow file is imported during runtime and looks for flow variables so if you define the Flow is the main guard, it won’t show up
For multiprocessing, it will be a lot better if you use LocalDaskExecutor instead + mapping your tasks, which spins up a multiprocessing pool also