Carlos Cueto
06/08/2022, 4:15 PMFailed to load and execute Flow's environment: ValueError('No flows found in file.')
This is the flow definition:
if __name__ == '__main__':
with Flow('Scouter-Solr-Script') as flow:
snowflake_user = PrefectSecret('snowflake_usr')
snowflake_pwd = PrefectSecret('snowflake_pwd')
snowflake_to_solr(snowflake_user, snowflake_pwd)
flow.run_config = LocalRun(labels=['SVRNAME1'])
flow.storage = Git(repo="Prefect-Flows", flow_path="Python/Scouter/snowflake_to_solr.py", git_clone_url_secret_name="azure_devops_clone_url")
flow.register(project_name='Scouter')
I'm assuming it has to do with the if __name__ == '__main__'
part on top of the Flow class definition, but I don't know how to go about fixing this. I need that for multiprocessing that happens within the main task of the flow.Kevin Kho
Carlos Cueto
06/08/2022, 4:45 PMif __main__
but now it leads to an error with the multiprocessing that goes on inside the snowflake_to_solr
task defined inside the flow:
_pickle.PicklingError: Can't pickle <function convertToRecords at 0x000001E0C09137F0>: attribute lookup convertToRecords on main failedKevin Kho
cloudpicke
. Could you move that inside the task instead?Carlos Cueto
06/08/2022, 5:07 PMKevin Kho
convertToRecords
?Carlos Cueto
06/08/2022, 5:40 PM@task
decorator, with another sub-function inside of it called convertToRercords
. It was previously outside the task function.Kevin Kho
Carlos Cueto
06/09/2022, 1:38 PMKevin Kho
Faheem Khan
07/14/2022, 8:29 AM