Luis Muniz
06/28/2020, 10:40 PMfrom tasks.collect.games import *
from tasks.collect.streamers import *
from tasks.collect.streams import *
from tasks.enrich.game import *
from tasks.enrich.streamer import *
from tasks.enrich.stream import *
from tasks.store.game import *
from tasks.store.streamer import *
from tasks.store.stream import *
from tasks.util.common import *
with Flow("STRDATA POC") as strdata:
collected_games = collect_games()
enriched_games = enrich_game.map(collected_games)
collected_streamers = collect_streamers()
enriched_streamers = enrich_streamer.map(collected_streamers)
collected_streams = collect_streams_per_game.map(enriched_games, unmapped(enriched_streamers))
enriched_streams = enrich_stream.map(flatten(collected_streams))
store_game.map(enriched_games)
store_stream.map(enriched_streams)
store_streamer.map(enriched_streamers)
The Flow runs OK when I run it standalone, but when I register it in my local prefect server, I can see the following error in the dashboard:
Failed to load and execute Flow's environment: ModuleNotFoundError("No module named 'tasks'")
it seems to be similar to an issue I found about not being able to submit flows to prefect cloud because of some peculiarity with pickle?
https://github.com/PrefectHQ/prefect/issues/1742
But this was related to packaging the flow in a docker image, so I can't apply the solution to my case
The layout of my project is the following:
deploy
|_
prod
|_
register.py (contains flow.register)
flows
|_
strdata_poc.py (contains flow definition - see above)
tasks
|_
collect
|_
games.py
streamers.py
streams.py
enrich
|_
...
Chris White
PATH
.”
We’re working on trying to make this a little easier to manage within Prefect’s API but ultimately we have to be able to recreate your Flow object at runtime (which requires your imports be accessible)Luis Muniz
06/28/2020, 11:01 PMChris White
Luis Muniz
06/28/2020, 11:02 PMChris White
Luis Muniz
06/28/2020, 11:04 PMChris White
prefect agent start -p /path/to/my/project -p /maybe/another/path/within/my/project
(you can use as many -p
flags as you need, and each directory you pass is added to your import path for your process, and in your case taht should also cover the ephemeral dask workers)Luis Muniz
06/28/2020, 11:07 PMChris White
Luis Muniz
06/28/2020, 11:13 PMChris White
Luis Muniz
06/28/2020, 11:14 PMChris White
Anish Chhaparwal
09/30/2020, 11:31 AM