Pedro Machado02/25/2021, 4:56 PM
referencing the 'latest' version of that base image to have your repo be always up to date in your flows run environment
Pedro Machado02/25/2021, 5:58 PM
run in dbt cloud 2. runs python code that pulls data from snowflake, scores it using a model, and pushes it back to snowflake 3. runs
again They want to be able to change the code that is run in step 2 independent of the flow code just like they can update the dbt code in a separate repo and have dbt cloud fetch it before each run.
from prefect import Flow, task def download_files_from_repo(repo_path, local_path): # Here you'd setup your download pass @task def get_newest_utils(): # Utils should be `pip install -e /my_utilities` in the base image download_files_from_repo("my-repo/src/my_utilities", "/my_utilities") # Now we'll import the module dynamically to use the new files import my_utilities return my_utilities @task def do_something(utils): print(dir(utils)) with Flow("example") as flow: utilities = get_newest_utils() do_something(utilities) flow.run()
Pedro Machado02/25/2021, 6:21 PM