Matthew Blau
02/12/2021, 9:09 PMZanie
docker build
? Or writing a flow that uses docker storage?Matthew Blau
02/12/2021, 9:13 PMif __name__ == '__main__':
logging = prefect.context.get("logger")
with Flow(name="example",
#schedule=schedule,
state_handlers=[slack_notifier],
storage = Docker(dockerfile="/home/lookup/integration/Dockerfile",
ignore_healthchecks= False,
)) as flow:
result = write_all_files()
from prefect import config
flow.run_config = DockerRun(env={f"PREFECT__CONTEXT__SECRETS__{k}": v for k, v in config.context.secrets.items()})
flow.register(project_name="test")
Zanie
RUN pip install pandas
or similar in that Dockerfile?Matthew Blau
02/12/2021, 9:14 PMZanie
setup.py
in the module that you're putting into the docker container and using pip install -e /path/to/your/module
or similar.Docker
storage class which has a kwarg for this
python_dependencies (List[str], optional): list of pip installable dependencies for the image
Matthew Blau
02/12/2021, 9:16 PMpython3 integration.py
Zanie
Matthew Blau
02/12/2021, 9:20 PMZanie
flow.register()
needs to be able to run and python is going to complain if the module is not available. You can wrap the module imports in a try/except
block and ignore the exception when you're just registering the flow or you can put the imports into their respective tasks so they are not attempted until flow runtime.Matthew Blau
02/12/2021, 9:37 PMZanie