Thread
#prefect-community
    a

    Aaron Goodrich

    1 year ago
    Hey guys pretty new to prefect, probably an easy solution here, but I'm trying to move to prefect from our current system of standalone dockerized ETL integrations.   Normally I use docker compose and mount the directory to the image so I can get any csv's I generate back out of the container.   In prefect, I see I can throw my script into an image at runtime with
    flow.storage = Docker(
        path="my_flow.py",
        files={"/source/of/my_flow.py": "my_flow.py"},
        stored_as_script=True
    )
    but I don't see how I can get, say, my dynamically generated files back out.   Any suggestions?
    Jim Crist-Harif

    Jim Crist-Harif

    1 year ago
    In prefect, I see I can throw my script into an image at runtime with
    Just to clarify, this builds a new image that contains your script at flow registration time, this doesn't update live. With some careful configuration, you could mount a volume in the deployed flow runs on a docker agent, and configure the flow with a
    Local
    storage though.
    but I don't see how I can get, say, my dynamically generated files back out.
    You can mount volumes in a docker-executed flow run by passing the
    --volume
    flag when starting a docker agent: https://docs.prefect.io/orchestration/agents/docker.html#mounting-volumes. If you write your outputs to a mounted volume on the host system, you should have access to them on the host.
    a

    Aaron Goodrich

    1 year ago
    thank you!