https://prefect.io logo
Join Slack
Powered by
Hi! is it possible to pass source of flow that sto...
# ask-community
d
Hi! is it possible to pass source of flow that stores in local or docker storage on one machine to agent on the other machine?
k
You can copy the file over but Prefect has no mechanism for that
d
@Kevin Kho so the only one way to do that is use cloud storages like git or s3?
k
that’s the easiest but you can use Local Storage plus DockerRun if the flow files are in the Docker container
d
@Kevin Kho and how agent on differrent machine will see the image from another server?
k
Ah I see what you mean. Yeah you’d need to upload the image to a registry
👍 1
v
Hi, I'd like to use Local Storage + DockerRun + Docker Agent to run my flows in containers. All my sources are in the image so that seems to correspond to the setup you are suggesting @Kevin Kho. But I'm not sure how to do that. Also the docs suggest Local Storage can only be used with Local Agent, so I'm a little confused. At the moment I tried to do this :
Copy code
from time import sleep
import prefect
from prefect import task, Flow
from prefect.agent.docker import DockerAgent
from prefect.run_configs import DockerRun
from prefect.schedules import CronSchedule
from prefect.storage.local import Local

@task
def ping():
    logger = prefect.context.get("logger")
    for n in range(20):
        <http://logger.info|logger.info>(f"Ping n°{n}")
        sleep(3)

with Flow("dummy flow") as flow:
    ping()

flow.schedule = CronSchedule("* * * * *")
flow.storage = Local()
flow.run_config = DockerRun(
    image="<http://ghcr.io/mtes-mct/monitorfish/monitorfish-pipeline:v1.6.4|ghcr.io/mtes-mct/monitorfish/monitorfish-pipeline:v1.6.4>"
)
flow.register(project_name=PROJECT_NAME, build=False)
agent = DockerAgent(show_flow_logs=True)
agent.start()
In the server UI I see the flow that correctly gets registered, the Docker Agent queries the API, and flow runs are scheduled but never submitted nor executed.
Any hint would be welcome !
What I'm guessing is I would need to store my serialized flows in the 
.prefect/flows
folder inside the image.. Ideally I'd like have the serialized flows stored outside of the image, on the machine that runs the Docker Agent. When starting a container to run a flow, the 
.prefect/flows
folder of the host machine would therefore need to be bind mounted inside the flow runner container. Is there any way to do that with Prefect ?
k
If you are using DockerRun + LocalStorage, the ideal is to do 
Local(stored_as_script=True, path =…)
so you then point Prefect to the path inside the container. I have an example here If you want to mount, yes you need to bind. Try mounting the volume on the agent side and I think that should be enough
v
Hi, I forgot to reply but thanks, I got this working, exactly what I wanted ! In case an example cna help someone : • Flows configuration : https://github.com/MTES-MCT/monitorfish/blob/master/datascience/src/pipeline/flows_config.py • DockerFile to build the image that will run the flows : https://github.com/MTES-MCT/monitorfish/blob/master/infra/docker/Dockerfile.DataPipeline • Run script to register flows and start the DockerAgent : https://github.com/MTES-MCT/monitorfish/blob/master/datascience/main.py
k
I archived this answer here
👍 1
3 Views
Open in Slack
PreviousNext
Powered by