Ray Tang
04/18/2022, 3:25 PMprefect agent docker start --show-flow-logs
I have tried to run prefect register --project test-project --path hello_word_dockerized.py
, but the registration failed because I don't have pandas installed on my docker agent's host (and I am not intended to because I want to have isolated docker images per flows).
import prefect
from prefect import task, Flow
from prefect.run_configs import DockerRun
from prefect.storage import Local
import pandas
def get_logger():
from prefect.utilities.logging import get_logger
import logging
import sys
logger = get_logger()
logger.setLevel('INFO')
log_stream = logging.StreamHandler(sys.stdout)
logger.addHandler(log_stream)
return logger
logger = get_logger()
@task(log_stdout=True)
def hello_world():
<http://logger.info|logger.info>("Hello, World (Dockerized)")
with Flow("Hello World Dockerized") as flow:
hello_world()
IMAGE_NAME = "prefect-hello-world-dockerized"
IMAGE_TAG = "latest"
import os
print(os.path.basename(__file__))
flow.storage = Local(path="/opt/prefect/flows/hello_word_dockerized.py", stored_as_script=True)
flow.run_config = DockerRun(image=f"{IMAGE_NAME}:{IMAGE_TAG}", labels=["uat", "docker"])
Kevin Kho
04/18/2022, 3:27 PMRay Tang
04/18/2022, 3:37 PMKevin Kho
04/18/2022, 3:40 PMRay Tang
04/18/2022, 3:51 PMKevin Kho
04/18/2022, 3:55 PMtry-except
instead because prefect register …
opens the Python file and evaluates it to create that Flow objectRay Tang
04/18/2022, 4:00 PMwith Flow("Hello World Dockerized") as flow:
try:
from flow_common import hello_workld_task
hello_workld_task()
except:
pass
flow_common
is not available during registrationKevin Kho
04/18/2022, 4:24 PMAnurag Bajpai
04/19/2022, 6:07 AMfrom prefect.utilities.storage import extract_flow_from_file
@task(checkpoint=False,log_stdout=True,max_retries=3,retry_delay=dt.timedelta(seconds=10))
def register_flow(path, project_name):
flow = extract_flow_from_file(path)
flow.register(project_name=project_name)
with Flow("Register Flow", run_config=DockerRun(image='your-image-here')) as flow:
flow_path = Parameter("flow_path")
project_name = Parameter("project_name")
register_flow(flow_path, project_name)
Ray Tang
04/19/2022, 8:18 AMdocker run
$ docker run -it prefect-hello-world-dockerized:latest /bin/sh
# cd /opt/prefect/build/hello_world_dockerized/flows/
# ls
hello_word_dockerized.py register_flow.py
but when we run the register_flow.py (which is provided by @Anurag Bajpai), it throws FileNotFoundError: [Errno 2] No such file or directory:
DockerRun
is not running 🤔 (because when we put in some bad image name, no error was thrown)Kevin Kho
04/19/2022, 1:35 PMRay Tang
04/19/2022, 3:06 PMpython register_flow.py --project_name demo-project-dockerized/hello-world-dockerized --flow_path /opt/prefect/build/hello_world_dockerized/flows/ --image_name prefect-hello-world-dockerized:latested/flows/ --image_name aprefect-hello-world-dockerized:latest
it doesn't throw an error with a incorrect image_nameAnurag Bajpai
04/19/2022, 3:51 PMKevin Kho
04/19/2022, 3:53 PMflow.run()
does not respect RunConfig. Only agent backed runs do. But I think Ray is registering?flow.run()
uses executor but not RunConfig