Raviraja Ganta04/30/2022, 8:09 PM
module1,2 uses configs from config.yaml file and sample_flow import the code from modules, declares them as tasks and wraps them in a flow. I have some other custom inbuilt dependencies as well. Some of the modules needs to run on GPU. I want to run this on AWS EC2. • What is the best way to package the code? • What is the way to run agent that uses GPU on EC2?
main-project | src | | __init__.py | module1.py | module2.py | configs | | config.yaml | flows | | sample_flow.py
What is the way to run agent that uses GPU on EC2?You would need to use a local agent and match the agent label with your flow's run configuration
Raviraja Ganta05/01/2022, 6:23 PM
• https://medium.com/the-prefect-blog/the-simple-guide-to-productionizing-data-workflows-with-docker-31a5aae67c0aThank you for this 🙏
You would need to use a local agent and match the agent label with your flow's run configurationI am using run_config as DockerRun with the custom image. I can to use DockerAgent only right? (Failing in LocalAgent when tested in Local setup) I want to understand whether if a container is created with image provided in run_config and agent runs that. In that case, I have to build run_config image with GPU support right? In the Link you shared, there is nothing mentioned about run_config.
Raviraja Ganta05/01/2022, 9:16 PM
with this content:
[unix_http_server] file=/tmp/supervisor.sock [supervisord] loglevel=info [rpcinterface:supervisor] supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface [supervisorctl] serverurl=unix:///tmp/supervisor.sock ; use a unix:// URL for a unix socket [program:prefect-agent] command=/Users/YOU/opt/anaconda3/envs/your_venv/bin/prefect agent local start --label gpu
prefect agent local start --label gpu -p /Users/your_username/path/to/your_modules
Raviraja Ganta05/01/2022, 9:25 PM
Brad I05/03/2022, 5:39 PM
However, when I try to create the deployment it errors saying that it can’t find my custom modules:
DeploymentSpec( name="orion-test", flow=flow, flow_runner=KubernetesFlowRunner( image="<http://us.gcr.io/myproject/test-orion:latest|us.gcr.io/myproject/test-orion:latest>", namespace="orion", image_pull_policy=KubernetesImagePullPolicy.ALWAYS, ), )
Is there a github repo or other example of how to package and deploy Prefect 2.0 code with custom modules?
$ prefect deployment create test_orion/workflows/orion_test.py Loading deployments from python script at 'test_orion/workflows/orion_test.py'... ... from test_orion.tasks.utils import Request ... ModuleNotFoundError: No module named 'test_orion' Encountered exception while loading specifications from 'test_orion/workflows/orion_test.py'