Raviraja Ganta
04/30/2022, 8:09 PMmain-project
|
src
|
| __init__.py
| module1.py
| module2.py
|
configs
|
| config.yaml
|
flows
|
| sample_flow.py
module1,2 uses configs from config.yaml file and sample_flow import the code from modules, declares them as tasks and wraps them in a flow. I have some other custom inbuilt dependencies as well. Some of the modules needs to run on GPU. I want to run this on AWS EC2.
• What is the best way to package the code?
• What is the way to run agent that uses GPU on EC2?Anna Geller
What is the way to run agent that uses GPU on EC2?You would need to use a local agent and match the agent label with your flow's run configuration
Raviraja Ganta
05/01/2022, 6:23 PM• https://medium.com/the-prefect-blog/the-simple-guide-to-productionizing-data-workflows-with-docker-31a5aae67c0aThank you for this 🙏
You would need to use a local agent and match the agent label with your flow's run configurationI am using run_config as DockerRun with the custom image. I can to use DockerAgent only right? (Failing in LocalAgent when tested in Local setup) I want to understand whether if a container is created with image provided in run_config and agent runs that. In that case, I have to build run_config image with GPU support right? In the Link you shared, there is nothing mentioned about run_config.
Anna Geller
Raviraja Ganta
05/01/2022, 9:16 PMAnna Geller
supervisord.conf
with this content:
[unix_http_server]
file=/tmp/supervisor.sock
[supervisord]
loglevel=info
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[supervisorctl]
serverurl=unix:///tmp/supervisor.sock ; use a unix:// URL for a unix socket
[program:prefect-agent]
command=/Users/YOU/opt/anaconda3/envs/your_venv/bin/prefect agent local start --label gpu
prefect agent local start --label gpu -p /Users/your_username/path/to/your_modules
Raviraja Ganta
05/01/2022, 9:25 PMAnna Geller
Brad I
05/03/2022, 5:39 PMDeploymentSpec(
name="orion-test",
flow=flow,
flow_runner=KubernetesFlowRunner(
image="<http://us.gcr.io/myproject/test-orion:latest|us.gcr.io/myproject/test-orion:latest>",
namespace="orion",
image_pull_policy=KubernetesImagePullPolicy.ALWAYS,
),
)
However, when I try to create the deployment it errors saying that it can’t find my custom modules:
$ prefect deployment create test_orion/workflows/orion_test.py
Loading deployments from python script at 'test_orion/workflows/orion_test.py'...
...
from test_orion.tasks.utils import Request
...
ModuleNotFoundError: No module named 'test_orion'
Encountered exception while loading specifications from 'test_orion/workflows/orion_test.py'
Is there a github repo or other example of how to package and deploy Prefect 2.0 code with custom modules?Anna Geller