I have a question, I am new to prefect , I have configured prefect cloud to run the flow, the flow is created in an aws ec2 and I run the job and scheduled it , now its running on cloud ui
my question is the actual execution happens in my prefect cloud or in my aws ec2 ? is there any way i can run my prefect flow with docker in prefect cloud ?
01/21/2022, 12:18 PM
Your flow is running entirely on your infrastructure, in your case, your EC2 instance. You can read more about Prefect’s hybrid execution model here.
To run your flows in Docker containers, you can leverage a Docker agent which can be started the same way as a local agent on your EC2 instance.
01/21/2022, 1:01 PM
Thanks, but when I start with docker agent my flow not running its just shows scheduled only
one more question, in my code I am writing some debug messages to my local log file, but when the flow runs in cloud this log file not updating with the python logs, how can I get the python logs jn my local when flow run in cloud. when I run the flow in my local the log messages properly updating
01/21/2022, 1:08 PM
there are several reasons why your flow may be stuck in a scheduled state, but usually this happens when labels not set correctly between the agent and Flow. Did you assign a label to your agent and the same label to your flow in a run configuration? Here is an example:
prefect agent docker start --label dockerEC2
and in your flow:
from prefect.run_configs import DockerRun
flow.run_config = DockerRun(
Writing to a local log file is a bit more involved to set up with Docker agent. You would need to:
1. Mount a volume
2. Attach a FileHandler to your Prefect logger - here is an example