✅ thought for 302.4 seconds

Great choice. Let’s make your EC2 instance slightly less existential by giving it a proper job as a Prefect worker. Assumption: You’re on Prefect 3.x. High‑level best practices for an EC2 worker - Work pool type - Process pool: simplest and reliable for a single EC2 box. Runs your flows as local processes. Recommended unless you specifically need container isolation. - Docker/Kubernetes: use if you need containerized isolation or elastic scaling. For a single EC2, Process is fine. - Docs: Workers (Prefect 3) - Instance prep - Install system packages: git, python3.x, build tools needed by your flows. - Create a dedicated user (e.g.,

prefect

) and a virtualenv for reproducibility. - Install Prefect 3:

pip install prefect

- Auth to Prefect Cloud/Server - Set

PREFECT_API_URL

and

PREFECT_API_KEY

as environment variables (store securely; use a service account key if you’re on Cloud Pro/Enterprise). - API keys: Manage API keys - Network - Outbound HTTPS (443) to Prefect Cloud is required; inbound can usually be denied entirely. Use Security Groups to keep it tight. - Startup management - Run the worker under systemd so it restarts automatically on reboot/crash. - Security and secrets - Don’t bake secrets into images or shell history. Use AWS SSM Parameter Store/Secrets Manager or systemd Environment files with restricted perms. - Logging - Worker/stdout logs go to Prefect Cloud by default and to journald if using systemd. Adjust verbosity with Prefect settings if needed. Step 1 — Create a Process work pool Run this from anywhere that can reach the Prefect API (your laptop is fine):

prefect work-pool create my-ec2-pool --type process

Step 2 — Start a worker on the EC2 instance Export your Prefect connection settings and start the worker. You can test interactively first:

export PREFECT_API_URL="<https://api.prefect.cloud/api/accounts/<acct_id>/workspaces/<ws_id>>"
export PREFECT_API_KEY="pnu_****"

prefect worker start -p my-ec2-pool -n ec2-1

To keep it running, install a systemd unit:

# /etc/systemd/system/prefect-worker.service
[Unit]
Description=Prefect Worker
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=prefect
WorkingDirectory=/home/prefect
Environment=PREFECT_API_URL=<https://api.prefect.cloud/api/accounts/<acct_id>/workspaces/<ws_id>>
Environment=PREFECT_API_KEY=pnu_****
ExecStart=/home/prefect/.venv/bin/prefect worker start -p my-ec2-pool -n ec2-1
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target

Then:

sudo systemctl daemon-reload
sudo systemctl enable --now prefect-worker
sudo systemctl status prefect-worker

Step 3 — A simple project and deploy.py Below is a minimal example that registers a deployment which points to your Git repo. The EC2 worker will clone the repo and run the flow locally (process-based). Project layout (example):

.
├── flows/
│   └── hello.py
└── deploy.py

flows/hello.py

from prefect import flow, get_run_logger

@flow
def hello(name: str = "world"):
    logger = get_run_logger()
    <http://logger.info|logger.info>(f"Hello, {name}!")

deploy.py ``` from prefect import deploy from prefect.deployments.runner import RunnerDeployment from prefect.runner.storage import GitRepository # Configure where the worker should pull code from storage = GitRepository( url="https://github.com/your-org/your-repo.git", branch="main", # or pin commit_sha for determinism include_submodules=False ) deployment = RunnerDeployment( name="hello-deploy", flow_name="hello", # resolves from the entrypoint entrypoint="flows/hello.py:hello", # path within the repo parameters={"name": "EC2"}, work_pool_name="my-ec2-pool", storage=storage, ) if name == "__main__":

# Registers the deployment in your Prefect workspace deploy(deployment)

Register the deployment:

python deploy.py

Alternative if your code already lives on the EC2 box and won’t change often: you can omit `storage` and keep code local, but any worker that executes the run will need the same files on disk.

Step 4 — Trigger a run
Two easy ways:

- CLI

prefect deployment run hello/hello-deploy --param name=Marvin --watch

- Python API

from prefect.deployments import run_deployment flow_run = run_deployment( name="hello/hello-deploy", parameters={"name": "Marvin"}, ) print(flow_run.id) ``` Useful references - Workers overview: Workers (Prefect 3) - API keys and auth: Manage API keys - Work pools CLI: run

prefect work-pool --help

- Deployment run CLI:

prefect deployment run --help

If you prefer containers for isolation, we can switch this to a Docker work pool and use an image instead of a bare EC2 environment. Otherwise, the Process worker above is the least fussy and gets the job done. Would you like me to adapt the example to Docker?