Short answer: Automations can’t “run a block” directly. In Prefect 3.x (including self-hosted Server), the pattern is: - Create a tiny flow that loads your AWS Lambda block and invokes it - Deploy that flow - Create an Automation whose action is “Run a deployment” (optionally passing event data as parameters) Here’s how to set it up end-to-end. 1) Install the AWS integration where your worker runs - Ensure your worker environment has

prefect-aws

(and AWS creds available). Docs: Prefect AWS integration 2) Create an AWS Lambda block (once) - You can create it in the UI (Blocks → Add block → AWS → Lambda Function) or via Python. This block stores your Lambda function name and credentials. - Block docs: Blocks in Prefect 3) Write a tiny flow that invokes the block

from prefect import flow
from prefect_aws.lambda_function import LambdaFunction

@flow
def invoke_my_lambda(event: dict | None = None):
    # Load the AWS Lambda block you created in the UI (e.g., named "my-lambda")
    lf = LambdaFunction.load("my-lambda")

    # invocation_type:
    # - "Event" for async, fire-and-forget
    # - "RequestResponse" for synchronous (default)
    resp = lf.invoke(payload=event or {}, invocation_type="Event")
    return resp

Reference for the block’s invoke method (payload and options): prefect-aws docs - Method signature we use:

LambdaFunction.invoke(payload: Optional[dict] = None, invocation_type="RequestResponse", ...)

4) Create a deployment for the flow - In Prefect 3.x, use

.from_source(...).deploy(...)

(not the old 2.x commands)

from prefect.flows import Flow

# Point to your repo / path that contains the flow above
f = Flow.from_source(
    source="<https://github.com/your-org/your-repo>",
    entrypoint="flows/invoke_lambda.py:invoke_my_lambda",
)

# Deploy to a work pool with a worker running
f.deploy(
    name="invoke-lambda",
    work_pool_name="your-work-pool",
    parameters={"event": {}},   # default; will be overridden by automation if you want
)

5) Create the Automation to run the deployment - In the UI: Automations → New automation - Choose your trigger (e.g., a specific event or a flow-run state condition) - Action: “Run a deployment” - Select the deployment you created (“invoke-lambda”) - Optional: pass parameters from the triggering event using Jinja templates, e.g.: - Parameters JSON:

{
        "event": {{ event }}
      }

- Or if you want just a portion of the event:

{{ event.payload }}

Automation docs: - Creating automations: Create automations - Triggers and passing parameters: Deployment triggers and parameters - Actions reference: Automation actions API 6) Ensure a worker is running - Your worker (process/docker/k8s/etc.) must be attached to the selected work pool to pick up the run and execute the Lambda invocation. Alternative (no flow): Call the Lambda via HTTP - If your Lambda has a Function URL or is behind API Gateway, you can use the “Call a webhook” action in your automation to POST directly to that URL. This avoids running a flow, but you won’t be using the Lambda block in that case. - Webhook action docs: Create a webhook Would you like me to tailor the Automation trigger (which event/state) and the parameter mapping for your use case? If you can share: - The event you want to react to (e.g., a specific flow-run state, a custom event) - Whether you need to pass any data into the Lambda payload …I can provide the exact Automation configuration and flow code snippet to match.

Totally get it—spinning a Kubernetes pod for a tiny “glue” action is overkill. Key point: Automations can’t directly “run a block”. They can only run predefined actions. But you can avoid starting a flow/pod by having the Automation call your Lambda over HTTP instead of using the AWS block. Two practical options: A) No-flow, instant: Automation → Call a webhook → Lambda - Use Lambda Function URLs (auth = NONE) or API Gateway to expose your Lambda via HTTP. - In your Automation, choose the “Call a webhook” action and configure: - URL: your Lambda Function URL or API Gateway endpoint - Method: POST - Headers: add Content-Type and any shared-secret header you use - Body: JSON, optionally templated with event data - Example body that forwards the Prefect event to Lambda:

{
    "prefect_event": {{ event }},
    "source": "prefect-automation",
    "flow_run_id": "{{ event.resource.id if event.resource.type == 'prefect.flow-run' else '' }}"
  }

- Notes: - If you use Lambda Function URLs with

AWS_IAM

auth, Prefect can’t sign the request (no SigV4 in CallWebhook). Use: - Function URL with auth NONE + a shared secret header, and/or - API Gateway with an API key or custom authorizer - Make sure your self-hosted Prefect Server can egress to the URL. - Docs: Create automations and actions reference: Automation actions API B) Still want to use the AWS Lambda Block but avoid K8s cold starts - Run a small “utility” work pool that doesn’t create K8s jobs: - Process worker on a tiny VM (fastest startup; no image pulls) - Or Docker worker on a persistent node with the image pre-pulled - Deploy a tiny flow that loads the

LambdaFunction

block and calls

invoke

, and have your Automation run that deployment on the warm work pool. This keeps your main work in Kubernetes while avoiding pod spin-up for these micro-actions. - AWS integration docs: Prefect AWS integration If you’re open to Option A (webhook), I can give you exact header/body templates for your Automation. If you prefer Option B, I can sketch the minimal Process/Docker work pool setup and the tiny flow snippet wired to

LambdaFunction.invoke

. Which route fits your environment and security constraints (API Gateway vs Function URL, need for IAM-only access, etc.)?

Great question — you can absolutely keep everything inside Kubernetes and still avoid per-flow pod spin-ups by running a long-lived worker pod. Two patterns: 1) Process worker in a long-lived pod (simplest) - What happens: A single Kubernetes pod runs

prefect worker start

(type “process”). Your tiny “Lambda-invoker” flow runs as a subprocess inside this pod — no new pods, no image pulls per run. - Good for: fast, lightweight actions (like calling Lambda), where the worker image already has your Python deps (prefect, prefect-aws, boto3). - Notes: - Your code will be pulled from source at run time (e.g., Git) but no container orchestration is involved for each run. - Ensure AWS creds are available in the pod (env vars, IRSA on EKS, etc.). Example Kubernetes Deployment (process worker):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: prefect-process-worker
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prefect-process-worker
  template:
    metadata:
      labels:
        app: prefect-process-worker
    spec:
      serviceAccountName: your-sa # e.g., with IRSA in EKS for AWS creds
      containers:
        - name: worker
          # Build your own or use a base with Prefect + prefect-aws preinstalled
          image: your-registry/prefect-process:latest
          imagePullPolicy: IfNotPresent
          env:
            - name: PREFECT_API_URL
              value: "<http://your-prefect-server/api>"  # point to your self-hosted Server
            # If you use Prefect Cloud, set PREFECT_API_KEY instead
            # - name: PREFECT_API_KEY
            #   valueFrom: { secretKeyRef: ... }
            # AWS creds via env, IRSA, or mounted volume
          args:
            - prefect
            - worker
            - start
            - -p
            - utility-process-pool
          resources:
            requests:
              cpu: "100m"
              memory: "256Mi"
            limits:
              cpu: "500m"
              memory: "512Mi"

Build the image for this pod with the deps you need:

# Dockerfile
FROM python:3.11-slim
ENV PIP_NO_CACHE_DIR=1
RUN pip install "prefect>=3" "prefect-aws" boto3
# optional: app code for tiny flows if you don’t use from_source
CMD ["bash"]

2) Docker worker in a long-lived pod (with a Docker daemon available) - What happens: A single Kubernetes pod runs the Prefect Docker worker. It launches flow runs as Docker containers using a Docker daemon. In Kubernetes, that means either: - Sidecar Docker-in-Docker (DinD) container the worker talks to, or - Mounting the host’s Docker socket (only if your cluster actually runs Docker and you accept the security trade-off). - Good for: you want per-run container isolation and layer caching without creating Kubernetes Jobs per run. - Caveats: - Many clusters use containerd, so host Docker socket may not exist — DinD sidecar is the common approach. - Security: DinD requires a privileged sidecar. Cache persists only while the pod is alive unless you back it with a PVC. Example Kubernetes Deployment (docker worker + DinD sidecar): ``` apiVersion: apps/v1 kind: Deployment metadata: name: prefect-docker-worker spec: replicas: 1 selector: matchLabels: app: prefect-docker-worker template: metadata: labels: app: prefect-docker-worker spec: serviceAccountName: your-sa volumes: - name: dind-storage emptyDir: {} # or a PVC for persistent caching containers: - name: dind image: docker:24-dind securityContext: privileged: true args: - "--host=tcp://0.0.0.0:2375" - "--storage-driver=overlay2" volumeMounts: - name: dind-storage mountPath: /var/lib/docker resources: requests: cpu: "200m" memory: "512Mi" - name: worker image: your-registry/prefect-docker-worker:latest

env: - name: DOCKER_HOST value: tcp://localhost:2375 - name: PREFECT_API_URL value: "http://your-prefect-server/api" args: - prefect - worker - start - -p - utility-docker-pool resources: requests: cpu: "200m" memory: "512Mi"

Build the worker image with Docker support:

# Dockerfile FROM python:3.11-slim RUN pip install "prefect>=3" "prefect[docker]" CMD ["bash"] ``` Which should you choose? - Use Process worker if your tiny flow just calls AWS (Lambda/S3/etc.) and you’re okay running that code directly in the worker container. This gives you the fastest response and simplest setup. - Use Docker worker if you need each run in its own container but still want to avoid Kubernetes job spin-up. Accept the DinD or host-Docker trade-offs. How this helps your automation - Your Automation’s “Run a deployment” action targets the utility work pool (process or docker). Because the worker pod is already running and warm, the run starts immediately — no K8s job creation, no image pull per run. - For the Lambda use-case, the process worker is typically ideal: the tiny flow loads your

LambdaFunction

block and calls

invoke()

instantly. Helpful docs - Work pools and workers: Work pools & workers - AWS integration: Prefect AWS integration - Automations: Creating automations and actions: Automation actions If you share whether your cluster uses Docker or containerd, I can tailor the Docker worker setup (DinD vs host socket) — and provide a minimal image + Helm-style values if you prefer deploying via Helm.