Short answer: yes, pull steps can run arbitrary shell commands, so they can make HTTP requests. But the important caveat is: pull steps are not the right place to set infrastructure/job variables for the current run like CPU/memory requests/limits. Why? In Prefect 3.x, pull steps are primarily for retrieving/preparing flow code in the execution environment: - clone/pull code - set working directory - install dependencies - run shell scripts From the available step APIs, Prefect includes built-ins like: -

prefect.deployments.steps.pull.git_clone(...)

-

prefect.deployments.steps.pull.pull_from_remote_storage(...)

-

prefect.deployments.steps.utility.run_shell_script(...)

So if your question is “can I call an external service from pull steps?” the answer is yes, for example via

curl

in

run_shell_script

. However, if your question is “can that pull step return dynamic job variables that change the pod/container resources for this same flow run?” then generally no. Reason: Work pool / worker infrastructure is selected and created before the flow code is pulled and executed. CPU/memory requests/limits are part of the worker’s job configuration / infrastructure creation. By the time pull steps run, the worker has already launched the job with whatever variables/config it had for that run. So the sequence is roughly: 1. Flow run is created 2. Worker reads the deployment + work pool/job variables 3. Worker creates the infrastructure job 4. Inside that environment, Prefect executes pull steps to fetch/prepare code 5. Flow starts That means pull steps can affect: - code available in the runtime - files - env vars within the running job - dependency installation But they cannot retroactively resize the already-created infrastructure for that same run. What you can do instead 1. Set job variables before creating/submitting the flow run If your external service knows the right CPU/memory, the cleanest pattern is: - call your service first - create the flow run / trigger the deployment with those job variables already supplied 2. Use deployment/work pool variables templating if your trigger path can inject values If you trigger runs programmatically, pass per-run infrastructure overrides there. 3. Have your external service create/update deployments dynamically This is useful if sizing changes less frequently than every run. 4. Route runs to different work pools or queues For example: - small - medium - large Your service decides which one to use before submission. If you do just want to call an HTTP endpoint during pull steps for setup purposes, a shell step can do that, e.g.:

yaml
pull:
  - prefect.deployments.steps.utility.run_shell_script:
      script: |
        curl -X POST <https://your-service.example/api/prefect/pull-hook> \
          -H "Authorization: Bearer $SERVICE_TOKEN" \
          -H "Content-Type: application/json" \
          -d '{"deployment":"my-deployment"}'

But again, that is useful for runtime setup, not for changing Kubernetes/ECS/Docker resource settings for the current run. I wasn’t able to retrieve a matching GitHub discussion/issue for this exact pattern, but the relevant step APIs I verified are in: -

prefect.deployments.steps

-

prefect.deployments.steps.pull

-

prefect.deployments.steps.utility

If you want, I can help you design the correct pattern for your setup based on how you launch runs: - via UI - via API - via

prefect deploy

- via an automation - via Kubernetes/ECS worker If you tell me which worker type/work pool you’re using, I can sketch the best way to inject dynamic CPU/memory per run.

Yes — with a Kubernetes work pool, there’s no built-in way for a deployment to compute/update its own job variables for the current run unless those values are provided before the worker creates the job. So for your specific question: No, you cannot update CPU/memory job variables “from inside the deployment” for that same execution unless they were already supplied when the run was created. That’s because the k8s worker needs those values up front to build the Kubernetes Job spec. What this means in practice For a given flow run, resource values can come from: - deployment defaults - work pool defaults - values provided at run creation time, such as custom job variables But once the worker has started creating the pod/job, pull steps and flow code are too late to change: -

resources.requests.cpu

-

resources.requests.memory

-

resources.limits.cpu

-

resources.limits.memory

So if you want them to vary by dataset, Prefect needs that information before submission. So is there any way without passing them as explicit inputs? Not really for the current run’s infrastructure. You need some pre-run mechanism. Your options are basically: 1. Pass job variables at run creation time This is what your service is already doing, and it’s the most direct solution. 2. Use separate deployments for different resource profiles Example: -

my-flow-small

-

my-flow-medium

-

my-flow-large

3. Use separate work pools / queues for resource classes Your triggering logic chooses the target pool/queue. 4. Use deployment parameters as the source of truth, and have an external launcher translate them into job variables For example, dataset metadata implies a size class, and your launcher converts that to k8s resources. 5. Overprovision a bit and let the flow handle variability internally Less efficient, but simpler operationally. Important distinction: parameters vs job variables - Parameters are available to the flow code - Job variables are available to the worker when creating infrastructure Prefect does not automatically derive job variables from flow parameters at run time in the worker. So if your question is:

can Prefect inspect the parameter/data and then decide pod memory/cpu before starting the pod?

The answer is not by itself inside a single deployment execution path. Patterns that usually work best for this If you want to keep one logical deployment, the cleanest architecture is: 1. User/UI/API chooses dataset or passes dataset id 2. A lightweight launcher determines required resources 3. The launcher creates the actual run with the right

job_variables

That launcher could be: - your existing service - another Prefect flow - a small API wrapper around Prefect If you want users to keep using the Prefect UI directly, then the main limitation is exactly what you noticed: they must manually provide the job variable overrides unless you split things into multiple pre-sized deployment options. Practical recommendation For k8s, I’d usually recommend one of these two: Option A — keep your service as the launcher Best if you need fine-grained sizing per dataset. Option B — define a few resource tiers and expose separate deployments or queues Best if users often launch from the UI and you want a simpler experience. For example: -

dataset-processing / small

-

dataset-processing / medium

-

dataset-processing / large

Then your external sizing logic just maps data to a tier, instead of exact CPU/memory every time. If you want, I can show you: - how to pass

job_variables

when creating a run programmatically for a k8s work pool, or - how to model this as small/medium/large deployments in Prefect Cloud/UI.