https://prefect.io logo
Join Slack
Powered by
hiya <@ULVA73B9P> walk me through specifying `task...
# ask-marvin
e
hiya @Marvin walk me through specifying 
taskRoleArn
on my Flow definition in Prefect.yaml. does it go under 
parameters:
? at the root level? if you can spit out an example with it defined that would be super helpful.
m
✅ thought for 86.9 seconds
Yep — not under 
parameters:
and not at the root. For an ECS work pool in Prefect 3.x, 
taskRoleArn
is exposed as the snake_case job variable 
task_role_arn
, and it goes under:
Copy code
yaml
work_pool:
  job_variables:
    task_role_arn: ...
So in 
prefect.yaml
, it looks like this:
Copy code
yaml
deployments:
  - name: my-ecs-deployment
    entrypoint: flows/my_flow.py:my_flow
    work_pool:
      name: my-ecs-pool
      job_variables:
        task_role_arn: arn:aws:iam::123456789012:role/MyPrefectTaskRole
A slightly fuller example:
Copy code
yaml
name: my-project
prefect-version: 3.0.0

deployments:
  - name: my-flow-dev
    entrypoint: flows/my_flow.py:my_flow

    work_pool:
      name: my-ecs-pool
      job_variables:
        task_role_arn: arn:aws:iam::123456789012:role/MyPrefectTaskRole
        cpu: 1024
        memory: 2048
        env:
          MY_ENV: dev

    parameters:
      some_input: hello
A couple important notes: * 
parameters:
is only for flow function parameters. * Example: if your flow is 
def my_flow(some_input: str)
, then 
some_input
belongs under 
parameters:
. * 
task_role_arn
is infrastructure configuration for the ECS worker/job, so it belongs under 
work_pool.job_variables
. Also worth distinguishing: * 
task_role_arn
= permissions inside the running ECS task * 
execution_role_arn
= permissions ECS uses to start the task (pull image, write logs, etc.) If you want, I can also show you: * a full 
prefect.yaml
for ECS/Fargate, or * how this lines up with the ECS work pool settings in the UI/CLI. Relevant docs: * Prefect docs * prefect.yaml deployment configuration * Deploy with ECS
e
oh I didn't realize it was in the 
work_pool
, that's unfortunate because we use an aliased 
work_pool
definition elsewhere in the yaml file. is there a way to merge the aliased block with this one setting, or do I have to define this one by hand?
hmm @Marvin oh I didn't realize it was in the 
work_pool
, that's unfortunate because we use an aliased 
work_pool
definition elsewhere in the yaml file. is there a way to merge the aliased block with this one setting, or do I have to define this one by hand?
m
✅ thought for 11.2 seconds
Yes — if you’re using YAML anchors/aliases, you can usually merge in the aliased 
work_pool
block and then override/add 
job_variables
. The main thing is that this is YAML behavior, not Prefect-specific behavior. Example with an anchor:
Copy code
yaml
definitions:
  ecs_work_pool: &ecs_work_pool
    name: my-ecs-pool
    job_variables:
      cpu: 1024
      memory: 2048
      env:
        MY_ENV: dev

deployments:
  - name: flow-a
    entrypoint: flows/flow_a.py:flow_a
    work_pool:
      <<: *ecs_work_pool

  - name: flow-b
    entrypoint: flows/flow_b.py:flow_b
    work_pool:
      <<: *ecs_work_pool
      job_variables:
        task_role_arn: arn:aws:iam::123456789012:role/MyTaskRole
But there’s an important gotcha: YAML merge keys are shallow. That means in the 
flow-b
example above, the second 
job_variables:
replaces the entire aliased 
job_variables
mapping — it does not automatically merge nested keys like 
cpu
, 
memory
, etc. So if you do it this way, 
flow-b
would end up with only:
Copy code
yaml
job_variables:
  task_role_arn: ...
and not the inherited 
cpu
, 
memory
, 
env
. To preserve the existing 
job_variables
, you have two good options. Option 1: Make 
job_variables
its own anchor too
Copy code
yaml
definitions:
  ecs_job_variables: &ecs_job_variables
    cpu: 1024
    memory: 2048
    env:
      MY_ENV: dev

  ecs_work_pool: &ecs_work_pool
    name: my-ecs-pool
    job_variables:
      <<: *ecs_job_variables

deployments:
  - name: flow-b
    entrypoint: flows/flow_b.py:flow_b
    work_pool:
      <<: *ecs_work_pool
      job_variables:
        <<: *ecs_job_variables
        task_role_arn: arn:aws:iam::123456789012:role/MyTaskRole
That’s usually the cleanest pattern. Option 2: Define this deployment’s 
work_pool
by hand If it’s only one special deployment, sometimes it’s simpler and more readable to just write out the full 
work_pool
block explicitly. So short answer: * Yes, you can merge the aliased block * But if you need to add one nested 
job_variables
field, you’ll probably want a separate anchor for 
job_variables
 to avoid overwriting the whole nested map If you want, paste your current aliased YAML pattern and I can rewrite it into the minimal working version.
e
@Marvin hmm is it possible to scope definitions to only certain YAML elements? we have something like:
Copy code
definitions:
  work_pools:
    foo: &foo
can I put my new 
job_variables
at the same level as my 
work_pools
then?
m
❌ Error: unhandled errors in a TaskGroup (1 sub-exception)
Sorry, I encountered an error while processing your request. Please try again.
e
@Marvin thanks for the help getting the 
task_role_arn
set in the job variables, that passed through successfully via the prefect.yaml deployment. however, now that it's set, the scheduled flows are not yet picking it up yet. our job variables for this flow contains only 
image
and 
task_role_arn
, and obviously 
image
is being parsed correctly since the flow actually runs, right? or is that not a valid flag as to whether my job variables are set correctly or not? my job variables looks like:
Copy code
{
  "image": "[ECR-image-path]",
  "task_role_arn": "[role-arn]"
}
m
✅ thought for 114.6 seconds
Yep — the fact that 
image
is being used does not by itself prove that 
task_role_arn
is being applied the way you expect. For ECS workers in Prefect 3.x, those two variables land in different places: - 
image
is used in the task definition/container definition - 
task_role_arn
is used in the runtime task request / overrides So it’s possible for: - the task to launch with the correct image - and the task role override to still not be taking effect, or not be visible where you're checking Your JSON shape is valid:
Copy code
{
  "image": "[ECR-image-path]",
  "task_role_arn": "[role-arn]"
}
So the next question is really where you're checking for 
task_role_arn
. Important distinction: - If you inspect the task definition, you may see the image and conclude vars were applied - But 
task_role_arn
is not necessarily something you’ll confirm there the same way; it’s associated with the ECS task at run time A few likely gotchas: 1. Scheduled runs use deployment job variables too - There isn’t special “scheduled runs ignore job variables” behavior in Prefect 3.x - So if the deployment has 
job_variables
, scheduled runs should inherit them 2. You may be looking at the wrong ECS artifact - 
image
showing up in the launched task is expected - To verify role behavior, check the actual ECS task/run details, not just whether the container started 3. The IAM role may not actually be assumable/usable by the task - Even if Prefect passes 
task_role_arn
, the ECS side can still fail to use it due to trust policy / permissions / launch constraints 4. If you’re reusing an existing task definition ARN, behavior can be confusing - In some ECS setups, task definition reuse can make it look like some config is “stuck” while other runtime settings vary If you want, I’d recommend checking the launched ECS task directly with AWS and confirming what task role the running task has. Also, if you share: - whether you’re using an ECS worker / ECS work pool - whether this is Fargate or EC2 - and how you’re verifying that the role is not being picked up (AWS console, task metadata, app permissions failure, etc.) …I can help narrow down exactly where it’s breaking. Relevant docs: - Work pools and job variables - Deploy with ECS - prefect.yaml deployments If you want, paste the relevant part of your 
prefect.yaml
deployment block and how you’re validating the task role, and I can sanity-check it.
e
@Marvin I have definitely traced the specific Fargate task ID. I see the overrides does specify 
taskRoleArn
, however since the base underlying task definition does not specify any role ARN, the override doesn't actually take effect. the net result is that our Prefect flows are missing the permissions defined in the role's policies.
m
✅ thought for 22.6 seconds
Ah — that’s a really useful detail, and it changes the diagnosis. If you can see 
taskRoleArn
in the ECS overrides for the launched Fargate task, but the task still runs without the expected permissions unless the task definition itself has a role set, then this points to an ECS/AWS behavior constraint more than Prefect “dropping” the field. In other words: - Prefect is passing 
task_role_arn
- ECS shows it in the run request / overrides - but for your setup, Fargate is not actually honoring that override unless the task definition already includes a task role So from a practical standpoint, you should not rely on 
task_role_arn
as the only source of task role configuration here. Recommended fix Set the role on the ECS task definition itself, not only as a runtime override. In Prefect terms, that usually means using the ECS work pool’s base job template / worker configuration so the generated task definition includes the role ARN directly, rather than only injecting it at run time. You likely want both of these concepts clearly set: - 
execution_role_arn
— lets ECS pull images / write logs / start the task - 
task_role_arn
— permissions available inside the running container If the base task definition currently has neither, that would line up with what you’re observing. Why 
image
worked but 
task_role_arn
didn’t This is the distinction we were circling around: - 
image
affects task definition/container config, so it’s naturally reflected in what launches - 
task_role_arn
is being sent as a runtime field, but AWS appears not to be applying it effectively in your case unless the task definition itself already declares a role So yes: 
image
being correct is not a valid indicator that the IAM role config is correct. What I’d do next 1. Open your ECS work pool’s job template in Prefect Cloud/UI 2. Ensure the base template includes role configuration for the task definition 3. Redeploy and launch a fresh scheduled run 4. Confirm in AWS that the task definition attached to the task includes the role you expect 5. Then verify the container can actually call the AWS APIs allowed by that role If you want a precise fix, send me either: - the ECS work pool job template, or - the relevant 
prefect.yaml
deployment plus whether the work pool itself defines any ECS defaults because the likely issue is that the role needs to be configured at the work pool template level, not only in deployment 
job_variables
. Relevant docs: - Prefect ECS deployment docs - Work pools and job variables - prefect.yaml deployments If you paste the work pool job template, I can point to the exact field that should hold the task role.
e
yes but @Marvin how do I control the definition of the runtime task? we are already specifying the name of this task definition, and we call it something like 
prefect-running-a-flow
to distinguish it from the Prefect ECS service task definition (I still have the scars from conflating the two). we define the Prefect ECS service task def concretely, but how do we specify the definition of this other task definition which Prefect uses at runtime?
m
✅ thought for 144.5 seconds
Yes — that runtime flow-run task definition is controlled by the ECS work pool’s base job template, not by the Prefect ECS service task definition. So the mental model is: - Prefect ECS service task definition = the ECS service/container running 
prefect worker start ...
- Flow-run task definition = the ECS task definition the worker creates or reuses when it launches a flow run Those are completely separate. How you control the flow-run task definition You have two main options: Option 1: Let Prefect generate the task definition from the work pool template - The ECS work pool has a base job template - That template contains: - the shape of the ECS 
task_definition
- the shape of the ECS 
task_run_request
- the variables deployments are allowed to override via 
job_variables
This is the normal path. Option 2: Point Prefect at an already-existing ECS task definition - Set 
task_definition_arn
in 
job_variables
- Then Prefect uses that existing task definition instead of generating one That is the cleanest option if you want full explicit AWS-side control. --- What you probably want in your case Since you’ve found that your Fargate tasks only behave correctly when the task definition itself has the role set, you likely want one of these approaches: Approach A: Use a prebuilt ECS task definition
Copy code
yaml
deployments:
  - name: my-flow
    entrypoint: flows/my_flow.py:my_flow
    work_pool:
      name: my-ecs-pool
      job_variables:
        task_definition_arn: arn:aws:ecs:us-east-1:123456789012:task-definition/prefect-running-a-flow:7
In that model: - you create/manage 
prefect-running-a-flow
in AWS yourself - Prefect just launches tasks from it If you want strict control over IAM/task definition contents, this is often the easiest route. --- Approach B: Customize the ECS work pool base job template If you want Prefect to generate the runtime task definition, then the source of truth is the work pool’s base job template. You can inspect the default ECS template with:
Copy code
bash
prefect work-pool get-default-base-job-template --type ecs
Verified CLI: - 
prefect work-pool get-default-base-job-template --type ecs
- 
prefect work-pool update <name> --base-job-template <file>
Then you can update your pool with a customized template:
Copy code
bash
prefect work-pool update my-ecs-pool --base-job-template ecs-template.json
In that template, the important section is the 
job_configuration.task_definition
object. That is the template Prefect uses to register the flow-run task definition. --- Important nuance for your role issue Out of the box, Prefect’s ECS worker template puts: - 
executionRoleArn
on the task definition - 
taskRoleArn
on the run request overrides But based on what you found, you want 
taskRoleArn
also/instead on the task definition itself. So you’d need to customize the base job template so the generated 
task_definition
includes it. Conceptually, you want the generated task definition to look more like:
Copy code
json
{
  "family": "{{ family }}",
  "cpu": "{{ cpu }}",
  "memory": "{{ memory }}",
  "executionRoleArn": "{{ execution_role_arn }}",
  "taskRoleArn": "{{ task_role_arn }}",
  "containerDefinitions": [
    {
      "name": "{{ container_name }}",
      "image": "{{ image }}"
    }
  ]
}
Then deployment-level 
job_variables.task_role_arn
can fill that value into the task definition itself. --- Where to edit this You can do this either: 1. In the UI Go to your ECS work pool and edit its base job template 2. Via CLI - export the default template - modify it - update the pool Example workflow:
Copy code
bash
prefect work-pool get-default-base-job-template --type ecs --file ecs-template.json
Edit 
ecs-template.json
, then:
Copy code
bash
prefect work-pool update my-ecs-pool --base-job-template ecs-template.json
--- What deployments can and cannot do
Your 
prefect.yaml
deployment 
job_variables
can only fill values into fields that the work pool template exposes. So if your work pool template does not include 
taskRoleArn
in the 
task_definition
section, then this deployment config:
Copy code
yaml
work_pool:
  name: my-ecs-pool
  job_variables:
    task_role_arn: arn:aws:iam::123456789012:role/my-flow-role
will only affect places where the template uses 
{{ task_role_arn }}
. That is why the work pool template is the real control point here. --- Recommendation Given your findings, I’d recommend one of these: - Fastest / most deterministic: use a pre-created ECS task definition via 
task_definition_arn
- More Prefect-native / flexible: customize the ECS work pool base job template so 
taskRoleArn
is present in 
job_configuration.task_definition
If you want, I can help you draft the exact 
ecs-template.json
diff so that: - the runtime-generated task definition includes 
taskRoleArn
- deployments can keep setting 
task_role_arn
from 
prefect.yaml
- and you avoid mixing this up with the Prefect worker/service task definition again Docs: - Work pools - ECS deployments - prefect.yaml deployments
e
@Marvin if we have over 33,000 task definition revisions of the running-a-flow task def, can I just delete the task def altogether and let a Prefect redeploy recreate it from prefect.yaml? it chooses a historical revision at random by default, right? I'm aware of the setting to use the latest revision, yes.
m
✅ thought for 102.3 seconds
Short answer: no, Prefect does not choose a historical revision at random by default. If Prefect is using a family name for the generated runtime task definition, AWS resolves that to the latest ACTIVE revision, not a random old one. So the behavior is: - not random - generally latest active revision - unless you explicitly point Prefect at a specific 
task_definition_arn
, in which case it uses that exact revision On deleting the task definition family entirely If you are not using 
task_definition_arn
, and instead letting Prefect generate the runtime task definition from the ECS work pool / 
prefect.yaml
config, then yes — on the next run, Prefect should register a new task definition again. So in that generated-task-definition model: - delete/deregister old revisions - next eligible flow run should cause Prefect to create a fresh revision from the work pool base job template + deployment 
job_variables
But there’s one important caveat: - 
prefect.yaml
itself does not directly create the ECS task definition in AWS at deploy time - the worker creates or resolves the runtime ECS task definition when a flow run is launched So the recreation happens on next run, not necessarily at 
prefect deploy
time. Important distinction 1. Generated task definition - Prefect builds task definition from work pool template - if missing, it can recreate it 2. Explicit 
task_definition_arn
override - Prefect expects that task definition to already exist - if you delete it, runs will fail until you provide a valid ARN again So whether you can safely delete depends on which model you’re using. For your case Since you said you’re specifying the name of the runtime family like 
prefect-running-a-flow
, that sounds like the generated task-definition path using 
family
, not a hardcoded 
task_definition_arn
. If that’s true, then yes, deleting/deregistering that family’s revisions should let Prefect recreate a new one on the next run. Recommendation before cleanup Given 33,000+ revisions, I’d be cautious and verify these first: - confirm you are not setting 
task_definition_arn
- confirm the ECS work pool base job template is valid - confirm the work pool template includes the task definition fields you now want, especially role settings - trigger one manual run after cleanup to confirm the new revision is created as expected One more note on 
match_latest_revision_in_family
 That setting affects whether Prefect tries to reuse the latest matching revision in the family instead of registering another one unnecessarily. It does not mean “choose some arbitrary old revision.” Without a specific ARN, the family lookup resolves to the latest active revision. Useful docs: - ECS deployments - Work pools - prefect.yaml deployments If you want, I can help you sanity-check the exact cleanup plan and tell you whether you should: - delete old revisions, - switch to 
task_definition_arn
, - or update the work pool template first so the recreated revision has the correct 
taskRoleArn
on the task definition.
2 Views
Open in Slack
PreviousNext
Powered by