Afternoon all, having a problem getting a Flow to run on

FargateTaskEnvironment

. I have a self hosted Prefect 0.12.2 instance running on AWS, and am trying to run a flow against Fargate, but getting the following error that seems to only relate to Cloud hosted versions. Any ideas? From fargate_task.py287

container_overrides = [
    {
        "name": "flow-container",
        "environment": [
            {
                "name": "PREFECT__CLOUD__AUTH_TOKEN",
                "value": config.cloud.agent.auth_token
                or config.cloud.auth_token,
            },
            {"name": "PREFECT__CONTEXT__FLOW_RUN_ID", "value": flow_run_id},
            {"name": "PREFECT__CONTEXT__IMAGE", "value": get_flow_image(flow)},
        ],
    }
]

Hi @Darragh Is this the first time you are running the flow in 0.12.2?

Hey @itay livni First time on 0.12.2 and using the

FargateTaskEnvironment

Yeah. I ran into a similar message upgrading. My "Environment": Cloud, FargateAgent. LocalExecutor. The issue may be your runner token.

Thanks @itay livni !! My case is slightly different in that I’m using self hosted rather than Cloud, so I’ve no idea why it’s looking for a cloud runner token…

@Darragh what is the error?

BoxKeyError(“‘Config’ object has no attribute ‘auth_token’“,)

Ah I see, will send in a fix. FWIW this environment is very old and is going to be deprecated soon for a better pattern so I wouldn’t expect it to be maintained much longer 🙂

Thanks Josh!!

This ticket outlines the idea: https://github.com/PrefectHQ/prefect/issues/2928

Hmm, interesting… so if I assume that the config for an agent [FargateAgent in my case] will allow for more flexibility, does that mean I should still be able to have flow specific config? Main reason I’m using the FargateTaskEnvironment is to be able to have 2 containers running side by side. A contains the Flow, and then reaches out to B to do “stuff”. No idea if you’d get that same level of flexibility just using a generic Agent?

Yep you will be able to configure each fargate task on a flow-specific level

Great! If the Enrionment is going away, what would replace it in this case such that I could do that config? Or is that still in planning? Not 100% clear to me from the PR..

Still in planning but the thought is that environment type specs (k8s manifests, fargate task defs, etc.) can all be loaded/templated onto the agent and then used when flow metadata matches. Things like executors may move to the flow object directly. Still being thought out though

Ok - but the agent would be using those templates for multiple Flows right? What if someone needed the flexibility that the current Env stuff goes, where I can have completely different specs per Flow?

Yep it will. In that issue it has it outlined in a way where you could have an agent that has two job specs on it and then when you register your flow you say

metadata["job_spec_1"]

or something and it would use that specific job spec

Right, then in my case that means I need to have a job spec for every Flow, cos they’re all going to be configured slightly differently

Ah didn’t see your message! I think you would need to do two things, pip install master locally where you build your flow and also set the prefect_version to master in the Docker storage

Great thanks! Make slif easier, I cna leave the server alone 😁

🤔 Interesting. I’m thinking that maybe there could be some Docker caching happening somewhere during your build step. Do you see any output during your build step from this command? https://github.com/PrefectHQ/prefect/blob/0b3ef62bc99c5901b1fb9ae77142f7e5c5c74187/src/prefect/environments/storage/docker.py#L162-L166

Yep, I get the following:

Step 3/9 : RUN pip show prefect || pip install git+<https://github.com/PrefectHQ/prefect.git@master#egg=prefect[kubernetes]>

What is the output following that line?

Full output here

Ah yeah if your base image

<http://registry.gitlab.com/STUFF/parallell:latest|registry.gitlab.com/STUFF/parallell:latest>

already has prefect baked into it then it won’t install from master. Kind of weird because I use a default prefect image which I feel like always has prefect in it already and it installs master 🤔 edit: I use the

python:3.7-slim

image as my base

Yeah didn’t spot it till after - we use a custom base so we can layer some extra pieces in for all of our flows. This is the current custom image base:

FROM prefecthq/prefect:0.12.2-python3.6

What would I need to change to so I can get master?

You could just change that base to

prefecthq/prefect:master

Hmm thought I’d tried that… probably not though 😄 Gimme a minute and I’ll confirm..

I believe it’s a string. Whatever boto3 normally expects https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html

Sorry I meant is there any escpaing etc… to be done on the session_token before passing it in? I’m not sure AWS is liking the formatting…

launch_type="FARGATE",
region="us-east-1",
aws_session_token="FwoGZXIvYXdzEOL//////////wEaDFSgRMmr3M07yXJ3gCKCAdZO6f/LRZc6b7DjDip0lrTvCa+FDQpFAGJEyB6Ka1tF9By3fTgKkbqSM6EnuHQgTEviQJrOn13E7wlvKKVV1++YGaa3gb1Pn9q12BxCN7I6SvQ8oBW9AE73Judo0tuYTdTc5eYC7m2PaYU/d5fkRIj29EJWp9EpO3+yq/S1saxiGvYosKGy+AUyKBOOerH4ymGN9lxo/5aprU5DXumyaC6yg2satDNNoUdPaSBQ2R8fNq0=",
cpu="256",
memory="512"

Tbh not sure, haven’t really used session tokens (don’t run much long running stuff on fargate)

Thanks Josh, no problem, I know some of my questions are a bit obscure 🙂 I’ll open it back up to the wider group!