Afternoon all, having a problem getting a Flow to ...
# ask-community
d
Afternoon all, having a problem getting a Flow to run on
FargateTaskEnvironment
. I have a self hosted Prefect 0.12.2 instance running on AWS, and am trying to run a flow against Fargate, but getting the following error that seems to only relate to Cloud hosted versions. Any ideas? From fargate_task.py287
Copy code
container_overrides = [
    {
        "name": "flow-container",
        "environment": [
            {
                "name": "PREFECT__CLOUD__AUTH_TOKEN",
                "value": config.cloud.agent.auth_token
                or config.cloud.auth_token,
            },
            {"name": "PREFECT__CONTEXT__FLOW_RUN_ID", "value": flow_run_id},
            {"name": "PREFECT__CONTEXT__IMAGE", "value": get_flow_image(flow)},
        ],
    }
]
i
Hi @Darragh Is this the first time you are running the flow in 0.12.2?
d
Hey @itay livni First time on 0.12.2 and using the
FargateTaskEnvironment
i
Yeah. I ran into a similar message upgrading. My "Environment": Cloud, FargateAgent. LocalExecutor. The issue may be your runner token.
I am looking for the thread that answered the same question
d
Thanks @itay livni !! My case is slightly different in that I’m using self hosted rather than Cloud, so I’ve no idea why it’s looking for a cloud runner token…
Still hitting the problem above, dunno if any of the Prefect guys can shed any light?
j
@Darragh what is the error?
d
BoxKeyError(“‘Config’ object has no attribute ‘auth_token’“,)
j
Ah I see, will send in a fix. FWIW this environment is very old and is going to be deprecated soon for a better pattern so I wouldn’t expect it to be maintained much longer 🙂
d
Thanks Josh!!
When you say deprecated, what’s gonna replace it? he says as he trembles in fear…
😆 1
j
d
Hmm, interesting… so if I assume that the config for an agent [FargateAgent in my case] will allow for more flexibility, does that mean I should still be able to have flow specific config? Main reason I’m using the FargateTaskEnvironment is to be able to have 2 containers running side by side. A contains the Flow, and then reaches out to B to do “stuff”. No idea if you’d get that same level of flexibility just using a generic Agent?
j
Yep you will be able to configure each fargate task on a flow-specific level
d
Great! If the Enrionment is going away, what would replace it in this case such that I could do that config? Or is that still in planning? Not 100% clear to me from the PR..
j
Still in planning but the thought is that environment type specs (k8s manifests, fargate task defs, etc.) can all be loaded/templated onto the agent and then used when flow metadata matches. Things like executors may move to the flow object directly. Still being thought out though
d
Ok - but the agent would be using those templates for multiple Flows right? What if someone needed the flexibility that the current Env stuff goes, where I can have completely different specs per Flow?
Possibility I’m missing a level of abstraction here 🙂
j
Yep it will. In that issue it has it outlined in a way where you could have an agent that has two job specs on it and then when you register your flow you say
metadata["job_spec_1"]
or something and it would use that specific job spec
d
Right, then in my case that means I need to have a job spec for every Flow, cos they’re all going to be configured slightly differently
@josh I see your cahnge got merged - how do I uptake? Do i do just pass
prefect_version="master"
in my DockerStorage, or other?
Hey @josh Sorry to annoy but how do I get that change working for me? I thought I just had to do the above with the prefect version, but no joy..
j
Ah didn’t see your message! I think you would need to do two things, pip install master locally where you build your flow and also set the prefect_version to master in the Docker storage
d
Great thanks! Make slif easier, I cna leave the server alone 😁
Hey @josh back again - that change doesn’t seem to have done much for me. Made the following changes with no effect: •
flow.storage = Docker(…., prefect_version="master")
• Local installation for build/register:
pip install git+<https://github.com/PrefectHQ/prefect.git@master>
• Server updatedL
pip install git+<https://github.com/PrefectHQ/prefect.git@master>
And still
Failed to load and execute Flow's environment: BoxKeyError("'Config' object has no attribute 'auth_token'",)
j
🤔 Interesting. I’m thinking that maybe there could be some Docker caching happening somewhere during your build step. Do you see any output during your build step from this command? https://github.com/PrefectHQ/prefect/blob/0b3ef62bc99c5901b1fb9ae77142f7e5c5c74187/src/prefect/environments/storage/docker.py#L162-L166
d
Yep, I get the following:
Step 3/9 : RUN pip show prefect || pip install git+<https://github.com/PrefectHQ/prefect.git@master#egg=prefect[kubernetes]>
j
What is the output following that line?
d
Full output here
Untitled
Seems to be a line in there suggesting it’s still using 0.12.1
j
Ah yeah if your base image
<http://registry.gitlab.com/STUFF/parallell:latest|registry.gitlab.com/STUFF/parallell:latest>
already has prefect baked into it then it won’t install from master. Kind of weird because I use a default prefect image which I feel like always has prefect in it already and it installs master 🤔 edit: I use the
python:3.7-slim
image as my base
d
Yeah didn’t spot it till after - we use a custom base so we can layer some extra pieces in for all of our flows. This is the current custom image base:
Copy code
FROM prefecthq/prefect:0.12.2-python3.6
What would I need to change to so I can get master?
j
You could just change that base to
prefecthq/prefect:master
d
Hmm thought I’d tried that… probably not though 😄 Gimme a minute and I’ll confirm..
Ok so that got me to next problem, missing parameter in taskDefinition, doesn’t tell me which one though…
@josh Any idea what format the aws_session_token should be? Keep getting
The security token included in the request is invalid'
j
I believe it’s a string. Whatever boto3 normally expects https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html
d
Sorry I meant is there any escpaing etc… to be done on the session_token before passing it in? I’m not sure AWS is liking the formatting…
Copy code
launch_type="FARGATE",
region="us-east-1",
aws_session_token="FwoGZXIvYXdzEOL//////////wEaDFSgRMmr3M07yXJ3gCKCAdZO6f/LRZc6b7DjDip0lrTvCa+FDQpFAGJEyB6Ka1tF9By3fTgKkbqSM6EnuHQgTEviQJrOn13E7wlvKKVV1++YGaa3gb1Pn9q12BxCN7I6SvQ8oBW9AE73Judo0tuYTdTc5eYC7m2PaYU/d5fkRIj29EJWp9EpO3+yq/S1saxiGvYosKGy+AUyKBOOerH4ymGN9lxo/5aprU5DXumyaC6yg2satDNNoUdPaSBQ2R8fNq0=",
cpu="256",
memory="512"
j
Tbh not sure, haven’t really used session tokens (don’t run much long running stuff on fargate)
d
Thanks Josh, no problem, I know some of my questions are a bit obscure 🙂 I’ll open it back up to the wider group!