https://prefect.io
Join Slack
Hi team, I noticed this error when trying to deploy my flow to ecs. How would I go about fixing this...
# ask-community
m
Hi team, I noticed this error when trying to deploy my flow to ecs. How would I go about fixing this? I already downloaded Docker onto my cluster
Copy code
[27 July 2021 11:01am]: Unsupported Storage type: Docker
k
Hey @Madison Schott, could you share your Flow storage and run config?
What agent are you using?
m
ecs
k
This error message comes from the local agent. Do you only have another agent that may have picked up the Flow?
m
is it my dockerfile path that's causing issues? Since it's a local path?
yeah maybe
how do I register a flow with a certain agent?
k
Using labels. Agents can only pick up Flows with matching labels. The agent labels must be a superset of flow labels to be able to pick it up
m
How do I set a label for my flow?
k
Like this
m
I believe I have that then
Copy code
with Flow("user_brand_campaigns", storage=STORAGE, run_config=RUN_CONFIG) as user_profile_flow:
the labels are set in RUN_CONFIG
k
What are the labels on the agents that you have running?
m
aren't they just the ones above? ecs and dev? or was I supposed to define these somewhere else as well?
k
So the Flow has labels, and the Agents also have labels. Agents pick up flows that have matching labels. How are you starting up your agents? Using the CLI?
m
using this script
Copy code
from prefect.agent.ecs.agent import ECSAgent

AGENT = ECSAgent(cluster="prefect-prod", labels=['ecs', 'dev'])

AGENT.start()
k
That looks all good. Let me take a look at some stuff.
How many agents do you have running now? Do you have a local agent by chance? Because the error message at the top comes from the local agent, but the ECS agent doesn’t have the same check? I am wondering if you see the flow is actually getting picked up by the ECS agent in the logs when you start the Flow Run
m
Yes I have a local agent also running
How can I stop the local agent and make sure it runs on the ecs agent?
k
You have to stop that process
m
How do I do that? There's no option to remove it
k
So the agent will continue to poll the API even if you remove it from the UI. You need to find where that process in running on your hardware and close it. Did you start it from the CLI or from a Python script also? What is your OS?
m
I had a run agent command in my python script
I just closed all of my terminals and it still has the local agent running
but now it gave me the option to remove from the ui
Now that I only have the ecs agent running, should I rerun the python file with my flow locally?
k
RunConfigs are not respected by 
flow.run()
so you need to do a real run
m
What is considered a real run?
k
Going to the UI and clicking the Quick Run
m
I am still getting the same error
it is still running on the local agent but I don't see any local agents running
k
After you removed the agent from the UI, did it re-appear?
Actually, do you want to try adding more specific labels and testing again so that we know that the ECS agent is really picking it up and we know if there is something wrong with the flow?
m
sure, so do I just rerun the script that creates the agent? And then also run the file where I made changes to the run_config in the flow?
k
Yes exactly with added labels that guarantee the ECSAgent picks up the Flow. Yes if you mean re-register by “run the file” after changing the run_config. Not 
flow.run()
m
Ok I created the new agent now I am running file that contains the register command
Is it a good sign when the flow takes awhile to register? it says 
prefect.Docker | Pushing image to the registry...
k
It’s needed because the Docker container is being built and pushed to ECR, where the flow on ECS will grab it from so it’s certainly not bad 👍
m
Well I know it's working now cause I just got this error 
An error occurred (ClientException) when calling the RegisterTaskDefinition operation: Role is not valid
k
It looks correctly defined in your 
execution_role_arn
. Did you create the role in AWS and is the arn matching?
m
I used the account alias name instead of the id number
so I'm gonna try reregistering using account number in the role
k
One sec we have the roles somewhere. Let me find it for you.
m
thanks!
k
I think this should be a good starting point, but you may need more depending on what resources for Flow hits.
About the ghost local agent though, what is your OS? We can try hunting down that process to kill it.
m
Ok cool so assigning that role that policy should fix this?
I'm on a mac
k
The role specifically did not have permissions to 
registerTaskDefinition
so this should at least get past this error yep. Maybe something else will pop up.
m
Got it 👍
2 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by