Madison Schott

    Madison Schott

    1 year ago
    Hi team, I noticed this error when trying to deploy my flow to ecs. How would I go about fixing this? I already downloaded Docker onto my cluster
    [27 July 2021 11:01am]: Unsupported Storage type: Docker
    Kevin Kho

    Kevin Kho

    1 year ago
    Hey @Madison Schott, could you share your Flow storage and run config?
    What agent are you using?
    Madison Schott

    Madison Schott

    1 year ago
    ecs
    Kevin Kho

    Kevin Kho

    1 year ago
    This error message comes from the local agent. Do you only have another agent that may have picked up the Flow?
    Madison Schott

    Madison Schott

    1 year ago
    is it my dockerfile path that's causing issues? Since it's a local path?
    yeah maybe
    how do I register a flow with a certain agent?
    Kevin Kho

    Kevin Kho

    1 year ago
    Using labels. Agents can only pick up Flows with matching labels. The agent labels must be a superset of flow labels to be able to pick it up
    Madison Schott

    Madison Schott

    1 year ago
    How do I set a label for my flow?
    Kevin Kho

    Kevin Kho

    1 year ago
    Like this
    Madison Schott

    Madison Schott

    1 year ago
    I believe I have that then
    with Flow("user_brand_campaigns", storage=STORAGE, run_config=RUN_CONFIG) as user_profile_flow:
    the labels are set in RUN_CONFIG
    Kevin Kho

    Kevin Kho

    1 year ago
    What are the labels on the agents that you have running?
    Madison Schott

    Madison Schott

    1 year ago
    aren't they just the ones above? ecs and dev? or was I supposed to define these somewhere else as well?
    Kevin Kho

    Kevin Kho

    1 year ago
    So the Flow has labels, and the Agents also have labels. Agents pick up flows that have matching labels. How are you starting up your agents? Using the CLI?
    Madison Schott

    Madison Schott

    1 year ago
    using this script
    from prefect.agent.ecs.agent import ECSAgent
    
    AGENT = ECSAgent(cluster="prefect-prod", labels=['ecs', 'dev'])
    
    AGENT.start()
    Kevin Kho

    Kevin Kho

    1 year ago
    That looks all good. Let me take a look at some stuff.
    How many agents do you have running now? Do you have a local agent by chance? Because the error message at the top comes from the local agent, but the ECS agent doesn’t have the same check? I am wondering if you see the flow is actually getting picked up by the ECS agent in the logs when you start the Flow Run
    Madison Schott

    Madison Schott

    1 year ago
    Yes I have a local agent also running
    How can I stop the local agent and make sure it runs on the ecs agent?
    Kevin Kho

    Kevin Kho

    1 year ago
    You have to stop that process
    Madison Schott

    Madison Schott

    1 year ago
    How do I do that? There's no option to remove it
    Kevin Kho

    Kevin Kho

    1 year ago
    So the agent will continue to poll the API even if you remove it from the UI. You need to find where that process in running on your hardware and close it. Did you start it from the CLI or from a Python script also? What is your OS?
    Madison Schott

    Madison Schott

    1 year ago
    I had a run agent command in my python script
    I just closed all of my terminals and it still has the local agent running
    but now it gave me the option to remove from the ui
    Now that I only have the ecs agent running, should I rerun the python file with my flow locally?
    Kevin Kho

    Kevin Kho

    1 year ago
    RunConfigs are not respected by
    flow.run()
    so you need to do a real run
    Madison Schott

    Madison Schott

    1 year ago
    What is considered a real run?
    Kevin Kho

    Kevin Kho

    1 year ago
    Going to the UI and clicking the Quick Run
    Madison Schott

    Madison Schott

    1 year ago
    I am still getting the same error
    it is still running on the local agent but I don't see any local agents running
    Kevin Kho

    Kevin Kho

    1 year ago
    After you removed the agent from the UI, did it re-appear?
    Actually, do you want to try adding more specific labels and testing again so that we know that the ECS agent is really picking it up and we know if there is something wrong with the flow?
    Madison Schott

    Madison Schott

    1 year ago
    sure, so do I just rerun the script that creates the agent? And then also run the file where I made changes to the run_config in the flow?
    Kevin Kho

    Kevin Kho

    1 year ago
    Yes exactly with added labels that guarantee the ECSAgent picks up the Flow. Yes if you mean re-register by “run the file” after changing the run_config. Not
    flow.run()
    Madison Schott

    Madison Schott

    1 year ago
    Ok I created the new agent now I am running file that contains the register command
    Is it a good sign when the flow takes awhile to register? it says
    prefect.Docker | Pushing image to the registry...
    Kevin Kho

    Kevin Kho

    1 year ago
    It’s needed because the Docker container is being built and pushed to ECR, where the flow on ECS will grab it from so it’s certainly not bad 👍
    Madison Schott

    Madison Schott

    1 year ago
    Well I know it's working now cause I just got this error
    An error occurred (ClientException) when calling the RegisterTaskDefinition operation: Role is not valid
    Kevin Kho

    Kevin Kho

    1 year ago
    It looks correctly defined in your
    execution_role_arn
    . Did you create the role in AWS and is the arn matching?
    Madison Schott

    Madison Schott

    1 year ago
    I used the account alias name instead of the id number
    so I'm gonna try reregistering using account number in the role
    Kevin Kho

    Kevin Kho

    1 year ago
    One sec we have the roles somewhere. Let me find it for you.
    Madison Schott

    Madison Schott

    1 year ago
    thanks!
    Kevin Kho

    Kevin Kho

    1 year ago
    I think this should be a good starting point, but you may need more depending on what resources for Flow hits.
    About the ghost local agent though, what is your OS? We can try hunting down that process to kill it.
    Madison Schott

    Madison Schott

    1 year ago
    Ok cool so assigning that role that policy should fix this?
    I'm on a mac
    Kevin Kho

    Kevin Kho

    1 year ago
    The role specifically did not have permissions to
    registerTaskDefinition
    so this should at least get past this error yep. Maybe something else will pop up.
    Madison Schott

    Madison Schott

    1 year ago
    Got it 👍