https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-community
  • d

    Dev Dasgupta

    01/31/2022, 5:13 PM
    Hi All, I'm new to prefect and after making a few refactoring to my existing flow.py. Initially, the folder structure looked the following and it was getting registered successfully (
    prefect register --project my_project -p bar
    )
    foo_root_folder
    ├── README.md
    ├── bar
    │   └── flow.py
    ├── pyproject.toml
    ├── requirements.txt
    ├── setup.cfg
    Since the file
    flow.py
    was cluttered with all utils functions required for the flow execution I refactored it into a separate python package and made calls to those functions in
    flow.py
    from bar.utils import get_some_useful_function
    The resultant folder looked like the following:
    foo_root_folder
    ├── README.md
    ├── bar
    │   ├── __init__.py
    │   ├── utils.py
    │   └── flow.py
    ├── pyproject.toml
    ├── requirements.txt
    ├── setup.cfg
    When I try to register now I am getting error as
    ModuleNotFoundError: No module named 'bar'
    Can anyone please let me know if anyone faced similar issues before and how they resolved it? Thanks
    k
    2 replies · 2 participants
  • a

    Andrew Lawlor

    01/31/2022, 5:38 PM
    im trying out prefect for our etl pipelines. we use GCP as our cloud provider. Id like to use prefect cloud and run agents in GCP. is the best option for doing this the vertex agent? If I do use that, what is the process for getting started? do i just create the agents at the beginning via cli and then leave them running, or do i have to continuously update them? is there a way to set up ci/cd for my flow code?
    k
    g
    108 replies · 3 participants
  • k

    Keith Veleba

    01/31/2022, 5:53 PM
    Does anyone know how to run an AWS batch job and wait for completion?
    k
    13 replies · 2 participants
  • d

    David Yang

    01/31/2022, 6:46 PM
    Hi All, anyone can get flows from Azure DevOps service successfully using git storage? I got this error "Failed to load and execute Flow's environment: HTTPUnauthorized('No valid credentials provided')". It seems that git storage doesn't support private access token to access azure devops? This is my code: flow.storage = Git(     repo="xxxxxxx/prefect/_git/prefect",                            # name of repo     flow_path ="./flows/azuredevops.py",                    # location of flow file in repo     branch_name ="main",     repo_host="xxxxx@dev.azure.com",                    # repo host name     git_token_secret_name ="azuresecret" )
    k
    x
    14 replies · 3 participants
  • k

    Keith Veleba

    01/31/2022, 7:16 PM
    when you put AWS_CREDENTIALS in via the UI as a Secret, do you use the text or the json options, and do you put it in as a JSON snippet?
    a
    k
    22 replies · 3 participants
  • c

    Chris Hemphill

    01/31/2022, 8:05 PM
    Hey everyone! Chris Hemphill from Actium Health
    👋 5
    k
    c
    +1
    4 replies · 4 participants
  • d

    Dave

    01/31/2022, 8:21 PM
    Hi team, our team has recently deployed prefect on gcp kubernete. and it keeps saying
    Couldn't connect to *Prefect Server* at *<http://34.82.119.23:4200/graphql>
    ,* can anyone help?
    k
    m
    +2
    106 replies · 5 participants
  • l

    Leon Kozlowski

    01/31/2022, 9:10 PM
    What will happen if I deploy a new version of a flow while a previous version is in a running state?
    a
    6 replies · 2 participants
  • d

    Daniel Kornhauser

    01/31/2022, 10:26 PM
    Can anybody point to a relatively large project that uses prefect in github to checkout their directory tree structure ? Wondering how they group and name the python files of their flows and tasks when they have a lot of them.
    k
    5 replies · 2 participants
  • y

    Yorrick Mendes

    01/31/2022, 10:34 PM
    Hey prefectors, I’ve got a rather basic question. Is it possible to add 2 flows as tasks/dependencies to a main flow? It doesn’t look like it’s possible besides unpacking each flow and adding all the tasks and dependencies individually. The other way to go about it would be to add two tasks that each run a flow locally but that doesn’t seem like the cleanest solution and I’d probably lose a lot of the UI sugar with that approach
    k
    2 replies · 2 participants
  • s

    Suresh R

    02/01/2022, 7:40 AM
    Hi! Each mapped task is getting executed twice in my flow, What might be the reason?
    INFO
    CloudTaskRunner
    Task 'collate-quarterly-crdlotcookie[5]': Starting task run...
    INFO
    CloudTaskRunner
    Task 'collate-quarterly-crdlotcookie[5]': Starting task run...
    a
    8 replies · 2 participants
  • v

    Vinoth Kumar

    02/01/2022, 8:03 AM
    Hi unable to install prefect Orion . I am getting ImportError: cannot import name 'ThreadedChildWatcher' from 'asyncio'
    a
    31 replies · 2 participants
  • t

    Tony Waddle

    02/01/2022, 12:03 PM
    Sorry if I'm missing something obvious - how do you cancel a running agent? Can't see it in docs, -h or ui
    a
    3 replies · 2 participants
  • m

    Muddassir Shaikh

    02/01/2022, 12:31 PM
    The slack alerts sent by state handlers
    handler = slack_notifier(only_states=[Failed])
    do not contain details of the flow name and the taskname, how can we add that?
    a
    2 replies · 2 participants
  • k

    Konstantin

    02/01/2022, 12:41 PM
    Hello everyone, I can't figure out how to use native libraries in Prefect, for example "prefect.tasks.gsheets"
    Give an example of the code for using this class
    a
    26 replies · 2 participants
  • m

    Muddassir Shaikh

    02/01/2022, 12:54 PM
    The search bar is not able to find the taskname when the task name are created from a function. Example: PREFECT_SPLIT_FILES..... is a taskname but when searched doesn't appear. Does it have something to do with task_name and task_run_name ?
    a
    j
    +1
    26 replies · 4 participants
  • m

    Madhup Sukoon

    02/01/2022, 1:57 PM
    Hello! I am extremely new to Prefect, so apologies if this is a rookie question or has already been answered. I have a repository containing a bunch of scripts, folders and modules. I want to import a subset of the functions defined here and build a Flow in a new
    flow.py
    file and then run it via a Docker Agent. I also have some external (pip-installable) dependencies that would need to be present for the flow to run. Is it possible to do this? Does someone have an example Flow that uses Github Storage and Docker Run?
    k
    a
    14 replies · 3 participants
  • v

    Vladimir Bolshakov

    02/01/2022, 2:09 PM
    Hi! Is it possible to customise base URL of Orion UI in static builds? I have started standalone Orion server (without agent) and UI is not refreshed, because of CORS errors. UI app try to fetch data from
    localhost:4200
    . I think the reason is here: https://github.com/PrefectHQ/prefect/blob/orion/orion-ui/src/plugins/api.ts#L198
    a
    3 replies · 2 participants
  • m

    Madhup Sukoon

    02/01/2022, 5:24 PM
    Hi! Just wondering if it's possible (maybe using
    env
    or
    host_config
    ) to have the Docker Agent install packages in the docker container at runtime? (Looking for something similar to Docker Storage's
    python_dependencies
    , but for runtime)
    k
    4 replies · 2 participants
  • c

    Constantino Schillebeeckx

    02/01/2022, 5:44 PM
    I've got a question about the ECS Agent documentation, in this section, there is a "Agent role policy" example, but there is no mention (I think) of where that gets used. The documentation only goes through the "execution role" and the "task role", where does "agent role" come into play?
    a
    k
    12 replies · 3 participants
  • m

    Madhup Sukoon

    02/01/2022, 6:16 PM
    I am using Prefect Cloud. My task Goes to
    Scheduled
    >
    Submitted
    and then nothing happens. The only entry in logs is:
    Submitted for execution: Container ID: 143cabe06c721.....
    My flow uses GitHub Storage and DockerRun, and uses some secrets. Any way I could get more verbose logs on what might be happening? This is what my
    flow.py
    looks like:
    from main import planner, creator, publisher
    
    planner = task(planner)
    creator = task(creator)
    publisher = task(publisher)
    
    foo = Secret('foo')
    bar = Secret('bar')
    
    with Flow(
        "myflow",
        run_config = DockerRun(
            env={"EXTRA_PIP_PACKAGES": "requests pyyaml"}
        ),
        storage = GitHub(
            repo="me/myrepo",
            path="flow.py",
            access_token_secret="GITHUB_ACCESS_TOKEN"
        )
        
    ) as flow:
        plan = planner()
        media = creator(foo, bar, plan)
        post = publisher(foo, bar, media)
    main
    is an adjacent python file (
    main.py
    )
    k
    a
    56 replies · 3 participants
  • d

    Davoud Hejazi

    02/01/2022, 9:22 PM
    I want to add slack notifications in case there is an error while running the flow. I have the lines below as part of my code:
    from prefect.utilities.notifications import slack_notifier
    from prefect.engine.state import Failed
    SLACK_WEBHOOK_URL = "the slack secret webhook url"
    # os.environ["PREFECT__CONTEXT__SECRETS__SLACK_WEBHOOK_URL"] = "<https://hooks.slack.com/services/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>"
    handler = slack_notifier(only_states=[Failed])
    prefect.context.setdefault("secrets", {})
    prefect.context.secrets["SLACK_WEBHOOK_URL"] = "the slack secret webhook url"
    I have even created config.toml file in .predict directory with the content below:
    [context.secrets]
    SLACK_WEBHOOK_URL = "the slack secret webhook url"
    My task stats with:
    @task(nout=2, max_retries=1, retry_delay=timedelta(seconds=1), checkpoint=False, state_handlers=[handler])
    I also tried to add prefect app to slack. But when running my flows I am seeing this error (part of the error): see thread To make it clear, when I remove the
    state_handlers=[handler]
    from my task definition the flow runs fine. When I visit the prefect app in slack I see this too, which I am not sure if it connected to my problem or not. Any comments on how I can solve the slack notification problem?
    k
    46 replies · 2 participants
  • k

    Kevin Mullins

    02/01/2022, 10:33 PM
    Any advice on how to troubleshoot a job being scheduled but not submitted to Kubernetes? I have Prefect Core 0.15.11 with the agent installed in k8s. I ran a first flow which is running still, and then triggered a separate flow that got scheduled in the prefect UI but not submitted to K8S. I don’t say any logs from the agent saying it is deploying the flow run with that ID.
    k
    9 replies · 2 participants
  • s

    Shaoyi Zhang

    02/01/2022, 11:49 PM
    Is there a timeout for submitted flow runs with Kubernetes agent when there’s not enough resources in the k8s namespace?
    k
    m
    6 replies · 3 participants
  • h

    Horatiu Bota

    02/02/2022, 8:37 AM
    hi community & thanks for the help! i've got a parameters json file that i want to pass to a prefect flow using
    flow.run(parameters=json.loads(open(file).read()))
    -- the parameters file looks something like this:
    {
        "other_params": {},
        "company_params": {
            "company_id": "ID1",
            "company_name": "Company",
            "datasets": [
                {
                    "dataset_label": "label",
                    "dataset_url": "URL"
                },
            ]
        }
    }
    is there any way to access nested values in the actual flow? something like:
    with Flow("flow") as flow:
            company_params = Parameter("company_params")
            datasets = company_params.get("datasets")
    a
    1 reply · 2 participants
  • s

    Suresh R

    02/02/2022, 10:45 AM
    Hi!, I have a mapped task with 21 childs, also set task concurrency of 5, But only 2 child tasks are scheduled at a time, What might be the cause?
    a
    k
    35 replies · 3 participants
  • c

    Chris Arderne

    02/02/2022, 12:09 PM
    @task
    decorator has
    log_stdout=True
    but is there any way to add
    stderr
    to logging?
    a
    m
    41 replies · 3 participants
  • s

    shijas km

    02/02/2022, 2:50 PM
    Hi I am facing an issue I need to add a shell task before my python tasks in a flow but its not working , throwing some error like flow outside cant be called but when i comment out all task except the shell task it works can I combine both shell task and python task in a single flow? any example??
    k
    n
    9 replies · 3 participants
  • c

    Constantino Schillebeeckx

    02/02/2022, 3:36 PM
    is there some sort of API endpoint that we can hit which serves as a health check? I'm wondering how I can programmatically check that the agent is healthy
    m
    a
    +1
    17 replies · 4 participants
  • d

    David Wang

    02/02/2022, 4:02 PM
    Hi everyone, I am currently trying to setup an ECS agent with aws. However, the task seems to stop and exit with the error
    essential container in task exited
    . The prefect ecs agent also does not appear in the UI. I have been able to start the ECS agent locally, but am unable to start it through aws. Another thing to mention is that when I try to add the logConfiguration to try to see what could be going wrong with the service it will give me an error of
    ResourceInitializationError: failed to validate logger args: : signal: killed
    . I’ve double checked with devOps that the IAM roles and network configurations should be correct too. Any ideas on how to debug this or why this is happening?
    k
    a
    +1
    41 replies · 4 participants
Powered by Linen
Title
d

David Wang

02/02/2022, 4:02 PM
Hi everyone, I am currently trying to setup an ECS agent with aws. However, the task seems to stop and exit with the error
essential container in task exited
. The prefect ecs agent also does not appear in the UI. I have been able to start the ECS agent locally, but am unable to start it through aws. Another thing to mention is that when I try to add the logConfiguration to try to see what could be going wrong with the service it will give me an error of
ResourceInitializationError: failed to validate logger args: : signal: killed
. I’ve double checked with devOps that the IAM roles and network configurations should be correct too. Any ideas on how to debug this or why this is happening?
k

Kevin Kho

02/02/2022, 4:05 PM
First time I have seen the
ResourceInitializationError
myself. Are you using the base Prefect image or your own? Are you just following the docs or do you have any added configuration?
d

David Wang

02/02/2022, 4:06 PM
I followed the docs and the tutorial at https://towardsdatascience.com/how-to-cut-your-aws-ecs-costs-with-fargate-spot-and-prefect-1a1ba5d2e2df
only difference I made was not using the aws parameter store for now
k

Kevin Kho

02/02/2022, 4:15 PM
I think we need the logging to figure out the ECS issue and it might be a connectivity issue or the log group doesn’t exist so I would check for those?
d

David Wang

02/02/2022, 4:29 PM
I’ve confirmed that the log group exists. I did ask devOps about the connectivity earlier, but they said that I shouldnt be having trouble connecting to CloudWatch. I will try asking again about it though
k

Kevin Kho

02/02/2022, 4:31 PM
The most common cause for the container exiting though is image incompatibility.
d

David Wang

02/02/2022, 4:32 PM
current image that is specified in the container definitions is
"image": "prefecthq/prefect:latest-python3.8",
k

Kevin Kho

02/02/2022, 4:34 PM
Ah ok that should be good
d

David Wang

02/02/2022, 10:20 PM
I talked to devOps and they confirmed that connectivity is not the problem, as there would be a different error message. I also rechecked the log configuration but there doesn’t seem to be anything wrong.
"logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-group": "$ECS_LOG_GROUP_NAME",
                    "awslogs-region": "$AWS_REGION",
                    "awslogs-stream-prefix": "ecs",
                    "awslogs-create-group": "true"
                }
            }
is there any other way to add the logs to the task definition like in the aws console?
k

Kevin Kho

02/02/2022, 10:23 PM
Not quite cuz the Task Definition tab just gives an error but no logs if you don’t setup CloudWatch. It won’t fix your logging issue, but the
task exited
can also be due to a lack of IAM permissions
Like not able to pull the image
d

David Wang

02/07/2022, 4:21 PM
I did once get a
CannotPullContainerError: inspect image has been retried 5 time(s): failed to resolve ref "<http://docker.io/prefecthq/prefect:latest-python3.8|docker.io/prefecthq/prefect:latest-python3.8>": failed to do request: Head <https://registry-1.docker.io/v2/prefecthq/prefect/manifests/latest-python3.8>: dial tcp ...
error while I was messing around with the security groups
I did also double check the IAM permissions and I should have the correct permissionss
a

Anna Geller

02/07/2022, 5:08 PM
it looks like this setting might be missing:
assignPublicIp=ENABLED
You can set up as part of the network configuration:
aws ecs create-service \
    --service-name $ECS_SERVICE_NAME\
    --task-definition $ECS_SERVICE_NAME:1 \
    --desired-count 1 \
    --launch-type FARGATE \
    --platform-version LATEST \
    --cluster $ECS_CLUSTER_NAME \
    --network-configuration awsvpcConfiguration="{subnets=[$SUBNET1, $SUBNET2, $SUBNET3],assignPublicIp=ENABLED}" --region $AWS_REGION
To explain it a bit more - my intuition (not 100% sure) is that your flow run container doesn't have access to the Internet in order to pull the container image from Dockerhub
d

David Wang

02/07/2022, 6:50 PM
I do have that setting enabled, however the VPC and subnets I’m using is not a default subnet, nor does it allow auto-assign public ipv4 or 6 addresses - would that affect it?
the VPC and subnet does have internet connection through a NAT gateway
a

Anna Geller

02/07/2022, 6:57 PM
that may be the issue indeed since you need to make sure that the CONTAINER within that instance also has access to the internet through that NAT gateway, You can test this out by launching a simple ECS task with a container getting the image from some public Dockerhub repo. Have you tried launching any ECS task within that container without Prefect? Did that work using the same network configuration?
d

David Wang

02/07/2022, 6:59 PM
I have not launched any task without prefect - is there any simple ecs task I can run to test this out?
a

Anna Geller

02/07/2022, 7:04 PM
there are a bunch of tutorials online, here are some you can use: •

https://www.youtube.com/watch?v=o_qSS4S1g34▾

•

https://www.youtube.com/watch?v=eq4wL2MiNqo▾

• https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ECS_AWSCLI_Fargate.html • https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-cli-tutorial-ec2.html
d

David Wang

02/07/2022, 9:06 PM
just tested out launching an ecs task using the aws docs tutorial, and it was able to run with no problems
a

Anna Geller

02/08/2022, 2:43 PM
Did you run it in the same subnets as your Perfect flows? Did you assign public IPs? Did the setup for this differ in any way from your Prefect ECSRun?
d

David Wang

02/08/2022, 4:08 PM
Yes, I ran it in the same subnets as the prefect flows. I don’t think I assigned public IPs as I followed the part of the tutorial where the example was using a private subnet. I also reran the prefect flows with the exact same command as the tutorial besides the task definition name and it returned the
ResourceInitializationError: failed to validate logger args: : signal: killed
error again. And even if I remove the log configuration part to see if it will run it will give the
essential container in task exited
k

Kevin Kho

02/08/2022, 4:25 PM
Hey David, unfortunately I think there isn’t more advice we can give without really going into the AWS account and looking at the setup. We don’t offer that here in community support, but can always connect you to Prefect’s professional services team.
Or I dunno if Anna may have more ideas. She’s out sick though for now. Sorry about that.
d

David Wang

02/08/2022, 10:33 PM
I finally got it the logs to work - turns out I had to delete an existing log endpoint. DevOps had not tried this because he did not think that was the problem. So now the ECS agent shows up in the UI and is up and running. Only problem now is that when I try to run an example flow like this it gives me an error of
Parameter validation failed: Missing required parameter in networkConfiguration.awsvpcConfiguration: "subnets" Unknown parameter in networkConfiguration.awsvpcConfiguration: "Subnets", must be one of: subnets, securityGroups, assignPublicIp
. Do I need to specify network configurations again somewhere in the run config?
k

Kevin Kho

02/08/2022, 11:09 PM
Glad you got that figured out. So this error is because Prefect can infer the default VPC, but if you have custom ones, you need to specify them with a job template. It seems like your agent was able to start though? How did you start your agent in the right VPC?
d

David Wang

02/09/2022, 4:19 PM
i got it to run by adding the --run-task-kwargs command in the task definition like in this post
k

Kevin Kho

02/09/2022, 4:24 PM
So agent works but Flow still does not?
d

David Wang

02/09/2022, 4:24 PM
yes thats correct
k

Kevin Kho

02/09/2022, 4:31 PM
So I think there are two places to put this. One is on the flow like this where you add a
task_definition_path
. That links to here , and then you can specify those subnet and security group there. The ECS agent also takes a definition upon starting . You can pass
--task_definition_path
. You just need to make sure that these live somewhere the agent can pull during runtime (like an S3 bucket it has access to). The agent task_definition serves as a default for the Flows that it runs, but the RunConfig can override it. The default one the agent uses can be found here
d

David Wang

02/09/2022, 8:10 PM
what would be the correct format to specify the subnets and security groups? This is what I currently have but it says networkConfiguration parameter doesn’t exist.
networkMode: awsvpc
cpu: 512
memory: 1024
containerDefinitions:
  - name: prefectEcsAgent
networkConfiguration:
  awsvpcConfiguration:
    Subnets:
      - subnet-xxx
    securityGroups:
      - sg-xxx
    assignPublicIp: DISABLED
k

Kevin Kho

02/09/2022, 8:21 PM
Let me look around. The quickest I found is this
Got my ECS set up. About to try this
Ok I think my understanding was wrong. Per the AWS ECS here , you need to specify it at run time instead of the task definition that gets registered. So custom VPCs need to be specified in task definition
run_task_kwargs
or on the agent
If you specify the awsvpc network mode, the task is allocated an elastic network interface, and you must specify a NetworkConfiguration when you create a service or run a task with the task definition.
This is so painful I dunno why you can’t do it as part of the task definition
d

David Wang

02/09/2022, 10:17 PM
Yep it works after I specified the subnets and security group inside the run_task_kwargs argument in the runConfig for the flow. Interesting that I have to since I already specified the subnets/security groups with the run_task_kwargs parameter when starting the agent.
k

Kevin Kho

02/09/2022, 10:22 PM
The kwargs on the agent should propagate. Could you share your syntax?
d

David Wang

02/10/2022, 3:57 PM
{
  "family": "$ECS_SERVICE_NAME",
  "requiresCompatibilities": [
    "FARGATE"
  ],
  "networkMode": "awsvpc",
  "cpu": "512",
  "memory": "1024",
  "taskRoleArn": "arn:aws:iam::xxx:role/prefectTaskRole",
  "executionRoleArn": "arn:aws:iam::xxx:role/prefectECSAgentTaskExecutionRole",
  "containerDefinitions": [
    {
      "name": "$ECS_SERVICE_NAME",
      "image": "prefecthq/prefect",
      "essential": true,
      "command": [
        "prefect",
        "agent",
        "ecs",
        "start",
        "--run-task-kwargs",
        "<s3://xxx-test-bucket/david/ecs-config.yaml>"
      ],
      "environment": [
        {
          "name": "PREFECT__CLOUD__API_KEY",
          "value": "xxx"
        },
        {
          "name": "PREFECT__CLOUD__AGENT__LABELS",
          "value": "['dev']"
        },
        {
          "name": "PREFECT__CLOUD__AGENT__LEVEL",
          "value": "INFO"
        },
        {
          "name": "PREFECT__CLOUD__API",
          "value": "<https://api.prefect.io>"
        }
      ],
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "$ECS_LOG_GROUP_NAME",
          "awslogs-region": "$AWS_REGION",
          "awslogs-stream-prefix": "ecs",
          "awslogs-create-group": "true"
        }
      }
    }
  ]
}
and inside the yaml file
networkConfiguration:
  awsvpcConfiguration:
    Subnets:
      - subnet-xxx
    securityGroups:
      - sg-xxx
    assignPublicIp: DISABLED
I also have the network configuration setup when creating the service with aws ecs create-service
aws ecs create-service \
    --service-name $ECS_SERVICE_NAME\
    --task-definition $ECS_SERVICE_NAME:1 \
    --desired-count 1 \
    --launch-type FARGATE \
    --cluster $ECS_CLUSTER_NAME \
    --network-configuration awsvpcConfiguration="{subnets=[$SUBNET1],securityGroups=[$SECURITYGROUP]}" --region $AWS_REGION
To get the flow to run I just put in the networkConfiguration parameter in the RUN_CONFIG
RUN_CONFIG = ECSRun(
    labels=["dev"],
    task_role_arn="arn:aws:iam::xxx:role/prefectTaskRole",
    run_task_kwargs=dict(cluster="prefectEcsCluster",networkConfiguration={
            'awsvpcConfiguration': {
                'subnets': [
                    'subnet-xxx'
                ],
                'securityGroups': [
                    'sg-xxx',
                ],
                'assignPublicIp': 'DISABLED'
            }
        }),
)
c

Christopher

02/11/2022, 6:53 AM
This might be unhelpful advice at this stage but I was reading through trying to get my own ECS set up. Your yaml file has "Subnets" with a capital S which I think is wrong. That what this error message is saying anyway: https://prefect-community.slack.com/archives/CL09KU1K7/p1644359607318099?thread_ts=1643817770.525689&amp;cid=CL09KU1K7
:upvote: 3
d

David Wang

02/11/2022, 4:28 PM
Looks like you were right! The flow now works normally without any extra added networkConfigurations needed.
🙌 1
View count: 8