Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Hi!

A question: How do I configure AWS Security Groups and Subnets for a Push Workpool? I can see in the UI that there is an option to add a VPC ID, but I cannot find anything allowing me to set subnet and/or security group.

I see that ECSTask class supports VPC, Custom Subnets and Custom Security Groups (<https://prefecthq.github.io/prefect-aws/ecs/>).

Is this what is used under-the-hood? If so, can I use `task_customizations` to patch in  my desired configuration options?

hey, you can define this under `Network Configuration` under the default tab.

Alternatively you can go to the advanced tab and modify the `task_run_request` directly and modify it anyway you like to be passed to: <https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_RunTask.html#API_RunTask_RequestSyntax>

Hi <@U02EBC3N05N> that link to the RunTask Request Syntax is super helpful. Thanks!

Sadly, there seems to be no "Network Configuration" field in the "Base Job Template" when you edit a Push Workpool.

I'll see what I can do by editing the template JSON under the "Advanced" tab

Update: I'm not seeing anything in the job configuration template JSON which suggests support for setting subnets or security groups.

This is quite a serious blocker to our deployment: we want to set up our ELT jobs using Prefect Push Workpools. However, access to our backend databases requires configuring task vpc, subnet and security group settings. Unless I can do that, the entire effort is dead-in-the-water

hey, if you create a new ecs push work pool do you see the `Network Configuration` under defaults? It was added within the last month to the default job template, so depending on when you created your work pool, yours may not have it.

The job_configuration template is entirely configurable, you can add anything you'd like yourself

Screenshot 2023-11-29 at 8.58.52 AM.png

right. the workpool was created a few weeks ago...

&amp; good to hear the job configuration template is highly customisable. If you have a collection of examples which can be shared, that would be awesome.

I'm going to try to create a new workpool now, then, and see if i can get my networking configured.

ok - good news. network config is indeed present. I'll work on this, and report back later.

<@U02EBC3N05N> many thanks for the pointers!

<@U02EBC3N05N> thanks again for the info - it proved to be the key.

First, a potential bug: setting network configuration under `Network Configuration` does *not* work (sadly).

I tried setting the following code (actual ids masked out) but the configurations were not detected:
```{
  "networkConfiguration":{
    "awsvpcConfiguration":{
	  "Subnets":["subnet-asdfasdf1","subnet-asdfasdf2"],
	  "AssignPublicIp":"False",
	  "SecurityGroups":["sg-asdfasdf1","sg-asdfasdf2"]
	}
  }
}```
However, hacking the raw JSON via the "Advanced" tab does work: inserting the network configuration block as shown in your snippet above, in the task_run_request section, works. I can confirm that my task ran on my ECS cluster with the correct security groups set.

I second <@U060Y4EAW2Z>’s statement. Setting the `Network Configuration` in the UI for an ECS Push work pool does *not* have any effect, but changing the JSON in the "Advanced" tab had the desired effect.