Hi all Can someone elaborate on this error Cannot provide `t Prefect Community #ask-community

Hi all Can someone elaborate on this error “Cannot...

David

05/19/2021, 10:53 AM

Hi all Can someone elaborate on this error “Cannot provide

task_definition_arn

when using

Docker

storage”? I have a flow that I want to register with docker (ECR) but I dont want to generate_task_definition every deploy (because besides the docker itself nothing changed) Why cant I pass task arn when using docker storage? Thanks

David

05/19/2021, 11:32 AM

Moreover, Why does the ecs agent calls register_task_definition for every execution and not searching for an existing task definition (like fargate agent)? am I missing something?

Kevin Kho

05/19/2021, 1:47 PM

Hi @David, on the first one, the ECS task’s image can only be configured as part of the task definition so I don’t think there’s a way to use the image from Docker storage with that task definition. On the second one, the previous behavior was to search for an existing task definition, but this wouldn’t work for RunConfigs that changed from run to run (think Env variables changing).

David

05/20/2021, 8:10 AM

Actually I dont undetstand why the ecs tasks image can only be configured as part of the task definition. If I understand correctly when I am calling flow.register I create a new docker image which will be pushed to my registry. If I created manually task definition in ECS that pulls the last image with my env vars and all, and I am passing its arn in the run config, why would you make a new image? sounds like everything is in place

David

05/20/2021, 8:15 AM

I will ask differently, is there a way to use docker storage with a single task definition without calling register every flow deployment?

Kevin Kho

05/20/2021, 12:48 PM

Oh my bad. I understand what you’re saying now. I was thinking uploading an image from local, not building to a registry, For the DockerStorage + ECS, I will have to ask the person on the team who wrote that so I’ll get back to you on that. The answer to the second question is no, but I’ll double check to be sure and get a more detailed answer for you on that.

Kevin Kho

05/20/2021, 1:32 PM

So for the last question you can do

flow.register(build=False)

. Still checking for the ECS question.

David

05/20/2021, 8:55 PM

Thank you for checking 🙏

David

05/20/2021, 8:55 PM

If I use the flow.register build = False It wont update the flow docker? I will look at that

Kevin Kho

05/21/2021, 2:50 PM

Hey, I didn’t get a response but I did dig through the code. I think what happened here is that the --task-definition-arn image can’t be changed so there is an expectation of some users that the image they upload would be applied, and we can’t do that. That said, DockerStorage would only work under the right circumstances so there was a decision to not make it available to avoid misuse and confusion. I think you might be able to use the

ECSRun

config and pass the image there but I’m not 100% sure on this either

David

05/23/2021, 6:45 AM

mm I understand that, but its a bummer that this is not a flag I can skip. I dont think passing the image to the

ECSRun

solves my issue cause I am trying to avoid registering new task definition

Kevin Kho

05/23/2021, 2:05 PM

You’re welcome to open an issue and the core team will see it.

2 Views

Open in Slack

Previous Next