Hello - I was wondering if there is a way to prevent DockerStorage from being built / pushed when th...

Vincent

07/13/2021, 9:08 PM

Hello - I was wondering if there is a way to prevent DockerStorage from being built / pushed when the Flow is already registered with a known idempotency_key. Ideally, the storage should only be built/pushed when the Flow has changed.

Kevin Kho

07/13/2021, 9:10 PM

Hey @Vincent, what version are you on and how are you registering? CLI or

flow.register()

Vincent

07/13/2021, 9:11 PM

I am on 14.22 and I am registering using

flow.register(..., idempotency_key=flow.serialized_hash())

Kevin Kho

07/13/2021, 9:19 PM

~~How long are your builds taking? It should be pretty fast because we made a change that respected the Docker cache.~~ I guess the upload may take a while.

Vincent

07/13/2021, 9:26 PM

The issue is not a question of speed / upload time. I want to freeze the image used at registration, and only update it when the flow changes.

Kevin Kho

07/13/2021, 9:29 PM

Will ask the team about this.

Kevin Kho

07/13/2021, 9:39 PM

The image will still build even with the idempotency key. So if the Flow is the only reason causing the builds, the suggestion is to use another form of Storage like S3 or Github, and then pair that with

DockerRun

DockerRun

can pull the image to run on, and then the Flow will be retrieved from Storage and run on top of that image.

Vincent

07/13/2021, 9:49 PM

Thanks for the suggestions.

4 Views

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.