https://prefect.io
Join Slack
Design question: Why isn’t `idempotency_key=flow.serialized_hash()` the default? <https://docs.pref...
# ask-community
n
Design question: Why isn’t 
idempotency_key=flow.serialized_hash()
the default? https://docs.prefect.io/orchestration/concepts/flows.html#core-client I can’t think of why someone would want the Flow version to change if the Flow definition hasn’t.
j
Backwards compatibility reasons. Note that if you use the new 
prefect register
cli it is the default (since this was a new CLI method, there was no backwards compatibility to break).
n
1. Will it become the default in 1.0? 2. I see 
prefect register
create a new version every time. And I don’t see how to configure the CLI’s behavior in this regard. Did I miss something?
j
It shouldn't create a new version every time, that sounds like a bug. Can you provide a reproducible example?
n
If I can draft one, I’ll report it on the issue tracker. 👍
🙏 1
Maybe I’m misunderstanding the design. If I call
Copy code
prefect register --project "my project" --path path/to/flow.py
multiple times, are multiple versions expected to be created? Maybe 
prefect register
only avoids creating new versions when it’s running via 
--watch
?
j
No, the use you've described above should only register once.
I suspect it has to do with whatever storage/run_config settings you've used, as I'm unable to reproduce your issue. A reproducible example would be useful here.
n
Yup, trying to boil it down now. Just wanted to make sure I wasn’t misunderstanding the intended use.
I’ve boiled down my flow to something very simple, and yet 
prefect register
still creates a new version every time I run it. Here’s the boiled down flow definition:
Copy code
with Flow(
    "some-test",
    run_config=UniversalRun(labels=["my-company"]),
    storage=S3(bucket="some-bucket"),
) as flow:
    pass
And I’m running:
Copy code
prefect register --project "my-project" --path path/to/myflow.py
And yet every time I run it I get a new Flow version. Is there anything I should try before I report this on the issue tracker?
j
Ah, that makes sense.
Your flow is configured to upload to a different s3 key every time you build it. Try specifying a key as well. 
storage=S3(bucket="some-bucket", key="some-key")
n
Ohhhhhhhhhhhh
Let me read up on them docs, now. 😅
j
This is mostly for legacy reasons, I'd like to change this behavior as it's not usually what a user wants now, but it is the current behavior.
sorry about all that, hopefully you can get things working now :)
n
Thanks for your guidance. I will come back to whine some more if it still isn’t working as I expect. 😄
https://docs.prefect.io/api/latest/storage.html#s3 Would you consider a PR that changes this (doc + corresponding behavior)
If this key is not provided the Flow upload name will take the form 
slugified-flow-name/slugified-current-timestamp
.
to this?
If this key is not provided the Flow upload name will take the form 
slugified-flow-name/serialized-flow-hash
.
Or would that be a backwards-incompatible change?
j
Hmmm, that's an interesting idea. Lets hold off on a PR for now, but I'll raise an issue so we can think about the implications.
@Marvin open "Would it make sense to use the flow hash as part of the auto-generated flow file names"
m
m
Hi, I observed the similar behaviour of having a new version every time with 
Azure
being used as storage (providing 
container
parameter and using 
AZURE_STORAGE_CONNECTION_STRING
env). Is there a similar option here (as I don’t see 
key
as one of the 
Azure
parameters here) or I have to use 
idempotency_key
every time while registering (which I already am doing but bored to have to change it for every flow change)? Using 
flow.serialized_hash
looks like a good solution for me, but have to try it out 🙂 Thanks!
14 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by