https://prefect.io
Join Slack
Hey, I am running a flow locally and have another flow deployed in the cloud. I loop over somethin...
# ask-community
b
Hey, I am running a flow locally and have another flow deployed in the cloud. I loop over something and create multiple sub flows.
Copy code
create_flow_run.run(
            flow_name="advanced_cleaning",
            project_name="modelling",
            run_name=f"data_update={data_update}",
            parameters=dict(data_update=data_update),
        )
When I run this it is called multiple times, but only one sometimes two flows are triggered in the cloud, is this a possible bug? Do run names need to be unique? This is being called from within a task fyi
I see there is a StartFlowRun also, but this gives me the same issues 🤷
in fact the loop, every time it calls the task to create a new flow run, returns the exact same flow run url 🤷
k
This has to do with the idempotency key. These tasks use the context task run id as the default idempotency key. Multiple invocations will use the same key and only trigger one flow run, so you need to supply multiple idempotency keys to generate multiple runs
b
are there any examples of this? I dont really get what you are saying, where do I get the idempotency key from ? And why does it work fine when it is running from the cloud flow and not locally?
k
You can just supply any string as the idempotency key so you can do something like datetime.now() as a string and it will trigger multiple runs
I guess this is an example but it’s the client not as task
It should behave the same for both in that 1 run is created
b
got it, thanks mate
Fixed the issue. Thanks @Kevin Kho 🙏
👍 1
Hey @Kevin Kho, this ran fine when triggering locally but when it runs on the schedule in the cloud, I get a 
Error during execution of task: TypeError("create_flow_run() got an unexpected keyword argument 'idempotency_key'")
Locally I am using prefect 
0.15.5
and my prefect core version is the same on the UI. Any ideas?
k
That was added actually let me see the changelog
You need 0.15.8
b
ah ok, any breaking changes I should be aware of if I upgrade?
k
From 0.15.5 to 0.15.8 should be absolutely nothing
b
incredible
k
You seen to be up super late?
b
it is 7.40am 😆
You can never get my timezone right lol
k
Ah lol cuz you normally ping later in the day
b
yeah, I usually start work, then get stuck and you save me lol
😆 1
6 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by