https://prefect.io
Join Slack
it seems that by not setting an environment the default environment is used. The default environment...
# ask-community
c
it seems that by not setting an environment the default environment is used. The default environment loads the default executor which is the localExecutor which the docs say
An executor that runs all functions synchronously and immediately in the main thread. To be used mainly for debugging purposes.
still I have 3 parallell flow runs
with
Copy code
flow.executor = LocalDaskExecutor(scheduler="threads", num_workers=1)
k
The executor is for a flow run. Not across flow runs. So each of those Flows has an executor
Prefect Cloud has Flow Run Concurrency limits, which is a separate thing if you need it
c
I'm running the flows (each of the 3 is the same flow file registered ) via graphql / the Ui (Run button)
so you are saying I cannot limit the amount of flows an agent is processing in parrallel? (in OS prefect)
k
The concurrency limit is not on the agent level. It’s on a Flow level. If you are using Server, you need to write additional code because Server doesn’t have the concurrency limiting.
One flow can have multiple invocations (flow runs). This is desirable when you have like multiple parameter sets you want to run concurrently
c
ok thank you this is already 90% of what I want I guess - where is this code going, inside prefect-core ? I guess instead of canceling I'm able to re-schedule, the runs take arount 15 min so I could re-schedule and then check again for how I understand this
k
That code snippet would go in the Flow definition. It utilizes flow state handler
Yes you can do that
Ah well. You need to do both cancel the current and re-schedule the next in the same state handler.
✅ 1
c
perfect - thank you for taking the time to point me to this
k
Of course!
c
btw state paused has this
`start_time (datetime)`: time at which the task is scheduled to resume; defaults to 10 years from now if not provided.
should be Cancel + Schedule in once from my understanding
k
I saw your response on the Github issue. Good catch and thanks for mentioning!
2 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by