https://prefect.io
Join Slack
1. Is there a way to pass parameters to via dask executor to flow? The requirement here is to run a ...
# ask-community
m
1. Is there a way to pass parameters to via dask executor to flow? The requirement here is to run a flow for different datasets which could be configured either from dask or the caller. And the execution of flows of different datasets could be running in parallel. 2. Is there a way to start the run of a flow using a Flask api? and also can the cancellation of flow be controlled by an api? (I tried doing this but i was getting mutiprocessor errors)
n
Hi @Minakshi - the answer to your first question is "sort-of" and ties into your second question. The reason it's not resoundingly yes is that executors aren't currently configured at runtime, which means you register a flow with an executor context that's read at runtime but isn't modified. That said, flow runs can be launched from any client with access to GraphQL (including 
prefect.Client
), which means you can use a Flask or any other API to kick off a flow run, provided it has network access to the API (whether Server or Cloud).
For example, you could do this with the 
prefect.Client
with the following code:
Copy code
import prefect
client = prefect.Client()

client.create_flow_run(flow_id="your_flow_id", parameters={"some_parameters": "that_change_per_run"})
m
will this run synchronously or asynchronously?
n
These are HTTP request wrappers, so would be run asynchronously
m
the flow i am working with is time consuming and has subprocess calls, if it synchronous, it blocks the current thread and hence the http request goes in a limbo..
will try it out. thanks
n
Ah sorry - are you using just Prefect Core? Or are you using Server/Cloud?
m
prefect core
n
Got it - in which case disregard the advice I gave above; 
flow.run()
is a synchronous call, which will lock your current thread but will let you pass in parameters and an executor at runtime.
m
hmm ok
n
In order to make this happen asynchronously (or without blocking the main thread), you'll need to use normal Pythonic ways of spinning off new threads
Your final question about cancellation will require that you have some sort of infrastructure/API setup, which would mean you need a persistence layer like Prefect Server/Prefect Cloud
m
okay
3 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by