Thread
#prefect-community
    m

    Minakshi

    2 years ago
    1. Is there a way to pass parameters to via dask executor to flow? The requirement here is to run a flow for different datasets which could be configured either from dask or the caller. And the execution of flows of different datasets could be running in parallel. 2. Is there a way to start the run of a flow using a Flask api? and also can the cancellation of flow be controlled by an api? (I tried doing this but i was getting mutiprocessor errors)
    nicholas

    nicholas

    2 years ago
    Hi @Minakshi - the answer to your first question is "sort-of" and ties into your second question. The reason it's not resoundingly yes is that executors aren't currently configured at runtime, which means you register a flow with an executor context that's read at runtime but isn't modified. That said, flow runs can be launched from any client with access to GraphQL (including
    prefect.Client
    ), which means you can use a Flask or any other API to kick off a flow run, provided it has network access to the API (whether Server or Cloud).
    For example, you could do this with the
    prefect.Client
    with the following code:
    import prefect
    client = prefect.Client()
    
    client.create_flow_run(flow_id="your_flow_id", parameters={"some_parameters": "that_change_per_run"})
    m

    Minakshi

    2 years ago
    will this run synchronously or asynchronously?
    nicholas

    nicholas

    2 years ago
    These are HTTP request wrappers, so would be run asynchronously
    m

    Minakshi

    2 years ago
    the flow i am working with is time consuming and has subprocess calls, if it synchronous, it blocks the current thread and hence the http request goes in a limbo..
    will try it out. thanks
    nicholas

    nicholas

    2 years ago
    Ah sorry - are you using just Prefect Core? Or are you using Server/Cloud?
    m

    Minakshi

    2 years ago
    prefect core
    nicholas

    nicholas

    2 years ago
    Got it - in which case disregard the advice I gave above;
    flow.run()
    is a synchronous call, which will lock your current thread but will let you pass in parameters and an executor at runtime.
    m

    Minakshi

    2 years ago
    hmm ok
    nicholas

    nicholas

    2 years ago
    In order to make this happen asynchronously (or without blocking the main thread), you'll need to use normal Pythonic ways of spinning off new threads
    Your final question about cancellation will require that you have some sort of infrastructure/API setup, which would mean you need a persistence layer like Prefect Server/Prefect Cloud
    m

    Minakshi

    2 years ago
    okay