Thread
#prefect-community
    Nicolas Bigaouette

    Nicolas Bigaouette

    1 year ago
    Hi all! I'm exploring prefect for a specifc use case but I am not sure if it can perform what I need. Maybe someone can tell if it's possible? I have a web backend that sometimes needs to perform some commands sequences. Looking at a
    Flow
    seems to support this. But I have a specific requirement where a one (or more) command(s) are actually RPCs. Those RPCs are customs; the remote command to be executed is added to a queue and the remote worker polls that queue. After having executed a command, the remote worker will
    POST
    its result to the backend. There is multiple instances of the backend running (for redundancy and scaling) so the result will probably not be received by the process that created the RPC call. This creates a discontinuity in the code; The task driving the workflow has to stop when enqueuing its remote command and another instance will pick up the work at a later time. I can probably model my sequences with prefect just fines, except I am not sure how to handle the discontinuity described above. I though I could raise a "pause" signal in the task that initiated the custom RPC. Then the backend instance receiving the
    POST
    could send a "continue" signal to the paused task. Is that even possible? Can such a workflow be modeled by prospect? Thanks!!
    nicholas

    nicholas

    1 year ago
    Hi @Nicolas Bigaouette - yes, a `Pause`/`Resume` is definitely possible with Prefect, you can read more about that here. I think that will fit your use case so long as your remote backend is able to keep track of the task run id, which can then be resumed using this mutation:
    mutation SetTaskRunStates {
      set_task_run_states(input: { states: [
       task_run_id: <<TASK_RUN_ID>>,
                      state: {
                        type: 'Resume',
                        message: <<you can use this field to store some message about why it was resumed/what resumed it>>
                      }
    ] }) {
        states {
          id
        }
      }
    }
    Nicolas Bigaouette

    Nicolas Bigaouette

    1 year ago
    Thanks! So if I understand it correctly, the prefect server pauses a task. One of my own backend receives the
    POST
    results and calls the graphql mutation on the prefect server to resume the task. I could either embed the
    POST
    result in the message or go through my DB for that information, keeping track of of the task id at every step.
    nicholas

    nicholas

    1 year ago
    Exactly @Nicolas Bigaouette. In addition, you can always ask Prefect Server for all your
    Paused
    task runs in case you want to handle them without storing the
    task_run_id
    first:
    query PausedTaskRuns {
      task_run(where: { state: _eq: "Paused" }) {
        id
        name
      }
    }
    Nicolas Bigaouette

    Nicolas Bigaouette

    1 year ago
    Great! Thanks a lot! 🙏
    nicholas

    nicholas

    1 year ago
    Happy to help! let us know if you run into any trouble 🙂