@Marvin I am looking to create a ci/cd pipeline that executes a prefect flow but I want to use webhooks to tell me when the flow is finished running How can I achieve this

To create a CI/CD pipeline that executes a Prefect flow and uses webhooks to notify you when the flow finishes running, you can follow these steps: 1. Set Up CI/CD Integration: You can use Prefect's GitHub Actions for authentication and deployment creation. Here's an example workflow using GitHub Actions:

yaml
   name: Deploy Prefect flow
   on:
     push:
       branches:
         - main
   jobs:
     deploy:
       name: Deploy
       runs-on: ubuntu-latest
       steps:
         - name: Checkout
           uses: actions/checkout@v4
         - name: Log in to Docker Hub
           uses: docker/login-action@v3
           with:
             username: ${{ secrets.DOCKER_USERNAME }}
             password: ${{ secrets.DOCKER_PASSWORD }}
         - name: Setup Python
           uses: actions/setup-python@v5
           with:
             python-version: "3.11"
         - name: Prefect Auth
           uses: PrefectHQ/actions-prefect-auth@v1
           with:
             prefect-api-key: ${{ secrets.PREFECT_API_KEY }}
             prefect-workspace: ${{ secrets.PREFECT_WORKSPACE }}
         - name: Prefect Deploy
           run: prefect deployment build ./flow.py:my_flow -n my-deployment && prefect deployment apply my-deployment.yaml

Reference: Prefect CI/CD Guide 2. Set Up State Change Hooks: You'll want to use state change hooks to send a notification when the flow finishes running. This can be either a success or a failure notification. Here's an example using a Slack webhook to notify upon completion:

python
   from prefect import flow, task
   from prefect.blocks.notifications import SlackWebhook
  
   webhook = SlackWebhook.load("my-slack-webhook")

   @task
   def notify_slack_success(flow_name, state, flow_run):
       webhook.notify(
           content=f"Flow {flow_name} finished with state {state.name}. Check details here: {flow_run.url}"
       )

   @task
   def notify_slack_failure(flow_name, state, flow_run):
       webhook.notify(
           content=f"Flow {flow_name} failed with state {state.name}. Check details here: {flow_run.url}"
       )

   @flow
   def my_flow():
       # Your flow logic here

   if __name__ == "__main__":
       my_flow(
           on_success=[notify_slack_success],
           on_failure=[notify_slack_failure]
       )

The

notify_slack_success

and

notify_slack_failure

functions use the Slack webhook to send a notification depending on the flow's final state. Reference: Prefect State Change Hooks By following these steps, integrating your Prefect flows with CI/CD pipelines and setting up webhooks for notifications can be achieved seamlessly. If you need specific examples of how to configure your flow tasks or deployment scripts, you can dive further into the Prefect documentation and related guides.

@Marvin Can we block the PR until the flow completes successfulyy

To block a PR until a Prefect flow completes successfully, you can use GitHub Actions to integrate Prefect flow execution with your CI/CD pipeline. Here’s how you can set it up: 1. Set Up GitHub Actions Workflow: Create a GitHub Actions workflow that executes your Prefect flow. Use state change hooks to monitor the flow's progress. 2. Set Up Blocking Mechanism: Use the

checks

API of GitHub to create a blocking check. This check will run your Prefect flow and block the PR until the flow completes successfully. Here is an example workflow:

yaml
name: Run Prefect Flow and Block PR

on:
  pull_request:
    branches:
      - main

jobs:
  run-prefect-flow:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v2

      - name: Set up Python
        uses: actions/setup-python@v2
        with:
          python-version: '3.8'
      
      - name: Install dependencies
        run: pip install -r requirements.txt

      - name: Run Prefect Flow
        run: |
          prefect auth login --key ${{ secrets.PREFECT_API_KEY }}
          PREFECT_FLOW_RUN_ID=$(prefect run --name my-flow)
          echo "PREFECT_FLOW_RUN_ID=$PREFECT_FLOW_RUN_ID" >> $GITHUB_ENV
      
      - name: Wait for flow completion
        run: |
          while :; do
            STATUS=$(prefect flow-run get --id $PREFECT_FLOW_RUN_ID | jq -r .state)
            if [ "$STATUS" == "Completed" ]; then
              echo "Flow run completed successfully"
              exit 0
            elif [ "$STATUS" == "Failed" ]; then
              echo "Flow run failed"
              exit 1
            fi
            echo "Waiting for flow completion..."
            sleep 30
          done

      - name: Create Check Run
        if: ${{ success() }}
        run: |
          curl -X POST -H "Authorization: token ${{ secrets.GITHUB_TOKEN }}" \
          -H "Content-Type: application/json" \
          --data '{ "name": "prefect-flow-check", "head_sha": "${{ github.sha }}", "status": "completed", "conclusion": "success" }' \
          <https://api.github.com/repos/$>{{ github.repository }}/check-runs

      - name: Create Check Run
        if: ${{ failure() }}
        run: |
          curl -X POST -H "Authorization: token ${{ secrets.GITHUB_TOKEN }}" \
          -H "Content-Type: application/json" \
          --data '{ "name": "prefect-flow-check", "head_sha": "${{ github.sha }}", "status": "completed", "conclusion": "failure" }' \
          <https://api.github.com/repos/$>{{ github.repository }}/check-runs

This workflow will: 1. Run on Pull Requests: The workflow triggers on pull requests to the

main

branch. 2. Execute Prefect Flow: It runs the Prefect flow and stores the flow run ID. 3. Wait for Flow Completion: It monitors the flow run's status and waits for it to complete or fail. 4. Create Check Run: It uses GitHub’s checks API to create a check run that reports the status of the flow. This approach effectively blocks the PR until the Prefect flow completes successfully. If the flow fails, the check run will fail, and the PR will not be merged. For more details, refer to: - GitHub Actions Documentation - Prefect CI/CD Guide - Prefect State Change Hooks Guide

@Marvin I'd prefer to have two actions, one that tirggers the flow and one that triggers after the flow has finished I do not ewant the Github action to wait as this uses actions runner minutes 🙂

To achieve this in a more efficient way without consuming GitHub Actions runner minutes, you can set up two separate GitHub Actions workflows. One workflow will trigger the Prefect flow, and another workflow will monitor the flow's completion. Prefect's automations and notifications can be used to trigger the second GitHub Action upon completion. Here's how you can proceed: ### Workflow 1: Trigger Prefect Flow Create a GitHub Actions workflow to trigger the Prefect flow:

yaml
name: Trigger Prefect Flow

on:
  pull_request:
    branches:
      - main

jobs:
  trigger-flow:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v2

      - name: Set up Python
        uses: actions/setup-python@v2
        with:
          python-version: '3.8'
      
      - name: Install dependencies
        run: pip install -r requirements.txt

      - name: Run Prefect Flow
        run: |
          prefect auth login --key ${{ secrets.PREFECT_API_KEY }}
          PREFECT_FLOW_RUN_ID=$(prefect run --name my-flow)
          echo "PREFECT_FLOW_RUN_ID=$PREFECT_FLOW_RUN_ID" >> $GITHUB_ENV

### Workflow 2: Monitor Flow Completion Set up a Prefect automation to trigger the second GitHub Action when the flow completes. This requires setting up a webhook to GitHub Actions. #### Prefect Automation Use the Prefect Cloud UI to create an automation that triggers an event when the flow state changes to `Completed`: 1. Go to Automations: In the Prefect Cloud UI, navigate to Automations. 2. Create New Automation: Click on "Create Automation" and configure it to listen for flow run completion events. 3. Trigger Webhook: - Use the webhook URL for GitHub Actions as the action in your automation. - The webhook URL for GitHub Actions can be found under the "Webhook" trigger in a new GitHub Actions workflow.

json
{
  "type": "sequence",
  "within": 7200,
  "triggers": [
    {
      "type": "event",
      "posture": "Reactive",
      "expect": ["prefect.flow-run.Completed"],
      "match_related": {
        "prefect.resource.name": "my-flow",
        "prefect.resource.role": "flow"
      }
    }
  ]
}

### Workflow 2: GitHub Action Triggered by Webhook Create a GitHub Actions workflow triggered by the Prefect webhook (you'll need to enable the "repository_dispatch" event):

yaml
name: Check Flow Completion

on:
  repository_dispatch:
    types: [prefect_flow_completed]

jobs:
  check-flow:
    runs-on: ubuntu-latest
    steps:
      - name: Print Flow Status
        run: |
          echo "Prefect flow has completed"
          # Add any additional steps you need here, like updating a status check

### Enabling Webhook in Prefect Automation 1. Generate a Personal Access Token: - Go to GitHub settings and create a new Personal Access Token with appropriate permissions. 2. Webhook URL: - The webhook URL will look something like:

<https://api.github.com/repos/:owner/:repo/dispatches>

- In your Prefect automation, set the webhook payload like:

json
     {
       "event_type": "prefect_flow_completed",
       "client_payload": {}
     }

3. Set Headers: - Include the GitHub Personal Access Token in the headers of your webhook with Authorization set to

token YOUR_PERSONAL_ACCESS_TOKEN

. By setting up separate workflows, the PR will only be merged when the second workflow confirms that the flow run is completed successfully. This way, you efficiently manage CI/CD resources without consuming unnecessary GitHub Actions runner minutes.

Legend thank you @Marvin

You're welcome! If you have any more questions or need further assistance, feel free to ask. Good luck with your CI/CD setup! 🚀