Jeff Hale
11/16/2022, 10:24 PMJack
11/17/2022, 1:11 PMJean-Michel Provencher
11/17/2022, 9:27 PMgit clone <https://github.com/jmprovencher/prefect-tutorial>
and follow the steps in README.md
and TUTORIAL.md
Jeff Hale
11/21/2022, 5:35 PMRyan Peden
11/23/2022, 4:59 PMAnna Geller
Khuyen Tran
11/29/2022, 4:19 PMNick Acosta
11/29/2022, 11:18 PMJeff Hale
11/30/2022, 2:45 PMMadison Schott
11/30/2022, 5:55 PMBen Muller
12/05/2022, 12:19 AMwait_for_flow_run
so that I can run sub flows concurrently without introducing all the async stuff I was trying to avoid. I ended up using the rest api. Would there have been an easier way to go about this? seems to do the job β
import time
import requests
from prefect import get_run_logger, task
from prefect.deployments import run_deployment
from aws_helpers import Ssm
@task
def wait_for_deployment(flow_run_id: str, log_polling_interval: int = 25, max_run_time: int = 1000):
logger = get_run_logger()
api_key = Ssm.get_secret("/prefect/api_key")
api_url = Ssm.get_secret("/prefect/api_url")
while True:
flow_run = request_prefect_api(path=f"/flow_runs/{flow_run_id}", api_key=api_key, api_url=api_url)
<http://logger.info|logger.info>(f"{flow_run['name']} : {flow_run['state_type']}")
state = flow_run["state_type"]
if state == "COMPLETED":
break
if state == "FAILED":
raise Exception(f"Deployment: {flow_run['name']}, Flow run failed: {flow_run['state']['message']}")
if flow_run["total_run_time"] > max_run_time:
raise Exception(f"Flow run exceeded max run time of {max_run_time} seconds: marking as failed")
time.sleep(log_polling_interval)
def request_prefect_api(path: str, api_key: str, api_url: str):
url = f"{api_url}{path}"
headers = {"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"}
return requests.get(url, headers=headers).json()
Chris Reuter
12/06/2022, 3:22 PMJeff Hale
12/07/2022, 7:18 PMJeff Hale
12/07/2022, 7:51 PMAnna Geller
Khuyen Tran
12/09/2022, 6:01 PMAnna Geller
Madison Schott
12/29/2022, 4:51 PMJeff Hale
01/03/2023, 4:21 PMKhuyen Tran
01/03/2023, 4:26 PMAnna Geller
Anna Geller
Danilo Drobac
01/09/2023, 1:58 PMBryan Whiting
01/09/2023, 8:36 PMStephen
01/11/2023, 4:14 AMAnna Geller
Henning Holgersen
01/13/2023, 6:55 PMRyan Peden
01/14/2023, 9:10 PMKhuyen Tran
01/16/2023, 2:18 AMDanilo Drobac
01/16/2023, 9:21 AM