Sander
05/17/2022, 7:38 AMAnna Geller
import requests
import time
# Import the Prefect flow module.
from prefect import flow
def get_real_time_data():
data = requests.get("<http://127.0.0.1:8000/orders>").json()
print(data)
# Decorate the main function with @flow.
@flow(name="Streaming Pipeline")
def main():
get_real_time_data()
if __name__ == "__main__":
while True:
main()
time.sleep(5)
this post discusses it in depthLooks like killing the agent doesn't clean up the universal flow runs on the Linux box.exactly, this may be problematic because data flows are not intended to be running forever and what you are likely looking for when using Prefect is more visibility into what happens within each of those runs - in that case treating the flow as the actual unit of observable business logic (with optional tasks to add even more visibility) and moving the while loop out of the flow may be beneficial and easier to troubleshoot
Sander
05/17/2022, 11:02 AMAnna Geller
Sander
05/17/2022, 11:08 AMAnna Geller
Sander
05/17/2022, 11:16 AMAnna Geller
if the orchestration layer is able to hard stop flow runsthis is an extremely hard problem in the hybrid execution model as many processes can't be killed just via an API call - canceling work on a remote infrastructure is NOT on the immediate roadmap, only marking runs as Completed/Failed from the UI is
Sander
05/17/2022, 11:23 AMAnna Geller
Sander
05/17/2022, 11:32 AMAnna Geller
@flow
Sander
05/17/2022, 12:07 PMAnna Geller
Sander
05/17/2022, 12:38 PMAnna Geller
Sander
05/17/2022, 1:30 PMAnna Geller
Sander
05/17/2022, 3:32 PMAnna Geller
Sander
06/09/2022, 10:54 AM@task
Def run_task():
Print(‘hi’)
@flow
def my_flow()
While True:
Run_task()
I can start this using a deployment but it doesn't stop. What will be the preferred method for stopping this flow?Anna Geller
prefect flow-run delete UUID
Sander
06/09/2022, 11:10 AMAnna Geller
import pendulum
now = pendulum.now().time()
if pendulum.time(hour=10, minute=0) > now > pendulum.time(hour=20, minute=0):
# your logic here
or schedule with RRule between those times every 10 seconds?Sander
06/09/2022, 1:33 PMAnna Geller
Sander
06/09/2022, 1:52 PMAnna Geller
Sander
06/09/2022, 2:08 PMAnna Geller