Mohamed Alaa

08/04/2022, 10:07 AM
Hello Guys! I have a couple of questions about prefect's scheduler and agent 1. Is it possible to make Flow run Instances Dependent on each other. By that I mean two things a. Trigger another Deployment based on the final state of a current Deployment Flow run instance. For example, if I have an ETL flow run that loads data to a database, if successful, i want it to trigger another flow that would read the data from this database and proceed with other logic. b. Making the Run of a Scheduled Deployment dependent on previous runs, so that if i have a Deployment scheduled to run every 15 mins, the 11:00 AM run, and other proceeding runs, would halt if the 10:45 AM run failed. 2. I've been running prefect for a PoC for my use-case and have been facing difficulties with the agent. It seems that the agent would suddenly crash and print out a large error message in the middle of a flow run. This is part of the error message
Traceback (most recent call last):
  File "/usr/lib/python3.8/asyncio/", line 848, in _read_ready__data_received
    data = self._sock.recv(self.max_size)
ConnectionResetError: [Errno 104] Connection reset by peer
From my understanding, this means that there was some connection problems with a TCP/IP connection and it wasn't able to send an acknowledgment message, however, I dont understand why it happens and it seems to me that it happens randomly (probably not true). Does it have to do with the size of data being transmitted (I am extracting data from an S3 bucket for processing)? Thanks in Advance!

Taylor Curran

08/04/2022, 2:30 PM
OK, I am not sure on 2, but regarding 1 have you checked out our API resources? Particularly client.create_flow_run() and also options like get_run_context()?

Mohamed Alaa

08/07/2022, 8:27 AM
Great!, Thanks for your reply
:party-parrot: 1