hey guys, got a basic questions. Just found out about prefect, and started messing around with it today. I've been following along the docs. And I've tried using this locally, and using the dask executor. I think I'm missing something. I've made a simple job to move file from a database into a sftp. However, when I run the file, I get this message. If I ctrl-c out of it, it crashes the program. Do i have to do it in nohup, or is there a way i can execute the script in the background. So it just runs, and follows the schedule? If there is any documentation on this, that would be of great help. Thanks!
Copy code
[2019-12-05 18:38:18,226] INFO - prefect.Flow: sf -> sftp | Waiting for next scheduled run at 2019-12-05T21:00:00+00:00
j
josh
12/05/2019, 6:46 PM
What is the schedule you’re attaching to your Flow?
josh
12/05/2019, 6:50 PM
Ultimately the schedule needs to run in a process so you are able to spin up however you want! Whether it’s in a subprocess, docker container, etc…
a
ARun
12/05/2019, 6:51 PM
The value is dynamically generated from some options
So the reason your schedule sits there is because it’s waiting for the scheduled run at 21:00 and the reason why it ends when you quit the process is because the schedule is attached to the flow. When the process ends the Flow object that you created no longer exists. I recommend using something like supervisor if you want to run this in the background https://pypi.org/project/supervisor/
💯 1
a
ARun
12/05/2019, 6:55 PM
Ah, I see. Thanks
ARun
12/05/2019, 6:56 PM
Is there a way to manage all the subprocess via prefect?
j
josh
12/05/2019, 6:59 PM
All of the subprocesses on your machine running schedules would be outside of prefect’s control
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.