https://prefect.io logo
Title
s

Stephen Mugisha

03/08/2021, 5:02 PM
Hello, Noob question here, How do I execute an agent run (docker agent) on a server so that if I exit/logout of the server, the process/agent stays running/executing scheduled flows? Your help is much appreciated.
z

Zanie

03/08/2021, 5:04 PM
Hey @Stephen Mugisha --
screen
is one of the simplest ways to do this
screen -S my-agent
-->
prefect agent docker start
--> Ctrl A, d (exit screen)
s

Stephen Mugisha

03/08/2021, 5:06 PM
Thanks very much @Zanie
m

merlin

03/08/2021, 6:56 PM
another noob question: is this something we would need to do usually? I was thinking the container configs would start the agent along with everything else... but I haven't got that far yet.
z

Zanie

03/08/2021, 6:58 PM
If you're using
prefect server start
the agent is started separately; if you're using the Prefect Server helm chart, you can install the agent to your cluster at the same time; if you're using Prefect Cloud, you'll need to start an agent somewhere. The agent is in charge of spawning the containers to run flows.
🙌 1
m

merlin

04/30/2021, 5:08 PM
So the agent can run on the k8s cluster, which also hosts the server and , UI and apollo endpoint. I guess Hasura, Graphql and towel could also be on the cluster, but we would want postgress to be persistent as RDS or at a minimum getting regular snapshots saved to s3.
My planned architecture is on kubernetes cluster in AWS. I'm expecting to use prefect to orchestrate data engineering pipelines to manage a persistent data store for a medium sized team on AWS. Is it important to persist graphQL data? I assume the postgres DB has important flow history and execution history that will be critical to store.
z

Zanie

04/30/2021, 5:12 PM
Happy to hear from you 🙂
Are you separating "GraphQL data" from the "postgres data" ?
m

merlin

04/30/2021, 5:23 PM
Hi! I don't know how those relate yet, I haven't actually started building yet (I'm working out some integration questions with our existing scheduler). I guess what i need to know is, when setting up prefect server what is needed to ensure that my data flow history metadata is preserved. For example there will be hundreds of weekly/daily jobs, if the cluster dies I need to launch a new one and not lose the history of my data store. Related question: For Prefect Cloud, is there a way to export/backup that job run history in the case where we someday migrated off of Cloud?
z

Zanie

04/30/2021, 6:24 PM
All the persistent data is stored in postgres
There is some example IAC code at https://github.com/PrefectHQ/server/pull/171 for deploying server with an external database to Azure/GCP; might be helpful
We don't have a way to export data right now -- the Cloud database has additional things that the Server database does not so they're not strictly compatible.
You can query for your run data via GraphQL and pipe it to somewhere else, we just have row limits so you'd have to have multiple scoped queries.
m

merlin

04/30/2021, 7:47 PM
cool thats good insight, i'm doing probably too much thinking ahead.