Hello Noob question here How do I execute an agent run docke

Question

Hello Noob question here How do I execute an agent run docker agent on a server so that if I exit logout of the server the process agent stays running executing scheduled flows Your help is much appreciated

Zanie · Answer

Hey < Stephen Mugisha> `screen` is one of the simplest ways to do this

Zanie · Answer

Zanie · Answer

`screen S my agent` gt `prefect agent docker start` gt Ctrl A d exit screen

Stephen Mugisha · Answer

Thanks very much < Zanie>

merlin · Answer

another noob question is this something we would need to do usually I was thinking the container configs would start the agent along with everything else but I haven t got that far yet

Zanie · Answer

If you re using `prefect server start` the agent is started separately if you re using the Prefect Server helm chart you can install the agent to your cluster at the same time if you re using Prefect Cloud you ll need to start an agent somewhere The agent is in charge of spawning the containers to run flows

merlin · Answer

So the agent can run on the k8s cluster which also hosts the server and UI and apollo endpoint I guess Hasura Graphql and towel could also be on the cluster but we would want postgress to be persistent as RDS or at a minimum getting regular snapshots saved to s3

merlin · Answer

My planned architecture is on kubernetes cluster in AWS I m expecting to use prefect to orchestrate data engineering pipelines to manage a persistent data store for a medium sized team on AWS Is it important to persist graphQL data I assume the postgres DB has important flow history and execution history that will be critical to store

Zanie · Answer

Happy to hear from you slightly smiling face

Zanie · Answer

Are you separating GraphQL data from the postgres data

merlin · Answer

Hi I don t know how those relate yet I haven t actually started building yet I m working out some integration questions with our existing scheduler I guess what i need to know is when setting up prefect server what is needed to ensure that my data flow history metadata is preserved For example there will be hundreds of weekly daily jobs if the cluster dies I need to launch a new one and not lose the history of my data store Related question For Prefect Cloud is there a way to export backup that job run history in the case where we someday migrated off of Cloud

Zanie · Answer

All the persistent data is stored in postgres

Zanie · Answer

There is some example IAC code at <https github com PrefectHQ server pull 171> for deploying server with an external database to Azure GCP might be helpful

Zanie · Answer

We don t have a way to export data right now the Cloud database has additional things that the Server database does not so they re not strictly compatible

Zanie · Answer

You can query for your run data via GraphQL and pipe it to somewhere else we just have row limits so you d have to have multiple scoped queries

merlin · Answer

cool thats good insight i m doing probably too much thinking ahead