https://prefect.io logo
a

Alberto de Santos

10/22/2020, 2:14 PM
Hi community, I have a question around launching Agents and keeping them working. I understand, one of the most handy options is just using
nohup
, however, in order to do that, I have to execute everything with the
flow.run_agent()
option and then comment that line of code to execute the flows. The alternative is through CLI, however, I couldn’t make it work (it doesn’t file the libraries I need to execute, despite of using the
-p
param) Which is your experience/view here?
r

Raphaël Riel

10/22/2020, 2:18 PM
Hi Alberto! On my side I built my own custom CLI (w/Click) to launch the Agent. I had to in order to build Dynamic labels and recover my Tokens from AWS SecretsManager.
Copy code
echo "Starting Agent"
pipenv run ./cli.py start-agent "${AGENT_NAME}" --token-secret-id=${AGENT_RUNNER_TOKEN_SECRET_ID}
This way I can keep my flow file intact for Local Dev.
a

Alberto de Santos

10/22/2020, 2:19 PM
BEAUTIFUL!!! This is precisely the inspiration I was looking for!
r

Raphaël Riel

10/22/2020, 2:19 PM
Poke me if you need more detail on my implementation!
I’m also registering my flows the same way. And I keep my Flows-Inventory in Python-code.
d

Dylan

10/22/2020, 2:21 PM
Raphaël I like that script! 💪
😊 1
r

Raphaël Riel

10/22/2020, 2:21 PM
Kinda primitive for now, but I’m expecting to add more Metadata in a near future
d

Dylan

10/22/2020, 2:23 PM
The only thing I’ll add is: if you’re looking to run Prefect in a production environment, you may want to consider moving to a containerized setup of some kind sooner rather than later
There’s a
DockerAgent
available as well as for Fargate and Kubernetes
r

Raphaël Riel

10/22/2020, 2:26 PM
I’m already running under Fargate/Docker. A) I’m building the image on my computer using either stable version or “dirty” github repo. B) Pushing Image to AWS ECR C) Launching a Fargate (Using AWS CDK) for my agent. D) The agent register all of its flow under his own label+project and E) Starts itself with the same labels.
d

Dylan

10/22/2020, 2:28 PM
Those comments were directed more toward Alberto since I recognize him as a newer community member from a week or two ago 😄
But that sounds like a robust setup to me! 💯
r

Raphaël Riel

10/22/2020, 2:28 PM
Oh! I tought these were for me 😛
d

Dylan

10/22/2020, 2:28 PM
haha no worries!
a

Alberto de Santos

10/22/2020, 5:37 PM
I am new, yes! Quite glad to be here, actually 🙂
marvin 1
p

Pedro Machado

10/22/2020, 11:24 PM
I am considering doing something similar for our set up in which we are running a DockerAgent in a Linux (ubuntu) VM. I had some docker permission issues running
prefect agent
with
supervisord
and was considering an alternative. Would docker-compose be a good way to run the agent inside of a container using a `cli`similar to Raphaël's? Is docker-compose with a restart policy a good way to restart the agent if it dies unexpectedly?
r

Raphaël Riel

10/23/2020, 12:11 PM
@Pedro Machado My Setup relies on Fargate to restart the Docker in the event of the agent exiting. The way I’ve setuped the Docker’s CMD, my CLI Script that boots the Agent is PID 1 (Using
dumb-init
) IMO this is a perfectly valid setup if you’re trusting docker-compose!
👍 1
a

Alberto de Santos

10/23/2020, 8:41 PM
@Raphaël Riel Being inspired by your script, do you mind sharing a bit more your
set_temporary_function
, please? My only purpose is to learn a way to do it. I did it already by my own (and it works), but wanted to know how you managed to do it with the
with set_temporary_function(params):
approach. Please, feel free not to answer 🙂
r

Raphaël Riel

10/26/2020, 11:48 AM
This method is built-in with Prefect. The Prefect’s agent CLI is using this method. I have no credit for this one 😛 See
from prefect.utilities.configuration import set_temporary_config
a

Alberto de Santos

10/26/2020, 11:48 AM
LOL 🙂
2 Views