Hi. I have a Linux VM hosted on GCP that I 'm using as a single node for running Prefect Server + a LocalAgent. I added a unit file for each to auto-start them. I SSH'ed into the VM, installed docker + docker-compose, then _pip install_ed prefect. However, prefect is now only available under my user (GCP SSH logs into your GCP user by default). (

which prefect

==

/home/staljaard/.local/bin/prefect

) How would you recommend installing/configuring my prefect CLI and _pip install_ed prefect (agent) to be available for all users SSH'ing into the VM?

Hi @Stéphan Taljaard, I think the general question here is how to get any python library available to all users on Linux. Based on this , the solutions seem a bit ugly. In general though, Prefect saves flows and results in the user directory that I feel you may run into inconsistencies if you try to share stuff. I think the recommendation would be to make a common user.

You can also configure the Prefect home directory to be anywhere though 🙂

Hi, I've installed Prefect on a VM and having some issues... I can see the dashboard, but have issues with the agents. running

prefect agent docker start

, I see

INFO - agent | Waiting for flow runs...

but I do not see any agent when going to

<VM IP>:8080/?agents=

• How did you run the agent in a way that it is shows in the

<VM IP>:8080/?agents=

? • How did you setup the Prefect connection in python code running on different machines (other VMs) to connect to that specific instance? Thanks

Hey @YD, I assume you did

prefect backend server

already? And also configured your API endpoint to that VM? Were you able to register a flow and have it appear?

@Kevin Kho I did run

prefect backend server

and

prefect server start --detach

not sure where are the instruction on how to configure my API endpoint to that VM. can you please point me to them ? flow does not appear and neither that agent

Check this . You want this configuration

apollo_url="http://<your-ip-address>:4200/graphql"

Current ports

Just confirming that you have server on the VM and you’re trying to to register flows spin up an agent on a separate machine that hits the API on the VM right? The

config.toml

will be on the agent side/registration machine side. You can create it if it doesn’t exist. Let me find a template. Are you using Docker for the agent?

I have a server on a VM 1. for testing: I am trying to run basic code on that VM. just like I did when I ran Prefect on my laptop. 2. for usage: I want to run python code that uses Prefect server, on different machine, another VM or my laptop When I ran it on my laptop. in the python code I had

LocalAgent().start()

, this started an agent that stayed on. On the VM, running this basic code does not appear to work. I tried starting the agent from the python code and from the command line, both with and without docker. it looks like it starting an agent, but it just waits for flow run, and does not show up in the dashboard, and does not persist

I see ok. So a couple of things. First, you normally would split out the flow and the agent starting. So in this case:

def main():
    with Flow("hello_world") as flow:
        name = Parameter('name')
        say_hello(name='world')
    flow.run()
    # flow.register(project_name="Test")

flow.run()

would be for local testing.

flow.register()

would be to register to Cloud/Server when ready. Then you can start your agent with another script:

from prefect.agent.local import LocalAgent
LocalAgent().start()

This would then start your agent if you call it with

python ____.py

. The reason is that the previous setup spins up an agent every time you register. You want to separate these. An agent can run multiple flows. You can also do

prefect local agent start

with the command line instead of the script. Now there are two things to test. You said that the agent is not showing in the UI. If you do

python _____.py

to register the flow, does it show in the UI? If the registered flow does not show in the UI, and the agent is running but does not show in the UI, then it means the API endpoint that these are hitting is not the one you want. By default with server, it will hit

localhost:4200

. You want it to hit

VM IP:4200

instead. This is why you need the

config.toml

in your

.prefect

folder in the home directory of the machine the agent is running on to point to the right API endpoint. You file should look like

backend = "server"

[server]
host = "<http://SERVER-VM-IP>"
port = "4200"
host_port = "4200"

This will make the agent and

flow.register()

send requests to the right API.

I still doing something wrong I want to run the Agent on the Prefect VM server (is this OK ?) I created

config.toml

in

~/.prefect

with the following content:

[server]
host = "<http://aws31-x01-my01>"
port = "4200"
host_port = "4200"

[server.ui]
apollo_url="<http://aws31-x01-my01:4200/graphql>"

then I ran

prefect backend server

prefect server stop

prefect server start -d

but it looks like it ignores the config file

You need the agent on your laptop configured to hit the API endpoint on the VM. The agent checks for flows to run every 10 seconds. If it finds flows to run, it will create the process on the laptop (or container or somewhere else). The agent is responsible for creating the process. You need to configure the agent with the IP address similar to what you did previously

So to make sure I understand.. On every machine that I want to use the Prefect server I need to • install prefect • setup the config.toml with Prefect server ip • start agent (do I need to do it for each user on the VM ?) I should not run

prefect backend server

or

prefect server start

or install Docker, correct?

You should run

prefect backend server

because the default is to use the Prefect Cloud API end point. You should not do

prefect server start

. You would need to install docker if your flows depend on it

Seams to be working... Thanks... thought I do see this in the Agent page

This means the agent is off. Did you turn off the process?

No.. it appears to be running

I think you mean your server is running, but I’m saying that agent is not running? That’s why the flows are late here. There is not agent to pick them up.

OK, will run the following on my laptop

% prefect agent docker start

Since it’s yellow, I assume this was able to communicate with your server for a bit. Did you turn off the process? It has to continue running

I did not turn off anything, as far as I am aware. not on my laptop and not on the Prefect server VM

Could you show be what the agent is saying? Do you get something like this?

I do notice that it did not create a new agent when I started it on my laptop. the agent in the dashboard is the one I started on the Prefect server VM is this expected ?

Yeah there is none. You need to use something like supervisor to keep the process running. The agent is meant to be a long running process

OK... checking this out

Just realized your profile picture LOL

I am big fun of the hitchhiker :) I am aware... this is my next step... our company is strict on security, so first I want to see I can use it on prem. then I will try the cloud. BTW, when I use the cloud service, will I also need to use

supervisor

?

Yes for the agents. That’s right

it looks like when using

supervisor

I must provide label, is this expected ?

Not expected, labels should be separate from supervisor. Supervisor should just be for running the process. You may need matching labels to run your flows though. Are you familiar with labels?

When I started agent without

supervisor

I did not have to provide a label, but when I use

supervisor

I had to add

--label  <computer name>

for it to run. and it created a new agent on the Prefect server

Oh ok I see what you are saying now. Yes I guess you need to provide the label then. Seems like it doesn’t attach the default label. I would also encourage you to add labels with the multiple machine setup to that the right machines pick up the corresponding flows