Channels
#prefect-community
Title
m
Michael Fichtinger
08/04/2021, 8:36 AMQuick question from a newbie in prefect.
What is the easiest way to set up prefect and the agents to distribute flows to different machines in the local network (see picture).
Maybe you can give me some advice and share some readings regarding this.
Thanks
k
Kevin Kho
08/04/2021, 2:04 PM
Hey @Michael Fichtinger, this setup can’t quite be done in Prefect Server because there is no mechanism that lets you do what you want with Flow 3 where the agent that is available will grab it.
In Prefect Cloud, it would involve setting up a Flow Run Concurrency Limit on flows 1, 2, and 3 so that only one could run at a time. Then Flow 3 will naturally be queued if you have flow 1 and 2 already running, and then it will run on the next available machine. Flow Run Concurrency is not in Server, so we have limited options.
1. You can just attach Flow 3 as a dependency for either flow 1 or 2. This is the simplest and then you just live with the delay.
2. You can start flow 3 with either flow 1 or 2. You start it at the end. Do a quick check to see if a copy of flow 3 is already running before you trigger a run. You can do this with the API.
3. You can provide an idempotency key to flow 3 when you start it. This is good for 24 hours and runs that share the same key will not trigger multiple copies. So only the first run will trigger. This will only work if your flow is ran at a greater interval than 24 hours.
m
Michael Fichtinger
08/04/2021, 2:31 PMHi @Kevin Kho
Thanks for the quick response. 🙂
I fear my question is a more basic one.
My question was more about how to set up different local machines to run some flows.
If I start the prefect server locally, I can easily run my flows. That was easy and is already quite cool.
The next thought which came to my mind was how do I run flows on different machine. My simple approach was to just start local agents on different machines and then label the flows accordingly. But that does not work since Prefect complains that it can't find the flows, because they are stored in ~/.prefect/flows/ on the machine where I started the server.
k
Kevin Kho
08/04/2021, 2:32 PM
Ohh…you can’t use local Storage then. You need to use something like S3 storage or Github storage. Just something that lets those other machine pull the flow down. Storage docs.
m
Michael Fichtinger
08/04/2021, 2:34 PMI thought the same and currently trying that with github, but I get an error when setting this up.
Failed to load and execute Flow's environment: UnknownObjectException(404, {'message': 'Not Found', 'documentation_url': 'https://docs.github.com/rest'},
I'm not quite sure how to pass the token as a secret and if I register the flow correctly.
k
Kevin Kho
08/04/2021, 2:37 PM
I have this demo but it’s public. You could even register it and it would pull it down from this repo.
You pass in the secret name, and then it would retrieve the secret. Are you on Prefect Cloud?
m
Michael Fichtinger
08/04/2021, 2:39 PM Copy code
No, I'm not.k
Kevin Kho
08/04/2021, 2:39 PM
I think that’s fine. You just need to set a local secret.
m
Michael Fichtinger
08/04/2021, 2:39 PM Copy code
import prefect
from prefect.core import Flow
from prefect.storage import GitHub
from prefect.utilities.tasks import task
from prefect.tasks.shell import ShellTask
from prefect.client.secrets import Secret
shell_task = ShellTask(helper_script="cd ~", return_all=True, stream_output=True)
@task()
def print_result(result):
logger = prefect.context.get("logger")
<http://logger.info|logger.info>(result)
with Flow("test flow github") as flow:
content = shell_task(command="echo Test")
print_result(content)
p = Secret('token')
flow.storage = GitHub(
repo="<https://github.com/repo/flows>", # name of repo
path="my_flows/test1_flow.py", # location of flow file in repo
access_token_secret=p.get(), # name of personal access token secret
)That's what I have. But it's a private repo.
k
Kevin Kho
08/04/2021, 2:40 PM
Try
access_token_secret="token"m
Michael Fichtinger
08/04/2021, 2:54 PMThat did the trick. Thanks @Kevin Kho
👍 1
4 Views