m

    Michael Fichtinger

    1 year ago
    Quick question from a newbie in prefect. What is the easiest way to set up prefect and the agents to distribute flows to different machines in the local network (see picture). Maybe you can give me some advice and share some readings regarding this. Thanks
    Kevin Kho

    Kevin Kho

    1 year ago
    Hey @Michael Fichtinger, this setup can’t quite be done in Prefect Server because there is no mechanism that lets you do what you want with Flow 3 where the agent that is available will grab it. In Prefect Cloud, it would involve setting up a Flow Run Concurrency Limit on flows 1, 2, and 3 so that only one could run at a time. Then Flow 3 will naturally be queued if you have flow 1 and 2 already running, and then it will run on the next available machine. Flow Run Concurrency is not in Server, so we have limited options. 1. You can just attach Flow 3 as a dependency for either flow 1 or 2. This is the simplest and then you just live with the delay. 2. You can start flow 3 with either flow 1 or 2. You start it at the end. Do a quick check to see if a copy of flow 3 is already running before you trigger a run. You can do this with the API. 3. You can provide an idempotency key to flow 3 when you start it. This is good for 24 hours and runs that share the same key will not trigger multiple copies. So only the first run will trigger. This will only work if your flow is ran at a greater interval than 24 hours.
    m

    Michael Fichtinger

    1 year ago
    Hi @Kevin Kho Thanks for the quick response. 🙂 I fear my question is a more basic one. My question was more about how to set up different local machines to run some flows. If I start the prefect server locally, I can easily run my flows. That was easy and is already quite cool. The next thought which came to my mind was how do I run flows on different machine. My simple approach was to just start local agents on different machines and then label the flows accordingly. But that does not work since Prefect complains that it can't find the flows, because they are stored in ~/.prefect/flows/ on the machine where I started the server.
    Kevin Kho

    Kevin Kho

    1 year ago
    Ohh…you can’t use local Storage then. You need to use something like S3 storage or Github storage. Just something that lets those other machine pull the flow down. Storage docs.
    m

    Michael Fichtinger

    1 year ago
    I thought the same and currently trying that with github, but I get an error when setting this up. Failed to load and execute Flow's environment: UnknownObjectException(404, {'message': 'Not Found', 'documentation_url': 'https://docs.github.com/rest'},
    I'm not quite sure how to pass the token as a secret and if I register the flow correctly.
    Kevin Kho

    Kevin Kho

    1 year ago
    I have this demo but it’s public. You could even register it and it would pull it down from this repo.
    You pass in the secret name, and then it would retrieve the secret. Are you on Prefect Cloud?
    m

    Michael Fichtinger

    1 year ago
    No, I'm not.
    Kevin Kho

    Kevin Kho

    1 year ago
    I think that’s fine. You just need to set a local secret.
    m

    Michael Fichtinger

    1 year ago
    import prefect
    from prefect.core import Flow
    from prefect.storage import GitHub
    from prefect.utilities.tasks import task
    from prefect.tasks.shell import ShellTask
    from prefect.client.secrets import Secret
    
    shell_task = ShellTask(helper_script="cd ~", return_all=True, stream_output=True)
    
    
    @task()
    def print_result(result):
        logger = prefect.context.get("logger")
        <http://logger.info|logger.info>(result)
    
    
    with Flow("test flow github") as flow:
        content = shell_task(command="echo Test")
        print_result(content)
    
    p = Secret('token')
    
    flow.storage = GitHub(
        repo="<https://github.com/repo/flows>",  # name of repo
        path="my_flows/test1_flow.py",  # location of flow file in repo
        access_token_secret=p.get(),  # name of personal access token secret
    )
    That's what I have. But it's a private repo.
    Kevin Kho

    Kevin Kho

    1 year ago
    Try
    access_token_secret="token"
    . Prefect will pull the Secret with that name later
    m

    Michael Fichtinger

    1 year ago
    That did the trick. Thanks @Kevin Kho