https://prefect.io logo
s

Simone Cittadini

06/23/2020, 8:02 AM
On a side note, I've noticed that if you call register on the same flow ( no code changes ) multiple times it creates multiple versions, is it considered a bug or that is how it works ? I can't let anyone register whatever they want from the laptop, so in my plan this is how it will work: • code is made in a branch of a common "all flows" repository • once approved is merged and the pipeline deploys the package in the agent, which proceeds to register everything If I dumbly register everything all the n flow already existing and untouched will bump their revision, not optimal ( not a problem if this is just how it works, I'll make the scan of code less dumb with some hash checking, but of course if it's just a matter of waiting a fix I'll gladly just wait 😁 )
j

Jeremiah

06/23/2020, 11:41 AM
Hi @Simone Cittadini - that is how it works 😄 Prefect automatically versions all flows, and does a “best-guess” that a flow registered to the same project and with the same name as an existing flow is a new version of that flow (you can control this behavior by providing a versioning id). Prefect currently randomly assigns slugs to tasks for tracking, which has lots of advantages but means that re-running a flow generation script will result in different (random) slugs. A few days ago, we merged the first PR in a series of enhancements designed to make ID assignment stable across multiple runs of the script. One of our use cases is to use (for example) a hash of task slugs to identify whether a flow has changed. This makes it possible, for example, to have hot-reloading flows without re-registering. tldr; that is how it works, but starting in the next release you’ll have more tools for checking and customizing this behavior yourself!
upvote 1
j

Joe Schmid

06/23/2020, 12:42 PM
Hi @Simone Cittadini, here's how we address Flow registration: 1. For deployment in our staging and prod environments, our CICD pipeline registers Flows in a Prefect project for that environment, e.g. "DS-prod" (for Data Science production) 2. For local development and testing from laptops, engineers and data scientists use their own Prefect project dedicated to them. We just have them use first name and last initial, e.g. "JoeS" For #2, they use a .env file with an entry like:
Copy code
PREFECT_PROJECT_NAME=JoeS
And then we use a docker-compose.yml that let's them easily start a Prefect Local Agent using a label that matches their personal project name.
Copy code
services:
  flows:
    image: <image>
    container_name: flows
    command: "bash -c \"prefect auth login -t $PREFECT__CLOUD__USER__AUTH_TOKEN && prefect agent start local --label $PREFECT_PROJECT_NAME\""
    network_mode: "host"
    volumes:
      - ".:/opt/srm-ds-flows"
They can then either run Flows directly in that Docker container, register Flows with cloud, and run Flows from Cloud which then get executed on that Local Agent. All of this works because Flows with identical names but in different Prefect projects are treated as distinct Flows.
upvote 1