https://prefect.io logo
d

delphi

07/29/2020, 9:35 AM
I have a beginner's question. Can it be possible to share the registered flows among team members? The scenario is as follows: 1. There are 2 or more members in the team, they are all either flow developer or flow runner 2. One member develop a flow according to the agent environment on his desktop PC and register the flow to Prefect Server 3. Another member can run this registered flow on his own desktop PC env 4. Due to different env parameters for different desktop PC's, e.g. different logged in account at the desktop, we cannot centralize the agent run even though the FLOW itself is the same 5. Even though the agents are distributed, we still want to make the launch of flows centered in Prefect Server UI The question is : from the document I can see there is no actual execution information exchange between agent and Server which I thought the Python scripts are also kept local. Then how I can achieve the effects above? The question can be divided into two: 1. How to share scripts between different members? Maybe Git or other version control is right solution which I want to confirm it 2. The flow register process --- Is it a must for different agents on different PC to register their own Flows to Server ? More background: This environment is a Prefect Server installed in an ECS server which can be accessed by each team member. I don't know whether it's feasible for us to do like this. The document only describe local scenario for Prefect Server. Thank you for your kind help!
j

josh

07/29/2020, 11:21 AM
Hi @delphi take a look into the other flow storage options outside of the default Local storage: https://docs.prefect.io/orchestration/execution/storage_options.html https://docs.prefect.io/api/latest/environments/storage.html For your questions: 1. Yeah for sharing scripts something like Git would be a good way to go 2. Agents pull flows with matching labels https://docs.prefect.io/orchestration/agents/overview.html#flow-affinity-labels and agents do not register the flows but users do (either manually or through some sort of CI/CD process)
d

Daniel

07/29/2020, 1:48 PM
+1 for using git to collaborate and version your code
a

Adam

07/29/2020, 1:56 PM
Hi @delphi, we use Git to store our code and collaborate. Above this, our flows have a method like:
Copy code
if __name__ == "__main__":
    flow.run()
This allows developers to just do
python flow.py
and have it run locally (they use the
prefect server start
etc commands to run it locally (i.e. when developing and testing). We also have a CI pipeline that when they commit their code its sent to our different environments. We have a separate
build.py
script that imports the above
flow.py
and does something like:
Copy code
flow.storage = Docker(
    registry_url="<http://gcr.io/registry/flows|gcr.io/registry/flows>",
    python_dependencies=["pandas", "prefect[google,kubernetes]"],
)
flow.register(project_name="prefect-test-1")
The CI server then calls
python build.py
instead, which then registers the flow rather than running it. Hope that makes sense
upvote 2
d

delphi

07/29/2020, 4:07 PM
@Adam Thank you Adam for sharing your practice, I will try your way!
@Chris White Thanks for the detailed explanation. Finally I realize that I have not read the whole documentation. I took the https://docs.prefect.io/core/... docs as the high level part. And take the https://docs.prefect.io/api/... docs as the detailed information. I just lost the whole Orchestration part, which I thought is just some guides for deployment. In fact it's the core concepts for the developers. Now I know the detailed information of Prefect concepts, e.g. the labels of an executor environment should fit with the Flow's, it solved my problem of how to allocate different execution resources according to different environment. Thank you!
💯 1
🚀 1