Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Title
t
Tilak Maddy
12/06/2021, 10:54 AMUserWarning: A flow with the same name is already contained in storage; if you changed your Flow since the last build, you might experience unexpected issues and should re-create your storage object.
serialized_flow = flow.serialize(build=build) # type: Any
Flow URL: <https://cloud.prefect.io/XXX/flow/ZZZ>
└── ID: XXXX
└── Project: test_proj_2
└── Labels: ['dev']flow.register(project_name)a
Anna Geller
12/06/2021, 10:55 AMall the flows have the same nameThis is not allowed, flow name must be unique across a project
t
Tilak Maddy
12/06/2021, 10:56 AMSorry I meant I have a flow with the same flow name under different projects
a
Anna Geller
12/06/2021, 10:57 AMwhat storage do you use?
t
Tilak Maddy
12/06/2021, 10:57 AMGithub
All the flows have the same storage too
a
Anna Geller
12/06/2021, 10:57 AMit looks like the warning is useful and warns you that you may be overriding storage of one flow by a flow with the same name but from a different project
t
Tilak Maddy
12/06/2021, 10:58 AMI am okay with all flows having the same storage but while registering I register them with a different schedule (which is metadat ofc that cloud picks up) to different projects. Is it okay ?
(Actually I need all the flows to have the same storage aka the same flow file)
a
Anna Geller
12/06/2021, 11:00 AMI know you said you wish to reuse the same flow name across different project, but I would highly encourage you to change that strategy. The flow names usually identify a flow and keeping the same name (and the same storage) for multiple flows across projects may bite you at some point.
Can you explain the reason for reusing the same flow name but a different project? Perhaps it’s just a matter of finding the right abstraction to make your code reusable across those use cases (regardless of whether those use cases are identified by a flow name or by a flow’s project).
@Tilak Maddy how do you reuse the same flow file in Github storage to register the same flow twice with a different metadata? Is it just parameter values and schedules that differ across projects?
t
Tilak Maddy
12/06/2021, 11:47 AMIs it just parameter values and schedules that differ across projects?Yes you are spot on ! I will surely explain more, I had gone away for a while ... in a meeting.
👍 1
So this was my situation https://prefect-community.slack.com/archives/CL09KU1K7/p1638483576263700
As you can see @Anna Geller they have given me really * HARD * solutions to implement that work for production, like flow builder, a new flow that checks for flows, etc, etc. So ....what I decided was that for each ETL pipeline I'll create a new project and then register the same flow name with different schedule to its corresponding project. This way I just have to update storage in one place and call
flow.register(pipeline_id)a
Anna Geller
12/06/2021, 12:02 PMI see. If there is really nothing else that differs across flows than the parameter values and schedule, then this solution should work, provided that you set the parameter defaults and schedules using either the UI or GraphQL API: you can see below the same flow in project p and xyz but with different parameter defaults set through UI
❤️ 1
t
Tilak Maddy
12/06/2021, 12:13 PMHey thanks @Anna Geller I'll read up on that - default parameters things and get back to you in a while. Point to note: I need to also think how do I keep track of the all the flow runs of a particular pipeline. For instance if it was a separate project then I would have a flow ID and it would just be a matter of getting all the flow runs of that ID . Like at least we may need to query like average time that the last 5 runs have taken, etc etc. ALSO for now I am not too sure if I really need all that information. .So yeah..I'll think about and see ya !
On a slight tangent , why don't use DARK MODE 😡 ?
a
Anna Geller
12/06/2021, 12:21 PM@Tilak Maddy this should be doable with GraphQL, the duration or run statistics would either way have to be calculated through flow runs and task runs, not flows. You would then probably put your flow name and project name into the where conditions
There is an experimental dark mode you can toggle here
t
Tilak Maddy
12/06/2021, 12:24 PMHey i use dark mode, i was just wondering why you weren't 😛 Not really angry - I am terrible emoji user (always fail to communicate the right thing)
a
Anna Geller
12/06/2021, 12:30 PMI don’t spend that much time in the UI so normal white screen doesn’t bother me at all there tbh 🙂
but in a code editor that’s a different story, I totally understand you