how can two projects share code? I have some `utils` I want to be common to all projects, but i want...

Andy Dienes

04/27/2023, 1:40 PM

how can two projects share code? I have some

utils

I want to be common to all projects, but i want to develop and deploy

projA

and

projB

separately. I'm using the s3 recipe

flapili

04/27/2023, 1:41 PM

a git remote ?

flapili

04/27/2023, 1:41 PM

or private package

Andy Dienes

04/27/2023, 1:42 PM

there's no way to do this inside a repo?

flapili

04/27/2023, 1:42 PM

then pip install git+https://[...]

flapili

04/27/2023, 1:42 PM

it's better to do versionning and install it locally

Andy Dienes

04/27/2023, 1:43 PM

no I mean my common code doesn't really make sense to factor out into a new remote or package at all

flapili

04/27/2023, 1:43 PM

you could keep it in a monorepos

Andy Dienes

04/27/2023, 1:43 PM

I'm hoping to make a repo like "my-projects" and have a couple different subdirs for each project

Andy Dienes

04/27/2023, 1:43 PM

but they can share some common source code

flapili

04/27/2023, 1:44 PM

but in my opinion sharing code without vesionning = fail

Andy Dienes

04/27/2023, 1:44 PM

ok, well what I'm doing now is just copying

utils

into each project subdir, so code duplication also = fail 🙂

flapili

04/27/2023, 1:45 PM

you could create a custom block too

flapili

04/27/2023, 1:45 PM

in fact there are a lot of solution

flapili

04/27/2023, 1:45 PM

you could do an "http" storage block as example

flapili

04/27/2023, 1:46 PM

which would return only what needed according to the context

Andy Dienes

04/27/2023, 1:46 PM

Copy code

src
  -flows
    -project_1
       deployment.yaml
       flow1.py
    -project_2
       deployment.yaml
       flow2.py
  -utils
    file.py

Andy Dienes

04/27/2023, 1:46 PM

what is the absolute simplest least-boilerplate way to give each flow the ability to

import file

Andy Dienes

04/27/2023, 1:47 PM

not trying to be adversarial I definitely appreciate the tips!

flapili

04/27/2023, 1:47 PM

without sharing the whole I don't know

flapili

04/27/2023, 1:47 PM

hmm

flapili

04/27/2023, 1:47 PM

is you issue more about import ?

flapili

04/27/2023, 1:48 PM

or just about "1 don't need flow2 in project A" ?

Andy Dienes

04/27/2023, 1:48 PM

sorry, those are projects not flows

flapili

04/27/2023, 1:48 PM

didn't played yet with project sorry 😅

Andy Dienes

04/27/2023, 1:48 PM

somehow need to include

utils

in the

push

block?

Andy Dienes

04/27/2023, 1:48 PM

ah ok, nw

Andy Dienes

04/27/2023, 1:51 PM

maybe a github block is the way to go, just pointing to

src/utils

in the same repo? not sure how to make that work with projects

flapili

04/27/2023, 1:56 PM

I have no idea at all about prefect's project 😅

alex

04/27/2023, 2:04 PM

@Andy Dienes how is your code organized? Do your two projects and utils all live in different repositories?

Andy Dienes

04/27/2023, 2:07 PM

ideally I will have them like the file structure sketched above, with a monorepo, a common source code dir, and then a separate directory for various projects

Andy Dienes

04/27/2023, 2:08 PM

I just want to separate "core" logic from "flow/glue" logic

redsquare

04/27/2023, 2:09 PM

we just shutil.copytree helper files from a root folder to a child folder of the flow

redsquare

04/27/2023, 2:10 PM

on deployment *

Andy Dienes

04/27/2023, 2:11 PM

is there somewhere I can specify that in the

prefect.yaml

or the

deployment.yaml

or are you doing it manually?

redsquare

04/27/2023, 2:11 PM

our deployments are in python

redsquare

04/27/2023, 2:12 PM

we use github actions to deploy

redsquare

04/27/2023, 2:12 PM

each flow has a separate deployment file

redsquare

04/27/2023, 2:12 PM

cookie cuttered up

redsquare

04/27/2023, 2:13 PM

we tried with packages but it was just friction everywhere

alex

04/27/2023, 2:13 PM

If your project is housed in one repo, you should be able to pull it down with a single

git_clone_project

step. It sounds like you’d need multiple deployments for this project, and each deployment would have a different entrypoint to the flow that it wraps. A UX for declaring multiple deployments in a project should be out later today, but you can check out this PR to see what the experience will be like.

Andy Dienes

04/27/2023, 2:17 PM

you should be able to pull it down with a single
git_clone_project
step

so this would go in my

pull

section? I'm using the

s3

recipe so right now it's pulling from s3

Andy Dienes

04/27/2023, 2:17 PM

I also don't want to have to check in every change just to test a deployment

alex

04/27/2023, 2:19 PM

Gotcha, that should still work as long as all your code is in one project. The key is that you push and pull the entire project to and from somewhere. You can then deploy multiple flows from that project by changing the entrypoint and other configs between deployments.

Andy Dienes

04/27/2023, 2:24 PM

oh I see now

Andy Dienes

04/27/2023, 2:25 PM

so instead of multiple projects I'll have one project for the whole repo and then multiple deployments ?

👍 1

Andy Dienes

04/27/2023, 2:27 PM

but can I specify a list of folders? so in the example above https://prefect-community.slack.com/archives/C03D12VV4NN/p1682603181745109?thread_ts=1682602803.506389&cid=C03D12VV4NN

Andy Dienes

04/27/2023, 2:27 PM

the one deployment could push

flow1.py

and all of

utils

, and the second deployment would push only

flow2.py

and all of

utils

alex

04/27/2023, 2:31 PM

Yeah, that should be possible. Each deployment can have its own

push

and

pull

steps so you could push and pull each deployment to different locations. Each deployment would also need its own

.prefectignore

file to prevent uploading other flow files during deployment. It’s possible, but it also introduces additional complexity. What problem are you solving for by excluding files on a per-deployment basis?

86 Views

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.