https://prefect.io logo
Title
a

Andy Dienes

04/27/2023, 1:40 PM
how can two projects share code? I have some
utils
I want to be common to all projects, but i want to develop and deploy
projA
and
projB
separately. I'm using the s3 recipe
f

flapili

04/27/2023, 1:41 PM
a git remote ?
or private package
a

Andy Dienes

04/27/2023, 1:42 PM
there's no way to do this inside a repo?
f

flapili

04/27/2023, 1:42 PM
then pip install git+https://[...]
it's better to do versionning and install it locally
a

Andy Dienes

04/27/2023, 1:43 PM
no I mean my common code doesn't really make sense to factor out into a new remote or package at all
f

flapili

04/27/2023, 1:43 PM
you could keep it in a monorepos
a

Andy Dienes

04/27/2023, 1:43 PM
I'm hoping to make a repo like "my-projects" and have a couple different subdirs for each project
but they can share some common source code
f

flapili

04/27/2023, 1:44 PM
but in my opinion sharing code without vesionning = fail
a

Andy Dienes

04/27/2023, 1:44 PM
ok, well what I'm doing now is just copying
utils
into each project subdir, so code duplication also = fail 🙂
f

flapili

04/27/2023, 1:45 PM
you could create a custom block too
in fact there are a lot of solution
you could do an "http" storage block as example
which would return only what needed according to the context
a

Andy Dienes

04/27/2023, 1:46 PM
src
  -flows
    -project_1
       deployment.yaml
       flow1.py
    -project_2
       deployment.yaml
       flow2.py
  -utils
    file.py
what is the absolute simplest least-boilerplate way to give each flow the ability to
import file
?
not trying to be adversarial I definitely appreciate the tips!
f

flapili

04/27/2023, 1:47 PM
without sharing the whole I don't know
hmm
is you issue more about import ?
or just about "1 don't need flow2 in project A" ?
a

Andy Dienes

04/27/2023, 1:48 PM
sorry, those are projects not flows
f

flapili

04/27/2023, 1:48 PM
didn't played yet with project sorry 😅
a

Andy Dienes

04/27/2023, 1:48 PM
somehow need to include
utils
in the
push
block?
ah ok, nw
maybe a github block is the way to go, just pointing to
src/utils
in the same repo? not sure how to make that work with projects
f

flapili

04/27/2023, 1:56 PM
I have no idea at all about prefect's project 😅
a

alex

04/27/2023, 2:04 PM
@Andy Dienes how is your code organized? Do your two projects and utils all live in different repositories?
a

Andy Dienes

04/27/2023, 2:07 PM
ideally I will have them like the file structure sketched above, with a monorepo, a common source code dir, and then a separate directory for various projects
I just want to separate "core" logic from "flow/glue" logic
r

redsquare

04/27/2023, 2:09 PM
we just shutil.copytree helper files from a root folder to a child folder of the flow
on deployment *
a

Andy Dienes

04/27/2023, 2:11 PM
is there somewhere I can specify that in the
prefect.yaml
or the
deployment.yaml
or are you doing it manually?
r

redsquare

04/27/2023, 2:11 PM
our deployments are in python
we use github actions to deploy
each flow has a separate deployment file
cookie cuttered up
we tried with packages but it was just friction everywhere
a

alex

04/27/2023, 2:13 PM
If your project is housed in one repo, you should be able to pull it down with a single
git_clone_project
step. It sounds like you’d need multiple deployments for this project, and each deployment would have a different entrypoint to the flow that it wraps. A UX for declaring multiple deployments in a project should be out later today, but you can check out this PR to see what the experience will be like.
a

Andy Dienes

04/27/2023, 2:17 PM
you should be able to pull it down with a single
git_clone_project
step
so this would go in my
pull
section? I'm using the
s3
recipe so right now it's pulling from s3
I also don't want to have to check in every change just to test a deployment
a

alex

04/27/2023, 2:19 PM
Gotcha, that should still work as long as all your code is in one project. The key is that you push and pull the entire project to and from somewhere. You can then deploy multiple flows from that project by changing the entrypoint and other configs between deployments.
a

Andy Dienes

04/27/2023, 2:24 PM
oh I see now
so instead of multiple projects I'll have one project for the whole repo and then multiple deployments ?
👍 1
the one deployment could push
flow1.py
and all of
utils
, and the second deployment would push only
flow2.py
and all of
utils
a

alex

04/27/2023, 2:31 PM
Yeah, that should be possible. Each deployment can have its own
push
and
pull
steps so you could push and pull each deployment to different locations. Each deployment would also need its own
.prefectignore
file to prevent uploading other flow files during deployment. It’s possible, but it also introduces additional complexity. What problem are you solving for by excluding files on a per-deployment basis?