how can two projects share code? I have some `util...
# best-practices
a
how can two projects share code? I have some
utils
I want to be common to all projects, but i want to develop and deploy
projA
and
projB
separately. I'm using the s3 recipe
f
a git remote ?
or private package
a
there's no way to do this inside a repo?
f
then pip install git+https://[...]
it's better to do versionning and install it locally
a
no I mean my common code doesn't really make sense to factor out into a new remote or package at all
f
you could keep it in a monorepos
a
I'm hoping to make a repo like "my-projects" and have a couple different subdirs for each project
but they can share some common source code
f
but in my opinion sharing code without vesionning = fail
a
ok, well what I'm doing now is just copying
utils
into each project subdir, so code duplication also = fail 🙂
f
you could create a custom block too
in fact there are a lot of solution
you could do an "http" storage block as example
which would return only what needed according to the context
a
Copy code
src
  -flows
    -project_1
       deployment.yaml
       flow1.py
    -project_2
       deployment.yaml
       flow2.py
  -utils
    file.py
what is the absolute simplest least-boilerplate way to give each flow the ability to
import file
?
not trying to be adversarial I definitely appreciate the tips!
f
without sharing the whole I don't know
hmm
is you issue more about import ?
or just about "1 don't need flow2 in project A" ?
a
sorry, those are projects not flows
f
didn't played yet with project sorry 😅
a
somehow need to include
utils
in the
push
block?
ah ok, nw
maybe a github block is the way to go, just pointing to
src/utils
in the same repo? not sure how to make that work with projects
f
I have no idea at all about prefect's project 😅
a
@Andy Dienes how is your code organized? Do your two projects and utils all live in different repositories?
a
ideally I will have them like the file structure sketched above, with a monorepo, a common source code dir, and then a separate directory for various projects
I just want to separate "core" logic from "flow/glue" logic
r
we just shutil.copytree helper files from a root folder to a child folder of the flow
on deployment *
a
is there somewhere I can specify that in the
prefect.yaml
or the
deployment.yaml
or are you doing it manually?
r
our deployments are in python
we use github actions to deploy
each flow has a separate deployment file
cookie cuttered up
we tried with packages but it was just friction everywhere
a
If your project is housed in one repo, you should be able to pull it down with a single
git_clone_project
step. It sounds like you’d need multiple deployments for this project, and each deployment would have a different entrypoint to the flow that it wraps. A UX for declaring multiple deployments in a project should be out later today, but you can check out this PR to see what the experience will be like.
a
you should be able to pull it down with a single
git_clone_project
step
so this would go in my
pull
section? I'm using the
s3
recipe so right now it's pulling from s3
I also don't want to have to check in every change just to test a deployment
a
Gotcha, that should still work as long as all your code is in one project. The key is that you push and pull the entire project to and from somewhere. You can then deploy multiple flows from that project by changing the entrypoint and other configs between deployments.
a
oh I see now
so instead of multiple projects I'll have one project for the whole repo and then multiple deployments ?
👍 1
the one deployment could push
flow1.py
and all of
utils
, and the second deployment would push only
flow2.py
and all of
utils
a
Yeah, that should be possible. Each deployment can have its own
push
and
pull
steps so you could push and pull each deployment to different locations. Each deployment would also need its own
.prefectignore
file to prevent uploading other flow files during deployment. It’s possible, but it also introduces additional complexity. What problem are you solving for by excluding files on a per-deployment basis?