https://prefect.io logo
Powered by
Title
a

Andy Dienes

04/27/2023, 1:40 PM
how can two projects share code? I have some 
utils
I want to be common to all projects, but i want to develop and deploy 
projA
and 
projB
separately. I'm using the s3 recipe
f

flapili

04/27/2023, 1:41 PM
a git remote ?
or private package
a

Andy Dienes

04/27/2023, 1:42 PM
there's no way to do this inside a repo?
f

flapili

04/27/2023, 1:42 PM
then pip install git+https://[...]
it's better to do versionning and install it locally
a

Andy Dienes

04/27/2023, 1:43 PM
no I mean my common code doesn't really make sense to factor out into a new remote or package at all
f

flapili

04/27/2023, 1:43 PM
you could keep it in a monorepos
a

Andy Dienes

04/27/2023, 1:43 PM
I'm hoping to make a repo like "my-projects" and have a couple different subdirs for each project
but they can share some common source code
f

flapili

04/27/2023, 1:44 PM
but in my opinion sharing code without vesionning = fail
a

Andy Dienes

04/27/2023, 1:44 PM
ok, well what I'm doing now is just copying 
utils
into each project subdir, so code duplication also = fail 🙂
f

flapili

04/27/2023, 1:45 PM
you could create a custom block too
in fact there are a lot of solution
you could do an "http" storage block as example
which would return only what needed according to the context
a

Andy Dienes

04/27/2023, 1:46 PM
src
  -flows
    -project_1
       deployment.yaml
       flow1.py
    -project_2
       deployment.yaml
       flow2.py
  -utils
    file.py
what is the absolute simplest least-boilerplate way to give each flow the ability to 
import file
?
not trying to be adversarial I definitely appreciate the tips!
f

flapili

04/27/2023, 1:47 PM
without sharing the whole I don't know
hmm
is you issue more about import ?
or just about "1 don't need flow2 in project A" ?
a

Andy Dienes

04/27/2023, 1:48 PM
sorry, those are projects not flows
f

flapili

04/27/2023, 1:48 PM
didn't played yet with project sorry 😅
a

Andy Dienes

04/27/2023, 1:48 PM
somehow need to include 
utils
in the 
push
block?
ah ok, nw
maybe a github block is the way to go, just pointing to 
src/utils
in the same repo? not sure how to make that work with projects
f

flapili

04/27/2023, 1:56 PM
I have no idea at all about prefect's project 😅
a

alex

04/27/2023, 2:04 PM
@Andy Dienes how is your code organized? Do your two projects and utils all live in different repositories?
a

Andy Dienes

04/27/2023, 2:07 PM
ideally I will have them like the file structure sketched above, with a monorepo, a common source code dir, and then a separate directory for various projects
I just want to separate "core" logic from "flow/glue" logic
r

redsquare

04/27/2023, 2:09 PM
we just shutil.copytree helper files from a root folder to a child folder of the flow
on deployment *
a

Andy Dienes

04/27/2023, 2:11 PM
is there somewhere I can specify that in the 
prefect.yaml
or the 
deployment.yaml
or are you doing it manually?
r

redsquare

04/27/2023, 2:11 PM
our deployments are in python
we use github actions to deploy
each flow has a separate deployment file
cookie cuttered up
we tried with packages but it was just friction everywhere
a

alex

04/27/2023, 2:13 PM
If your project is housed in one repo, you should be able to pull it down with a single 
git_clone_project
step. It sounds like you’d need multiple deployments for this project, and each deployment would have a different entrypoint to the flow that it wraps. A UX for declaring multiple deployments in a project should be out later today, but you can check out this PR to see what the experience will be like.
a

Andy Dienes

04/27/2023, 2:17 PM
you should be able to pull it down with a single 
git_clone_project
step
so this would go in my 
pull
section? I'm using the 
s3
recipe so right now it's pulling from s3
I also don't want to have to check in every change just to test a deployment
a

alex

04/27/2023, 2:19 PM
Gotcha, that should still work as long as all your code is in one project. The key is that you push and pull the entire project to and from somewhere. You can then deploy multiple flows from that project by changing the entrypoint and other configs between deployments.
a

Andy Dienes

04/27/2023, 2:24 PM
oh I see now
so instead of multiple projects I'll have one project for the whole repo and then multiple deployments ?
👍 1
but can I specify a list of folders? so in the example above https://prefect-community.slack.com/archives/C03D12VV4NN/p1682603181745109?thread_ts=1682602803.506389&cid=C03D12VV4NN
the one deployment could push 
flow1.py
and all of 
utils
, and the second deployment would push only 
flow2.py
and all of 
utils
a

alex

04/27/2023, 2:31 PM
Yeah, that should be possible. Each deployment can have its own 
push
and 
pull
steps so you could push and pull each deployment to different locations. Each deployment would also need its own 
.prefectignore
file to prevent uploading other flow files during deployment. It’s possible, but it also introduces additional complexity. What problem are you solving for by excluding files on a per-deployment basis?
75 Views
#best-practices-coordination-planeJoin Slack
Powered by