https://prefect.io logo
#prefect-community
Title
# prefect-community
d

Dylan

05/10/2022, 7:21 PM
Is there a convenient way to override the flow execute command?
I’m installing my project using
pdm
and other tools like poetry and venv behave similarly, in that they don’t expect projects to be installed globally, and are instead expected to be scoped to an environment on the host.
I’d like to install the project in a docker image
pdm install
where prefect is a project dependency, among many.
If I install prefect globally with
pip install prefect
I expect that the flow run will fail, as the prefect invocation is outside the bounded scope of a project installed with pdm. This is similar to how govendor and bundler work.
Ideally I’d be able to specify a working directory and then something to the effect of
pdm run prefect execute flow-run
so that the flow-run scopes to the installed project
As an alternative to modifying the execute command, I’m going to use pdm to install the project globally and see if that works.
Nope
No dice
k

Kevin Kho

05/10/2022, 7:56 PM
I am thinking here. Will respond in a bit
d

Dylan

05/10/2022, 8:03 PM
Thanks, let me know if my ask and context are clear. I don’t think I’m doing anything out of the ordinary here. It seems pretty well aligned with project builds to use a tool to build and install a project. Packaging has always been a bit painful in python. Circle CI uses poetry in their python image for example, and pdm is like poetry but allows for environment variable expansion, making is a great choice for CI pipelines that need to pull from private github repo’s. I wouldn’t expect
prefect
to be on the global path of any image but the prefect image, and if I’m installing a project globally (basically at the OS level) I’m breaking modularity and the setup for that has to deviate substantially from installing a project locally for development work.
I decided to take a short cut and take on a bit of tech debt for the time being, just installing stuff globally with pip. pyproject.toml support is ideal, and lockfiles help pin shas of packages so we avoid MITM attacks. I’ll have to update packages in two places for now, which isn’t terrible but it’ll trip someone up eventually.
In the prefect OSS stuff I see you all use setup utils and old school requirements files, would be cool if there were some blog post about how to set this up. I assume the documentation always covers the easiest use case, but in this case the example is actually how you have to do it.
PEP 517 and 518 for reference are the motivations behind poetry and pdm
Anyway, I’m unblocked but would be cool to close the loop on this
Gentle nudge wrt above ^^
@Kevin Kho
k

Kevin Kho

05/17/2022, 8:06 PM
Ok I went through this again and it’s clearer now, ultimately I think the only thing stopping you from using
pdm
in your container is the last of flexibility in the command attached to for
prefect execute flow-run
. I think the best way to go about this is to expose some functionality to change that. I don’t know what the sentiment around that will be. A lot of people request this in a different context: they want to add their own CustomFlowRunner class that is used. I will write up an issue for this later or tomorrow.
d

Dylan

05/17/2022, 8:37 PM
Thanks, that all sounds good.
k

Kevin Kho

05/20/2022, 3:59 PM
4 Views