https://prefect.io logo
Join Slack
Powered by
Is there a convenient way to override the flow exe...
# prefect-community
d
Is there a convenient way to override the flow execute command?
I’m installing my project using 
pdm
and other tools like poetry and venv behave similarly, in that they don’t expect projects to be installed globally, and are instead expected to be scoped to an environment on the host.
I’d like to install the project in a docker image 
pdm install
where prefect is a project dependency, among many.
If I install prefect globally with 
pip install prefect
I expect that the flow run will fail, as the prefect invocation is outside the bounded scope of a project installed with pdm. This is similar to how govendor and bundler work.
Ideally I’d be able to specify a working directory and then something to the effect of 
pdm run prefect execute flow-run
so that the flow-run scopes to the installed project
As an alternative to modifying the execute command, I’m going to use pdm to install the project globally and see if that works.
Nope
No dice
k
I am thinking here. Will respond in a bit
d
Thanks, let me know if my ask and context are clear. I don’t think I’m doing anything out of the ordinary here. It seems pretty well aligned with project builds to use a tool to build and install a project. Packaging has always been a bit painful in python. Circle CI uses poetry in their python image for example, and pdm is like poetry but allows for environment variable expansion, making is a great choice for CI pipelines that need to pull from private github repo’s. I wouldn’t expect 
prefect
to be on the global path of any image but the prefect image, and if I’m installing a project globally (basically at the OS level) I’m breaking modularity and the setup for that has to deviate substantially from installing a project locally for development work.
I decided to take a short cut and take on a bit of tech debt for the time being, just installing stuff globally with pip. pyproject.toml support is ideal, and lockfiles help pin shas of packages so we avoid MITM attacks. I’ll have to update packages in two places for now, which isn’t terrible but it’ll trip someone up eventually.
In the prefect OSS stuff I see you all use setup utils and old school requirements files, would be cool if there were some blog post about how to set this up. I assume the documentation always covers the easiest use case, but in this case the example is actually how you have to do it.
PEP 517 and 518 for reference are the motivations behind poetry and pdm
Anyway, I’m unblocked but would be cool to close the loop on this
Gentle nudge wrt above ^^
@Kevin Kho
k
Ok I went through this again and it’s clearer now, ultimately I think the only thing stopping you from using 
pdm
in your container is the last of flexibility in the command attached to for 
prefect execute flow-run
. I think the best way to go about this is to expose some functionality to change that. I don’t know what the sentiment around that will be. A lot of people request this in a different context: they want to add their own CustomFlowRunner class that is used. I will write up an issue for this later or tomorrow.
d
Thanks, that all sounds good.
k
4 Views
Open in Slack
PreviousNext
Powered by