Thread
#prefect-community
    d

    Dylan

    4 months ago
    Is there a convenient way to override the flow execute command?
    I’m installing my project using
    pdm
    and other tools like poetry and venv behave similarly, in that they don’t expect projects to be installed globally, and are instead expected to be scoped to an environment on the host.
    I’d like to install the project in a docker image
    pdm install
    where prefect is a project dependency, among many.
    If I install prefect globally with
    pip install prefect
    I expect that the flow run will fail, as the prefect invocation is outside the bounded scope of a project installed with pdm. This is similar to how govendor and bundler work.
    Ideally I’d be able to specify a working directory and then something to the effect of
    pdm run prefect execute flow-run
    so that the flow-run scopes to the installed project
    As an alternative to modifying the execute command, I’m going to use pdm to install the project globally and see if that works.
    Nope
    No dice
    Kevin Kho

    Kevin Kho

    4 months ago
    I am thinking here. Will respond in a bit
    d

    Dylan

    4 months ago
    Thanks, let me know if my ask and context are clear. I don’t think I’m doing anything out of the ordinary here. It seems pretty well aligned with project builds to use a tool to build and install a project. Packaging has always been a bit painful in python. Circle CI uses poetry in their python image for example, and pdm is like poetry but allows for environment variable expansion, making is a great choice for CI pipelines that need to pull from private github repo’s. I wouldn’t expect
    prefect
    to be on the global path of any image but the prefect image, and if I’m installing a project globally (basically at the OS level) I’m breaking modularity and the setup for that has to deviate substantially from installing a project locally for development work.
    I decided to take a short cut and take on a bit of tech debt for the time being, just installing stuff globally with pip. pyproject.toml support is ideal, and lockfiles help pin shas of packages so we avoid MITM attacks. I’ll have to update packages in two places for now, which isn’t terrible but it’ll trip someone up eventually.
    In the prefect OSS stuff I see you all use setup utils and old school requirements files, would be cool if there were some blog post about how to set this up. I assume the documentation always covers the easiest use case, but in this case the example is actually how you have to do it.
    PEP 517 and 518 for reference are the motivations behind poetry and pdm
    Anyway, I’m unblocked but would be cool to close the loop on this
    Gentle nudge wrt above ^^
    @Kevin Kho
    Kevin Kho

    Kevin Kho

    4 months ago
    Ok I went through this again and it’s clearer now, ultimately I think the only thing stopping you from using
    pdm
    in your container is the last of flexibility in the command attached to for
    prefect execute flow-run
    . I think the best way to go about this is to expose some functionality to change that. I don’t know what the sentiment around that will be. A lot of people request this in a different context: they want to add their own CustomFlowRunner class that is used. I will write up an issue for this later or tomorrow.
    d

    Dylan

    4 months ago
    Thanks, that all sounds good.
    Kevin Kho

    Kevin Kho

    4 months ago