Hey when setting Docker as the storage for a flow you can specify dependencies through the `python_d...

Maikel Penz

07/22/2021, 4:49 AM

Hey when setting Docker as the storage for a flow you can specify dependencies through the

python_dependencies

property. However, I’m doing it dynamically and reading from a flow configured with Poetry. I noticed that the format in Poetry doesn’t match what is expected in the

python_dependencies

E.g: In Poetry:

colorama = "^0.4.4"

What I tried: `python_dependencies=["colorama^0.4.4"]`<< doesn’t work How it should be:

python_dependencies=["colorama>=0.4.4"]

Before I write something ugly to convert Poetry to Requirements.txt style, is there something available on Prefect to convert it or Poetry support ?

Maikel Penz

07/22/2021, 4:51 AM

For reference this is a more complete piece of code:

Copy code

flow.storage = Docker(
            registry_url=f"{aws_account}.dkr.ecr.{aws_region}.<http://amazonaws.com|amazonaws.com>",
            image_name=flow.name,
            image_tag="latest",
            python_dependencies=["colorama^0.4.4"],
            env_vars={"PYTHONPATH": "/opt/prefect/flows"},
        )

Kevin Kho

07/22/2021, 4:53 AM

Hey @Maikel Penz, we don’t have anything for this…but I think it’s just a

replace

? Or does it get uglier? I am wondering if there would be interest for this as a contribution as I know we have poetry users, but this is my first time seeing this specific request.

Maikel Penz

07/22/2021, 4:57 AM

Hey Kevin ! I can put a replace for now but I see that Poetry has many variants using

We have users that use

Pipenv

as well so we need to handle both Poetry and Pipenv conversion. It would be great if there was something built in Prefect but you are right, needs to see if there’s interest from the community

Kevin Kho

07/22/2021, 5:01 AM

Oh man that’s a bunch of cases yeah. Have you seen this ? It might be able to convert for you.

Maikel Penz

07/22/2021, 5:17 AM

oh.. that looks promising.. let me give it a go and get back to you 💪

Maikel Penz

07/22/2021, 5:37 AM

hmmm interesting.. I have this configuration on Poetry

colorama = "^0.4.4"

I ran the library above to convert from Poetry to requirements.txt and it generated

colorama==0.*,>=0.4.4

which seems fine. If I install it through

pip install -r requirements.txt

it works perfectly. However, when passing this on prefect it doesn’t seen to like it

python_dependencies=["colorama==0.*,>=0.4.4"],

Error:

Copy code

Required-by: 
 ---> 54796cc71133
Step 5/9 : RUN pip install boto3 colorama==0.*,>=0.4.4 wheel
 ---> Running in 2d12af75d1f8
ERROR: Invalid requirement: 'colorama==0.*,'

Maikel Penz

07/22/2021, 5:38 AM

actually.. just noticed that it generates the command

pip install boto3 colorama==0.*,>=0.4.4 wheel

and it fails because of the comma if I put single quotes in addition to the double quotes around each dependency I might be fine. E.g:

python_dependencies=["'colorama==0.*,>=0.4.4'"],

Maikel Penz

07/22/2021, 5:52 AM

I might be able to use it. Thanks for sharing this Kevin !!

Kevin Kho

07/22/2021, 1:38 PM

Nice!

Maikel Penz

08/16/2021, 5:02 AM

@Leandro Mana

Leandro Mana

08/16/2021, 9:55 PM

👍

8 Views

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.