Hey all, I'm trying to switch over from using python deployment to the cli so I am able to use githu...

Mitch

07/19/2023, 6:53 PM

Hey all, I'm trying to switch over from using python deployment to the cli so I am able to use github storage. Within my CLI I do not have all of the required dependencies to run my python file as we use a docker image and PyCharm interpreter to run the code. When running the

prefect deploy

cli command I am getting a module not found error even though the image I included contains that dependency. Is there a way to resolve this?

✅ 1

Jake Kaplan

07/19/2023, 7:43 PM

hey, can you explain what you mean by "image I included contains the dependency"? To run

prefect deploy

, prefect has to be installed in the environment where you're running the command

Mitch

07/19/2023, 7:44 PM

Prefect is installed, however packages such as private pypi package we have (contained in the docker image I have created and stored on aws ECR) are not in the local environment ( powershell ). When running my flows locally, I use the pycharm interpreter as the env which spins up the docker container to "run" the flows.

Jake Kaplan

07/19/2023, 7:46 PM

what module isn't found? Is it one of your private packages that's not in your CLI env?

Mitch

07/19/2023, 7:46 PM

Yes

Mitch

07/19/2023, 7:47 PM

Since I don't use my cli env to run flows or python files. I was previously using the pythonic deployment so the flows were being deployed from the docker container

Mitch

07/19/2023, 7:48 PM

I figured since the flow code is using github as storage, it shouldn't depend on the local file whenI am deploying using

prefect deploy

, only the prefect.yaml and should fail on task run instead

Jake Kaplan

07/19/2023, 7:54 PM

ah I see what you mean

Jake Kaplan

07/19/2023, 7:55 PM

let me check something really quick

👍 1

Mitch

07/19/2023, 7:55 PM

Yeah it's kinda a unique situation... Is there a good location to see examples of prefect.yaml

Jake Kaplan

07/19/2023, 8:09 PM

So at the moment yes, to use the cli command you'll need to be able to load your python file successfully (which includes dependencies)

Jake Kaplan

07/19/2023, 8:12 PM

Two sort of work arounds I have for you: • you can use

client.create_deployment(...)

directly to make the request (

prefect deploy

is a big helper method around that) • you can defer your imports so your file can load and you can use the cli

Copy code

# instead of:
import my_private_package

@flow
def my_flow():
   my_private_package.func()

# do this:
@flow
def my_flow():
   import my_private_package
   my_private_package.func()

Jake Kaplan

07/19/2023, 8:13 PM

it is on the backlog to evaluate that kind of case that you just described though

Jake Kaplan

07/19/2023, 8:13 PM

so you don't need to have the code right there for a nice deployment experience

Mitch

07/19/2023, 8:13 PM

I'll give it a try. Still determining if we will use agents or workers due to our deployment method

Mitch

07/19/2023, 8:13 PM

Thanks Jake

👍 1

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.