https://prefect.io logo
Join Slack
Powered by
Hello everyone. I was experiencing an issue when c...
# ask-community
f
Hello everyone. I was experiencing an issue when creating a Deployment, which I documented here: https://discourse.prefect.io/t/error-when-creating-a-deployment-with-the-cli-modulenotfound/2426 Unfortunately it has not garnered very much attention from what I can tell. What it bols down to, is that creating a Deployment using the Prefect CLI yields:
Copy code
Script at 'data_exports/extract_data.py' encountered an exception: ModuleNotFoundError("No module named 'pandas'")
which kind of confuses me, since the host where I am creating the deployment is not where it will run. My questions boil down to: 1. Why does the CLI need to know 
pandas
(or any other library that I’m using)? 2. Does this mean that I should set up a different venv for every one of my deployments?
k
Ran into this issue a few times. I think whenever a deployment is created from a flow, the CLI / Python API needs to import the flow function. This can't be done without all the dependencies being resolved. The solution is to install 
pandas
wherever you're running your script
f
Yes, the solution is indeed to install whatever library is causing the issue. However, what if I have numerous deployments each with their own libraries/versions (which potentially conflict across deployments)? So, the general solution would be to have a venv per deployment?
k
That sounds like a pain tbf
c
Hi Filip - I deal with this in a CI/CD pipeline that is a shared system. My personal resolution was not to build_from_flow, but to build just a generic Deployment object, and passing the path / entrypoint. In other words, I don’t import the flow as an object, and the code isn’t run / loaded. The downsides of this, I have had to create my own parameter schema since it cannot be inferred, but that’s a work in progress
e.g.
Copy code
deployment = Deployment(
    name=f"Import-{FLOW_ENVIRONMENT}",
    flow_name="Import_flow",
    version=1,
    work_queue_name="dev",
    infrastructure=k8s_job,
    path="/opt/prefect/flows",
    parameters=params,
    entrypoint="flow.py:flow_import",
    parameter_openapi_schema=schema
)
f
Thanks a lot for the info @Christopher Boyd! I'm having a bit of trouble finding much info on the 
parameter_openapi_schema
, do you know of an example in this context that I might have a look at?
c
I can share an example
Copy code
schema = {
    'title': 'Parameters',
    'type': 'object',
    'properties': {
        'callback_url': {
            'type': 'string',
            'title': 'callback_url'
        },
        'trigger_source': {
            'type': 'string',
            'title': 'trigger_source'
        },
        'external_id': {
            'type': 'string',
            'title': 'external_id'
        },
        'debug': {
            'type': 'boolean',
            'title': 'debug',
            'default': 'False'
        },
        'gcp_logging_enabled': {
            'type': 'boolean',
            'title': 'gcp_logging_enabled',
            'default': 'False'
        },
        "duration": {
            "type": "array",
            "items": {
                "type": "integer"
            },
            "example": [
                1
            ],
            "title": "duration"
        },
    }
}
👍 1
properties would be where you define the actual parameters, type and title are the minimums you would need
f
Ah, I didn't realize that a dict would suffice! Do each of the individual properties' fields (
type
, 
title
, 
default
..) correspond to the 
pydantic.Field
kwargs?
Ah I see, that's just an OpenAPI/JsonSchema compatible schema. So, if I had default such as 
pendulum.yesterday('Europe/Berlin').date()
, I wouldn't be able to pass those sensibly and would need to go the 
pydantic.Field
route, correct?
c
Truthfully, I’m not 100% sure there - you’re right it is just an openapi compatible schema, I can’t speak knowledgeably to your question regarding pydantic.Field though unfortunatley
f
That's alright, the schema example already helped me tremendously. Thank you and have a nice weekend 🙂
Open in Slack
PreviousNext
Powered by