https://prefect.io logo
Title
f

Filip Panovski

03/06/2023, 8:46 AM
Hello everyone. I was experiencing an issue when creating a Deployment, which I documented here: https://discourse.prefect.io/t/error-when-creating-a-deployment-with-the-cli-modulenotfound/2426 Unfortunately it has not garnered very much attention from what I can tell. What it bols down to, is that creating a Deployment using the Prefect CLI yields:
Script at 'data_exports/extract_data.py' encountered an exception: ModuleNotFoundError("No module named 'pandas'")
which kind of confuses me, since the host where I am creating the deployment is not where it will run. My questions boil down to: 1. Why does the CLI need to know
pandas
(or any other library that I’m using)? 2. Does this mean that I should set up a different venv for every one of my deployments?
k

Kelvin DeCosta

03/06/2023, 9:00 AM
Ran into this issue a few times. I think whenever a deployment is created from a flow, the CLI / Python API needs to import the flow function. This can't be done without all the dependencies being resolved. The solution is to install
pandas
wherever you're running your script
f

Filip Panovski

03/06/2023, 9:02 AM
Yes, the solution is indeed to install whatever library is causing the issue. However, what if I have numerous deployments each with their own libraries/versions (which potentially conflict across deployments)? So, the general solution would be to have a venv per deployment?
k

Kelvin DeCosta

03/06/2023, 9:25 AM
That sounds like a pain tbf
c

Christopher Boyd

03/06/2023, 2:19 PM
Hi Filip - I deal with this in a CI/CD pipeline that is a shared system. My personal resolution was not to build_from_flow, but to build just a generic Deployment object, and passing the path / entrypoint. In other words, I don’t import the flow as an object, and the code isn’t run / loaded. The downsides of this, I have had to create my own parameter schema since it cannot be inferred, but that’s a work in progress
e.g.
deployment = Deployment(
    name=f"Import-{FLOW_ENVIRONMENT}",
    flow_name="Import_flow",
    version=1,
    work_queue_name="dev",
    infrastructure=k8s_job,
    path="/opt/prefect/flows",
    parameters=params,
    entrypoint="flow.py:flow_import",
    parameter_openapi_schema=schema
)
f

Filip Panovski

03/24/2023, 4:21 PM
Thanks a lot for the info @Christopher Boyd! I'm having a bit of trouble finding much info on the
parameter_openapi_schema
, do you know of an example in this context that I might have a look at?
c

Christopher Boyd

03/24/2023, 4:26 PM
I can share an example
schema = {
    'title': 'Parameters',
    'type': 'object',
    'properties': {
        'callback_url': {
            'type': 'string',
            'title': 'callback_url'
        },
        'trigger_source': {
            'type': 'string',
            'title': 'trigger_source'
        },
        'external_id': {
            'type': 'string',
            'title': 'external_id'
        },
        'debug': {
            'type': 'boolean',
            'title': 'debug',
            'default': 'False'
        },
        'gcp_logging_enabled': {
            'type': 'boolean',
            'title': 'gcp_logging_enabled',
            'default': 'False'
        },
        "duration": {
            "type": "array",
            "items": {
                "type": "integer"
            },
            "example": [
                1
            ],
            "title": "duration"
        },
    }
}
👍 1
properties would be where you define the actual parameters, type and title are the minimums you would need
f

Filip Panovski

03/24/2023, 4:40 PM
Ah, I didn't realize that a dict would suffice! Do each of the individual properties' fields (
type
,
title
,
default
..) correspond to the
pydantic.Field
kwargs?
Ah I see, that's just an OpenAPI/JsonSchema compatible schema. So, if I had default such as
pendulum.yesterday('Europe/Berlin').date()
, I wouldn't be able to pass those sensibly and would need to go the
pydantic.Field
route, correct?
c

Christopher Boyd

03/24/2023, 4:57 PM
Truthfully, I’m not 100% sure there - you’re right it is just an openapi compatible schema, I can’t speak knowledgeably to your question regarding pydantic.Field though unfortunatley
f

Filip Panovski

03/24/2023, 5:05 PM
That's alright, the schema example already helped me tremendously. Thank you and have a nice weekend 🙂