https://prefect.io
Join Slack
Hi, I have a 2.x deployment question - I want to deploy my flows using python deployment object, her...
# ask-community
i
Hi, I have a 2.x deployment question - I want to deploy my flows using python deployment object, here the code:
Copy code
cluster_config_block = KubernetesClusterConfig.load("data-engineering")

k8s = KubernetesJob(
    cluster_config=cluster_config_block
)

deployment = Deployment.build_from_flow(
    flow=test,
    name="test",
    version="0.0.1",
    tags={"from-python-object"},
    parameters={"name": "test-run"},
    infra_overrides={"env": {"PREFECT_LOGGING_LEVEL": "DEBUG"}},
    work_queue_name="test",
    infrastructure=k8s
)

if __name__ == "__main__":
    result = deployment.apply()
    print(result)
I don't see where/how to pass the AWS credentials to deployment, without which ofc the flow run results in 
401 Unauthorized
response
n
Hi @Igor Morgunov - where do you want to store your flow code? you'll need to specify remote storage to use k8s infra with deployments, like this example
i
riiiight! with the 
storage
block I then get this error:
Copy code
File system created with scheme 's3' from base path 's3://....' could not be created. You are likely missing a Python module required to use the given storage protocol.
I did have a look at this post, but doing 
pip install s3fs
doesn't solve the problem
n
yep, try 
pip install s3fs
in your venv
i
tried, but still getting the same error
n
oops, apologies I didn't see you said you tried that what version of prefect are you on? also it may help if you shared your storage block definition
i
Copy code
Version:             2.7.3
API version:         0.8.4
Python version:      3.9.15
n
we had a fix to deployments rolled out to cloud recently, which may be related, could you try upgrading to 2.7.7?
i
Upgraded to 2.7.7 but still getting the same error
n
Hmm if its still saying
Copy code
You are likely missing a Python module required to use the given storage protocol.
I have only ever seen that when s3fs is missing from a python environment, if this thrown for a different reason I'm not sure what it is 🤔 i am able to apply a deployment to s3 storage against cloud, are you also pointing at cloud? or hosted orion
i
hosted orion
n
just to double check, have you upgraded orion as well? or just local venv
i
just local env
fiddled a bit, getting a different now, which is my problem, which is good 🙂 thanks for pointing in the right direction
n
cool - sure thing! let us know if you run into anything else
i
I got excited too soon 😞 after re-creating the k8s block in the cloud which I only had locally, I'm back to the same error
n
getting a different now
if you meant you were getting a different error, what was that other error? that might help us discover the problem
i
I was missing a k8s config block in the cloud
2 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by