https://prefect.io logo
Powered by
Title
i

Igor Morgunov

01/09/2023, 2:55 PM
Hi, I have a 2.x deployment question - I want to deploy my flows using python deployment object, here the code: 
cluster_config_block = KubernetesClusterConfig.load("data-engineering")

k8s = KubernetesJob(
    cluster_config=cluster_config_block
)

deployment = Deployment.build_from_flow(
    flow=test,
    name="test",
    version="0.0.1",
    tags={"from-python-object"},
    parameters={"name": "test-run"},
    infra_overrides={"env": {"PREFECT_LOGGING_LEVEL": "DEBUG"}},
    work_queue_name="test",
    infrastructure=k8s
)

if __name__ == "__main__":
    result = deployment.apply()
    print(result)
I don't see where/how to pass the AWS credentials to deployment, without which ofc the flow run results in 
401 Unauthorized
response
n

Nate

01/09/2023, 3:22 PM
Hi @Igor Morgunov - where do you want to store your flow code? you'll need to specify remote storage to use k8s infra with deployments, like this example
i

Igor Morgunov

01/09/2023, 3:29 PM
riiiight! with the 
storage
block I then get this error: 
File system created with scheme 's3' from base path 's3://....' could not be created. You are likely missing a Python module required to use the given storage protocol.
I did have a look at this post, but doing 
pip install s3fs
doesn't solve the problem
n

Nate

01/09/2023, 3:30 PM
yep, try 
pip install s3fs
in your venv
i

Igor Morgunov

01/09/2023, 3:33 PM
tried, but still getting the same error
n

Nate

01/09/2023, 3:35 PM
oops, apologies I didn't see you said you tried that what version of prefect are you on? also it may help if you shared your storage block definition
i

Igor Morgunov

01/09/2023, 3:37 PM
Version:             2.7.3
API version:         0.8.4
Python version:      3.9.15
n

Nate

01/09/2023, 3:37 PM
we had a fix to deployments rolled out to cloud recently, which may be related, could you try upgrading to 2.7.7?
i

Igor Morgunov

01/09/2023, 3:45 PM
Upgraded to 2.7.7 but still getting the same error
n

Nate

01/09/2023, 4:22 PM
Hmm if its still saying 
You are likely missing a Python module required to use the given storage protocol.
I have only ever seen that when s3fs is missing from a python environment, if this thrown for a different reason I'm not sure what it is 🤔 i am able to apply a deployment to s3 storage against cloud, are you also pointing at cloud? or hosted orion
i

Igor Morgunov

01/09/2023, 4:23 PM
hosted orion
n

Nate

01/09/2023, 4:25 PM
just to double check, have you upgraded orion as well? or just local venv
i

Igor Morgunov

01/09/2023, 4:27 PM
just local env
fiddled a bit, getting a different now, which is my problem, which is good 🙂 thanks for pointing in the right direction
n

Nate

01/09/2023, 4:33 PM
cool - sure thing! let us know if you run into anything else
i

Igor Morgunov

01/09/2023, 4:40 PM
I got excited too soon 😞 after re-creating the k8s block in the cloud which I only had locally, I'm back to the same error
n

Nate

01/09/2023, 4:42 PM
getting a different now
if you meant you were getting a different error, what was that other error? that might help us discover the problem
i

Igor Morgunov

01/09/2023, 4:42 PM
I was missing a k8s config block in the cloud
3 Views
#prefect-communityJoin Slack
Powered by