Igor Morgunov
01/09/2023, 2:55 PMcluster_config_block = KubernetesClusterConfig.load("data-engineering")
k8s = KubernetesJob(
cluster_config=cluster_config_block
)
deployment = Deployment.build_from_flow(
flow=test,
name="test",
version="0.0.1",
tags={"from-python-object"},
parameters={"name": "test-run"},
infra_overrides={"env": {"PREFECT_LOGGING_LEVEL": "DEBUG"}},
work_queue_name="test",
infrastructure=k8s
)
if __name__ == "__main__":
result = deployment.apply()
print(result)
I don't see where/how to pass the AWS credentials to deployment, without which ofc the flow run results in 401 Unauthorized
responseNate
01/09/2023, 3:22 PMIgor Morgunov
01/09/2023, 3:29 PMstorage
block I then get this error:
File system created with scheme 's3' from base path 's3://....' could not be created. You are likely missing a Python module required to use the given storage protocol.
I did have a look at this post, but doing pip install s3fs
doesn't solve the problemNate
01/09/2023, 3:30 PMpip install s3fs
in your venvIgor Morgunov
01/09/2023, 3:33 PMNate
01/09/2023, 3:35 PMIgor Morgunov
01/09/2023, 3:37 PMVersion: 2.7.3
API version: 0.8.4
Python version: 3.9.15
Nate
01/09/2023, 3:37 PMIgor Morgunov
01/09/2023, 3:45 PMNate
01/09/2023, 4:22 PMYou are likely missing a Python module required to use the given storage protocol.
I have only ever seen that when s3fs is missing from a python environment, if this thrown for a different reason I'm not sure what it is 🤔
i am able to apply a deployment to s3 storage against cloud, are you also pointing at cloud? or hosted orionIgor Morgunov
01/09/2023, 4:23 PMNate
01/09/2023, 4:25 PMIgor Morgunov
01/09/2023, 4:27 PMNate
01/09/2023, 4:33 PMIgor Morgunov
01/09/2023, 4:40 PMNate
01/09/2023, 4:42 PMgetting a different nowif you meant you were getting a different error, what was that other error? that might help us discover the problem
Igor Morgunov
01/09/2023, 4:42 PM