https://prefect.io logo
Title
i

Igor Morgunov

01/16/2023, 11:56 AM
Hi - I’m trying to use GitHub storage block (private repo with access token) in my deployment using AWS k8s backend, but I’m not sure where the AWS credentials need to be passed (deployment succeeds but submitting a flow run returns 401)
github_block = GitHub.load("XXXXXX")

cluster_config_block = KubernetesClusterConfig.load("data-engineering")

k8s = KubernetesJob(
    cluster_config=cluster_config_block
)

deployment = Deployment.build_from_flow(
    flow=test,
    name="test",
    version="0.0.1",
    tags={"from-python-object"},
    parameters={"name": "test-run"},
    infra_overrides={"env": {"PREFECT_LOGGING_LEVEL": "DEBUG"}},
    work_queue_name="test",
    infrastructure=k8s,
    storage=github_block,
)

if __name__ == "__main__":
    result = deployment.apply()
a

Anna Geller

01/16/2023, 6:53 PM
there is AwsCredentials block in the prefect-aws collection but why do you need AWS credentials with GitHub block?
i

Igor Morgunov

01/17/2023, 10:18 AM
I want to execute the flow in our AWS EKS cluster - I created the k8s block from my local .kube/config file, but when I run the flow, I get an error:
Submission failed. kubernetes.client.exceptions.ApiException: (401) Reason: Unauthorized HTTP
a

Anna Geller

01/17/2023, 12:17 PM
hmm this is not prefect related usually, the steps you could take for a minimal PoC is: • use
eksctl
to create a minimal K8s cluster e.g. single node • use
prefect kubernetes manifest agent
, adapt if needed e.g. namespace and apply this deployment • then you have the agent - all is left is K8s job block and a deployment
I need to update this but perhaps it helps https://github.com/anna-geller/dataflow-ops-aws-eks