https://prefect.io
Join Slack
Hi - is anyone using Azure AKS Agent and Jobs with Prefect Cloud? I'm having a really difficult time...
# ask-community
k
Hi - is anyone using Azure AKS Agent and Jobs with Prefect Cloud? I'm having a really difficult time understanding how everything is tied together. Right now, I am trying to run a flow that is stored within Azure storage and runs as a kubernetes job (presumably within the same cluster my agent is in). But I am running into the following Exception('Azure connection string not provided. Set 
AZURE_STORAGE_CONNECTION_STRING
environment variable or save connection string as Prefect secret.')
When building the docker image, I am making sure to set that environment variable with the correct value
In general, I am struggling to understand how the agent, host os, and cloud interact. When do environment variables get resolved? When I register the flow or when the flow is called to run?
k
Hey @Kevin, the agent pings Prefect Cloud, grabs the metadata for the flow. Contained in that metadata is the storage so it will fetch the flow. It will then run it on top of the RunConfiguration (KubernetesRun). Could you show me how you set up the Flow storage?
k
Copy code
azure_store = Azure(container="prefect")
my interpretation of the Class was that if you do not pass the conn string or secret name it defaults to the environment variable
k
Ah ok. So you will need the 
connection_string_secret
when you declare that. From the doc string:
Copy code
connection_string_secret (str, optional): the name of a Prefect secret that contains an Azure connection string for communicating with Blob storage. If not provided the value set in the environment as AZURE_STORAGE_CONNECTION_STRING will be used
Ah I see what you mean
Can you try passing the environment variable to the agent then?
k
I'll give that a shot
k
prefect agent kubernetes start --env AZURE_STORAGE_CONNECTION_STRING=…..
k
It will take me some time. I am getting comfortable with aks at the same time as prefect so moving extra slow.
k
No worries at all
k
I think I may have identified what is causing this current issue
I built a custom image that i reference within the flow run config itself
but i was using the default output of the prefect cli to deploy the agent
which uses a prefecthq image
k
Oh so are you good now? Seems like it’s just a matter of getting the env variable to the agent?
I think as a Secret stored on Prefect Cloud might be easier though
k
Circling back on this after two weeks. I have made a ton of progress all across the project but am still running into an issue. It seems like the prefect image doesn't like Kubernetes secrets being used as environment variables. Have you run into any issues with using k8s secrets?
Copy code
- name: PREFECT__CONTEXT__SECRETS__AZURE_STORAGE_CONNECTION_STRING
          valueFrom:
            secretKeyRef:
              name: prefect-secrets
              key: az_connection_string
a
@Kevin do you need it for the flow storage? In that case, what works better is to put this connection string as Prefect Secret. Do you know how to put Secret in the UI? Once you stored this secret there, you can simply reference this Secret on the Azure storage class:
Copy code
STORAGE = Azure(
    container="flows",
    connection_string_secret="AZURE_STORAGE_CONNECTION_STRING",
)
so no need to put it as Kubernetes Secret
k
I'll give that a shot. But I also would love to understand why my method is not working. I like that Prefect Cloud offers a solution but would prefer to at least understand why mine is not working vs. just blindly going with it
a
@Kevin overall, the Kubernetes Secrets on Kubernetes Agent should mainly be used to pull container images from external registries like Azure Container Registry, ECR or GCR. I can share a couple of examples, maybe you can have a look: • using Azure storage + a custom image on Azure AKS https://github.com/anna-geller/packaging-prefect-flows/blob/master/flows/azure_kubernetes_run_custom_azure_image.py • using Github storage + custom image on Azure AKS: https://github.com/anna-geller/packaging-prefect-flows/blob/master/flows/azure_kubernetes_run_custom_azure_image.py
👍 1
k
thanks - i'll have a look!
this may be a stupid question - but when i register a flow that includes a reference to a secret or to an environment variable - does the flow compile those values when i register? meaning if i were to go and look at the python code in my storage - would those values be equal to the value that i set? or will it still just have the reference to the environment variable?
a
no, the FlowRunner retrieves the secret value at runtime rather than at registration time. So every time a new flow run is created for this flow, FlowRunner retrieves the value of the Secret. This way, if your Secret would change, you don’t need to reregister or change your code in any way.
and it’s a totally valid question!
k
i think i'm very close to a lightbulb moment with prefect
marvin 1
a
happy to hear that! 🙂
k
so that makes sense. and i understand that prefect cloud client is an option to store secrets. but its not out of the question to use a cloud provider, correct? if i were to create a AzureKeyVault client and AzureKeyvault secret, it in theory could perform the same behavior
a
correct, you could, in theory, subclass the Secret class and create your own task that would retrieve the secret’s value from your custom backend
k
thanks. very helpful
a
to be correct, you would subclass the 
SecretBase
class https://docs.prefect.io/api/latest/tasks/secrets.html#secretbase
k
yup. i remember looking at that far too early in my prefect journey haha
👍 1
2 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by