Hi. I'd really appreciate some help on connecting to a remote k8s cluster running in azure. I can execute flows in the cluster when using

flow.run(...)

but when I

flow.register(...

) and then run it from the UI, I get errors in the dask workers in k8s saying that cannot connect to

localhost:4200

. which makes sense since that localhost:4200 has to be replaced by the IP where the prefect server is running. how to do that? (https://github.com/PrefectHQ/prefect/issues/3185)

Hi @Nuno Silva you can directly set an API endpoint for your flow runs that the agent deploys with the

--api

flag

Hi @josh. Sounds good, can you elaborate a bit more, where and how can I set that flag?

prefect agent start --api <server_ip>

?

Exactly! When you call either

prefect agent start

or

prefect agent install

you can say

--api <server_ip>

ok, still having some issues: prefect agent start --api http://server_ip:4200/graphql I get the logs from the agent: [2020-08-19 150850,895] INFO - agent | Agent connecting to the Prefect API at http://vm-python-server1.westeurope.cloudapp.azure.com:4200/graphql [2020-08-19 150850,902] INFO - agent | Waiting for flow runs... [2020-08-19 150858,496] INFO - agent | Found 1 flow run(s) to submit for execution. [2020-08-19 150858,529] INFO - agent | Deploying flow run 3b06a5ef-5970-4fa0-8744-313cc66cdf93 [2020-08-19 150934,368] INFO - agent | Process PID 22615 returned non-zero exit code

Can you run the agent with the

--show-flow-logs

flag to see what the logs from the process are showing?

ok found it: HTTPConnectionPool(host=server_ip, port=4200): Max retries exceeded with url: /graphql/graphql

Not sure if this is it but are you appending

/graphql

to the

--api

endpoint you are passing in? It looks like it’s attempting to talk to

/graphql/graphql

(the client automatically adds it)

shall I try 1)

--api= server_ip

or 2)

--api= server_ip:4200

?

Do it with the

4200

port

same error, timeout cannot connect

Do you know if your dask cluster is permissioned to access that endpoint?

explicitly it's not, they're both in azure in the same resource-group. when I run from that server/endpoint the same flow with

flow.run

it works

flow.run

doesn’t make any calls to the server so I would expect that to succeed. Could you try the reregister w/o the executor?

yes

Oh interesting. On the same instance where you are starting the agent could you test this snippet:

from prefect import Client

# add the /graphql this time
c = Client(api_server=<endpoint>/graphql)

c.graphql("{hello}")

and see if it raises an error

ok, so if I run that from inside the server, it fails, from outside the server it works

By inside the server do you mean in a pod/container/etc. where the prefect server is running? Because agents should be started externally to the prefect server

I have a vm in azure (ubuntu server), I installed conda and then prefect. I

prefect start server

and then

prefect start agent

. the dask cluster (executor) is a dask-kubernetes cluster that I deployed myself in azure aks

When getting a success using the LocalEnvironment without an executor are you still setting the

--api

or no?

if i set it, it fails, if i dont it works

Okay that makes sense. So what appears to be happening is that your agent wants to communicate with the prefect server on localhost:4200 since it’s on the same machine but dask would need to communicate over that external endpoint. What I would recommend is trying one of these options: • Deploy your agent somewhere where it can talk to that external endpoint via the

--api

flag and then it will be propagated down to the workers • Keep your agent on that instance without setting

--api

and instead on your dask cluster somehow set the env var

PREFECT___CLOUD___API=<endpoint>

cool, let me work on that. Thanks a lot @josh

Yeah you could try running the agent on your individual machine with the api set and see if that results in a success

which ports does the VM need to have open? right now I have open 8080 and 4200

Yep those are the only ones the deployment exposes 👍

quick update: if on my local machine I have no prefect server running and I just do

prefect agent start --api server_ip:4200

it fails immediately with

requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=4200)

Oh that’s weird I think that may be a bug

that would also explain why when running the agent in the VM where the server is running, even with the

--api

flag it only fails when the flow is scheduled, since only then it tries to use it

Yeah it would!

it solves the issue of the agent not starting, now it starts if I set the env var before

The agent might not be running the flow due to the labels on the flow’s environment and on the agent. You should be able to see the labels on your flow in the UI and the agent’s labels at start.

the labels are indeed different

Anytime! There’s also other cloud-based storage options as well 🙂 https://docs.prefect.io/orchestration/execution/storage_options.html

I was trying the azure storage and good news it finds the flow immediately, but I get errors in the agent :

AttributeError: 'NoneType' object has no attribute 'rstrip'

in

python3.8/site-packages/azure/storage/blob/_shared/base_client.py", line 349, in parse_connection_str

setup.py has the requirement as

"azure-storage-blob >= 12.1.0, < 13.0"

, I don’t know too much about this storage integration so if there are some updates it needs due to azure API changes we welcome the contribution haha

dask:adlfs also struggles a lot with the breaking changes in azure.storage, so that is normal at the moment I guess. I'll contribute when I fix it for sure

Were you able to pinpoint where you got the error from?

AttributeError: 'NoneType' object has no attribute 'rstrip'