https://prefect.io logo
Join Slack
Powered by
Would anyone know why an azure function isn't able...
# ask-community
g
Would anyone know why an azure function isn't able to connect to the prefect cloud? I set the correct api_url and api_key in the az function config settings and it all worked well about a month ago but now I'm getting an error of what looks to be from prefect that the cloud request isn't returning anything.
c
do you have an example of the error? this is a bit ambiguous with no logs or screenshots
g
Sure, it's a big one.
Copy code
Result: Failure Exception: RuntimeError: no validator found for <class 'azure.functions._eventgrid.EventGridEvent'>, see `arbitrary_types_allowed` in Config Stack: File "/azure-functions-host/workers/python/3.9/LINUX/X64/azure_functions_worker/dispatcher.py", line 357, in _handle__function_load_request func = loader.load_function( File "/azure-functions-host/workers/python/3.9/LINUX/X64/azure_functions_worker/utils/wrappers.py", line 44, in call return func(*args, **kwargs) File "/azure-functions-host/workers/python/3.9/LINUX/X64/azure_functions_worker/loader.py", line 132, in load_function mod = importlib.import_module(fullmodname) File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 1030, in _gcd_import File "<frozen importlib._bootstrap>", line 1007, in _find_and_load File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 680, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 850, in exec_module File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed File "/home/site/wwwroot/bossEventTrigger/__init__.py", line 9, in <module> def main(event: func.EventGridEvent): File "/home/site/wwwroot/.python_packages/lib/site-packages/prefect/flows.py", line 659, in flow Flow( File "/home/site/wwwroot/.python_packages/lib/site-packages/prefect/context.py", line 176, in __register_init__ __init__(__self__, *args, **kwargs) File "/home/site/wwwroot/.python_packages/lib/site-packages/prefect/flows.py", line 192, in __init__ ValidatedFunction(self.fn, config=None) File "pydantic/decorator.py", line 126, in pydantic.decorator.ValidatedFunction.__init__ File "pydantic/decorator.py", line 264, in pydantic.decorator.ValidatedFunction.create_model File "pydantic/main.py", line 1027, in pydantic.main.create_model File "pydantic/main.py", line 198, in pydantic.main.ModelMetaclass.__new__ File "pydantic/fields.py", line 506, in pydantic.fields.ModelField.infer File "pydantic/fields.py", line 436, in pydantic.fields.ModelField.__init__ File "pydantic/fields.py", line 557, in pydantic.fields.ModelField.prepare File "pydantic/fields.py", line 831, in pydantic.fields.ModelField.populate_validators File "pydantic/validators.py", line 765, in find_validators
I believe it's an issue with the az function connecting to prefect cloud because the error message states that in line 1506 of orchestrate.py the response.json() is None.
This is the az function.
Copy code
from prefect.deployments import run_deployment
import azure.functions as func

def main(event: func.EventGridEvent):
    file_name = get_file_from_subject(event.subject)

    response = run_deployment(name='main-flow/dp_blob_ingestion_smart_rent_group',
                            flow_run_name= file_name,
                            parameters={
                                    'blob_path': event.subject,
                                    'target_schema': 'GALTMAN',
                                    'github_params': {
                                        'repo': 'my_repo',
                                        'branch': 'my_branch'
                                        },
                                    'snwf_params': {
                                        'account': 'my_account',
                                        'warehouse': 'my_wh',
                                        'db': 'TEMP',
                                        'schema': 'my_schema',
                                        'authenticator': 'snowflake'
                                        }
                                    },
                            timeout=0)
    
def get_file_from_subject(subject):
    return subject.split('/')[-1].replace('.json', '').upper()
But note, i've also tried it with just a simple block secret string get() and it still had the same error. The az function only fails when prefect code is there.
c
Copy code
Result: Failure Exception: RuntimeError: no validator found for <class 'azure.functions._eventgrid.EventGridEvent'>
It looks like it’s failing to validate the pydantic typing?
I don’t really see anything prefect wise failing here, and I don’t see the line you are referencing?
g
Sorry!!! That error msg was when I had the azure function decorated with @flow. here is the accurate error msg I was referring to once I removed the decorator.
Copy code
Exception: JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Stack:   File "/azure-functions-host/workers/python/3.9/LINUX/X64/azure_functions_worker/dispatcher.py", line 452, in _handle__invocation_request
    call_result = await self._loop.run_in_executor(
  File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/azure-functions-host/workers/python/3.9/LINUX/X64/azure_functions_worker/dispatcher.py", line 718, in _run_sync_func
    return ExtensionManager.get_sync_invocation_wrapper(context,
  File "/azure-functions-host/workers/python/3.9/LINUX/X64/azure_functions_worker/extension.py", line 215, in _raw_invocation_wrapper
    result = function(**args)
  File "/home/site/wwwroot/bossEventTrigger/__init__.py", line 10, in main
    response = run_deployment(name='main-flow/dp_blob_ingestion_smart_rent_group',
  File "/home/site/wwwroot/.python_packages/lib/site-packages/prefect/utilities/asyncutils.py", line 230, in coroutine_wrapper
    return run_async_in_new_loop(async_fn, *args, **kwargs)
  File "/home/site/wwwroot/.python_packages/lib/site-packages/prefect/utilities/asyncutils.py", line 181, in run_async_in_new_loop
    return anyio.run(partial(__fn, *args, **kwargs))
  File "/home/site/wwwroot/.python_packages/lib/site-packages/anyio/_core/_eventloop.py", line 70, in run
    return asynclib.run(func, *args, **backend_options)
  File "/home/site/wwwroot/.python_packages/lib/site-packages/anyio/_backends/_asyncio.py", line 292, in run
    return native_run(wrapper(), debug=debug)
  File "/usr/local/lib/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
    return future.result()
  File "/home/site/wwwroot/.python_packages/lib/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
    return await func(*args)
  File "/home/site/wwwroot/.python_packages/lib/site-packages/prefect/client/utilities.py", line 47, in with_injected_client
    return await fn(*args, **kwargs)
  File "/home/site/wwwroot/.python_packages/lib/site-packages/prefect/deployments.py", line 89, in run_deployment
    deployment = await client.read_deployment_by_name(name)
  File "/home/site/wwwroot/.python_packages/lib/site-packages/prefect/client/orion.py", line 1444, in read_deployment_by_name
    return schemas.responses.DeploymentResponse.parse_obj(response.json())
  File "/home/site/wwwroot/.python_packages/lib/site-packages/httpx/_models.py", line 756, in json
    return jsonlib.loads(self.text, **kwargs)
  File "/usr/local/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/lib/python3.9/json/decoder.py", line 355, in raw_decode
in this case, it's this line
Copy code
File "/home/site/wwwroot/.python_packages/lib/site-packages/prefect/deployments.py", line 89, in run_deployment
    deployment = await client.read_deployment_by_name(name)
  File "/home/site/wwwroot/.python_packages/lib/site-packages/prefect/client/orion.py", line 1444, in read_deployment_by_name
    return schemas.responses.DeploymentResponse.parse_obj(response.json())
c
does this run outside of an azure function, locally? also, what version are you using, 
orion.py
was removed as part of a recent version, although I don’t know if that has bearing here. can you do something even more simple, like just listing your deployments, or getting a health endpoint of your account, to simplify the problem, and determine if it’s azure or prefect
I can’t say that nothing has changed cloud side, but this seems to be a very specific issue, and 
read_deployment_by_name
would be used by many flow runs
g
@Christopher Boyd This exact script runs outside of an azure function successfully. Prefect 2.7.11 would you mind pointing me to a code snippet for checking the health endpoint of the account?
c
When you’re running it outside of an azure function, are they using the same prefect versions (local / azure)? You can test health endpoint which requires your api key just to make sure your networking and api key / url are good:
Copy code
curl -v -H 'Authorization: Bearer <replace with your api key>' <https://api.prefect.cloud/api/accounts/><your accound id>/workspaces/<workspace_id>/health
g
Yes they are using the same version both locally and in az. I am getting a powershell error when running that command. Sorry, not the best at powershell. I replaced my actual key with 'my_key'
Copy code
Invoke-WebRequest : Cannot bind parameter 'Headers'. Cannot convert the "Authorization: Bearer my_key" value of type 
"System.String" to type "System.Collections.IDictionary".
At line:1 char:12
+ curl -v -H 'Authorization: Bearer my_key ...
+            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidArgument: (:) [Invoke-WebRequest], ParameterBindingException
    + FullyQualifiedErrorId : CannotConvertArgumentNoMessage,Microsoft.PowerShell.Commands.InvokeWebRequestCommand
c
I’m not sure of the powershell syntax necessary here, this would be for bash / native curl, which I don’t think is functional in powershell
g
Gotcha, I went ahead and ran it in bash and it looks like it returns a 404. I swear my url and key are accurate though
Is the api_key specific to a workspace?
c
it would be specific to your user, if you authenticate with it, you would be able to select a workspace
what’s the URL you are using (minus the api key)
404 says the url is probably wrong, not the key
the key would be a 401 or 403
g
c
Copy code
< HTTP/2 403
< date: Fri, 03 Mar 2023 18:35:14 GMT
< server: istio-envoy
< content-length: 30
< content-type: application/json
< x-envoy-upstream-service-time: 3
< via: 1.1 google
< content-security-policy: default-src 'none'; object-src 'none'
< x-content-type-options: nosniff
< strict-transport-security: max-age=63072000; includeSubDomains; preload
< permissions-policy: accelerometer=(), ambient-light-sensor=(), autoplay=(), battery=(), camera=(), cross-origin-isolated=(), display-capture=(), document-domain=(), encrypted-media=(), execution-while-not-rendered=(), execution-while-out-of-viewport=(), fullscreen=(), geolocation=(), gyroscope=(), hid=(), idle-detection=(), magnetometer=(), microphone=(), midi=(), navigation-override=(), payment=(), picture-in-picture=(), publickey-credentials-get=(), screen-wake-lock=(), serial=(), sync-xhr=(), usb=(), web-share=(), xr-spatial-tracking=()
< x-permitted-cross-domain-policies: none
< referrer-policy: same-origin
< alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
<
* Connection #0 to host api.prefect.cloud left intact
{"detail":"Not authenticated"}%4
g
It's possible that I misinterpreted the response.
c
that’s with your url, with no key
g
This is the full response I get
Copy code
*   Trying 23.100.231.127:80...
* Connected to Bearer (23.100.231.127) port 80 (#0)
> GET / HTTP/1.1
> Host: Bearer
> User-Agent: curl/7.83.1
> Accept: */*
>
* Mark bundle as not supporting multiuse   
< HTTP/1.1 404 Not Found
< Content-Type: text/html; charset=us-ascii
< Server: Microsoft-HTTPAPI/2.0
< Date: Fri, 03 Mar 2023 18:36:12 GMT      
< Connection: close
< Content-Length: 315
<
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN""<http://www.w3.org/TR/html4/strict.dtd>">
<HTML><HEAD><TITLE>Not Found</TITLE>
<META HTTP-EQUIV="Content-Type" Content="text/html; charset=us-ascii"></HEAD>
<BODY><h2>Not Found</h2>
<hr><p>HTTP Error 404. The requested resource is not found.</p>
</BODY></HTML>
* Closing connection 0
* Could not resolve host: my_key'
* Closing connection 1
curl: (6) Could not resolve host: my_key'
*   Trying 34.111.36.90:443...
* Connected to api.prefect.cloud (34.111.36.90) port 443 (#2)
* schannel: disabled automatic use of client certificate     
* ALPN: offers http/1.1
* ALPN: server accepted http/1.1
> GET /api/accounts/bae78ed9-1e37-409a-b34e-9c99aca8d81e/workspaces/7f678f7c-d100-4c3f-a9e2-928136ba7022/health HTTP/1.1
> Host: api.prefect.cloud
> User-Agent: curl/7.83.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 403 Forbidden
< date: Fri, 03 Mar 2023 18:36:15 GMT
< server: istio-envoy
< content-length: 30
< content-type: application/json
< x-envoy-upstream-service-time: 4
< Via: 1.1 google
< Content-Security-Policy: default-src 'none'; object-src 'none'
< X-Content-Type-Options: nosniff
< Strict-Transport-Security: max-age=63072000; includeSubDomains; preload
< Permissions-Policy: accelerometer=(), ambient-light-sensor=(), autoplay=(), battery=(), camera=(), cross-origin-isolated=(), display-capture=(), document-domain=(), encrypted-media=(), execution-while-not-rendered=(), execution-while-out-of-viewport=(), fullscreen=(), geolocation=(), gyroscope=(), hid=(), idle-detection=(), magnetometer=(), microphone=(), midi=(), navigation-override=(), payment=(), picture-in-picture=(), publickey-credentials-get=(), screen-wake-lock=(), serial=(), sync-xhr=(), usb=(), web-share=(), xr-spatial-tracking=()     
< X-Permitted-Cross-Domain-Policies: none
< Referrer-Policy: same-origin
< Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
<
{"detail":"Not authenticated"}* Connection #2 to host api.prefect.cloud left intact
c
seems like maybe you have a proxy in the middle perhaps? I don’t know where this is coming from :
Copy code
curl: (6) Could not resolve host: my_key'
but the 403 says it’s a bad api key
here’s what it should look like if it were successful
Copy code
HTTP/2 200
date: Fri, 03 Mar 2023 18:42:10 GMT
server: istio-envoy
content-length: 4
content-type: application/json
x-envoy-upstream-service-time: 11
via: 1.1 google
content-security-policy: default-src 'none'; object-src 'none'
x-content-type-options: nosniff
strict-transport-security: max-age=63072000; includeSubDomains; preload
permissions-policy: accelerometer=(), ambient-light-sensor=(), autoplay=(), battery=(), camera=(), cross-origin-isolated=(), display-capture=(), document-domain=(), encrypted-media=(), execution-while-not-rendered=(), execution-while-out-of-viewport=(), fullscreen=(), geolocation=(), gyroscope=(), hid=(), idle-detection=(), magnetometer=(), microphone=(), midi=(), navigation-override=(), payment=(), picture-in-picture=(), publickey-credentials-get=(), screen-wake-lock=(), serial=(), sync-xhr=(), usb=(), web-share=(), xr-spatial-tracking=()
x-permitted-cross-domain-policies: none
referrer-policy: same-origin
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000

true%
g
I replaced my actual key with the string 'my_key' so that I could send the message.
@Christopher Boyd I have attempted to check the status using a python script. I don't get any error here so I assume the health check passes.
Copy code
import prefect
import anyio

# set up Prefect API URL and API key
api_url = "<https://app.prefect.cloud/account/bae78ed9-1e37-409a-b34e-9c99aca8d81e/workspace/141fc872-2acf-4c37-892a-d4aebeb77040>"
api_key = my_key


async def main():
    client = prefect.get_cloud_client(api_key=api_key)
    await client.api_healthcheck()

anyio.run(main)
Why would it succeed here but fail on the curl?
c
I don’t know that’s going to “fail” , a 404 is a valid response code, not a failure to retrieve a response code. A better test might be to use requests or httpx and print the response.status
I don’t know if this helping to resolve or investigate the error though, I would have to reproduce and attempt to run a prefect flow in an azure function to better understand expected behavior. It very well might be a bug on our end, but would need to test unless you had the ability to provide a clear example of working 1:1 local vs cloud and the tracebacks
I would still have to reproduce, I just don’t know I have the exact answer here or the ability to dig deep enough without trying to reproduce
g
I understand. I will try to make a concise and organized sample of local vs cloud and come back here for you to check out, worst case I'll use that to make a support ticket.
🙏 1
c
Thank you for understanding, sorry I don’t have the answer here - I can get some time to test this during the week
g
@Christopher Boyd here's a pdf of the problem summary I sent to support if you're interested.
az_func_prefect_problem.pdf
c
Im working on reproducing this - if you can share your trigger function, or that the full one? It looks like you are triggering the run deployment on blob upload?
g
Yea that's the full one. It's an event grid azure function, so I set the event subscription up to trigger the function whenever a blob gets uploaded under a specific blob path. The az function then triggers a deployment run
👍 1
42 Views
Open in Slack
PreviousNext
Powered by