https://prefect.io logo
Join Slack
Powered by
Hello. I’m currently trying to delete some deploym...
# prefect-community
s
Hello. I’m currently trying to delete some deployments and redeploy onto the prefect cloud, but I’m getting the following error:
Copy code
Client error '401 Unauthorized' for url '<https://api-beta.prefect.io/api/accounts/5c1f7ebd-7031-4f69-bc00-6d21a9b17548/workspaces/9a321454-aa9e-4fb8-8f4c-17e327a77a01/blocks/get_default_storage_block>'
For reference, I was trying to run some deployments on prefect cloud, and when that didnt work, I logged out of prefect cloud and activated a local orion server instead. The deployment command I am running is:
Copy code
prefect orion database reset && prefect deployment create manager.py
Any help on this would be much appreciated!
✅ 1
a
Looks like your API key expired - I don't think you need to recreate a deployment or change to local Orion - you would only need to create a new API key - you can set it to never expire
s
Ah I see ok
a
I think what caused the issue for you is that by default when you create an API key, it adds an expiration date for better security and to make it never expire, you need to select one more checkbox - LMK if this doesn't fix the issue for you
s
Hi. I’ve made a new key and tried to login, and it was fine. But I still seem to be getting the same error as before
Just to give a larger scope of the error :
Copy code
Traceback (most recent call last):
  File "/Users/sang.yn/miniforge3/envs/intel_env/lib/python3.8/site-packages/prefect/cli/_utilities.py", line 44, in wrapper
    return fn(*args, **kwargs)
  File "/Users/sang.yn/miniforge3/envs/intel_env/lib/python3.8/site-packages/prefect/utilities/asyncio.py", line 122, in wrapper
    return run_async_in_new_loop(async_fn, *args, **kwargs)
  File "/Users/sang.yn/miniforge3/envs/intel_env/lib/python3.8/site-packages/prefect/utilities/asyncio.py", line 69, in run_async_in_new_loop
    return anyio.run(partial(__fn, *args, **kwargs))
  File "/Users/sang.yn/miniforge3/envs/intel_env/lib/python3.8/site-packages/anyio/_core/_eventloop.py", line 70, in run
    return asynclib.run(func, *args, **backend_options)
  File "/Users/sang.yn/miniforge3/envs/intel_env/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 292, in run
    return native_run(wrapper(), debug=debug)
  File "/Users/sang.yn/miniforge3/envs/intel_env/lib/python3.8/asyncio/runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "/Users/sang.yn/miniforge3/envs/intel_env/lib/python3.8/asyncio/base_events.py", line 608, in run_until_complete
    return future.result()
  File "/Users/sang.yn/miniforge3/envs/intel_env/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
    return await func(*args)
  File "/Users/sang.yn/miniforge3/envs/intel_env/lib/python3.8/site-packages/prefect/cli/deployment.py", line 231, in create
    await spec.validate()
  File "/Users/sang.yn/miniforge3/envs/intel_env/lib/python3.8/site-packages/prefect/client.py", line 95, in with_injected_client
    return await fn(*args, **kwargs)
  File "/Users/sang.yn/miniforge3/envs/intel_env/lib/python3.8/site-packages/prefect/deployments.py", line 232, in validate
    self.flow_storage or await client.get_default_storage_block()
  File "/Users/sang.yn/miniforge3/envs/intel_env/lib/python3.8/site-packages/prefect/client.py", line 1269, in get_default_storage_block
    response = await <http://self._client.post|self._client.post>("/blocks/get_default_storage_block")
  File "/Users/sang.yn/miniforge3/envs/intel_env/lib/python3.8/site-packages/httpx/_client.py", line 1842, in post
    return await self.request(
  File "/Users/sang.yn/miniforge3/envs/intel_env/lib/python3.8/site-packages/httpx/_client.py", line 1527, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/Users/sang.yn/miniforge3/envs/intel_env/lib/python3.8/site-packages/prefect/client.py", line 233, in send
    response.raise_for_status()
  File "/Users/sang.yn/miniforge3/envs/intel_env/lib/python3.8/site-packages/httpx/_models.py", line 736, in raise_for_status
    raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Client error '401 Unauthorized' for url '<https://api-beta.prefect.io/api/accounts/5c1f7ebd-7031-4f69-bc00-6d21a9b17548/workspaces/9a321454-aa9e-4fb8-8f4c-17e327a77a01/blocks/get_default_storage_block>'
Just to show that the key has no expiration date:
a
this error indicates that your flow runner is not authenticated to retrieve flow code from storage - can you share the following? 1. your full DeploymentSpec of that flow 2. how did you configure the default storage 3. the output of 
prefect version
s
The prefect version shows as follows:
Copy code
Version:             2.0b5
API version:         0.3.1
Python version:      3.8.0
Git commit:          7b27c7cf
Built:               Tue, May 17, 2022 4:54 PM
OS/Arch:             darwin/x86_64
Profile:             default
Server type:         hosted
the default storage I made was local, I will get the deployment spec now..
What I find odd is that I am not trying to deploy onto the orion server instead of the prefect cloud, and yet ’https://api-beta.prefect.io/api… part of the error indicates that prefect infers that I am trying to deploy onto prefect cloud
a
I'm so confused - do you want to use Cloud 2.0 or self-host?
API keys are for Cloud 2.0, self hosted doesn't have API keys
s
No sorry - I was being a bit ambiguous so I apologize - I was testing the prefect cloud, and because that error was turning up, I switched to the local orion server
Because the error itself was referring to the api-beta, I thought the error originated from cloud
a
but what's your end goal - are you OK with still using Cloud 2.0? if so, you can create an API key which doesn't expire and configure it using:
Copy code
prefect cloud login -k yourkey -w youraccount/yourworkspace
s
The end goal is to use prefect cloud, yes. Let me try that and get back to you
a
to make it all less confusing, you can create multiple profiles:
Copy code
prefect profile create cloud
here configure cloud and to use local Orion, you can use the default profile:
Copy code
prefect profile use default
If I understood your intention, you want to develop locally on a local Orion server, but when you are ready to deploy it, you want to switch to the cloud, correct? this way, you can just switch the profile to cloud then:
Copy code
prefect profile use cloud
s
If I understood your intention, you want to develop locally on a local Orion server, but when you are ready to deploy it, you want to switch to the cloud, correct? Yes. I am also doing some benchmarking between prefect 1 and 2 to see what the pros/cons are between the two platforms, hence switching between them to compare and contrast. Sorry, I didnt give a context to what I was doing
a
I meant switching between local Prefect 2.0 and Cloud 2.0 - I didn't mean Prefect 1.0
I would recommend starting directly with Prefect 2.0 if you are starting with Prefect now
s
Yes. Ok, I feel like slack is making things lost in translation, and me being a numpty writing is causing no end to the confusion, and I am sorry. So just to clarify: I have been trying to run some deployments on the prefect cloud 2 and comparing/contrasting with prefect 1, which meant running some deployments on cloud 2 and testing and logging the features. As we have a large infrastructure already built in prefect 1, it was essential from my side to ensure that the people who put me in charge of this is ok with the changes (i.e. does the advantages of prefect 2 justify the effort put in to refactor?). Hence, coming to the present, the error was because I was playing around with deployments with prefect 2.
👍 1
I do still seem to be getting the error . I’ve tried to just view the prefect storage:
Copy code
prefect storage ls
and the error presists:
Copy code
Client error '401 Unauthorized' for url '<https://api-beta.prefect.io/api/accounts/5c1f7ebd-7031-4f69-bc00-6d21a9b17548/workspaces/9a321454-aa9e-4fb8-8f4c-17e327a77a01/blocks/get_default_storage_block>'
a
this error indicates that your flow runner is not authenticated to retrieve flow code from storage - can you share the following? 1. your full DeploymentSpec of that flow 2. how did you configure the default storage 3. the output of 
prefect version
totally understandable - if you are happy with Prefect 1.0, no need to change but at some point, you will have to migrate to Prefect 2.0 since this is the LTS version
s
So to answer the deployment. spec:
Copy code
FlowSchedule = rrule(
        freq = MINUTELY,
        interval= 5,
        dtstart = datetime.datetime(2020, 1, 1, 8, 31, 0),
        byhour = range(8, 9),
        byminute = range(30, 52),
    )

    return DeploymentSpec(
        flow=ng_daily_polling,
        name=f"test",
        schedule=RRuleSchedule.from_rrule(FlowSchedule),
        #tags=deploy_tags,                                                                                                                                                                                          
        #parameters=parameters,                                                                                                                                                                                     
        flow_runner=SubprocessFlowRunner(),
    )
As for the storage, following the documentation, I created the temporary storage https://orion-docs.prefect.io/concepts/storage/ Finally, for the version, I’m getting this:
Copy code
Version:             2.0b5
API version:         0.3.1
Python version:      3.9.13
Git commit:          7b27c7cf
Built:               Tue, May 17, 2022 4:54 PM
OS/Arch:             darwin/x86_64
Profile:             default
Server type:         hosted
The error only became apparent quite recently. Previously, the create deployment for this same batch of code was completely fine
a
the problem currently is that you want to use Cloud 2.0 but your profile doesn't point to Cloud - it points to a local configuration:
Copy code
Server type:         hosted
Can you build a new environment from scratch and take it all step by step? your deployment spec is fine but your installation is not if you want to use Cloud 2.0
s
Ok, I will get on that now
🙌 1
a
I think the easiest is to create a cloud profile, switch to it, then do 
prefect cloud login
on this cloud profile and this will make things much easier
s
I think switching to the ‘cloud’ profile fixed the bug. Thank you! By the way, I dont think this part of switching profiles is documented. Maybe I can assist in writing this? I dont know whether I know enough to do that but you never know haha
🙌 1
a
it's here but I agree, there's never enough docs + tutorials/examples
s
Ahhh ok, its just me being slow then hahaha
😄 1
Hi again. Just on the Deployment spec and the flows, for example in this particular flow:
Copy code
DeploymentSpec(
    name="test",
    flow=my_flow,
    flow_runner=SubprocessFlowRunner(env={"MY_VARIABLE": "FOO"}, condaenv="test"),
)
within the flow, is it possible to call upon the environmental variables as os.environ[‘MY_VARIABLE’]? Thanks
Hello, Sorry again. I’m struggling with the deployments on cloud again. For example. I created a profile called ‘cloudtest’, and the prefect version says:
Copy code
Version:             2.0b6
API version:         0.5.0
Python version:      3.10.4
Git commit:          502ee008
Built:               Tue, Jun 7, 2022 4:58 PM
OS/Arch:             darwin/x86_64
Profile:             cloud
Server type:         hosted
The server type says hosted, which means it isnt pointing to the cloud server, but looking at the config
Copy code
prefect -p 'cloudtest' config view
I seem to get
Copy code
PREFECT_PROFILE='cloudtest'
PREFECT_API_KEY='***"
PREFECT_API_URL='<http://beta.prefect.io/api>' (from profile)
The api_url is as shown in the docs, the api key is that which I did create, and yet, I seem to be getting this error:
Copy code
prefect.exceptions.PrefectHTTPStatusError: Redirect response '301 Moved Permanently' for url '<http://beta.prefect.io/api/block_documents/get_default_storage_block_document>'
Redirect location: '<https://beta.prefect.io/api/block_documents/get_default_storage_block_document>'
For more information check: <https://httpstatuses.com/301>
a
you can add env variables to the UniversalFlowRunner and all flow runners that inherit from it, incl. the SubprocessFlowRunner
Can you try creating new default storage?
if nothing else works, try recreating a deployment after you created new storage and assigned it as default
your deployment spec is fine
s
ok I’ll give it a try
👍 1
Ok, you seem to have the miracle touch. It seems the steps is 1. Ensure the local storage (or later, a S3 storage for example is sorted) is set with
Copy code
prefect storage create
2. Login to cloud with prefect cloud login --key *
Copy code
3. prefect deployment create **.py # for the creation of deployments
4. Profit! (New Prefect cloud UI is lovely by the way) The steps from 1 to 2, of the local prefect pointing to the cloud after that has been configured with the PREFECT_API_URL however, has been the bottleneck. Sometimes it seems to work, sometimes it doesnt.
a
I see - nice work figuring this out! keep us posted if the issue persists and thanks for the nice feedback
s
Just another question (sorry again!) - not related to the above issue - in prefect 1 the logging of states of a flow or task was handled with state_handlers, but in prefect 2 am I to assume that futures and states and calling them within the flow itself will be the equivalent operation?
a
what problem are you trying to solve this way? there is no concept of state handlers, but we will have a functionality to send e.g. alerts on failure
s
Oh nothing in particular - at the moment, I am using a combination of futures and client states to force a flow into failure, depending on a set criteria. It was just because I am looking at a prefect 1 version of the same code and prefect 2 version I made, and it was just a curiosity
👍 1
9 Views
Open in Slack
PreviousNext
Powered by