https://prefect.io logo
Powered by
Title
t

Tim-Oliver

01/04/2023, 2:08 PM
I have changed some things in my infrastructure block and now my flow runs crash with a 403 error: 
Crash detected! Execution was interrupted by an unexpected exception: httpx.HTTPStatusError: Client error '403 Forbidden' for url '<https://api.prefect.cloud/api/accounts/uuid/workspaces/uuid/block_types/a8b20987-a71e-4b43-9350-53a9e0d7d467>'
For more information check: <https://httpstatuses.com/403>
uuid contains my respective identifiers. I am a bit confused, because I cant find any block with the id 
a8b20987-a71e-4b43-9350-53a9e0d7d467
. Which is apparently what it want to access. Any tips?
When I request the block-types via REST API I get the following match: 
BlockType(id=UUID('a8b20987-a71e-4b43-9350-53a9e0d7d467'), name='Local File System', slug='local-file-system', logo_url=HttpUrl('<https://images.ctfassets.net/gm98wzqotmnx/EVKjxM7fNyi4NGUSkeTEE/95c958c5dd5a56c59ea5033e919c1a63/image1.png?h=250>', ), documentation_url=None, description='Store data as a file on a local file system.', code_example='Load stored local file system config:\n```python\nfrom prefect.filesystems import LocalFileSystem\n\nlocal_file_system_block = LocalFileSystem.load("BLOCK_NAME")\n```', is_protected=True)
This does not really resolve my confusion 🤨
I am using prefect 2.7.5.
I have another deployment which I changed today as well and this one works just fine.
n

Nate

01/04/2023, 3:35 PM
Hi @Tim-Oliver, can you share the details on your deployment that has started failing? e.g. the following info might be helpful • infrastructure block type • storage block type • what type of place your agent is running in
t

Tim-Oliver

01/04/2023, 3:39 PM
• Infrastructure block type is process
storage is github
and the agent runs on a local machine
n

Nate

01/04/2023, 3:45 PM
hmm, are you seeing that 403 in the agent logs or prefect UI? I'm curious if there's anything interesting logged before that is thrown
t

Tim-Oliver

01/04/2023, 3:47 PM
This is the agent log: 
16:15:41.526 | INFO    | prefect.agent - Submitting flow run '5d3175bf-02f7-4098-9f47-af166c34df44'
16:15:42.250 | INFO    | prefect.infrastructure.process - Opening process 'refreshing-stingray'...
16:15:42.476 | INFO    | prefect.agent - Completed submission of flow run '5d3175bf-02f7-4098-9f47-af166c34df44'
16:16:10.373 | INFO    | prefect.infrastructure.process - Process 'refreshing-stingray' exited cleanly.
And this is the only message in the prefect UI log: 
Crash detected! Execution was interrupted by an unexpected exception: httpx.HTTPStatusError: Client error '403 Forbidden' for url '<https://api.prefect.cloud/api/accounts/708fd0b2-7250-4540-b2fc-58f37a51ce1f/workspaces/2f075a5f-98b9-4a72-8cb0-cf9e37594b3e/block_types/a8b20987-a71e-4b43-9350-53a9e0d7d467>'
For more information check: <https://httpstatuses.com/403>
State message is the same as recorded in the log.
n

Nate

01/04/2023, 3:53 PM
hmm its strange to me that that the run appeared to complete - do you mind telling what you changed in your infra block leading up to this?
t

Tim-Oliver

01/04/2023, 3:55 PM
I changed the command from 
sbatch some_script.sh
to 
some_other_script.sh
. The whole process runs on a SLURM cluster and I moved from sbatch to srun. Further more I changed stream output from true to false.
I have another process block configured in the same way and this one works.
(Same up to conda env which is used in the back.)
I also tried to delete the deployment and upload it again from my machine.
n

Nate

01/04/2023, 7:44 PM
thanks for the info thus far - I'm having a difficult time reproducing, this may be worth opening an issue on, although I wonder if either: • recreating the problematic Process block instance • re-registering the actual block type with 
prefect block register -m prefect.infrastructure.process
would do anything. also, I'm curious what version of prefect your agent is running?
t

Tim-Oliver

01/05/2023, 7:17 AM
The agent is running on 2.7.5 as well.
I will try to re-create the blocks and also have to deploy something new.
I upgraded every prefect to 2.7.6 and removed the deployment, git-storage and process block and added everything again. I still encounter the same error.
Set the log-level to debug: 
Crash details:
Traceback (most recent call last):
  File "/miniconda3/envs/prefect-workflows-prefect-cloud/lib/python3.9/site-packages/prefect/engine.py", line 1574, in report_flow_run_crashes
    yield
  File "/miniconda3/envs/prefect-workflows-prefect-cloud/lib/python3.9/site-packages/prefect/engine.py", line 365, in begin_flow_run
    flow_run_context.result_factory = await ResultFactory.from_flow(
  File "/miniconda3/envs/prefect-workflows-prefect-cloud/lib/python3.9/site-packages/prefect/client/utilities.py", line 47, in with_injected_client
    return await fn(*args, **kwargs)
  File "/miniconda3/envs/prefect-workflows-prefect-cloud/lib/python3.9/site-packages/prefect/results.py", line 161, in from_flow
    return await cls.default_factory(
  File "/miniconda3/envs/prefect-workflows-prefect-cloud/lib/python3.9/site-packages/prefect/client/utilities.py", line 47, in with_injected_client
    return await fn(*args, **kwargs)
  File "/miniconda3/envs/prefect-workflows-prefect-cloud/lib/python3.9/site-packages/prefect/results.py", line 123, in default_factory
    return await cls.from_settings(**kwargs, client=client)
  File "/miniconda3/envs/prefect-workflows-prefect-cloud/lib/python3.9/site-packages/prefect/client/utilities.py", line 47, in with_injected_client
    return await fn(*args, **kwargs)
  File "/miniconda3/envs/prefect-workflows-prefect-cloud/lib/python3.9/site-packages/prefect/results.py", line 229, in from_settings
    storage_block_id, storage_block = await cls.resolve_storage_block(
  File "/miniconda3/envs/prefect-workflows-prefect-cloud/lib/python3.9/site-packages/prefect/results.py", line 257, in resolve_storage_block
    or await storage_block._save(is_anonymous=True, overwrite=True)
  File "/miniconda3/envs/prefect-workflows-prefect-cloud/lib/python3.9/site-packages/prefect/client/utilities.py", line 47, in with_injected_client
    return await fn(*args, **kwargs)
  File "/miniconda3/envs/prefect-workflows-prefect-cloud/lib/python3.9/site-packages/prefect/blocks/core.py", line 759, in _save
    await self.register_type_and_schema(client=client)
  File "/miniconda3/envs/prefect-workflows-prefect-cloud/lib/python3.9/site-packages/prefect/client/utilities.py", line 47, in with_injected_client
    return await fn(*args, **kwargs)
  File "/miniconda3/envs/prefect-workflows-prefect-cloud/lib/python3.9/site-packages/prefect/blocks/core.py", line 705, in register_type_and_schema
    await client.update_block_type(
  File "/miniconda3/envs/prefect-workflows-prefect-cloud/lib/python3.9/site-packages/prefect/client/orion.py", line 1049, in update_block_type
    await self._client.patch(
  File "/miniconda3/envs/prefect-workflows-prefect-cloud/lib/python3.9/site-packages/httpx/_client.py", line 1922, in patch
    return await self.request(
  File "/miniconda3/envs/prefect-workflows-prefect-cloud/lib/python3.9/site-packages/httpx/_client.py", line 1533, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/miniconda3/envs/prefect-workflows-prefect-cloud/lib/python3.9/site-packages/prefect/client/base.py", line 251, in send
    response.raise_for_status()
  File "/miniconda3/envs/prefect-workflows-prefect-cloud/lib/python3.9/site-packages/httpx/_models.py", line 745, in raise_for_status
    raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Client error '403 Forbidden' for url '<https://api.prefect.cloud/api/accounts/xxxxxxxxxxxxx/workspaces/xxxxxxxxxxxxx/block_types/a8b20987-a71e-4b43-9350-53a9e0d7d467>'
For more information check: <https://httpstatuses.com/403>
Same workflow run from a different workspace works just fine...
Okay, it is connect to service accounts. When I run the agent via my account it works. When I try to run it via service account it crashes.
Still confused why the other workflow runs via service account.
Alright, this is not related to service accounts but to role based access permissions. I have a specific role in which I did not activate "Create, edit and delete blocks". Now with this activated it works. But it still makes no sense to me why this workflow needs this permissions. Maybe this is a bug?
n

Nate

01/06/2023, 3:01 PM
ahh - so was it your service account (that did not have "Create, edit and delete blocks") credentials that were stored as env vars where the agent is running? if so, that would make sense, as agents would need to create an instance of the infrastructure block that it pulls from cloud in order to run the deployment, and if the underlying user can't do that then I see how you'd get a 403 - I definitely think that could more clear on the page where you select permissions for a given user though. the other workflow is running using the same service acct with same permissions / workspace?
t

Tim-Oliver

01/06/2023, 3:03 PM
Not sure, the service account I am talking about is a prefect service account.
Yes, the other flow was using the same prefect service account, same queue, same workspace.
n

Nate

01/06/2023, 3:06 PM
okay - so if you're still running the agent locally, do you know which user the 
PREFECT_API_KEY
env var in that process corresponds to?
t

Tim-Oliver

01/06/2023, 3:11 PM
I am not sure if we talk about the same thing when we say user. I run everything from my system-user (although, this should probably be changed too in the future). But I have my prefect-user-account and a prefect-service-account. My prefect-user-account role is Collaborator. For the prefect-service-account I made a new role with different permissions. When the permissions do not include "Create, edit and delete blocks" it fails.
n

Nate

01/06/2023, 3:25 PM
Thanks for the context! Apologies, I was using "user" to refer to a prefect cloud account which may be a human or service account, but I guess technically service accounts are non-user organization accounts. My guess as to what was happening was that your 
PREFECT_API_KEY
set in the environment where your agent is running belonged to your service account (which did not have "Create, edit and delete blocks"), which prevented the agent process from creating infrastructure blocks needed to run your deployment. However, if
Yes, the other flow was using the same prefect service account, same queue, same workspace.
... this is the case, I can't say I'm sure what's going on. I will check internally whether 
Create, edit and delete blocks
should be necessary in this case and whether this is expected behavior (again, if it is expected behavior, I do think it should be more clear which permissions are required) - will get back to this thread!
t

Tim-Oliver

01/06/2023, 3:26 PM
Thanks!
n

Nate

01/09/2023, 12:58 AM
Hey @Tim-Oliver - was doing a little more digging and I noticed in the traceback that the 403 is actually occurring while attempting to save the results associated with your flow run, which makes a little more sense in terms of 
Create, edit and delete blocks
permission, given that the flow result is created as as a Block behind the scenes two more questions 🙂: • have you configured any result-related settings for the flow that is failing? • is the flow that is marked as crashed spawning subflow runs?
t

Tim-Oliver

01/09/2023, 8:21 AM
Ah yes, retries is used. No subflows are created.
Thanks for the feedback!
#prefect-communityJoin Slack
Powered by