Tim-Oliver
01/04/2023, 2:08 PMCrash detected! Execution was interrupted by an unexpected exception: httpx.HTTPStatusError: Client error '403 Forbidden' for url '<https://api.prefect.cloud/api/accounts/uuid/workspaces/uuid/block_types/a8b20987-a71e-4b43-9350-53a9e0d7d467>'
For more information check: <https://httpstatuses.com/403>
uuid contains my respective identifiers. I am a bit confused, because I cant find any block with the id a8b20987-a71e-4b43-9350-53a9e0d7d467
. Which is apparently what it want to access. Any tips?BlockType(id=UUID('a8b20987-a71e-4b43-9350-53a9e0d7d467'), name='Local File System', slug='local-file-system', logo_url=HttpUrl('<https://images.ctfassets.net/gm98wzqotmnx/EVKjxM7fNyi4NGUSkeTEE/95c958c5dd5a56c59ea5033e919c1a63/image1.png?h=250>', ), documentation_url=None, description='Store data as a file on a local file system.', code_example='Load stored local file system config:\n```python\nfrom prefect.filesystems import LocalFileSystem\n\nlocal_file_system_block = LocalFileSystem.load("BLOCK_NAME")\n```', is_protected=True)
Nate
01/04/2023, 3:35 PMTim-Oliver
01/04/2023, 3:39 PMNate
01/04/2023, 3:45 PMTim-Oliver
01/04/2023, 3:47 PM16:15:41.526 | INFO | prefect.agent - Submitting flow run '5d3175bf-02f7-4098-9f47-af166c34df44'
16:15:42.250 | INFO | prefect.infrastructure.process - Opening process 'refreshing-stingray'...
16:15:42.476 | INFO | prefect.agent - Completed submission of flow run '5d3175bf-02f7-4098-9f47-af166c34df44'
16:16:10.373 | INFO | prefect.infrastructure.process - Process 'refreshing-stingray' exited cleanly.
Crash detected! Execution was interrupted by an unexpected exception: httpx.HTTPStatusError: Client error '403 Forbidden' for url '<https://api.prefect.cloud/api/accounts/708fd0b2-7250-4540-b2fc-58f37a51ce1f/workspaces/2f075a5f-98b9-4a72-8cb0-cf9e37594b3e/block_types/a8b20987-a71e-4b43-9350-53a9e0d7d467>'
For more information check: <https://httpstatuses.com/403>
Nate
01/04/2023, 3:53 PMTim-Oliver
01/04/2023, 3:55 PMsbatch some_script.sh
to some_other_script.sh
. The whole process runs on a SLURM cluster and I moved from sbatch to srun.
Further more I changed stream output from true to false.Nate
01/04/2023, 7:44 PMprefect block register -m prefect.infrastructure.process
would do anything.
also, I'm curious what version of prefect your agent is running?Tim-Oliver
01/05/2023, 7:17 AMCrash details:
Traceback (most recent call last):
File "/miniconda3/envs/prefect-workflows-prefect-cloud/lib/python3.9/site-packages/prefect/engine.py", line 1574, in report_flow_run_crashes
yield
File "/miniconda3/envs/prefect-workflows-prefect-cloud/lib/python3.9/site-packages/prefect/engine.py", line 365, in begin_flow_run
flow_run_context.result_factory = await ResultFactory.from_flow(
File "/miniconda3/envs/prefect-workflows-prefect-cloud/lib/python3.9/site-packages/prefect/client/utilities.py", line 47, in with_injected_client
return await fn(*args, **kwargs)
File "/miniconda3/envs/prefect-workflows-prefect-cloud/lib/python3.9/site-packages/prefect/results.py", line 161, in from_flow
return await cls.default_factory(
File "/miniconda3/envs/prefect-workflows-prefect-cloud/lib/python3.9/site-packages/prefect/client/utilities.py", line 47, in with_injected_client
return await fn(*args, **kwargs)
File "/miniconda3/envs/prefect-workflows-prefect-cloud/lib/python3.9/site-packages/prefect/results.py", line 123, in default_factory
return await cls.from_settings(**kwargs, client=client)
File "/miniconda3/envs/prefect-workflows-prefect-cloud/lib/python3.9/site-packages/prefect/client/utilities.py", line 47, in with_injected_client
return await fn(*args, **kwargs)
File "/miniconda3/envs/prefect-workflows-prefect-cloud/lib/python3.9/site-packages/prefect/results.py", line 229, in from_settings
storage_block_id, storage_block = await cls.resolve_storage_block(
File "/miniconda3/envs/prefect-workflows-prefect-cloud/lib/python3.9/site-packages/prefect/results.py", line 257, in resolve_storage_block
or await storage_block._save(is_anonymous=True, overwrite=True)
File "/miniconda3/envs/prefect-workflows-prefect-cloud/lib/python3.9/site-packages/prefect/client/utilities.py", line 47, in with_injected_client
return await fn(*args, **kwargs)
File "/miniconda3/envs/prefect-workflows-prefect-cloud/lib/python3.9/site-packages/prefect/blocks/core.py", line 759, in _save
await self.register_type_and_schema(client=client)
File "/miniconda3/envs/prefect-workflows-prefect-cloud/lib/python3.9/site-packages/prefect/client/utilities.py", line 47, in with_injected_client
return await fn(*args, **kwargs)
File "/miniconda3/envs/prefect-workflows-prefect-cloud/lib/python3.9/site-packages/prefect/blocks/core.py", line 705, in register_type_and_schema
await client.update_block_type(
File "/miniconda3/envs/prefect-workflows-prefect-cloud/lib/python3.9/site-packages/prefect/client/orion.py", line 1049, in update_block_type
await self._client.patch(
File "/miniconda3/envs/prefect-workflows-prefect-cloud/lib/python3.9/site-packages/httpx/_client.py", line 1922, in patch
return await self.request(
File "/miniconda3/envs/prefect-workflows-prefect-cloud/lib/python3.9/site-packages/httpx/_client.py", line 1533, in request
return await self.send(request, auth=auth, follow_redirects=follow_redirects)
File "/miniconda3/envs/prefect-workflows-prefect-cloud/lib/python3.9/site-packages/prefect/client/base.py", line 251, in send
response.raise_for_status()
File "/miniconda3/envs/prefect-workflows-prefect-cloud/lib/python3.9/site-packages/httpx/_models.py", line 745, in raise_for_status
raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Client error '403 Forbidden' for url '<https://api.prefect.cloud/api/accounts/xxxxxxxxxxxxx/workspaces/xxxxxxxxxxxxx/block_types/a8b20987-a71e-4b43-9350-53a9e0d7d467>'
For more information check: <https://httpstatuses.com/403>
Nate
01/06/2023, 3:01 PMTim-Oliver
01/06/2023, 3:03 PMNate
01/06/2023, 3:06 PMPREFECT_API_KEY
env var in that process corresponds to?Tim-Oliver
01/06/2023, 3:11 PMNate
01/06/2023, 3:25 PMPREFECT_API_KEY
set in the environment where your agent is running belonged to your service account (which did not have "Create, edit and delete blocks"), which prevented the agent process from creating infrastructure blocks needed to run your deployment. However, if
Yes, the other flow was using the same prefect service account, same queue, same workspace.... this is the case, I can't say I'm sure what's going on. I will check internally whether
Create, edit and delete blocks
should be necessary in this case and whether this is expected behavior (again, if it is expected behavior, I do think it should be more clear which permissions are required) - will get back to this thread!Tim-Oliver
01/06/2023, 3:26 PMNate
01/09/2023, 12:58 AMCreate, edit and delete blocks
permission, given that the flow result is created as as a Block behind the scenes
two more questions 🙂:
• have you configured any result-related settings for the flow that is failing?
• is the flow that is marked as crashed spawning subflow runs?Tim-Oliver
01/09/2023, 8:21 AM