Idan
01/10/2023, 3:50 PMos.seteuid(priv_user_id)
(and then later resets to original_user_id
).
This is run in a container, so the original_user_id
is root
, which entails PREFECT_HOME
is /root/.prefect/
.
Our tasks are long-running, so we also cache them. Then, every now and then, a task succeeds, but fails with:
Crash detected! Execution was interrupted by an unexpected exception: Traceback (most recent call last):
File "/usr/lib/python3.10/pathlib.py", line 1175, in mkdir
self._accessor.mkdir(self, mode)
PermissionError: [Errno 13] Permission denied: '/root/.prefect/storage'
During handling of the above exception, another exception occurred:
PermissionError: [Errno 13] Permission denied: '/root/.prefect/storage'
Any smart ideas how to facilitate both needs?Zanie
01/10/2023, 5:57 PMIdan
01/10/2023, 7:27 PMZanie
01/10/2023, 7:30 PMIdan
01/10/2023, 7:38 PMZanie
01/10/2023, 7:39 PMIdan
01/10/2023, 7:40 PMZanie
01/10/2023, 7:41 PMDockerContainer(env={"A_SETTING": "value"})
will override whatever the agent has set.mkdir
calls are happening concurrently and that there’s a race condition in the Python stdlib with exists_ok
? It’s weird that it’d manifest as a permission error though.Idan
01/10/2023, 7:45 PMos.access
then you can't verify the directory exists. Wonder if I could just start specific tasks/flows with a specific user ID 🤔Zanie
01/10/2023, 7:46 PMuser
from https://docker-py.readthedocs.io/en/stable/containers.htmlDockerContainer
abstraction)Idan
01/10/2023, 7:48 PMuser
field to DockerContainer
and then to _build_container_settings
?class DockerContainerWithUser(DockerContainer):
user: Optional[Union[str, int]] = Field(
default=None,
description='The username or user ID to use in the docker',
)
def _build_container_settings(
self,
docker_client: "DockerClient",
) -> Dict:
return {**super()._build_container_settings(docker_client=docker_client), "user": self.user}
But printing os.getuid()
in the beginning of the flow still prints 0
🤔Zanie
01/11/2023, 3:42 PMtype
string for your subclass (I have not played around with extending in that way yet).DockerContainer.run(command=[…])
for faster testing and that would avoid any confusion about if your class is used.Idan
01/11/2023, 5:55 PMtype
as suggested 👍 Now the agent complains (successfully?)
/usr/local/lib/python3.10/site-packages/prefect/agent.py:353: UserWarning: Block document has schema checksum sha256:b3f4df5ac83c5075a8d4b2ee8078c56c429e90af94e4cb3befd0bf6795a7176f which does not match the schema checksum for class 'DockerContainer'. This indicates the schema has changed and this block may not load.
infrastructure_block = Block._from_block_document(infra_document)
08:48:51.121 | ERROR | prefect.agent - Failed to get infrastructure for flow run 'f01c0ecc-8dda-4392-bfdf-5d7f9f5cf9e3'.
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/prefect/agent.py", line 372, in submit_run
infrastructure = await self.get_infrastructure(flow_run)
File "/usr/local/lib/python3.10/site-packages/prefect/agent.py", line 353, in get_infrastructure
infrastructure_block = Block._from_block_document(infra_document)
File "/usr/local/lib/python3.10/site-packages/prefect/blocks/core.py", line 554, in _from_block_document
block = block_cls.parse_obj(block_document.data)
File "pydantic/main.py", line 526, in pydantic.main.BaseModel.parse_obj
File "/usr/local/lib/python3.10/site-packages/prefect/blocks/core.py", line 175, in __init__
super().__init__(*args, **kwargs)
File "pydantic/main.py", line 342, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for DockerContainer
type
unexpected value; permitted: 'docker-container' (type=value_error.const; given=docker-container-w-user; permitted=('docker-container',))
08:52:34.454 | INFO | prefect.agent - Submitting flow run 'e4e9752b-f106-4548-b4e1-7b9ef095f1bd'
08:52:34.699 | ERROR | prefect.agent - Failed to get infrastructure for flow run 'e4e9752b-f106-4548-b4e1-7b9ef095f1bd'.
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/prefect/agent.py", line 372, in submit_run
infrastructure = await self.get_infrastructure(flow_run)
File "/usr/local/lib/python3.10/site-packages/prefect/agent.py", line 353, in get_infrastructure
infrastructure_block = Block._from_block_document(infra_document)
File "/usr/local/lib/python3.10/site-packages/prefect/blocks/core.py", line 554, in _from_block_document
block = block_cls.parse_obj(block_document.data)
File "pydantic/main.py", line 526, in pydantic.main.BaseModel.parse_obj
File "/usr/local/lib/python3.10/site-packages/prefect/blocks/core.py", line 175, in __init__
super().__init__(*args, **kwargs)
File "pydantic/main.py", line 342, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for DockerContainer
type
unexpected value; permitted: 'docker-container' (type=value_error.const; given=docker-container-w-user; permitted=('docker-container',))
user
argument natively, just the Python SDK does not 🤔Zanie
01/12/2023, 5:47 PMIdan
01/13/2023, 9:29 AMfrom prefect.utilities.dispatch import register_type
, but I imagine that has no effect on the agent itself?