Hi everyone, do others of you also struggle with segmentation faults? Unfortunately, we get them randomly on a regular basis. Does anyone have any ideas on how to fix them?

Terminated in Failed("Flow run encountered an exception. RuntimeError: Command failed with exit code 139:\n/usr/local/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 2 expired semaphore objects that need to be cleaned up on shutdown\n warnings.warn('resource_tracker: There appear to be %d '\n/tmp/prefect-mrwtc3pd: Line 1: 44084 segmentation fault (core dumped)

We are currently running Prefect on a self-hosted Kubernetes, but using

process

infrastructure type so far. From my layman's point of view, this could be due to the setup used. Can someone enlighten me here? Many thanks in advance!

yeah, whats stopping you from using jobs

that we cannot freely dispose of the infrastructure 🙈 for the reason and to still be able to make progress on this issue I'm currently testing it locally (setup: local Docker Desktop/Kubernetes, KubernetesJob, GCP Storage Bucket, Prefect Cloud). I can successfully create the deployment via

prefect deployment build flows\dbt\kafka_order\main_orders.py:main_flow -n k8s-kafka -sb gcs/gcsblock -ib kubernetes-job/k8s-kafka-job --apply

, see the files in the GCS bucket. But when I run the deployment in Prefect Cloud, I get the following error:

Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/fsspec/registry.py", line 211, in get_filesystem_class
    register_implementation(protocol, _import_class(bit["class"]))
  File "/usr/local/lib/python3.9/site-packages/fsspec/registry.py", line 234, in _import_class
    mod = importlib.import_module(mod)
  File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'gcsfs'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/prefect/filesystems.py", line 403, in filesystem
    self._filesystem = fsspec.filesystem(scheme, **self.settings)
  File "/usr/local/lib/python3.9/site-packages/fsspec/registry.py", line 251, in filesystem
    cls = get_filesystem_class(protocol)
  File "/usr/local/lib/python3.9/site-packages/fsspec/registry.py", line 213, in get_filesystem_class
    raise ImportError(bit["err"]) from e
ImportError: Please install gcsfs to access Google Storage

Actually

gcsfs

is already installed within the venv. So besides politics, I also have technical difficulties 🙂 Do you have any ideas about this? Addition: The containers are also created, but then jump directly into "Exited" due to the error mentioned above. Tried it before via gcs-bucket, but got "KeyError: "No class found for dispatch key 'gcs-bucket' in registry for type 'Block'."

in your k8s block you will need

"EXTRA_PIP_PACKAGES": "gcsfs"

Thanks a lot! Had to add some more, but it finally ran successfully 🎉