https://prefect.io logo
Title
m

Marius Vollmer

03/29/2023, 7:44 AM
Hi everyone, do others of you also struggle with segmentation faults? Unfortunately, we get them randomly on a regular basis. Does anyone have any ideas on how to fix them?
Terminated in Failed("Flow run encountered an exception. RuntimeError: Command failed with exit code 139:\n/usr/local/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 2 expired semaphore objects that need to be cleaned up on shutdown\n warnings.warn('resource_tracker: There appear to be %d '\n/tmp/prefect-mrwtc3pd: Line 1: 44084 segmentation fault (core dumped)
We are currently running Prefect on a self-hosted Kubernetes, but using
process
infrastructure type so far. From my layman's point of view, this could be due to the setup used. Can someone enlighten me here? Many thanks in advance!
r

redsquare

03/29/2023, 8:22 AM
yeah, whats stopping you from using jobs
m

Marius Vollmer

03/29/2023, 8:37 AM
that we cannot freely dispose of the infrastructure 🙈 for the reason and to still be able to make progress on this issue I'm currently testing it locally (setup: local Docker Desktop/Kubernetes, KubernetesJob, GCP Storage Bucket, Prefect Cloud). I can successfully create the deployment via
prefect deployment build flows\dbt\kafka_order\main_orders.py:main_flow -n k8s-kafka -sb gcs/gcsblock -ib kubernetes-job/k8s-kafka-job --apply
, see the files in the GCS bucket. But when I run the deployment in Prefect Cloud, I get the following error:
Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/fsspec/registry.py", line 211, in get_filesystem_class
    register_implementation(protocol, _import_class(bit["class"]))
  File "/usr/local/lib/python3.9/site-packages/fsspec/registry.py", line 234, in _import_class
    mod = importlib.import_module(mod)
  File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'gcsfs'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/prefect/filesystems.py", line 403, in filesystem
    self._filesystem = fsspec.filesystem(scheme, **self.settings)
  File "/usr/local/lib/python3.9/site-packages/fsspec/registry.py", line 251, in filesystem
    cls = get_filesystem_class(protocol)
  File "/usr/local/lib/python3.9/site-packages/fsspec/registry.py", line 213, in get_filesystem_class
    raise ImportError(bit["err"]) from e
ImportError: Please install gcsfs to access Google Storage
Actually
gcsfs
is already installed within the venv. So besides politics, I also have technical difficulties 🙂 Do you have any ideas about this? Addition: The containers are also created, but then jump directly into "Exited" due to the error mentioned above. Tried it before via gcs-bucket, but got "KeyError: "No class found for dispatch key 'gcs-bucket' in registry for type 'Block'."
r

redsquare

03/29/2023, 9:07 AM
in your k8s block you will need
"EXTRA_PIP_PACKAGES": "gcsfs"
can add in the ui e.g
m

Marius Vollmer

03/30/2023, 5:30 AM
Thanks a lot! Had to add some more, but it finally ran successfully 🎉
👍 1
Short update on this: We have since moved to kubernetes-job infrastructure (so no longer just in my local replica environment). However, we still regularly get the above-mentioned error message. If anyone has had similar experiences with the interaction between Prefect and dbt, I would be grateful for any feedback!
Finished in state Failed("Flow run encountered an exception. RuntimeError: Command failed with exit code 133:\n/usr/local/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 2 leaked semaphore objects to clean up at shutdown\n  warnings.warn('resource_tracker: There appear to be %d '\n/tmp/prefect-zy9ecizx: line 1:   168 Trace/breakpoint trap   (core dumped) dbt build --select dl_kafka__order_event_epp --profiles-dir /tmp/dbt --project-dir /tmp/dbt\n\n")