Hello we re running into an issue with running prefect 2 on Prefect Community #ask-community

Hello - we're running into an issue with running p...

Ted Collins

03/01/2024, 6:45 PM

Hello - we're running into an issue with running prefect 2 on a virtual environment on a macbook with the M2 chip. We're getting this error when we try to run a prefect command. Has anyone seen this before? Any recommendations? Thank you! After typing the command 'prefect' -- we get this error.

Copy code

ImportError: dlopen(/Users/Sara.Evans/Documents/git_repo/data_warehouse/dbt_runner/lib/python3.8/site-packages/_cffi_backend.cpython-38-darwin.so, 0x0002): tried: '/Users/Sara.Evans/Documents/git_repo/data_warehouse/dbt_runner/lib/python3.8/site-packages/_cffi_backend.cpython-38-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/Users/Sara.Evans/Documents/git_repo/data_warehouse/dbt_runner/lib/python3.8/site-packages/_cffi_backend.cpython-38-darwin.so' (no such file), '/Users/Sara.Evans/Documents/git_repo/data_warehouse/dbt_runner/lib/python3.8/site-packages/_cffi_backend.cpython-38-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64'))
thread '<unnamed>' panicked at /Users/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-0.18.3/src/err/mod.rs:790:5:
Python API call failed
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Traceback (most recent call last):
  File "/Users/Sara.Evans/Documents/git_repo/data_warehouse/dbt_runner/bin/prefect", line 5, in <module>
    from prefect.cli import app
  File "/Users/Sara.Evans/Documents/git_repo/data_warehouse/dbt_runner/lib/python3.8/site-packages/prefect/__init__.py", line 25, in <module>
    from prefect.states import State
  File "/Users/Sara.Evans/Documents/git_repo/data_warehouse/dbt_runner/lib/python3.8/site-packages/prefect/states.py", line 14, in <module>
    from prefect.client.schemas import State as State
  File "/Users/Sara.Evans/Documents/git_repo/data_warehouse/dbt_runner/lib/python3.8/site-packages/prefect/client/schemas.py", line 6, in <module>
    from prefect.server import schemas
  File "/Users/Sara.Evans/Documents/git_repo/data_warehouse/dbt_runner/lib/python3.8/site-packages/prefect/server/__init__.py", line 1, in <module>
    from . import models
  File "/Users/Sara.Evans/Documents/git_repo/data_warehouse/dbt_runner/lib/python3.8/site-packages/prefect/server/models/__init__.py", line 1, in <module>
    from . import (
  File "/Users/Sara.Evans/Documents/git_repo/data_warehouse/dbt_runner/lib/python3.8/site-packages/prefect/server/models/block_documents.py", line 14, in <module>
    from prefect.server.database.dependencies import inject_db
  File "/Users/Sara.Evans/Documents/git_repo/data_warehouse/dbt_runner/lib/python3.8/site-packages/prefect/server/database/dependencies.py", line 14, in <module>
    from prefect.server.database.interface import PrefectDBInterface
  File "/Users/Sara.Evans/Documents/git_repo/data_warehouse/dbt_runner/lib/python3.8/site-packages/prefect/server/database/interface.py", line 9, in <module>
    from prefect.server.database.orm_models import BaseORMConfiguration
  File "/Users/Sara.Evans/Documents/git_repo/data_warehouse/dbt_runner/lib/python3.8/site-packages/prefect/server/database/orm_models.py", line 27, in <module>
    from prefect.server.utilities.encryption import decrypt_fernet, encrypt_fernet
  File "/Users/Sara.Evans/Documents/git_repo/data_warehouse/dbt_runner/lib/python3.8/site-packages/prefect/server/utilities/encryption.py", line 7, in <module>
    from cryptography.fernet import Fernet
  File "/Users/Sara.Evans/Documents/git_repo/data_warehouse/dbt_runner/lib/python3.8/site-packages/cryptography/fernet.py", line 14, in <module>
    from cryptography.exceptions import InvalidSignature
  File "/Users/Sara.Evans/Documents/git_repo/data_warehouse/dbt_runner/lib/python3.8/site-packages/cryptography/exceptions.py", line 9, in <module>
    from cryptography.hazmat.bindings._rust import exceptions as rust_exceptions
pyo3_runtime.PanicException: Python API call failed

Ted Collins

03/01/2024, 6:46 PM

here is our requirements.txt file: anyio==3.7.1 dbt-core==1.7.8 dbt-snowflake==1.7.2 prefect==2.8.3 boto3==1.26.84 pendulum==2.1.2 prefect-aws==0.2.4 prefect-dbt[cli]==0.3.1 pandas==1.4.3 pyarrow==10.0.1 pydantic==1.10.6 imbalanced-learn==0.8.0 scikit-learn==0.24.1 matplotlib==3.5.2 xmltodict==0.12.0 PyDrive==1.3.1 python-pptx==0.6.21 seaborn==0.11.2 gspread==5.7.2 psycopg2-binary==2.8.6 requests==2.25.1 orjson==3.8.8 tabulate==0.9.0 cffi==1.15.1 snowflake-connector-python==3.4.0

Willem Kokke

03/01/2024, 6:53 PM

It looks like you have committed a virtual environment made on an intel mac to the git repository, and are now trying to use it on an apple silicon mac. You will have to recreate the virtual environment with that requirements.txt file on the new mac, it will install the packages for the correct processer architecture.

Willem Kokke

03/01/2024, 6:55 PM

I'm not sure if the virtual environment is committed to the repository, it could also be moved I suppose, but if it is, virtual environments are usually not committed to source control but recreated on each persons individual workstation.

Ted Collins

03/01/2024, 6:56 PM

thanks for your response. we did try to re-create the virtual env on the new mac's individual workstation.

Ted Collins

03/01/2024, 6:57 PM

This is the error that we get after having recreated the virtual environment on the arm 64 silicon mac.

Willem Kokke

03/01/2024, 7:01 PM

In that case you might have recreated the virtual environment using a shell that is running under rosetta2, which is apple's x86_64 emulation layer. That would cause it to install x86_64 binary python packages. If the environment is then later activated in a normal shell this error would occur. The only thing I know for sure is that it's definitely trying to load the x86_64 cffi dynamic library in a arm64 python.

Ted Collins

03/01/2024, 7:02 PM

okay thanks. are there alternatives to rosetta2?

Willem Kokke

03/01/2024, 7:07 PM

Ideally it would not be used at all unless you depend on python packages that are only available on intel macs and not yet on apple silicon ones. There should be very few of those left, and cffi is definitely not one of them. If you run the

arch

command in the shell you use to recreate the virtual environment you will be able to tell. If it is running under rosetta (which is not the case by default) it will say i386, else it will say arm64 as it should

10 Views

Open in Slack

Previous Next