https://prefect.io logo
Title
a

Antti Tupamäki

10/18/2022, 10:53 AM
Hi is there issue how prefect 1.4 .0 handles newest dask 2022.10.0 and python3.9 becouse ci/cd pipeline does not work now when I am using those but local machine with python 3.9.5 works fine.
Says ImportError: cannot import name ‘apply’ from ‘dask.compatibility’
r

Rob Freedy

10/19/2022, 3:29 PM
Hey @Antti Tupamäki! Are you able to share how you are registering the flow in.the ci/cd pipeline?
a

Antti Tupamäki

10/20/2022, 5:44 AM
actually just running it during test so no registration.
def test_customer_etl_flow(ftp_containerport, dbcontainer_port):

    target_dir = str(Path(__file__).parent)
    db_user = CustomSecrets("DBT__PGUSER").get_secret()
    db_password = CustomSecrets("DBT__PGPASSWORD").get_secret()

    customer_etl.flow.run(
        parameters=dict(
            target_dir=target_dir,
            hostname="localhost",
            raw_schema="public",
            download_folder="/upload",
            sftp_port=ftp_containerport,
            db_port=dbcontainer_port,
            database = "local"
        )
    )
    conn_string = get_sqlalchemy_db_uri_for_postgres(
        "localhost",
        "postgres",
        db_user,
        db_password,
        dbcontainer_port,
    )
    <http://logging.info|logging.info>("Connecting to database")
    conn = psycopg2.connect(conn_string)
    cursor = conn.cursor()
    <http://logging.info|logging.info>("Connected! Fetching data")
    cursor.execute(
        """SELECT table_name FROM information_schema.tables
           WHERE table_schema = 'public' AND table_type = 'BASE TABLE' """
    )
    assert len(tables) == 6
and flow itself
with Flow(
    name="customer  ETL",
    schedule=None,
    executor=DaskExecutor(cluster_kwargs={"n_workers": 1, "threads_per_worker": 1}),
) as flow: and here is too much customer specific stuff...
r

Rob Freedy

10/24/2022, 1:02 PM
How are you installing the dependencies before running the flow in the CI/CD pipeline?
a

Antti Tupamäki

10/25/2022, 11:13 AM
through poetry from poetry.lock
r

Rob Freedy

10/25/2022, 1:29 PM
Does your local poetry environment and your ci/cd poetry environment have the same dependencies and python versions installed? I have not been able to recreate the issue you are seeing, so if your local environments and ci/cd poetry environments are using the same python and dependency versions and you are still seeing that import error, I would submit a ticket here: https://github.com/PrefectHQ/prefect/issues/new/choose
a

Antti Tupamäki

10/26/2022, 5:37 AM
Ok I will do it thanks for trying the issue might be that mishmatch between local and ci/cd python version and some poetry dark magic.