Channels
#prefect-community
Title
a
Antti Tupamäki
10/18/2022, 10:53 AMHi is there issue how prefect 1.4 .0 handles newest dask 2022.10.0 and python3.9 becouse ci/cd pipeline does not work now when I am using those but local machine with python 3.9.5 works fine.
Says ImportError: cannot import name ‘apply’ from ‘dask.compatibility’
r
Rob Freedy
10/19/2022, 3:29 PMHey @Antti Tupamäki! Are you able to share how you are registering the flow in.the ci/cd pipeline?
a
Antti Tupamäki
10/20/2022, 5:44 AMactually just running it during test so no registration.
Copy code
def test_customer_etl_flow(ftp_containerport, dbcontainer_port):
target_dir = str(Path(__file__).parent)
db_user = CustomSecrets("DBT__PGUSER").get_secret()
db_password = CustomSecrets("DBT__PGPASSWORD").get_secret()
customer_etl.flow.run(
parameters=dict(
target_dir=target_dir,
hostname="localhost",
raw_schema="public",
download_folder="/upload",
sftp_port=ftp_containerport,
db_port=dbcontainer_port,
database = "local"
)
)
conn_string = get_sqlalchemy_db_uri_for_postgres(
"localhost",
"postgres",
db_user,
db_password,
dbcontainer_port,
)
<http://logging.info|logging.info>("Connecting to database")
conn = psycopg2.connect(conn_string)
cursor = conn.cursor()
<http://logging.info|logging.info>("Connected! Fetching data")
cursor.execute(
"""SELECT table_name FROM information_schema.tables
WHERE table_schema = 'public' AND table_type = 'BASE TABLE' """
)
assert len(tables) == 6and flow itself
Copy code
with Flow(
name="customer ETL",
schedule=None,
executor=DaskExecutor(cluster_kwargs={"n_workers": 1, "threads_per_worker": 1}),
) as flow: and here is too much customer specific stuff...r
Rob Freedy
10/24/2022, 1:02 PMHow are you installing the dependencies before running the flow in the CI/CD pipeline?
a
Antti Tupamäki
10/25/2022, 11:13 AMthrough poetry from poetry.lock
r
Rob Freedy
10/25/2022, 1:29 PMDoes your local poetry environment and your ci/cd poetry environment have the same dependencies and python versions installed? I have not been able to recreate the issue you are seeing, so if your local environments and ci/cd poetry environments are using the same python and dependency versions and you are still seeing that import error, I would submit a ticket here: https://github.com/PrefectHQ/prefect/issues/new/choose
a
Antti Tupamäki
10/26/2022, 5:37 AMOk I will do it thanks for trying the issue might be that mishmatch between local and ci/cd python version and some poetry dark magic.
3 Views