Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
pacc-clearcover-june-12-2023
pacc-may-31-2023
ppcc-may-16-2023
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Title
c
Chris Hart
04/03/2020, 4:45 PMI have started using the PostgresExecutor task and after reading the source and trying to understand how it works, I'm not clear on transactions and exactly how lots of separate tasks get rolled into them.. (https://github.com/PrefectHQ/prefect/blob/master/src/prefect/tasks/postgres/postgres.py#L174-L203)
I noticed that in a mapped insert task I ran, there were quite a few rejected records (on purpose), but all the remaining good ones seemed to be inserted correctly...
and yet, it seems that
commit=Falseis anyone aware of deeper dive docs on dask and psycopg?
j
josh
04/03/2020, 6:42 PMThe dask documentation is very good for a deeper dive into it https://docs.dask.org/en/latest/
As for the psycopg commit behavior I don’t know of any deeper dive resources but this is a nice page I found searching around which starts by explaining transactions https://www.postgresqltutorial.com/postgresql-python/transaction/
🙌 1
c
Chris Hart
04/03/2020, 8:22 PMah ok so the PostgresExecutor task does not support the connection.executemany() function inside the cursor.. this would be really helpful in mapped tasks that are doing data loading of fixed-size batches..
currently each run() call opens a separate connection/transaction per single query.. it would be possible to make a super giant querystring for the whole chunk, but executemany() makes it much friendlier
would this be something welcome as a PR? I can pretty easily extend the task class in my code, but might be useful to others... and by extension i wonder what the ambitions/roadmap are for postgres tasks
j
josh
04/03/2020, 8:26 PMAbsolutely welcome as a PR!
The task library is generally community driven so there are no formal roadmap items (yet) for the tasks in it. The goal is to have a library of easy to use tasks for performing basic functions that can also serve as templates/inspiration for others to use when making their own tasks 🙂
c
Chris Hart
04/03/2020, 8:33 PMawesome thanks