Hi, I'm trying to find out what is the recommended way to bulk load data from csv into the database (eg postgres) regularly in prefect. I understand I can do "insert into table ... " and use postgres.PostgresExecuteMany task, but I'm concerned that for very large files it might be slow. Is there a way to execute pg copy (or COPY), or "insert into" is how I should do it in prefect?

Since you can customise your environment that you run your task in (to the best of my understanding), you can add in

psql

there and use

\COPY

from your local worker node.

thanks, I think that's how I'll be doing it