Hi, I'm trying to find out what is the recommende...
# ask-community
m
Hi, I'm trying to find out what is the recommended way to bulk load data from csv into the database (eg postgres) regularly in prefect. I understand I can do "insert into table ... " and use postgres.PostgresExecuteMany task, but I'm concerned that for very large files it might be slow. Is there a way to execute pg copy (or COPY), or "insert into" is how I should do it in prefect?
h
Since you can customise your environment that you run your task in (to the best of my understanding), you can add in
psql
there and use
\COPY
from your local worker node.
That way it's batched.
m
thanks, I think that's how I'll be doing it