Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

*SqlServerExecuteMany(fast_executemany) performance vs SqlAlchemy(fast_Executemay) any ideas?*

Hey <@U02A67U1UUA>, where are you seeing the SqlAlchemy task? Or are you making your own?

Writing my own! I just wondered how Prefects task compared. We're an MS shop and do LOTTSSSSS of large inserts. I'm just trying to find the most performant way of doing it

Oh I see. We used Azure Data Factory in the last job to insert. It honestly sounds like you need to really benchmark cuz it’s important. I wouldn’t have a clue, but I think SqlAlchemy uses odbc too for those operations, it just has more features right? If it’s odbc under the hood, it should be just the same speed. I wonder if `pyarrow.flight` has a sql server connector. That may be a lot faster than odbc.

<https://github.com/sqlalchemy/sqlalchemy/issues/5089|this> would be like 10x faster if it existed.

I was hoping to keep it all in prefect, but i'll benchmark them both and see what comes out. Cheers again man!

I agree but ADF has parallelized copy so you can use that on parquet (of course you pay for extra compute) and speed up by 4x. Maybe you should benchmark using jdbc also.