a

    Adam Everington

    1 year ago
    SqlServerExecuteMany(fast_executemany) performance vs SqlAlchemy(fast_Executemay) any ideas?
    Kevin Kho

    Kevin Kho

    1 year ago
    Hey @Adam Everington, where are you seeing the SqlAlchemy task? Or are you making your own?
    a

    Adam Everington

    1 year ago
    Writing my own! I just wondered how Prefects task compared. We're an MS shop and do LOTTSSSSS of large inserts. I'm just trying to find the most performant way of doing it
    Kevin Kho

    Kevin Kho

    1 year ago
    Oh I see. We used Azure Data Factory in the last job to insert. It honestly sounds like you need to really benchmark cuz it’s important. I wouldn’t have a clue, but I think SqlAlchemy uses odbc too for those operations, it just has more features right? If it’s odbc under the hood, it should be just the same speed. I wonder if
    pyarrow.flight
    has a sql server connector. That may be a lot faster than odbc.
    this would be like 10x faster if it existed.
    a

    Adam Everington

    1 year ago
    I was hoping to keep it all in prefect, but i'll benchmark them both and see what comes out. Cheers again man!
    Kevin Kho

    Kevin Kho

    1 year ago
    I agree but ADF has parallelized copy so you can use that on parquet (of course you pay for extra compute) and speed up by 4x. Maybe you should benchmark using jdbc also.