Gaurang Katre

    Gaurang Katre

    7 months ago
    Hi, I am looking for a Prefect task for doing an BigQuery export to GCS. Something similar to BigQueryTask but instead of writing to a
    table_dest
    , to write to a GCS bucket. Is there a Task or a way to do this?
    alex

    alex

    7 months ago
    Hi @Gaurang Katre! You should be able to accomplish loading to a GCS bucket by using the BigQueryTask to query the data and then use the GCSUpload task to upload the queried data. The BigQueryTask also has the option to return a DataFrame which may make it easier to get your data in the desired format before uploading it.
    Gaurang Katre

    Gaurang Katre

    7 months ago
    My dataset is a very large raw data that i am trying to export and may not have adequate memory to stage that data. Was looking for a method to do a direct export to GCS
    If there isn’t a Prefect task for doing this, is there a way to implement the BigQuery EXPORT DATA statement as a Prefect Task?
    Any suggestions or recommendations would be helpful!
    @alex I was able to get the BigQueryTask working for the BigQuery export. I just replaced the “SELECT … ” with the export statement “EXPORT DATA … SELECT…“. This exported the query results to the GCS bucket
    Thanks!
    alex

    alex

    7 months ago
    That’s great to hear!