Hi, I am looking for a Prefect task for doing an BigQuery export to GCS. Something similar to BigQueryTask but instead of writing to a
, to write to a GCS bucket. Is there a Task or a way to do this?
02/08/2022, 10:31 PM
Hi @Gaurang Katre! You should be able to accomplish loading to a GCS bucket by using the BigQueryTask to query the data and then use the GCSUpload task to upload the queried data. The BigQueryTask also has the option to return a DataFrame which may make it easier to get your data in the desired format before uploading it.
02/08/2022, 10:38 PM
My dataset is a very large raw data that i am trying to export and may not have adequate memory to stage that data. Was looking for a method to do a direct export to GCS
If there isn’t a Prefect task for doing this, is there a way to implement the BigQuery `EXPORT DATA` statement as a Prefect Task?
Any suggestions or recommendations would be helpful!
@alex I was able to get the BigQueryTask working for the BigQuery export. I just replaced the “SELECT … ” with the export statement “EXPORT DATA … SELECT…“. This exported the query results to the GCS bucket