Hi I am looking for a Prefect task for doing an BigQuery exp

Question

Hi I am looking for a Prefect task for doing an BigQuery export to GCS Something similar to but instead of writing to a `table dest` to write to a GCS bucket Is there a Task or a way to do

alex · Answer

Hi < Gaurang Katre> You should be able to accomplish loading to a GCS bucket by using the BigQueryTask to query the data and then use the task to upload the queried data The BigQueryTask also has the option to return a DataFrame which may make it easier to get your data in the desired format before uploading it

Gaurang Katre · Answer

My dataset is a very large raw data that i am trying to export and may not have adequate memory to stage that data Was looking for a method to do a direct export to GCS

Gaurang Katre · Answer

If there isn t a Prefect task for doing this is there a way to implement the `EXPORT DATA` as a Prefect Task

Gaurang Katre · Answer

Any suggestions or recommendations would be helpful

Gaurang Katre · Answer

< alex> I was able to get the working for the BigQuery export I just replaced the SELECT with the export statement EXPORT DATA SELECT This exported the query results to the GCS bucket

Gaurang Katre · Answer

Thanks

alex · Answer

That s great to hear