Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Hi, I am looking for a Prefect task for doing an BigQuery export to GCS. Something similar to <https://docs.prefect.io/api/latest/tasks/gcp.html#bigquerytask|BigQueryTask> but instead of writing to a `table_dest` , to write to a GCS bucket. Is there a Task or a way to do <https://cloud.google.com/bigquery/docs/reference/standard-sql/other-statements#export_data_statement|this>?

Hi <@U02QHCT3MKQ>! You should be able to accomplish loading to a GCS bucket by using the BigQueryTask to query the data and then use the <https://docs.prefect.io/api/latest/tasks/gcp.html#gcsupload|GCSUpload> task to upload the queried data. The BigQueryTask also has the option to return a DataFrame which may make it easier to get your data in the desired format before uploading it.

My dataset is a very large raw data that i am trying to export and may not have adequate memory to stage that data. Was looking for a method to do a direct export to GCS

If there isn’t a Prefect task for doing this, is there a way to implement the <https://cloud.google.com/bigquery/docs/reference/standard-sql/other-statements#export_data_statement|BigQuery >`EXPORT DATA`<https://cloud.google.com/bigquery/docs/reference/standard-sql/other-statements#export_data_statement| statement> as a Prefect Task?

Any suggestions or recommendations would be helpful!

<@U02GE39K551> I was able to get the <https://docs.prefect.io/api/latest/tasks/gcp.html#bigquerytask|BigQueryTask> working for the BigQuery export. I just replaced the “SELECT … ” with the export statement “EXPORT DATA … SELECT…“. This exported the query results to the GCS bucket