https://prefect.io
Join Slack
Hi, I am looking for a Prefect task for doing an BigQuery export to GCS. Something similar to <BigQu...
# ask-community
g
Hi, I am looking for a Prefect task for doing an BigQuery export to GCS. Something similar to BigQueryTask but instead of writing to a 
table_dest
, to write to a GCS bucket. Is there a Task or a way to do this?
a
Hi @Gaurang Katre! You should be able to accomplish loading to a GCS bucket by using the BigQueryTask to query the data and then use the GCSUpload task to upload the queried data. The BigQueryTask also has the option to return a DataFrame which may make it easier to get your data in the desired format before uploading it.
g
My dataset is a very large raw data that i am trying to export and may not have adequate memory to stage that data. Was looking for a method to do a direct export to GCS
If there isn’t a Prefect task for doing this, is there a way to implement the BigQuery `EXPORT DATA` statement as a Prefect Task?
Any suggestions or recommendations would be helpful!
@alex I was able to get the BigQueryTask working for the BigQuery export. I just replaced the “SELECT … ” with the export statement “EXPORT DATA … SELECT…“. This exported the query results to the GCS bucket
👍 1
Thanks!
a
That’s great to hear!
5 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by