hi everybody m trying to write a csv file to the GCS Does an Prefect Community #ask-community

hi everybody, 'm trying to write a csv file to the...

Laura Vaida

07/06/2021, 7:48 AM

hi everybody, 'm trying to write a csv file to the GCS. Does anybody know how to do that as result? I just can define the result object but don't know how to write to that.. thanks!

Copy code

@task(log_stdout=True)
def write_order_data (dataframe):
    current_date = dt.today().strftime("%Y_%m_%d")
    #GCS_Result = GCSResult(bucket='uwg-mail', location = 'orders_import_sf' + '_' + current_date + '.csv')
    dataframe.to_csv('<gs://uwg-mail/orders_import_sf.csv>', dataframe, header=True)

✅ 1

Marko Herkaliuk

07/06/2021, 9:07 AM

hi, try add result to task decorator

Marko Herkaliuk

07/06/2021, 9:08 AM

Copy code

@task(
    result=S3Result(
        bucket="",
        boto3_kwargs={
            "aws_access_key_id": "",
            "aws_secret_access_key": "",
            "endpoint_url": ""
        },
        location=f".csv"
    )
)

here s3 example

👍 1

Laura Vaida

07/06/2021, 1:13 PM

heey, thanks for the answer, my problem is with the to_csv method i have to define a path rightaway...

Kevin Kho

07/06/2021, 1:42 PM

You can use the PandasSerializer along with the S3Result or GCSResult. Wait, why do you need the path right away? Is your

to_csv

call right? `

Copy code

dataframe.to_csv('<gs://uwg-mail/orders_import_sf.csv>', dataframe, header=True)

You shouldn’t need to pass dataframe here right?

Marko Herkaliuk

07/06/2021, 1:52 PM

with the to_csv method i have to define a path rightaway.

no, u don’t just make

Copy code

return df.to_csv(index=False)

and path put in result

Laura Vaida

07/07/2021, 7:59 AM

thanks guys! i just worked around it now like that:

Copy code

@task(log_stdout=True)
def write_order_data (dataframe):
    dataframe.to_csv('<gs://xyz/file.csv>',  header=True, storage_options = {"token" : Secret("GCP_CREDENTIALS").get()})

🖖 1

👍 1

Open in Slack

Previous Next