https://prefect.io logo
Powered by
Title
j

James Phoenix

09/07/2022, 5:49 PM
What am I doing incorrectly? šŸ˜„
āœ… 1
n

Nate

09/07/2022, 6:04 PM
Hey @James Phoenix when you run your flow, how do you do it?
j

James Phoenix

09/07/2022, 6:06 PM
I am currently testing locally just by running python3 file.py
I tried awaiting the function by putting it in asyncio.run(anaylse_youtube_video) but no luck there
n

Nate

09/07/2022, 6:13 PM
did you call 
asyncio.run(anaylse_youtube_video)
or 
asyncio.run(anaylse_youtube_video())
?
j

James Phoenix

09/07/2022, 6:24 PM
@flow
async def analyse_youtube_video():
    # 1. Download the video from YouTube and upload it to GCS
    gcs_block = GCS.load("buffalo-raw-video-files")

    # Make some fake JSON data:
    data = {"name": "John", "age": 30, "city": "New York"}
    return await gcs_block.write_path(
        "video.mp4", data)

asyncio.run(analyse_youtube_video)
This triggers:
ValueError: a coroutine was expected, got <prefect.flows.Flow object at 0x7f952f49a3a0>
n

Nate

09/07/2022, 6:25 PM
that makes sense, you need to give 
asyncio.run()
a coroutine like 
asyncio.run(analyse_youtube_video())
j

James Phoenix

09/07/2022, 6:31 PM
Cool that ran šŸ‘
How do I fix this write error?
async def analyse_youtube_video():
    # 1. Download the video from YouTube and upload it to GCS
    gcs_block = await GCS.load("buffalo-raw-video-files")

    # Make some fake JSON data:
    data = {"name": "John", "age": 30, "city": "New York"}
    return await gcs_block.write_path(
        "video.mp4", json.dump(data))

asyncio.run(analyse_youtube_video())
I pip installed gcfs which fixed the path error, now Iā€™m running into: 
TypeError: a bytes-like object is required, not 'str'
n

Nate

09/07/2022, 6:44 PM
GCS.write_path
accepts a string 
path
argument and a 
content
argument of type bytes it looks like you're trying to pass 
json.dump(data)
as 
content
, which won't work because 
json.dump
is for writing to files and doesn't return 
bytes
. you could replace 
json.dump(data)
with 
json.dumps(data).encode('utf-8')
j

James Phoenix

09/07/2022, 6:49 PM
I can confirm this works.
I do get this message though:
UserWarning: Block document has schema checksum sha256:1daf535b60cf9304103bcb4015280a30fd6bdd9686aa0c059eb9715082b5b3ec which does not match the schema checksum for class 'GCS'.
Thanks @Nate šŸ‘
n

Nate

09/07/2022, 6:51 PM
Don't worry about that warning! it just means that behind the scenes the implementation of the GCS block class has changed, it doesn't reflect anything you did wrong - we'll be silencing this warning soon glad I could help!
d

Dominik Wagner

09/09/2022, 10:54 AM
Iā€™m trying to do something similar, and following your example it seems to get stuck during the 
write_path
call forever with no error: 
def test_upload():
    gcs_block = GCS.load("gcs-block")
    content = "abcd".encode('utf-8')

    asyncio.run(gcs_block.write_path("test.txt", content))

test_upload()
Any ideas what could be the cause here, or even ideas on how to debug this?
Okay I have no idea what was causing it to hang, but I restarted my computer and now it works šŸ˜•
j

James Phoenix

09/09/2022, 11:15 AM
Change def test_upload() to 
async def test_upload():
Then await 
gcs_block.write_path("test.txt", content)
rather than using asyncio.run() inside of it
Wrap test_upload with a @task or @flow
Then call asyncio.run() outside of your flow/task
@Dominik Wagner I decided to just write some of my own Google Cloud Storage CRUD anyway instead of using a block:
gcs_crud.py
gcs_crud.py
d

Dominik Wagner

09/09/2022, 11:35 AM
Works either way - I was previously using the native google upload too, but now relying on the storage block to point to the correct bucket - figured I can still use the google functions and get the bucket with 
GCS.load(block).bucket_path
šŸ‘
šŸ™Œ 1
āœ… 1
j

James Phoenix

09/09/2022, 11:36 AM
šŸ‘
#prefect-communityJoin Slack
Powered by