Is it possible for prefect to fail with `ValueError Path Use Prefect Community #prefect-cloud

Is it possible for prefect to fail with `ValueErro...

Nikita Demir

01/27/2024, 2:21 AM

Is it possible for prefect to fail with

ValueError Path /Users/ndemir/.prefect/storage/14e82144d46844828ffe4cb2a7ce27cf does not exist

if the

result_storage

is set to GCS? Because it sure is happening to me occasionally, and is super frustrating because there doesn't seem to be any rhyme or reason for it. Most of the time it properly checks gcs, and maybe like 10% of the time it throws the above error

Nikita Demir

01/27/2024, 2:28 AM

This is actually pretty reproducible for me and maybe less random that I thought. Is it possible for a "quick run" through the UI of a deployment to not be picking up the

result_storage

? Because it seems to be happening from ui quick runs every single time

Nikita Demir

01/27/2024, 2:33 AM

Happens when I try to run the deployment from the cli as well. So confused why it would be looking locally when I very explicitly have defined

result_storage=GCS.load(

and the scheduled runs don't have issues

Nikita Demir

01/27/2024, 2:35 AM

The number of issues I've had with caching is so crazy. Do folks actually use it? Or is it better to just manually implement it?

Nate

01/27/2024, 7:51 PM

this happens when you run a task without remote result storage while testing or something, and then we save the result storage key and then when you run it somewhere else, its looking for the cached result at a place it no longer has access to

Nate

01/27/2024, 7:52 PM

i would guess that you ran it locally to test or something without the result storage specification?

Nate

01/27/2024, 7:58 PM

it may be helpful to use

Copy code

PREFECT_DEFAULT_RESULT_STORAGE_BLOCK=gcs/my-block

so that you can set it as an env var appropriately for dev / prod environments

7 Views

Open in Slack

Previous Next