This message was deleted.

hi @Andy Irvine you can set

PREFECT_DEFAULT_RESULT_STORAGE_BLOCK=gcs/my-gcs-block

To address the issue of Kubernetes pods trying to access data from the filesystem of other pods, integration of Google Cloud Storage (GCS) as shared storage can be a solution. While specific documentation on configuring GCS as default storage for work pools or servers in Prefect 2.x was not found in my initial search, there is general guidance on how to work with GCS in Prefect flows. Here is what you can do to move forward with GCS integration: 1. Ensure that your Prefect flows are configured to use GCS for storing any necessary files. Flows can push data to and pull data from GCS as needed. 2. Create a GCP service account with the required permissions to access GCS and download the JSON key file for authentication. 3. Store the service account credentials as a Prefect Secret or a Credentials block that can be referenced by your flows. 4. Use the GCS client within your flow functions to interact with your buckets as needed. This may involve creating custom tasks or utilizing existing tasks that support GCS operations. 5. Customize your Kubernetes job template to include the necessary environment variables that allow the worker to authenticate with GCS at runtime. If the problem persists, and you need more detailed documentation on how to accomplish these steps, consider revisiting the Prefect documentation or exploring the Prefect community resources for more information. The relevant guidance can be found in the Prefect Storage Guide where the process of creating a service account and referencing credentials is outlined. Moreover, details about configuring work pools can be looked up in the Prefect Work Pools documentation. Please note that as Prefect continues to evolve, the best practices and methods of integration can change. It's always a good idea to check the latest Prefect documentation or reach out to the Prefect community for the most up-to-date information.

Ah thanks @Nate was just deleting as did not seem to have a reply from Marvin but now I have both.

yeah marvin just takes a second

And the way to do this is to use a storage block configured for gcs? The documentation suggests that is legacy which seemed a bit confusing as no reference to storage in the work pool section

the idea of a work pool is sort of separate from result storage work pool is infra config for your flow runs result storage is where to store serialized

return

values for flow runs, independent of any work pool

To use Google Cloud Storage (GCS) with Prefect, you would install

prefect-gcp

using pip (

pip install prefect-gcp

) and then create a

GcsBucket

. However, the example I found doesn't specifically utilize

GcsBucket

, but it does demonstrate a basic Prefect flow structure. For a

GcsBucket

example, I would need to run another search or check the Prefect GCP integration repository or documentation. Nonetheless, the example code here is a simple Prefect deployment that uses

serve.py

and looks like it's meant to provide an illustration of a Python script serving a Prefect flow and mapping tasks:

python
import time
from prefect import flow, task
from pydantic import BaseModel, Field

class Data(BaseModel):
    key: str
    value: dict = Field(default_factory=dict)

@task
def fetch_data_for_key(key: str) -> Data:
    """A task that simulates fetching data from some source"""
    return Data(key=key, value={"data": "some data"})

@task
def process_data(data: Data) -> Data:
    """A task that simulates processing data"""
    print(f"Processing {data!r} (I swear)")
    time.sleep(3)
    return data

@flow
def work_i_need_to_do_sometimes(keys: list[str]):
    """Work you might need to do ever so often"""
    
    raw_data_futures = fetch_data_for_key.map(keys)

    process_data.map(raw_data_futures)

if __name__ == "__main__":
    work_i_need_to_do_sometimes.serve(
        name="sporradic-deployment",
        parameters={"keys": ["foo", "bar", "baz"]},
    )

The original code doesn't include a

GcsBucket

. If you need to see how to work with

GcsBucket

specifically, I'd suggest searching for more examples related to

GcsBucket

in GCP integrations. The example provided can be found in this GitHub repository.

Thanks for the links. Will give it a go