Hi everyone, Can someone explain to me why workers are preferred over agents but do not support sto...

Fabrice

11/08/2023, 2:15 PM

Hi everyone, Can someone explain to me why workers are preferred over agents but do not support storage blocks yet? We were migrating from agents to workers and got the following error:

Copy code

ValueError: Flow run UUID('1731bd60-1498-4283-be30-bd7b7a3bc6fd') was created from deployment 'DEPLOYMENT' which is configured with a storage block. Workers currently only support local storage. Please use an agent to execute this flow run.

We are on the latest version of Prefect

2.14.3

Thanks!

Taylor Curran

11/08/2023, 3:53 PM

You should be able to reference blocks in your prefect.yaml file: https://docs.prefect.io/2.14.3/guides/prefect-deploy/?h=templating#templating-options

Taylor Curran

11/08/2023, 3:53 PM

For more summary information — see this page: https://docs.prefect.io/2.14.3/guides/upgrade-guide-agents-to-workers/

Fabrice

12/06/2023, 1:24 PM

@Taylor Curran we do not use the prefect.yaml but we use the Prefect SDK. There we have the option to provide the storage block: https://docs.prefect.io/2.14.3/api-ref/prefect/deployments/deployments/#prefect.deployments.deployments.Deployment. Right now we moved back to agents but this is not the preferred way.

Fabrice

12/06/2023, 1:25 PM

example:

Fabrice

12/06/2023, 1:25 PM

Copy code

>>> deployment = Deployment.build_from_flow(
...     flow=my_flow,
...     name="s3-example",
...     version="2",
...     tags=["aws"],
...     storage=storage,
...     infra_overrides=dict("env.PREFECT_LOGGING_LEVEL"="DEBUG"),
>>> )

Fabrice

12/06/2023, 1:25 PM

this allows us to provide a storage block (storage=storage)

2 Views

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.