Can someone explain to me why this is part of the worker class? Surely there is a way to run docker workers and store deployments remotely

async def _check_flow_run(self, flow_run: "FlowRun") -> None:
        """
        Performs a check on a submitted flow run to warn the user if the flow run
        was created from a deployment with a storage block.
        """
        if flow_run.deployment_id:
            deployment = await self._client.read_deployment(flow_run.deployment_id)
            if deployment.storage_document_id:
                raise ValueError(
                    f"Flow run {flow_run.id!r} was created from deployment"
                    f" {deployment.name!r} which is configured with a storage block."
                    " Workers currently only support local storage. Please use an"
                    " agent to execute this flow run."
                )

Hello and thanks for the question! We need to update that language. You can definitely use a worker and store your code remotely. Using prefect.yaml is one way to set that up. We also just released a new python based deployment creation method - we’d love feedback on how that meets your use case and your experience in getting things set up.

I will look into this and get back to you. For the time being is there a way to silence this error? I would like to be able to store my deployments on a remote file system and use workers and a work-pool to run them inside of docker containers, but it seems like no matter how i slice and dice it, I need a RemoteFileSystem block to deploy.

You can use flow.from_source

So to clarify, the issue is that i am calling

deployment.upload_to_storage()

on a deployment that has a storage block specified. Not that the flow code lives in remote storage

Ah ok. Can you say a bit more? Is there a reason you need the upload to storage? I think I’d need to check with the team about if that will work with newer style deployments

I was looking to spin up containerized env for each of the flows, marking the dependancies for each flow inside of a docker infra object. I think a better pattern for this is, as you mentioned above, using something like a prefect.yaml file and defining the build/push/and pull steps. I think this can all be achieved with the docker steps provided in prefect-docker. What is still unclear to me is where/how i can programatically load this .yaml file (to deploy the flow programatically)

The new flow.deploy I shared is aimed at helping with that. There’s a little more about how you can use and customize that in the docs: https://docs.prefect.io/latest/tutorial/workers/ Let us know if there’s something that you need that’s not set out there.

It does not look like the same flexibility is offered through the deploy function that would otherwise be offered in the prefect.yaml or through using blocks and

Deployment.build_from_flow

,

.upload_to_storage

, and

.apply

For instance how could you define a pull step (or multiple pull steps) I would really love to see something that is along the lines of Deployment.from_yaml() similar to

<http://Deployment.to|Deployment.to>_yaml()

By pull step(s) in your yaml you mean where the flow code is pulled from? That’s what from_source lets you set out. I think there’s less flexibility in that it’s docker based. But that sounds like your use case? (For simpler local deployments, there’s flow.serve)