Join Slack

I can see the Pull steps in the Web UI. But they are missing completely in the python API

# prefect-kubernetes

brokoli

11/20/2023, 5:05 PM

I can see the Pull steps in the Web UI. But they are missing completely in the python API

Kevin Grismore

11/20/2023, 5:44 PM

You can use

from_source

to either select a block defining the location from which you want to pull code:

Copy code

from prefect import flow
from prefect_aws.s3 import S3Bucket

if __name__ == "__main__":
    flow.from_source(
        source=S3Bucket.load("my-code-storage-bucket"), entrypoint="flows.py:my_flow"
    ).deploy(name="my-deployment", work_pool_name="above-ground")

or use the

fsspec

compatible

RemoteStorage

class:

Copy code

from prefect import flow
from prefect.runner.storage import RemoteStorage

if __name__ == "__main__":
    flow.from_source(
        source=RemoteStorage(url="<az://my-container/my-folder>", account_name="my-account-name"),
        entrypoint="flows.py:my_flow",
    ).serve(name="deployment-from-remote-flow")

Nate

11/20/2023, 7:27 PM

@brokoli the pull steps live on the object you get back from the client when you read deployments

brokoli

11/21/2023, 10:00 AM

Tnx for the help Hmm, if I reference the S3Bucket in as storage I get

AttributeError: 'S3Bucket' object has no attribute 'set_base_path'

. If I do

fsspec

then for every flow I want to deploy, the contents of the (in my case) same bucket/path gets downloaded ... for 14 flows or so.

brokoli

11/21/2023, 10:02 AM

Which kind a defeats the purpose of the

deploy

function that is there to deploy multiple flows.

brokoli

11/21/2023, 10:05 AM

takes around 2 minutes

brokoli

11/21/2023, 10:37 AM

I tried to hack in the pull steps into the RunnerDeployment, but I can't figure how to do it

'RunnerDeployment' object has no attribute 'pull_steps'

Nate

11/21/2023, 2:33 PM

hey @brokoli - can you give an example of what you're trying to do? ideally with code 🙂

brokoli

11/21/2023, 3:06 PM

brokoli

11/21/2023, 3:08 PM

So I need to deploy and run

flows. In my CI pipeline I build the docker image (if requirements.txt change) and upload the flow code to S3. Then I want to register all the flows as a new deployment on prefect cloud

brokoli

11/21/2023, 3:08 PM

I have a worker in k8s that then create Jobs and the Jobs then pull the code from S3

brokoli

11/21/2023, 3:09 PM

One way to do it is to template the

prefect.yaml

to have all the flows and the other is to use

fsspec

brokoli

11/21/2023, 3:10 PM

I do that in CI/CD pipeline

brokoli

11/21/2023, 3:11 PM

I haven't found a way to do it from the CLI (I guess that only works for agents, not workers)

brokoli

11/21/2023, 3:15 PM

Does that usecase make sense or am I doing something fundamentally wrong?

Nate

11/21/2023, 5:01 PM

the use case makes a lot of sense!

Nate

11/21/2023, 5:02 PM

if you know how many deployments you need to make, using

prefect.yaml

makes a lot of sense, otherwise if you just have a monorepo situation and you just wanna deploy them all after you upload, you could run something like this in CI

deploy.py

Nate

11/21/2023, 5:04 PM

which you might want to modify a bit to search the remote dir / someplace else for flows, but the rough idea • find all the flows • create a flow object

from_source

• create a deployment from that • pass them all to

deploy

🙌 2

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.