m

    matt_innerspace.io

    2 years ago
    One simple thing i'm not grasping here - how do i deploy a flow that exists in a
    hello_world.py
    script? Lots of examples of what can happen inside the python script, but i'm missing how it would be deployed as a flow. I can see the flow runs inside a docker container, on an agent. I can see how the
    prefect
    cli tool can run a flow, but it seems the deployment and running are linked somehow? For comparison, i use openfaas python functions which also run in docker containers and are deployed wherever has room to run it, but there is an explicit
    faas-cli push ...
    to deploy to the cloud/cluster when you're ready.
    Zachary Hughes

    Zachary Hughes

    2 years ago
    Hi Matt, Using an example
    hello_world.py
    file containing a hello world flow as an example:
    from prefect import task, Flow
    
    @task
    def hello_world():
        print("Hello world!")
    
    
    with Flow("Hello") as flow:
        hello = hello_world()
    You can then import, run, and/or register it the same way you would as if it were in the same script:
    from hello_world import flow
    flow.run()
    flow.register(project_name=YOUR_PROJECT_NAME)
    Does that help at all?
    m

    matt_innerspace.io

    2 years ago
    ok, so the deployment happens within a python script... and that python code is pickled and stored in the cloud, then distributed to agents somehow.
    Zachary Hughes

    Zachary Hughes

    2 years ago
    Yep! Once your flow is registered with Cloud, it'll be available for you to create runs. Once a run is created, your agent will pick it up, presuming the labels on the agent and the flow match. If you'd like to test this out all in one step, you can run:
    flow.run_agent()
    As described here: https://docs.prefect.io/cloud/tutorial/first.html#run-flow-with-prefect-cloud
    Jeremiah

    Jeremiah

    2 years ago
    With one caveat - the code itself is never pickled or sent to Prefect. We don’t (and can’t) receive code; it always remains on your infrastructure. What’s sent to us is metadata about the flow so that Cloud knows enough to be able to orchestrate the various tasks; but you’ll need to make sure your agent has access to the code (which is made easy by using one of Prefect Core’s
    Environments
    - local, docker, etc.)
    Zachary Hughes

    Zachary Hughes

    2 years ago
    ^^ Excellent point.
    m

    matt_innerspace.io

    2 years ago
    Environments
    and
    Storage
    is where I'm getting tripped up - storage (local, s3, docker, etc) is where the code is deployed, which wasn't clear to me. I reworked the example above in a gist to deploy to my own docker repo, which still seems to fail... I'm missing something simple here - https://gist.github.com/qmnonic/921f5847af4bd7e5808a0eca61aeb6ed
    Chris White

    Chris White

    2 years ago
    Hi Matt, while this error message is very uninformative, I believe it’s occurring whenever your docker daemon attempts to push your image. If you are trying to push to docker hub, you might need to update your docker configuration to:
    Docker(registry_url="innerspace", image_name="a-repo-within-your-dockerhub")
    every container registry type interacts with docker’s naming conventions differently
    we’ll take a look at improving the quality of that message
    m

    matt_innerspace.io

    2 years ago
    ok, that worked.. thanks!
    Chris White

    Chris White

    2 years ago
    @Marvin archive “Building Docker Storage for DockerHub fails with error”
    Marvin

    Marvin

    2 years ago