Thread
#prefect-community
    Miguel Angel

    Miguel Angel

    1 year ago
    Have anyone worked with a local environment for testing in a, lets say, docker-compose fashion? Suppose that you have an aws ECS cluster to execute your flows and Prefect Cloud as backend.
    Kyle McChesney

    Kyle McChesney

    1 year ago
    I have been successful with locally running flows either in a headless fashion (directly) or registering my local machine as a local agent on the backend (not prefect cloud). What are you trying to test?
    Kevin Kho

    Kevin Kho

    1 year ago
    Hey @Miguel Angel, I think this can be done with
    DockerRun
    and the Docker agent if you want to spin a Flow using a container.
    Miguel Angel

    Miguel Angel

    1 year ago
    @Kyle McChesney I've testing my flows exactly like you said, but I want it something more running context agnostic e.g. switch between local an ECS context. I'm interested in testing locally also my Dask cluster, and custom agent. I'm not sure if that would be possible though.
    @Kevin Kho I've been trying, without success, to run my flow.Module inside a container for local testing. What I'm copying inside the container to point for local server is the following file.
    # debug mode
    debug = true
    
    # base configuration directory (typically you won't change this!)
    home_dir = "~/.prefect"
    
    backend = "server"
    
    [server]
    host = "<http://172.17.0.1>"
    port = "4200"
    host_port = "4200"
    endpoint = "${server.host}:${server.port}"
    
    [registry]
    endpoint = "172.17.0.1:5000"
    But the connection is refused.
    Kevin Kho

    Kevin Kho

    1 year ago
    I am a bit confused here. Do you have a Docker in container with everything inside that runs a Prefect container for the Flow? Because this leads to issues. Or are you putting everything inside a container, and then doing
    flow.reigster()
    and you get the error?
    Miguel Angel

    Miguel Angel

    1 year ago
    I followed basically all the steps in this repo which basically lead me to 3 different docker-compose files, Each of these files were made to setup: first, Prefect local server, them build a basic agent, and finally put the flow code inside a docker file. With all of that I'd be good, but couldn't re-create the last step to expose my Module inside the container
    Kevin Kho

    Kevin Kho

    1 year ago
    When you say you couldn’t recreate the last step for expose the Module, is that the same as the connection being refused?
    Miguel Angel

    Miguel Angel

    1 year ago
    Yes, same error, when I try somethin like this as entrypoint:
    prefect register -m "my-package.flows" --project my-project
    Kyle McChesney

    Kyle McChesney

    1 year ago
    Not 100% following, but
    my-package.flows
    is not a valid python module as far as I know. Does it allow the dash?
    Kevin Kho

    Kevin Kho

    1 year ago
    That is a good thought @Kyle McChesney, but I think the the error is using a Docker container to hit the API right? I think running this out of the container works right?
    Kyle McChesney

    Kyle McChesney

    1 year ago
    I haven’t tried running “everything” locally and in a container. I just made a prefect venv for flow registration and manual flow running. I was able to use the included docker-compose file to run the server locally in docker. You should be able to do something like:
    env PREFECT__SERVER__HOST=localhost:##### prefect register --project "test" -m "myproject.flows"
    That would get your flow module registered onto the server instance running on your local docker. Just replace the #### with whatever port you are exposing (pretty sure its the graphql endpoint you want)
    I would try this in a virtualenv “locally” to your dev box first, to make sure its all working
    if you really want to do everything in docker, I suppose you’d want to start up a new container with an interactive shell, and then make sure its on the same network as the compose app. Assuming you mount your code, I guess you could run
    prefect register
    and
    prefect run
    from that container
    Maybe add this to your compose file
    dev:
        image: prefecthq/prefect:0.15.3
        tty: true
        networks:
    prefect-server
    And then exec into this container, you should at least have the network
    Miguel Angel

    Miguel Angel

    1 year ago
    I'll try that approach. Thanks for the tips