https://prefect.io
Join Slack
Have anyone worked with a local environment for testing in a, lets say, docker-compose fashion? Supp...
# ask-community
m
Have anyone worked with a local environment for testing in a, lets say, docker-compose fashion? Suppose that you have an aws ECS cluster to execute your flows and Prefect Cloud as backend.
k
I have been successful with locally running flows either in a headless fashion (directly) or registering my local machine as a local agent on the backend (not prefect cloud). What are you trying to test?
k
Hey @Miguel Angel, I think this can be done with 
DockerRun
and the Docker agent if you want to spin a Flow using a container.
m
@Kyle McChesney I've testing my flows exactly like you said, but I want it something more running context agnostic e.g. switch between local an ECS context. I'm interested in testing locally also my Dask cluster, and custom agent. I'm not sure if that would be possible though.
@Kevin Kho I've been trying, without success, to run my flow.Module inside a container for local testing. What I'm copying inside the container to point for local server is the following file.
Copy code
# debug mode
debug = true

# base configuration directory (typically you won't change this!)
home_dir = "~/.prefect"

backend = "server"

[server]
host = "<http://172.17.0.1>"
port = "4200"
host_port = "4200"
endpoint = "${server.host}:${server.port}"

[registry]
endpoint = "172.17.0.1:5000"
But the connection is refused.
k
I am a bit confused here. Do you have a Docker in container with everything inside that runs a Prefect container for the Flow? Because this leads to issues. Or are you putting everything inside a container, and then doing 
flow.reigster()
and you get the error?
m
I followed basically all the steps in this repo which basically lead me to 3 different docker-compose files, Each of these files were made to setup: first, Prefect local server, them build a basic agent, and finally put the flow code inside a docker file. With all of that I'd be good, but couldn't re-create the last step to expose my Module inside the container
k
When you say you couldn’t recreate the last step for expose the Module, is that the same as the connection being refused?
m
Yes, same error, when I try somethin like this as entrypoint:
Copy code
prefect register -m "my-package.flows" --project my-project
k
Not 100% following, but 
my-package.flows
is not a valid python module as far as I know. Does it allow the dash?
k
That is a good thought @Kyle McChesney, but I think the the error is using a Docker container to hit the API right? I think running this out of the container works right?
k
I haven’t tried running “everything” locally and in a container. I just made a prefect venv for flow registration and manual flow running. I was able to use the included docker-compose file to run the server locally in docker. You should be able to do something like: 
env PREFECT__SERVER__HOST=localhost:##### prefect register --project "test" -m "myproject.flows"
That would get your flow module registered onto the server instance running on your local docker. Just replace the #### with whatever port you are exposing (pretty sure its the graphql endpoint you want)
👍 2
I would try this in a virtualenv “locally” to your dev box first, to make sure its all working
if you really want to do everything in docker, I suppose you’d want to start up a new container with an interactive shell, and then make sure its on the same network as the compose app. Assuming you mount your code, I guess you could run 
prefect register
and 
prefect run
from that container
Maybe add this to your compose file
Copy code
dev:
    image: prefecthq/prefect:0.15.3
    tty: true
    networks:
prefect-server
And then exec into this container, you should at least have the network
m
I'll try that approach. Thanks for the tips
9 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by