https://prefect.io logo
Join Slack
Powered by
Hi, I am trying to register a simple flow in a ser...
# ask-community
c
Hi, I am trying to register a simple flow in a server that I run in a docker locally. I first built a docker image on top of 
prefecthq/prefect:latest
image, which also copied my python script 
/src/test.py
and ran 
prefect backend server
. Then in the 
docker-compose.yaml
generated by 
prefect server config
, I added a service 
client
whose command is 
python /src/test/py
. After running 
docker-compose up
, I got the following error from the 
client
service:
Copy code
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=4200): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f24823ea0d0>: Failed to establish a new connection: [Errno 111] Connection refused'))
Meanwhile, I can access UI and on UI it’s connected to Apollo endpoint successfully. It seems like 
client
fails to connect to server at localhost:4200?
Copy code
import prefect
from prefect import task, Flow

@task
def hello_task():
    logger = prefect.context.get("logger")
    <http://logger.info|logger.info>("Hello world!")

with Flow("hello-flow") as flow:
    hello_task()

flow.register(project_name="tutorial")
# flow.run()
my python script 
/src/test.py
, where the error is thrown at 
flow.register()
line
Copy code
FROM prefecthq/prefect:latest
COPY src /src
RUN prefect backend server
my Dockerfile
Copy code
client:
    command: python /src/test.py
    depends_on:
      - ui
    build: .
    networks:
      prefect-server: null
The extra service i added in 
docker-compose.yaml
Copy code
client_1    | Traceback (most recent call last):
client_1    |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 175, in _new_conn
client_1    |     (self._dns_host, self.port), self.timeout, **extra_kw
client_1    |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 96, in create_connection
client_1    |     raise err
client_1    |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 86, in create_connection
client_1    |     sock.connect(sa)
client_1    | ConnectionRefusedError: [Errno 111] Connection refused
client_1    | 
client_1    | During handling of the above exception, another exception occurred:
client_1    | 
client_1    | Traceback (most recent call last):
client_1    |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 706, in urlopen
client_1    |     chunked=chunked,
client_1    |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 394, in _make_request
client_1    |     conn.request(method, url, **httplib_request_kw)
client_1    |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 239, in request
client_1    |     super(HTTPConnection, self).request(method, url, body=body, headers=headers)
client_1    |   File "/usr/local/lib/python3.7/http/client.py", line 1281, in request
client_1    |     self._send_request(method, url, body, headers, encode_chunked)
client_1    |   File "/usr/local/lib/python3.7/http/client.py", line 1327, in _send_request
client_1    |     self.endheaders(body, encode_chunked=encode_chunked)
client_1    |   File "/usr/local/lib/python3.7/http/client.py", line 1276, in endheaders
client_1    |     self._send_output(message_body, encode_chunked=encode_chunked)
client_1    |   File "/usr/local/lib/python3.7/http/client.py", line 1036, in _send_output
client_1    |     self.send(msg)
client_1    |   File "/usr/local/lib/python3.7/http/client.py", line 976, in send
client_1    |     self.connect()
client_1    |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 205, in connect
client_1    |     conn = self._new_conn()
client_1    |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 187, in _new_conn
client_1    |     self, "Failed to establish a new connection: %s" % e
client_1    | urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f24823ea0d0>: Failed to establish a new connection: [Errno 111] Connection refused
client_1    | 
client_1    | During handling of the above exception, another exception occurred:
client_1    | 
client_1    | Traceback (most recent call last):
client_1    |   File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
client_1    |     timeout=timeout
client_1    |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 756, in urlopen
client_1    |     method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
client_1    |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 574, in increment
client_1    |     raise MaxRetryError(_pool, url, error or ResponseError(cause))
client_1    | urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=4200): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f24823ea0d0>: Failed to establish a new connection: [Errno 111] Connection refused'))
client_1    | 
client_1    | During handling of the above exception, another exception occurred:
client_1    | 
client_1    | Traceback (most recent call last):
client_1    |   File "/src/test.py", line 12, in <module>
client_1    |     flow.register(project_name="tutorial")
client_1    |   File "/usr/local/lib/python3.7/site-packages/prefect/core/flow.py", line 1734, in register
client_1    |     idempotency_key=idempotency_key,
client_1    |   File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 1122, in register
client_1    |     project = self.graphql(query_project).data.project  # type: ignore
client_1    |   File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 554, in graphql
client_1    |     retry_on_api_error=retry_on_api_error,
client_1    |   File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 458, in post
client_1    |     retry_on_api_error=retry_on_api_error,
client_1    |   File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 738, in _request
client_1    |     session=session, method=method, url=url, params=params, headers=headers
client_1    |   File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 606, in _send_request
client_1    |     timeout=prefect.context.config.cloud.request_timeout,
client_1    |   File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 590, in post
client_1    |     return self.request('POST', url, data=data, json=json, **kwargs)
client_1    |   File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 542, in request
client_1    |     resp = self.send(prep, **send_kwargs)
client_1    |   File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 655, in send
client_1    |     r = adapter.send(request, **kwargs)
client_1    |   File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 516, in send
client_1    |     raise ConnectionError(e, request=request)
client_1    | requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=4200): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f24823ea0d0>: Failed to establish a new connection: [Errno 111] Connection refused'))
the complete stacktrace
k
Hey @Christopher Chong Tau Teng, I think I’m pretty confused here. A couple of questions 1. Does registering the flow outside of Docker on the same machine work? Can it communicate with localhost:4200? 2. You shouldn;t have to make the service. Using the Prefect Docker Agent will spin up your flow in a container. 3. The Docker Storage will provide a much easier interface so you don’t have to create your own Dockerfile for this case.
c
Hi @Kevin Kho, thanks for your reply. I am quite new to Docker so pardon me haha. 1. By outside of Docker, you mean my 
client
is built from a different image (
prefecthq/prefect:latest
) compared to 
towel
and 
graphql
(
prefecthq/server:core-0.15.7
), hence 
client
is outside* ? 2. Sorry I am confused on how Docker Server and Docker Agent work together. How do I use Docker Agent to run my flow python script so that the flow is registered in the Server? 3. Same question here. After I write my flow script, I understand I can use Docker Storage to store the flow in an image and push to some container registry. How do I then run this image in my Server?
k
No worries at all. 1. Don’t think that there is a 
client
. Using 
flow.register()
and then running the Python script with 
python myflow.py
to register will send an API request at your configured location (server) and then register the flow there. This doesn’t need to run in any container. 2. Server and Cloud are what we call backends. The Agent is a lightweight process that polls the backend for every 10 seconds. It looks for flows to run and if it finds any, it kicks them off. If you use the Docker Agent (which doesnt have to run in a container), it looks to start those flows inside a configured container. 3. The Storage creates and uploads the image. The RunConfiguration (Docs ) is what determines the run settings. You want the DockerRun specifically. This is where you specify the image, and this is registered along with you flow. And then the agent finds your flow, and starts your flow run on the image. For a simple hello world flow, you can store your flow on Github and then register with Github Storage + DockerRun. The agent will pull the flow from Github and run in a container. This will look like
Copy code
import prefect
from prefect import task, Flow

@task
def hello_task():
    logger = prefect.context.get("logger")
    <http://logger.info|logger.info>("Hello world!")

with Flow("hello-flow") as flow:
    hello_task()

flow.storage = GitHub(
        repo="org/repo",
        path="flows/my_flow.py",
    )
flow.run_config = DockerRun(image="prefecthq/prefect")

flow.register(project_name="tutorial")
# flow.run()
Here I used Github storage without authentication. This assumes the repo will be public.
I have an example of Github Storage and DockerRun here . You can copy this python code and register it too. Executing it will fetch it from this repo.
c
Hi @Kevin Kho thanks for your detailed explanation. I understand how different part stitches together now. I have one question: I am planning to use GCS as Storage and DockerRun as run_config, which will pull a custom image that I build and store in GCR. All these are defined in my 
flow.py
script which lives on my GCP VM that runs my Prefect Server. Normally, I just need to do 
python flow.py
to start the execution. However, as part of my company’s CICD we deploy application to VM as containers. This means that I will need to build an image that has Python, Prefect dependencies and run a container that executes 
python flow.py
and subsequently 
prefect agent docker start
. In this way, Docker Agent is creating a container within a container, and I am having the same error as posted above.. Also, do you have any alternative idea as to how we can deploy this application?
a
As Kevin mentioned, once you register your flow to the backend (in your case Prefect Server), it is then already deployed. To get your use case working, you need to: 1) Authenticate with your GCR Docker registry using your GCR credentials, as described here: 
docker login -u oauth2accesstoken -p "$(gcloud auth print-access-token)" <https://HOSTNAME>
2) Start the docker agent with your labels e.g. prefect agent docker start --label YOUR-LABEL 3) Add GCS storage and DockerRun run configuration to your flow e.g.
Copy code
from prefect import task, Flow
from prefect.storage.gcs import GCS
from prefect.run_configs import DockerRun


@task(log_stdout=True)
def hello_world():
    print("hello world")


with Flow(
    "gcs-flow",
    storage=GCS(bucket="your bucket"),
    run_config=DockerRun(labels=["your-label"], image="your.gcr.image"),
) as flow:
    hw = hello_world()

if __name__ == "__main__":
    flow.register(project_name="community")
c
hi @Anna Geller my issue right now is at the 
flow.register
step, it fails to reach the server endpoint.
Copy code
requests.exceptions.ConnectionError: HTTPConnectionPool(host='0.0.0.0', port=4200): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7efe1b0e6380>: Failed to establish a new connection: [Errno 111] Connection refused'))
This happens when I run a container that runs the following python script. It doesn’t happen when I run this script directly on my machine without using container.
Copy code
import prefect
from prefect import task, Flow
from prefect.storage import GCS
from prefect.run_configs import DockerRun

@task
def hello_task():
    logger = prefect.context.get("logger")
    <http://logger.info|logger.info>("Hello world!")

with Flow("hello-flow") as flow:
    hello_task()

flow.storage = GCS(bucket="XXX")
flow.run_config = DockerRun()
flow.register(project_name="tutorial1")
a
You really shouldn’t have to run 
flow.register()
in a container. You just have to do it once to deploy your flow (and later you may need to re-register if your flow structure changes). Running 
flow.register()
from a local machine is perfectly fine.
c
@Anna Geller I see, but our company has a strict requirement of only using container to deploy to VM. Is there any way I can make 
flow.register
work in a container?
I have found this similar to this issue, however the solution there doesn’t solve my problem: https://github.com/PrefectHQ/prefect/issues/2322
a
You need to make sure that your Server endpoint can be reached from this container from which you register the flows. Did you define the Server endpoint (
~/.prefect/config.toml
) in both, your Server and the container from which you register? I think you need to configure it in both so that your “registration container” knows which Server endpoint to use to register flows. Example: 1) on your Server:
Copy code
[server]
  [server.ui]
    apollo_url = "<http://YOUR_MACHINES_PUBLIC_IP:4200/graphql>"
2) on your registration container, or agents:
Copy code
[server]
endpoint = "YOUR_MACHINES_PUBLIC_IP:4200/graphql"
c
@Anna Geller i am currently testing on my local machine, so I just defined PREFECT__SERVER__ENDPOINT: http://localhost:4200/graphql in my registration container, but still, it fails to connect..
@Kevin Kho any idea on this?
k
What happens when you go to the UI and query
Copy code
query{
    hello
}
c
i can access UI perfectly fine
k
Is this still your current error message?
Copy code
requests.exceptions.ConnectionError: HTTPConnectionPool(host='0.0.0.0', port=4200): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7efe1b0e6380>: Failed to establish a new connection: [Errno 111] Connection refused'))
I'm wondering why the host is 0.0.0.0, I think it should say localhost is that environment variable is set correctly. What happens if you try 
prefect agent local start
also? I imagine you get the same error message?
Also I think the endpoint value should not have http. Should just be IP:4200/graphql
c
If I set 
localhost:4200/graphql
Copy code
requests.exceptions.InvalidSchema: No connection adapters were found for 'localhost:4200/graphql'
If I set http://localhost:4200/graphql
Copy code
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=4200): Max retries exceeded with url: /graphql (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f0c477a6410>: Failed to establish a new connection: [Errno 111] Connection refused'))
If I do 
prefect agent local start
Copy code
equests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=4200): Max retries exceeded with url: /graphql (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f6cf099e620>: Failed to establish a new connection: [Errno 111] Connection refused'))
@Kevin Kho @Anna Geller I think I got it working omg. I was running this registration container in the same docker-compose file as the Prefect server one — i just moved it to a different docker-compose file and ran it and i think it’s connected to the backend server!
a
Great! thanks for the update, @Christopher Chong Tau Teng!
638 Views
Open in Slack
PreviousNext
Powered by