Hello all, I am using prefect docker agent, for the flow run containers I need to expose some ports,...

Prasanth Kothuri

11/09/2021, 6:30 PM

Hello all, I am using prefect docker agent, for the flow run containers I need to expose some ports, how can I do this ? e.g

-p 12001:12001

Kevin Kho

11/09/2021, 6:50 PM

Hey @Prasanth Kothuri, looking into this

Kevin Kho

11/09/2021, 6:57 PM

I guess that would be the best bet. Since the Docker syntax is like:

Copy code

client.api.create_host_config(port_bindings={
    12001:12001
})

In Prefect it would be like:

Copy code

DockerRun(...,host_config={"port_bindings" = {12001:12001}})

Prasanth Kothuri

11/09/2021, 6:59 PM

👍

Prasanth Kothuri

11/10/2021, 4:36 PM

@Kevin Kho I am doing the following

Copy code

# flow run time configuration
f.run_config = DockerRun(
    labels=["dt4dev_osdataproc"],
    image="<http://gitlab-registry.internal.sanger.ac.uk/pam-dt4/spark-service/spark-service-docker/spark-app:v0.3|gitlab-registry.internal.sanger.ac.uk/pam-dt4/spark-service/spark-service-docker/spark-app:v0.3>",
    host_config={"port_bindings":{12001:12001}},
    env={"rpc_port": "12001","bm_port": "12101"}
)

when I execute the flow I don't see the port mapping on docker ps

Copy code

root@dt4-prefect:~/prefect# docker ps
CONTAINER ID   IMAGE                                                                                             COMMAND                  CREATED          STATUS                 PORTS                            NAMES
079f58a79463   <http://gitlab-registry.internal.sanger.ac.uk/pam-dt4/spark-service/spark-service-docker/spark-app:v0.3|gitlab-registry.internal.sanger.ac.uk/pam-dt4/spark-service/spark-service-docker/spark-app:v0.3>   "/usr/bin/entrypoint…"   10 seconds ago   Up 9 seconds                                            pearl-tiger

Kevin Kho

11/10/2021, 4:51 PM

Ok I’m not confident this is exposed now. The example here says it can be done without supplying the ports though. (check under port bindings). But in Prefect, the

ports

kwarg is not exposed. If what you tried doesn’t work, I think we need to open an issue

Copy code

client.api.create_container(
    'busybox', 'ls', ports=[1111, 2222],
    host_config=client.api.create_host_config(port_bindings={
        1111: 4567,
        2222: None
    })
)

Kevin Kho

11/10/2021, 5:02 PM

I’ll open an issue for this

Prasanth Kothuri

11/10/2021, 5:27 PM

ok, thank you, let me see if I use host networking in the meantime like this

host_config={"network_mode":"host"},

this should make all the ports reachable ?

Kevin Kho

11/10/2021, 5:29 PM

I guess that would yeah

Kevin Kho

11/10/2021, 6:02 PM

https://github.com/PrefectHQ/prefect/issues/5130

👍 1

Kevin Kho

11/22/2021, 5:07 PM

https://github.com/PrefectHQ/prefect/pull/5159

3 Views

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.