Hi all I want to use dependency between docker images For ex

Question

Hi all I want to use dependency between docker images For example I have 1 container `myapp` and another container which is actually the prefect `agent` that will be running the flows I figured out the `agent` needs to have the code that is there in `myapp` So I set up the flow like below ```flow storage = Docker dockerfile= path of Dockerfile inside myapp this will ensure that Dockerfile I used to build myapp will be used to build agent flow storage build this step fails with the eroor FileNotFound I looked at the prefect code in the github repo and this should work but fails flow register labels= some label ``` Is there a gap in my understanding of how the Docker storage works

josh · Answer

Hey < Vikram Iyer> can you verify that the place you are calling this build from has access to the Dockerfile in the same path that you pass to `dockerfile= ` Also as a side note you might not need to build the storage directly because `flow register` has a `build` kwarg which defaults to `True`

Vikram Iyer · Answer

The place I am calling from is ` app somedir code py` and The place where the Dockerfile is is ` app Dockerfile` Ideally the parent dir so should be accessible What do you think

josh · Answer

Could you post the full traceback that you get when you run ` build`

Vikram Iyer · Answer

```myapp | usr local lib python3 7 site packages prefect client client py 619 UserWarning No result handler was specified on your Flow Cloud features such as input caching and resuming task runs from failure may not work properly myapp | No result handler was specified on your Flow Cloud features such as myapp | usr local lib python3 7 site packages prefect core flow py 1371 UserWarning A flow with the same name is already contained in storage if you changed your Flow since the last build you might experience unexpected issues and should re create your storage object myapp | A flow with the same name is already contained in storage if you myapp | 2020 07 02 00 43 18 +0530 50 ERROR Error handling request api v1 metric add myapp | Traceback most recent call last myapp | File usr local lib python3 7 site packages urllib3 connectionpool py line 677 in urlopen myapp | chunked=chunked myapp | File usr local lib python3 7 site packages urllib3 connectionpool py line 392 in make request myapp | conn request method url httplib request kw myapp | File usr local lib python3 7 http client py line 1252 in request myapp | self send request method url body headers encode chunked myapp | File usr local lib python3 7 http client py line 1298 in send request myapp | self endheaders body encode chunked=encode chunked myapp | File usr local lib python3 7 http client py line 1247 in endheaders myapp | self send output message body encode chunked=encode chunked myapp | File usr local lib python3 7 http client py line 1026 in send output myapp | self send msg myapp | File usr local lib python3 7 http client py line 966 in send myapp | self connect myapp | File usr local lib python3 7 site packages docker transport unixconn py line 43 in connect myapp | sock connect self unix socket myapp | FileNotFoundError Errno 2 No such file or directory myapp | myapp | During handling of the above exception another exception occurred myapp | myapp | Traceback most recent call last myapp | File usr local lib python3 7 site packages requests adapters py line 449 in send myapp | timeout=timeout myapp | File usr local lib python3 7 site packages urllib3 connectionpool py line 725 in urlopen myapp | method url error=e pool=self stacktrace=sys exc info 2 myapp | File usr local lib python3 7 site packages urllib3 util retry py line 403 in increment myapp | raise six reraise type error error stacktrace myapp | File usr local lib python3 7 site packages urllib3 packages six py line 734 in reraise myapp | raise value with traceback tb myapp | File usr local lib python3 7 site packages urllib3 connectionpool py line 677 in urlopen myapp | chunked=chunked myapp | File usr local lib python3 7 site packages urllib3 connectionpool py line 392 in make request myapp | conn request method url httplib request kw myapp | File usr local lib python3 7 http client py line 1252 in request myapp | self send request method url body headers encode chunked myapp | File usr local lib python3 7 http client py line 1298 in send request myapp | self endheaders body encode chunked=encode chunked myapp | File usr local lib python3 7 http client py line 1247 in endheaders myapp | self send output message body encode chunked=encode chunked myapp | File usr local lib python3 7 http client py line 1026 in send output myapp | self send msg myapp | File usr local lib python3 7 http client py line 966 in send myapp | self connect myapp | File usr local lib python3 7 site packages docker transport unixconn py line 43 in connect myapp | sock connect self unix socket myapp | urllib3 exceptions ProtocolError Connection aborted FileNotFoundError 2 No such file or directory myapp | myapp | During handling of the above exception another exception occurred myapp | myapp | Traceback most recent call last myapp | File usr local lib python3 7 site packages docker api client py line 207 in retrieve server version myapp | return self version api version=False ApiVersion myapp | File usr local lib python3 7 site packages docker api daemon py line 181 in version myapp | return self result self get url json=True myapp | File usr local lib python3 7 site packages docker utils decorators py line 46 in inner myapp | return f self args kwargs myapp | File usr local lib python3 7 site packages docker api client py line 230 in get myapp | return self get url self set request timeout kwargs myapp | File usr local lib python3 7 site packages requests sessions py line 543 in get myapp | return self request GET url kwargs myapp | File usr local lib python3 7 site packages requests sessions py line 530 in request myapp | resp = self send prep send kwargs myapp | File usr local lib python3 7 site packages requests sessions py line 643 in send myapp | r = adapter send request kwargs myapp | File usr local lib python3 7 site packages requests adapters py line 498 in send myapp | raise ConnectionError err request=request myapp | requests exceptions ConnectionError Connection aborted FileNotFoundError 2 No such file or directory myapp | myapp | During handling of the above exception another exception occurred myapp | myapp | Traceback most recent call last myapp | File usr local lib python3 7 site packages gunicorn workers sync py line 134 in handle myapp | self handle request listener req client addr myapp | File usr local lib python3 7 site packages gunicorn workers sync py line 175 in handle request myapp | respiter = self wsgi environ resp start response myapp | File usr local lib python3 7 site packages flask app py line 2464 in call myapp | return self wsgi app environ start response myapp | File usr local lib python3 7 site packages flask app py line 2450 in wsgi app myapp | response = self handle exception e myapp | File usr local lib python3 7 site packages flask app py line 1867 in handle exception myapp | reraise exc type exc value tb myapp | File usr local lib python3 7 site packages flask compat py line 39 in reraise myapp | raise value myapp | File usr local lib python3 7 site packages flask app py line 2447 in wsgi app myapp | response = self full dispatch request myapp | File usr local lib python3 7 site packages flask app py line 1952 in full dispatch request myapp | rv = self handle user exception e myapp | File usr local lib python3 7 site packages flask app py line 1821 in handle user exception myapp | reraise exc type exc value tb myapp | File usr local lib python3 7 site packages flask compat py line 39 in reraise myapp | raise value myapp | File usr local lib python3 7 site packages flask app py line 1950 in full dispatch request myapp | rv = self dispatch request myapp | File usr local lib python3 7 site packages flask app py line 1936 in dispatch request myapp | return self view functions rule endpoint req view args myapp | File usr local lib python3 7 site packages flask classful py line 268 in proxy myapp | response = view request view args myapp | File usr local lib python3 7 site packages flask classful py line 239 in inner myapp | return fn args kwargs myapp | File app outlier detection api v1 metric py line 35 in add myapp | response = runtime flow data dict myapp | File app outlier detection flows runtime flow py line 68 in runtime flow myapp | flow register labels= l1 myapp | File usr local lib python3 7 site packages prefect core flow py line 1527 in register myapp | no url=no url myapp | File usr local lib python3 7 site packages prefect client client py line 662 in register myapp | serialized flow = flow serialize build=build type Any myapp | File usr local lib python3 7 site packages prefect core flow py line 1375 in serialize myapp | storage = self storage build type Optional Storage myapp | File usr local lib python3 7 site packages prefect environments storage docker py line 304 in build myapp | self build image push=push myapp | File usr local lib python3 7 site packages prefect environments storage docker py line 341 in build image myapp | client = self get client myapp | File usr local lib python3 7 site packages prefect environments storage docker py line 531 in get client myapp | base url=self base url version= auto tls=self tls config myapp | File usr local lib python3 7 site packages docker api client py line 190 in init myapp | self version = self retrieve server version myapp | File usr local lib python3 7 site packages docker api client py line 215 in retrieve server version myapp | Error while fetching server API version 0 format e myapp | docker errors DockerException Error while fetching server API version Connection aborted FileNotFoundError 2 No such file or directory myapp | 192 168 192 1 02 Jul 2020 00 43 18 +0530 POST api v1 metric add HTTP 1 1 500 0 in 10929µs```

josh · Answer

Oh you re attempting to register the flow using Docker inside of a Docker container and it cannot find a daemon to connect to

josh · Answer

`Docker` storage has a `base url` kwarg in which you can pass in the url of a Docker daemon in order to build the storage

josh · Answer

Optionally you could also mount your local daemon to your `myapp` container to achieve this <https medium com better programming about var run docker sock 3bfd276e12fd>

josh · Answer

Is there a reason why you are registering your flow inside of a container instead of on your local machine

Vikram Iyer · Answer

Oh Cool I will try using the base url Just curious is registering a flow from within a container using Docker not recommended

josh · Answer

It s not that it isn t recommended but it requires some extra setup slightly smiling face and if you already have Docker running on your machine it would make more sense to register there

Vikram Iyer · Answer

Short answer To be able to deploy anywhere Long answer So I have this huge docker compose that has all myapp + other services along with prefect ones ui scheduler apollo etc So this myapp service will accept the flow details and register them

Vikram Iyer · Answer

I hope this makes sense I will try drawing an HLD to explain if not

Vikram Iyer · Answer

Something like this

josh · Answer

Yeah I follow what you re attempting In this case where you re trying to build an image from within a container you should do one of the options from above mounting daemon providing `base url` etc Otherwise you could look into using a different storage option in conjunction with your docker based agent <https docs prefect io orchestration execution storage options html non docker storage for containerized environments>

Vikram Iyer · Answer

Sure The base url should be something like hostip docker daemon port right

josh · Answer

Yep

Vikram Iyer · Answer

Hey < josh> I thought I d let you know I mounted the same image for both myapp and the agent and things worked just fine Did not have to use Docker since I did not want a new container to be up rather wanted the agent container to take care of it Thanks for all the help slightly smiling face

Martin · Answer

< Vikram Iyer> I m trying to do something similar as you workflow but I have some doubts about it How did you register the image on the agent Every agent is registered for a single flow or could be reutilized for different ones I have a docker compose with all the stack for prefect serving the ui and apollo with traefik 2 and I would like to create differents flows but dettached for that docker compose stack

Vikram Iyer · Answer

Hey < Martin> I built the image for myapp and pointed the agent container to refer to the same image This way both myapp and agent containers have the same image and hence the code within If the current directory on your host has the code you can use `build ` in docker compose to say that both myapp and agent should have the current dir built into them As a final step make sure to mount the code directory inside both of them Does that answer your question

Martin · Answer

Yes for sure thanks So one agent per flow no

Martin · Answer

Do you have a docker compose example with that flow

Vikram Iyer · Answer

Not actually There is an agent running that can accept any flow Yep let me share a snapshot of the compose file

Vikram Iyer · Answer

You might have the original docker compose To that you can add below 2 services 1 is the app and the other is agent like so ```myapp build command command to run your application service networks prefect server environment PREFECT BACKEND server PREFECT SERVER ENDPOINT PREFECT SERVER GRAPHQL URL agent build command prefect agent start label anylabel networks prefect server environment PREFECT SERVER GRAPHQL URL PREFECT SERVER ENDPOINT restart always ```

Martin · Answer

ohh nice thanks