https://prefect.io logo
#prefect-community
Title
# prefect-community
s

Sushma Adari

06/10/2022, 2:36 PM
Hi all, we have named volumes to which the agents are mounted to in prefect. We are facing an issue whenever a Flow Fails/Task fails involving a docker mount that the full server error message is returned with the full server values including themount location, user, address, and password. Obviously this is a massive security issue. Is there anyway to circumvent this output? SCRIPT ALERT: Intraday Strategy Greeks entered state Failed with message 500 Server Error for `http+docker://localhost/v1.41/containers/416e8db53c290dc44f8b1621c027e1930f07606101c82ed0ed059a367bcd8c13/start`: Internal Server Error ("error while mounting volume '/var/lib/docker/volumes/output/_data': failed to mount local volume: mount //XXXXXX.XXX.XXX/XXX:/XXX/lib/docker/volumes/output/_data, data: addr=XX.XXX.X.XX, username=XXXX, password=XXXXXX: no such file or directory"). (edited)
k

Kevin Kho

06/10/2022, 2:43 PM
This is a docker-py log so we can’t control that, but I think what you can do to prevent sending this to us is attaching a Filter to your Python logger like this
s

Sushma Adari

06/10/2022, 3:16 PM
The error isn't coming from Docker-py it is coming direclty from the docker.exe -> https://github.com/docker/cli/blob/21c5391cee93f90ca17fb841b60a8aa72373723c/opts/mount_test.go
k

Kevin Kho

06/10/2022, 3:18 PM
Ah I see but I still think the best approach is the filter. Or does it fail before the Flow starts?
s

Sushma Adari

06/13/2022, 5:06 PM
Yes as far as I know it is failing even before the flow begins
When the flow is using that container it is mounting the volume but if the mount fails prefect is logging the error aabove
k

Kevin Kho

06/13/2022, 5:09 PM
This is a bit tricky. I think you need to use script based storage and then attach a Python logging filter before the Flow so this log gets hidden. Or you could disable Prefect logs too.
s

Sushma Adari

06/13/2022, 5:22 PM
I see that this allows writing files to docker storage but is there ability to read files from the docker storage as well?
k

Kevin Kho

06/13/2022, 5:45 PM
Yeah. Do
Copy code
build=False
when you register like this
s

Sushma Adari

06/13/2022, 5:49 PM
Can we pass multiple storage params to the flow? Since our script is currently stored in CodeCommit but we want the outputs /inputs coming from mounting the volumes
k

Kevin Kho

06/13/2022, 5:51 PM
Like if you want to use the Prefect interface to create an image, you can do:
Copy code
docker = Docker(...)
docker.build()
and I think that will make the image. And then you just continue to use the CodeCommit storage on top of DockerRun. That’s what you mean right? But if you have the mounts also, the Flow should be able to just read them already
s

Sushma Adari

06/13/2022, 5:53 PM
We already have the images created, we want to avoid the error message that is logged in prefect when starting a flow and the docker mount fails\
k

Kevin Kho

06/13/2022, 5:58 PM
So to follow the filter suggestion on CodeCommit, you would attach the Python filter at the start of that Flow and hopefully it blocks that error log.
s

Sushma Adari

06/13/2022, 5:59 PM
The error is not coming from Python it is a direct message from docker.exe. The python logger won't catch it
At least from what we know, we haven't managed to suppress it from python
k

Kevin Kho

06/13/2022, 6:11 PM
I agree, but if it appears in the Prefect logs, it must have gone through a Python logger so I think
docker-py
surfaced it.
1
s

Sushma Adari

06/13/2022, 6:15 PM
Thanks for your patience through this. We will try a few more things.
So the error is coming from the command line when the prefect agent is spinning up. On occasion, when the agent goes to run a flow the docker mount is failing from the prefect agent even before it starts the flow from python. Python is not starting before it fails and returns the logged message. The agent pulls the docker image that needs to be used and that is where it fails. So in the prefect logs it shows that the agent itself is returning the error not the Flow.
k

Kevin Kho

06/13/2022, 6:36 PM
Ohh ok that is a lot harder unfortunately. You may need to start the agent programatically instead of the CLI, and then before starting it, at a filter to the root logger? But then you don’t know why this fails either I think
5 Views