I am getting

[3 January 2022 8:26pm]: [Errno 24] Too many open files

error , what is the recommended value for open files ?

it looks like you don’t close files after opening them. Do you have some unclosed files in your flows? this post explains the problem

we read and write from s3 in prefect task, nothing apart from that, the default were too low (1024) and I increased them for the time being to resolve the issue