any ideas for some logging, etc. I could maybe turn on to get a better detail would be helpful! Otherwise, I'll keep digging and let you know if i figure it out

I honestly don’t have a good idea right now about what logs to add here --> could you simply try increasing the memory on the container? Alternatively, do your flows benefit from parallelism / distributed compute, or could you run them using the

LocalExecutor

?

the mystery is solved

💯 🎉 💯 was it an aptible setting?

so, the issue is as I expected, nothing to do with prefect, but aptible's deploy environment. TLDR: • containers are only accessible through tcp load-balancers • I tried to get clever and open a bunch of ports on the shared docker image, and on the ELB, and set the entrypoint for the containers to be subtly different per-container that we scaled with each worker listening on a distinct port • i expected/hoped that the requests through the elb would either be routed directly to the right container (not a reasonable assumption), or they'd multicast to all the containers and whichever one was listening on that port would act and the others would ignore it • Actually, aptible will hear a request on the elb at port e.g. 1234, and randomly pick from the containers behind the elb and pass through on that same port

ahhhhh nice work!!

aptible is architected under the assumption that all containers will run identical processes and be a bit "dumber" than dask workers are expected to be

yea absolutely thanks for sharing the solution and digging deep on this!