Dockerfile-workers: reduce the amount we install (#12464)

This is an attempt to reduce the rebuild time. In short, we reduce the amount
of stuff that the dockerfile installs, so as to give a faster startup.