The requirements file generated by `poetry export` isn't correctly processed by `pip install -r requirements.txt`. It contains twisted and treq, both pinned to 22.2.0.

When `pip` installs treq, it notices that `Twisted[tls]` is required. It then tries to acquire the latest twisted release, only to fail (because this hash isn't listed in the requirements file).From e.g. https://github.com/matrix-org/synapse/runs/5977154990?check_suite_focus=true

> ```
> #15 9.204 Collecting Twisted[tls]>=18.7.0
> #15 9.205 ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
> #15 9.205     Twisted[tls]>=18.7.0 from https://files.pythonhosted.org/packages/db/99/38622ff95bb740bcc991f548eb46295bba62fcb6e907db1987c4d92edd09/Twisted-22.4.0-py3-none-any.whl#sha256=f9f7a91f94932477a9fc3b169d57f54f96c6e74a23d78d9ce54039a7f48928a2 (from treq==22.2.0->-r /synapse/requirements.txt (line 724))
> #15 ERROR: executor failed running [/bin/sh -c pip install --prefix="/install" --no-warn-script-location -r /synapse/requirements.txt]: exit code: 1
> ```

The underlying pip issue is https://github.com/pypa/pip/issues/9644. A comment notes that one can avoid this behaviour with by `pip install`ing with the `--no-deps` flag. Let us do so.

(At first glance, the problem looks like https://github.com/python-poetry/poetry/issues/5311, but that was a bug in `poetry install`; this is `poetry export`, whose behaviour is fine AFAICS).

* Two scripts are basically entry_points already
* Move and rename scripts/* to synapse/_scripts/*.py
* Delete sync_room_to_group.pl
* Expose entry points in setup.py
* Update linter script and config
* Fixup scripts & docs mentioning scripts that moved

Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>

The driver for this is to stop Complement complaining about it, but as far as I can tell it was pointless and needed to go away anyway.

I'm a bit unclear about what exactly VOLUME does, but I think what it means is that, if you don't override it with an explicit -v argument, then docker run will create a temporary volume, and copy things into it. The temporary volume is then deleted when the container finishes.

That only sounds useful if your image has something to copy into it (otherwise you may as well just use the default root filesystem), and our image notably doesn't copy anything into /data.

So... this wasn't doing anything, except annoying Complement?

Having spent much of the last week attempting to run complement tests from somewhere with damp string instead of internet... something had to be done.

* Add healthcheck startup delay by 5secs and reduced interval check to 15s
to reduce waiting time for docker aware edge routers bringing an
instance online

They don't make any sense on the intermediate builder image. The final
images needs them to be of use for anyone.

Signed-off-by: Johannes Wienke <languitar@semipol.de>

Make pip install faster in Docker build for [Complement](https://github.com/matrix-org/complement) testing.

If files have changed in a `COPY` command, Docker will invalidate all of the layers below. So I changed the order of operations to install all dependencies before we `COPY synapse /synapse/synapse/`. This allows Docker to use our cached layer of dependencies even when we change the source of Synapse and speed up builds dramatically! `53.5s` -> `3.7s` builds 🤘

As an alternative, I did try using BuildKit caches but this still took 30 seconds overall on that step. 15 seconds to gather the dependencies from the cache and another 15 seconds to `Installing collected packages`.

Fix https://github.com/matrix-org/synapse/issues/9364

Cf. https://github.com/opencontainers/image-spec/blob/master/annotations.md#pre-defined-annotation-keys



Signed-off-by: Johannes Wienke <languitar@semipol.de>

Co-authored-by: Will Hunt <willh@matrix.org>
Co-authored-by: Erik Johnston <erik@matrix.org>

This is needed to build the cryptography library, since it does not
provide wheels for ARMv7.

Fixes #9403

Signed-off-by: Dan Callahan <danc@element.io>

This removes the version pin of the `prometheus_client` dependency, in direct response to #8831. If merged, this will close #8831 

As far as I can tell, no other changes are needed, but as I'm no synapse expert, I'm relying heavily on CI and maintainer reviews for this. My very primitive test of synapse with prometheus_client v0.9.0 on my home server didn't bring up any issues, so we'll see what happens.

Signed-off-by: Jordan Bancino

Short-term fix for https://github.com/matrix-org/synapse/issues/8766.

This bumps us closer to current Python without going all the way to 3.9.

Fixes #8674

Signed-off-by: Dan Callahan <danc@element.io>

Signed-off-by: Christopher May-Townsend <chris@maytownsend.co.uk>

As mentioned in #7397, switching to a debian base should help with multi-arch work to save time on compiling. This is unashamedly based on #6373, but without the extra functionality. Switch python version back to generic 3.7 to always pull the latest. Essentially, keeping this as small as possible. The image is bigger though unfortunately.

* Add libwebp dependency to Dockerfile

Signed-off-by: Juho Vanhanen <juho@vanhanen.io>

Alpine Linux 3.8 is still supported, but it seems like
it's quite outdated now.

While Python should be the same on both, all other libraries, etc.,
are much newer in Alpine 3.9 and 3.10.

Signed-off-by: Slavi Pantaleev <slavi@devture.com>

Signed-off-by: Amir Zarrinkafsh <nightah@me.com>

Python 3.7 is apparently faster than 3.6, and should be mature enough.

Fixes #5467.

Also:

* Fix wrapping in docker readme
* Clean up some docs on the docker image
* a workaround for #4554

There are two reasons this is a good thing:

 * first, it means that you don't end up with stuff kicking around your working
   copy ending up in the build image by mistake (which can upset the pip
   install process)

 * second: it means that the docker image cache is more effective, and we can
   reuse docker images when iterating on the docker stuff.

Rewrite the dockerfile as a multistage build: this means we can get rid of a whole load of cruft which we don't need.

This addresses #3224

This matches docker best practices.

Signed-off-by: Benedikt Heine <bebe@bebehei.de>

Signed-off-by: Paul Tötterman <paul.totterman@iki.fi>