The existing implementation of the `python_twisted_reactor_tick_time` metric is pretty useless, because it *only* 
measures the time taken to execute timed calls and callbacks from threads. That neglects everything that 
happens off the back of I/O, which is obviously quite a lot for us.

To improve this, I've hooked into a different place in the reactor - in particular, where it calls `epoll`. That call is 
the only place it should wait for something to happen - the rest of the loop *should* be quick.

I've also removed `python_twisted_reactor_pending_calls`, because I don't believe anyone ever looks at it, and
it's a nuisance to populate.

Always add state.room_id after the configurable ORDER BY. Otherwise,
for any sort, certain pages can contain results from
other pages. (Especially when sorting by creator, since there may
be many rooms by the same creator)

* Document different order direction of numerical fields

"joined_members", "joined_local_members", "version" and "state_events"
are ordered in descending direction by default (dir=f). Added a note
in tests to explain the differences in ordering.

Signed-off-by: Daniël Sonck <daniel@sonck.nl>

* Fix sample_config.yaml in regards track_puppeted_user_ips

Closes #11741

Signed-off-by: Jason Robinson <jasonr@matrix.org>

This should be (slightly) more efficient and it is simpler
to have a single method for inserting multiple values.

documentation claims that you can use the %(app)s variable in password_reset and email_validation subjects, but if you do you end up with an error 500

Co-authored-by: br4nnigan <10244835+br4nnigan@users.noreply.github.com>

Co-authored-by: Brendan Abolivier <babolivier@matrix.org>

Fix a bug introduced in Synapse v1.50.0rc1 whereby outbound federation could fail because too many EDUs were produced for device updates. (#11730)

Co-authored-by: David Robertson <davidr@element.io>

* change spec url in config files

* Create 11739.txt

* .txt -> .doc

Rather than hooking into the reactor loop, just add a timed task that runs every 100 ms to do the garbage collection.

Part 1 of a quest to simplify the reactor monkey-patching.

Currently when puppeting another user, the user doing the puppeting is
tracked for client IPs and MAU (if configured).

When tracking MAU is important, it becomes necessary to be possible to
also track the client IPs and MAU of puppeted users. As an example a
client that manages user creation and creation of tokens via the Synapse
admin API, passing those tokens for the client to use.

This PR adds optional configuration to enable tracking of puppeted users
into monthly active users. The default behaviour stays the same.

Signed-off-by: Jason Robinson <jasonr@matrix.org>

Fix a bug introduced in Synapse v1.0.0 whereby device list updates would not be sent to remote homeservers if there were too many to send at once. (#11729)

Co-authored-by: Brendan Abolivier <babolivier@matrix.org>

* Deal with mypy errors w/ type-hinted pynacl 1.5.0

Fixes #11644.

I really don't like that we're monkey patching pynacl SignedKey
instances with alg and version objects. But I'm too scared to make the
changes necessary right now.

(Ideally I would replace `signedjson.types.SingingKey` with a runtime class which
wraps or inherits from `nacl.signing.SigningKey`.) C.f. https://github.com/matrix-org/python-signedjson/issues/16

Signed-off-by: Sebastian Hasler <sebastian.hasler@stuvus.uni-stuttgart.de>

Signed-off-by: Sebastian Hasler <sebastian.hasler@stuvus.uni-stuttgart.de>

Having spent much of the last week attempting to run complement tests from somewhere with damp string instead of internet... something had to be done.

* Deal with mypy errors w/ type-hinted pynacl 1.5.0

Fixes #11644.

I really don't like that we're monkey patching pynacl SignedKey
instances with alg and version objects. But I'm too scared to make the
changes necessary right now.

(Ideally I would replace `signedjson.types.SingingKey` with a runtime class which
wraps or inherits from `nacl.signing.SigningKey`.) C.f. https://github.com/matrix-org/python-signedjson/issues/16

* Prettier complement logs

* Changelog

By returning all of the m.space.child state of the space, not just
the first 50. The number of rooms returned is still capped at 50.

For the federation API this implies that the requesting server will
need to individually query for any other rooms it is not joined to.

* Optionally use an on-disk sqlite db in tests

When debugging a test it is sometimes useful to inspect the state of the
DB. This is not easy when the db is in-memory: one cannot attach the
sqlite CLI to another process's DB.

With this change, if SYNAPSE_TEST_PERSIST_SQLITE_DB is set, we use
 `_trial_temp/test.db` as our sqlite database. One can then use 
`sqlite3 _trial_temp/test.db` and query to your heart's content.

The DB is destroyed and recreated between different test cases.

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>

This makes the serialization of events synchronous (and it no
longer access the database), but we must manually calculate and
provide the bundled aggregations.

Overall this should cause no change in behavior, but is prep work
for other improvements.

This was removed from MSC2674 before that was approved
and is not used by any known clients.

On Python 2, indexing a byte-string gives back a byte-string,
while on Python 3 it gives back the ASCII equivalent as an int.

* add some tests to verify we are stripping unauthorized fields out of unsigned

* add function to strip unauthorized fields from the unsigned object of event

* newsfragment

* update newsfragment number

* add check to on_send_membership_event

* refactor tests

* fix lint error

* slightly refactor tests and add some comments

* slight refactor

* refactor tests

* fix import error

* slight refactor

* remove unsigned filtration code from synapse/handlers/federation_event.py

* lint

* move unsigned filtering code to event base

* refactor tests

* update newsfragment

* requested changes

* remove unused retun values

Work around Mjolnir compatibility issue by adding an import for `glob_to_regex` in `synapse.util`, where it moved from. (#11696)

Make room creations denied by `user_may_create_room` cause an `M_FORBIDDEN` error to be returned, not `M_UNKNOWN` (#11672)

Co-authored-by: reivilibre <olivier@librepush.net>