When the local homeserver is already joined to a room and wants to
perform another remote join, we may find it useful to do a non-partial
state join if we already have the full state for the room.

Signed-off-by: Sean Quah <seanq@matrix.org>

This stops us from incrementing the presence stream position for no-op updates.

Fix up #14812 to avoid introducing a reference to the reactor.

Signed-off-by: Sean Quah <seanq@matrix.org>

* Fix missing field in AS documentation

The [AS Configuration Snippet](https://matrix-org.github.io/synapse/latest/application_services.html) is missing `id` field, without it Synapse will fail to load:

```
synapse-synapse-main-0 synapse 2023-01-13 23:05:25,450 - synapse.storage.databases - 84 - INFO - main - [database config 'master']: Starting 'main' database
synapse-synapse-main-0 synapse 2023-01-13 23:05:25,452 - synapse.config.appservice - 79 - ERROR - main - Failed to load appservice from '/as/synapse-hookshot-as/registration.yaml'
synapse-synapse-main-0 synapse 2023-01-13 23:05:25,452 - synapse.config.appservice - 80 - ERROR - main - "Required string field: 'id' (/as/synapse-hookshot-as/registration.yaml)"
synapse-synapse-main-0 synapse Traceback (most recent call last):
synapse-synapse-main-0 synapse   File "/usr/local/lib/python3.9/site-packages/synapse/config/appservice.py", line 57, in load_appservices
synapse-synapse-main-0 synapse     appservice = _load_appservice(hostname, yaml.safe_load(f), config_file)
synapse-synapse-main-0 synapse   File "/usr/local/lib/python3.9/site-packages/synapse/config/appservice.py", line 91, in _load_appservice
synapse-synapse-main-0 synapse     raise KeyError(
synapse-synapse-main-0 synapse KeyError: "Required string field: 'id' (/as/synapse-hookshot-as/registration.yaml)"
synapse-synapse-main-0 synapse 2023-01-13 23:05:25,452 - synapse.app._base - 207 - ERROR - main - Exception during startup
synapse-synapse-main-0 synapse Traceback (most recent call last):
synapse-synapse-main-0 synapse   File "/usr/local/lib/python3.9/site-packages/synapse/app/homeserver.py", line 340, in setup
synapse-synapse-main-0 synapse     hs.setup()
synapse-synapse-main-0 synapse   File "/usr/local/lib/python3.9/site-packages/synapse/server.py", line 310, in setup
synapse-synapse-main-0 synapse     self.datastores = Databases(self.DATASTORE_CLASS, self)
synapse-synapse-main-0 synapse   File "/usr/local/lib/python3.9/site-packages/synapse/storage/databases/__init__.py", line 93, in __init__
synapse-synapse-main-0 synapse     main = main_store_class(database, db_conn, hs)
synapse-synapse-main-0 synapse   File "/usr/local/lib/python3.9/site-packages/synapse/storage/databases/main/__init__.py", line 139, in __init__
synapse-synapse-main-0 synapse     super().__init__(database, db_conn, hs)
synapse-synapse-main-0 synapse   File "/usr/local/lib/python3.9/site-packages/synapse/storage/databases/main/events_bg_updates.py", line 98, in __init__
synapse-synapse-main-0 synapse     super().__init__(database, db_conn, hs)
synapse-synapse-main-0 synapse   File "/usr/local/lib/python3.9/site-packages/synapse/storage/databases/main/devices.py", line 1584, in __init__
synapse-synapse-main-0 synapse     super().__init__(database, db_conn, hs)
synapse-synapse-main-0 synapse   File "/usr/local/lib/python3.9/site-packages/synapse/storage/databases/main/devices.py", line 89, in __init__
synapse-synapse-main-0 synapse     super().__init__(database, db_conn, hs)
synapse-synapse-main-0 synapse   File "/usr/local/lib/python3.9/site-packages/synapse/storage/databases/main/roommember.py", line 1494, in __init__
synapse-synapse-main-0 synapse     super().__init__(database, db_conn, hs)
synapse-synapse-main-0 synapse   File "/usr/local/lib/python3.9/site-packages/synapse/storage/databases/main/room.py", line 1827, in __init__
synapse-synapse-main-0 synapse     super().__init__(database, db_conn, hs)
synapse-synapse-main-0 synapse   File "/usr/local/lib/python3.9/site-packages/synapse/storage/databases/main/room.py", line 1365, in __init__
synapse-synapse-main-0 synapse     super().__init__(database, db_conn, hs)
synapse-synapse-main-0 synapse   File "/usr/local/lib/python3.9/site-packages/synapse/storage/databases/main/room.py", line 119, in __init__
synapse-synapse-main-0 synapse     super().__init__(database, db_conn, hs)
synapse-synapse-main-0 synapse   File "/usr/local/lib/python3.9/site-packages/synapse/storage/databases/main/registration.py", line 2158, in __init__
synapse-synapse-main-0 synapse     super().__init__(database, db_conn, hs)
synapse-synapse-main-0 synapse   File "/usr/local/lib/python3.9/site-packages/synapse/storage/databases/main/presence.py", line 67, in __init__
synapse-synapse-main-0 synapse     super().__init__(database, db_conn, hs)
synapse-synapse-main-0 synapse   File "/usr/local/lib/python3.9/site-packages/synapse/storage/databases/main/presence.py", line 48, in __init__
synapse-synapse-main-0 synapse     super().__init__(database, db_conn, hs)
synapse-synapse-main-0 synapse   File "/usr/local/lib/python3.9/site-packages/synapse/storage/databases/main/transactions.py", line 73, in __init__
synapse-synapse-main-0 synapse     super().__init__(database, db_conn, hs)
synapse-synapse-main-0 synapse   File "/usr/local/lib/python3.9/site-packages/synapse/storage/databases/main/state.py", line 666, in __init__
synapse-synapse-main-0 synapse     super().__init__(database, db_conn, hs)
synapse-synapse-main-0 synapse   File "/usr/local/lib/python3.9/site-packages/synapse/storage/databases/main/state.py", line 82, in __init__
synapse-synapse-main-0 synapse     super().__init__(database, db_conn, hs)
synapse-synapse-main-0 synapse   File "/usr/local/lib/python3.9/site-packages/synapse/storage/databases/main/state.py", line 470, in __init__
synapse-synapse-main-0 synapse     super().__init__(database, db_conn, hs)
synapse-synapse-main-0 synapse   File "/usr/local/lib/python3.9/site-packages/synapse/storage/databases/main/event_federation.py", line 2007, in __init__
synapse-synapse-main-0 synapse     super().__init__(database, db_conn, hs)
synapse-synapse-main-0 synapse   File "/usr/local/lib/python3.9/site-packages/synapse/storage/databases/main/media_repository.py", line 148, in __init__
synapse-synapse-main-0 synapse     super().__init__(database, db_conn, hs)
synapse-synapse-main-0 synapse   File "/usr/local/lib/python3.9/site-packages/synapse/storage/databases/main/media_repository.py", line 68, in __init__
synapse-synapse-main-0 synapse     super().__init__(database, db_conn, hs)
synapse-synapse-main-0 synapse   File "/usr/local/lib/python3.9/site-packages/synapse/storage/databases/main/push_rule.py", line 330, in __init__
synapse-synapse-main-0 synapse     super().__init__(database, db_conn, hs)
synapse-synapse-main-0 synapse   File "/usr/local/lib/python3.9/site-packages/synapse/storage/databases/main/event_push_actions.py", line 1938, in __init__
synapse-synapse-main-0 synapse     super().__init__(database, db_conn, hs)
synapse-synapse-main-0 synapse   File "/usr/local/lib/python3.9/site-packages/synapse/storage/databases/main/metrics.py", line 68, in __init__
synapse-synapse-main-0 synapse     super().__init__(database, db_conn, hs)
synapse-synapse-main-0 synapse   File "/usr/local/lib/python3.9/site-packages/synapse/storage/databases/main/event_push_actions.py", line 249, in __init__
synapse-synapse-main-0 synapse     super().__init__(database, db_conn, hs)
synapse-synapse-main-0 synapse   File "/usr/local/lib/python3.9/site-packages/synapse/storage/databases/main/end_to_end_keys.py", line 1181, in __init__
synapse-synapse-main-0 synapse     super().__init__(database, db_conn, hs)
synapse-synapse-main-0 synapse   File "/usr/local/lib/python3.9/site-packages/synapse/storage/databases/main/search.py", line 426, in __init__
synapse-synapse-main-0 synapse     super().__init__(database, db_conn, hs)
synapse-synapse-main-0 synapse   File "/usr/local/lib/python3.9/site-packages/synapse/storage/databases/main/search.py", line 137, in __init__
synapse-synapse-main-0 synapse     super().__init__(database, db_conn, hs)
synapse-synapse-main-0 synapse   File "/usr/local/lib/python3.9/site-packages/synapse/storage/databases/main/account_data.py", line 64, in __init__
synapse-synapse-main-0 synapse     super().__init__(database, db_conn, hs)
synapse-synapse-main-0 synapse   File "/usr/local/lib/python3.9/site-packages/synapse/storage/databases/main/push_rule.py", line 114, in __init__
synapse-synapse-main-0 synapse     super().__init__(database, db_conn, hs)
synapse-synapse-main-0 synapse   File "/usr/local/lib/python3.9/site-packages/synapse/storage/databases/main/appservice.py", line 76, in __init__
synapse-synapse-main-0 synapse     self.services_cache = load_appservices(
synapse-synapse-main-0 synapse   File "/usr/local/lib/python3.9/site-packages/synapse/config/appservice.py", line 57, in load_appservices
synapse-synapse-main-0 synapse     appservice = _load_appservice(hostname, yaml.safe_load(f), config_file)
synapse-synapse-main-0 synapse   File "/usr/local/lib/python3.9/site-packages/synapse/config/appservice.py", line 91, in _load_appservice
synapse-synapse-main-0 synapse     raise KeyError(
synapse-synapse-main-0 synapse KeyError: "Required string field: 'id' (/as/synapse-hookshot-as/registration.yaml)"
synapse-synapse-main-0 synapse ******************************************************************************
synapse-synapse-main-0 synapse  Error during initialisation:
synapse-synapse-main-0 synapse     "Required string field: 'id' (/as/synapse-hookshot-as/registration.yaml)"
synapse-synapse-main-0 synapse  There may be more information in the logs.
synapse-synapse-main-0 synapse ******************************************************************************
```

* Changelog

* Also use stable name in SendJoinResponse struct

follow-up to #14832

* Changelog

* Fix a rename I missed

* Run black

* Update synapse/federation/federation_client.py

Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>

Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>

* Bump regex from 1.7.0 to 1.7.1

Bumps [regex](https://github.com/rust-lang/regex) from 1.7.0 to 1.7.1.
- [Release notes](https://github.com/rust-lang/regex/releases)
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/regex/compare/1.7.0...1.7.1

)

---
updated-dependencies:
- dependency-name: regex
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Changelog

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: GitHub Actions <github-actions[bot]@users.noreply.github.com>

* Use new query param when requesting a partial join

* Read new query param when serving partial join

* Provide new field names when serving partial joins

* Read new field names from partial join response

* Changelog

Fixes #13852

Closes: #13643

When there are many synchronous requests waiting on a
`_PerHostRatelimiter`, each request will be started recursively just
after the previous request has completed. Under the right conditions,
this leads to stack exhaustion.

A common way for requests to become synchronous is when the remote
client disconnects early, because the homeserver is overloaded and slow
to respond.

Avoid stack exhaustion under these conditions by deferring subsequent
requests until the next reactor tick.

Fixes #14480.

Signed-off-by: Sean Quah <seanq@matrix.org>

* Re-enable some linting

* Newsfile

* Remove comment

Fixes #14814

Faster remote room joins (worker mode): do not populate external hosts-in-room cache when sending events as this requires blocking for full state. (#14749)

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
Co-authored-by: Sean Quah <seanq@matrix.org>

Add index to improve performance of the `/timestamp_to_event` endpoint used for jumping to a specific date in the timeline of a room. (#14799)

Back-out some changes from 7e582a25
(#14786) which skipped necessary logic to calculate device lists properly.

Fixes: #8780

* Add `tag` to `listeners` documentation

* newsfile

Two parts to this:

 * Bundle the whole of the replacement with any edited events. This is backwards-compatible so I haven't put it behind a flag.
 * Optionally, inhibit server-side replacement of edited events. This has scope to break things, so it is currently disabled by default.

* Add missing worker settings to shared configuration

* newsfile

* update docs after review

* more update for doc

* This -> These

Co-authored-by: David Robertson <david.m.robertson1@gmail.com>

* Remove undocumented device from pushrules

* Add changelog

* Update changelog.d/14727.misc

* Rename 14727.misc to 14727.bugfix

Co-authored-by: David Robertson <davidr@element.io>

* Update link to towncrier in contribution guide

* newsfile