Separates REST layer code from the actual URL previewing.

Make `configure_workers_and_start` script used in Complement tests compatible with older versions of Python. (#15275)

With Redis commands do not need to be re-issued by the main
process (they fan-out to all processes at once) and thus it is no
longer necessary to worry about them reflecting recursively forever.

Signed-off-by: Tulir Asokan <tulir@maunium.net>

Refresh remote profiles that have been marked as stale, in order to fill the user directory. [rei:userdirpriv] (#14756)

* Scaffolding for background process to refresh profiles

* Add scaffolding for background process to refresh profiles for a given server

* Implement the code to select servers to refresh from

* Ensure we don't build up multiple looping calls

* Make `get_profile` able to respect backoffs

* Add logic for refreshing users

* When backing off, schedule a refresh when the backoff is over

* Wake up the background processes when we receive an interesting state event

* Add tests

* Newsfile

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>

* Add comment about 1<<62

---------

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>

Preparatory work to fix the user directory assuming that any remote membership state events represent a profile change. [rei:userdirpriv] (#14755)

* Remove special-case method for new memberships only, use more generic method

* Only collect profiles from state events in public rooms

* Add a table to track stale remote user profiles

* Add store methods to set and delete rows in this new table

* Mark remote profiles as stale when a member state event comes in to a private room

* Newsfile

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>

* Simplify by removing Optionality of `event_id`

* Replace names and avatars with None if they're set to dodgy things

I think this makes more sense anyway.

* Move schema delta to 74 (I missed the boat?)

* Turns out these can be None after all

---------

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>

Co-authored-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>

It is not necessary to reach out to the database to check some
parameters if the auto-join rooms are not configured, or (in some cases)
if auto-create rooms is not configured.

* Bump cryptography from 39.0.1 to 39.0.2

Bumps [cryptography](https://github.com/pyca/cryptography) from 39.0.1 to 39.0.2.
- [Release notes](https://github.com/pyca/cryptography/releases)
- [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst)
- [Commits](https://github.com/pyca/cryptography/compare/39.0.1...39.0.2

)

---
updated-dependencies:
- dependency-name: cryptography
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Changelog

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: GitHub Actions <github-actions[bot]@users.noreply.github.com>

* Reproduce bad scenario in test
* Avoid catchup optimisation for partial state rooms

* Install rust during Stage 0 of docker build

Thanks to @atomdmac for spotting the fix.
Fixes #15179.

* Changelog

* Bump hiredis from 2.2.1 to 2.2.2

Bumps [hiredis](https://github.com/redis/hiredis-py) from 2.2.1 to 2.2.2.
- [Release notes](https://github.com/redis/hiredis-py/releases)
- [Changelog](https://github.com/redis/hiredis-py/blob/master/CHANGELOG.md)
- [Commits](https://github.com/redis/hiredis-py/compare/v2.2.1...v2.2.2

)

---
updated-dependencies:
- dependency-name: hiredis
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Changelog

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: GitHub Actions <github-actions[bot]@users.noreply.github.com>

* Bump serde from 1.0.152 to 1.0.155

Bumps [serde](https://github.com/serde-rs/serde) from 1.0.152 to 1.0.155.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.152...v1.0.155

)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Changelog

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: GitHub Actions <github-actions[bot]@users.noreply.github.com>

* Bump pysaml2 from 7.2.1 to 7.3.1

Bumps [pysaml2](https://github.com/IdentityPython/pysaml2) from 7.2.1 to 7.3.1.
- [Release notes](https://github.com/IdentityPython/pysaml2/releases)
- [Changelog](https://github.com/IdentityPython/pysaml2/blob/v7.3.1/CHANGELOG.md)
- [Commits](https://github.com/IdentityPython/pysaml2/compare/v7.2.1...v7.3.1

)

---
updated-dependencies:
- dependency-name: pysaml2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Changelog

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: GitHub Actions <github-actions[bot]@users.noreply.github.com>

* Bump msgpack from 1.0.4 to 1.0.5

Bumps [msgpack](https://github.com/msgpack/msgpack-python) from 1.0.4 to 1.0.5.
- [Release notes](https://github.com/msgpack/msgpack-python/releases)
- [Changelog](https://github.com/msgpack/msgpack-python/blob/main/ChangeLog.rst)
- [Commits](https://github.com/msgpack/msgpack-python/compare/v1.0.4...v1.0.5

)

---
updated-dependencies:
- dependency-name: msgpack
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Changelog

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: GitHub Actions <github-actions[bot]@users.noreply.github.com>

* Bump gitpython from 3.1.30 to 3.1.31

Bumps [gitpython](https://github.com/gitpython-developers/GitPython) from 3.1.30 to 3.1.31.
- [Release notes](https://github.com/gitpython-developers/GitPython/releases)
- [Changelog](https://github.com/gitpython-developers/GitPython/blob/main/CHANGES)
- [Commits](https://github.com/gitpython-developers/GitPython/compare/3.1.30...3.1.31

)

---
updated-dependencies:
- dependency-name: gitpython
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Changelog

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: GitHub Actions <github-actions[bot]@users.noreply.github.com>

* Tweak docstring and type hint

* Flip logic and provide better name

* Separate decision from action

* Track a set of strings, not EventBases

* Require explicit boolean options from callers

* Add explicit option for partial state rooms

* Changelog

* Rename param

Fix missing conditional for registering `on_remove_user_third_party_identifier` module api callbacks (#15227

* Give PyCharm some help with `@cache_in_self`

* Changelog

* Fix import for old python versions

When pushing events in partial state rooms down incremental /sync, we
try to find the `m.room.member` state event for their senders by digging
through their auth events, so that we can present the membership to the
client. Events usually have a membership event in their auth events,
with the exception of the `m.room.create` event and a user's first join
into the room.

When implementing #13477, we took the case of a user's first join into
account, but forgot to handle the `m.room.create` case. This change
fixes that.

Signed-off-by: Sean Quah <seanq@matrix.org>