Debug for #8631.

I'm having a hard time tracking down what's going wrong in that issue.
In the reported example, I could see server A sending federation traffic
to server B and all was well. Yet B reports out-of-sync device updates
from A.

I couldn't see what was _in_ the events being sent from A to B. So I
have added some crude logging to track

- when we have updates to send to a remote HS
- the edus we actually accumulate to send
- when a federation transaction includes a device list update edu
- when such an EDU is received

This is a bit of a sledgehammer.

Updates complement.sh to read the ref from an environment
variable (defaulting to master) when downloading a complement
bundle for testing.

`FederationClient.get_room_hierarchy()` caches its return values, so
refactor the code to avoid modifying the returned room summary.

A couple of surprises for me here, so thought I'd document them

By scraping Open Graph information from the HTML even
when an autodiscovery endpoint is found. The results are
then combined to capture as much information as possible
from the page.

Per updates to MSC3440.

This is implement as a separate method since it needs to be cached
on a per-user basis, instead of a per-thread basis.

I've never found this terribly useful. I think it was added in the early days
of Synapse, without much thought as to what would actually be useful to log,
and has just been cargo-culted ever since.

Rather, it tends to clutter up debug logs with useless information.

Add a flag to the `synapse_review_recent_signups` script to ignore and filter appservice users. (#11675)

* Warn against using Let's Encrypt certs for encrypted TURN

This helps to avoid client-side issues:
* https://github.com/vector-im/element-android/issues/1533
* https://github.com/vector-im/element-ios/issues/2712



Signed-off-by: Andrew Ferrazzutti <fair@miscworks.net>

Fixes #11741

... and a minor thinko fix in the sample config.

The existing implementation of the `python_twisted_reactor_tick_time` metric is pretty useless, because it *only* 
measures the time taken to execute timed calls and callbacks from threads. That neglects everything that 
happens off the back of I/O, which is obviously quite a lot for us.

To improve this, I've hooked into a different place in the reactor - in particular, where it calls `epoll`. That call is 
the only place it should wait for something to happen - the rest of the loop *should* be quick.

I've also removed `python_twisted_reactor_pending_calls`, because I don't believe anyone ever looks at it, and
it's a nuisance to populate.

Always add state.room_id after the configurable ORDER BY. Otherwise,
for any sort, certain pages can contain results from
other pages. (Especially when sorting by creator, since there may
be many rooms by the same creator)

* Document different order direction of numerical fields

"joined_members", "joined_local_members", "version" and "state_events"
are ordered in descending direction by default (dir=f). Added a note
in tests to explain the differences in ordering.

Signed-off-by: Daniël Sonck <daniel@sonck.nl>

* Fix sample_config.yaml in regards track_puppeted_user_ips

Closes #11741

Signed-off-by: Jason Robinson <jasonr@matrix.org>

This should be (slightly) more efficient and it is simpler
to have a single method for inserting multiple values.

documentation claims that you can use the %(app)s variable in password_reset and email_validation subjects, but if you do you end up with an error 500

Co-authored-by: br4nnigan <10244835+br4nnigan@users.noreply.github.com>

Co-authored-by: Brendan Abolivier <babolivier@matrix.org>

Fix a bug introduced in Synapse v1.50.0rc1 whereby outbound federation could fail because too many EDUs were produced for device updates. (#11730)

Co-authored-by: David Robertson <davidr@element.io>

* change spec url in config files

* Create 11739.txt

* .txt -> .doc

Rather than hooking into the reactor loop, just add a timed task that runs every 100 ms to do the garbage collection.

Part 1 of a quest to simplify the reactor monkey-patching.

Currently when puppeting another user, the user doing the puppeting is
tracked for client IPs and MAU (if configured).

When tracking MAU is important, it becomes necessary to be possible to
also track the client IPs and MAU of puppeted users. As an example a
client that manages user creation and creation of tokens via the Synapse
admin API, passing those tokens for the client to use.

This PR adds optional configuration to enable tracking of puppeted users
into monthly active users. The default behaviour stays the same.

Signed-off-by: Jason Robinson <jasonr@matrix.org>

Fix a bug introduced in Synapse v1.0.0 whereby device list updates would not be sent to remote homeservers if there were too many to send at once. (#11729)

Co-authored-by: Brendan Abolivier <babolivier@matrix.org>