upgrade.md

Upgrading Synapse

Before upgrading check if any special steps are required to upgrade from the version you currently have installed to the current version of Synapse. The extra instructions that may be required are listed later in this document.

• Check that your versions of Python and PostgreSQL are still supported.

  Synapse follows upstream lifecycles for Python and PostgreSQL, and removes support for versions which are no longer maintained.

  The website https://endoflife.date also offers convenient summaries.

• If Synapse was installed using prebuilt packages, you will need to follow the normal process for upgrading those packages.

• If Synapse was installed using pip then upgrade to the latest version by running:

  pip install --upgrade matrix-synapse

• If Synapse was installed from source, then:

  1. Obtain the latest version of the source code. Git users can run git pull to do this.

  2. If you're running Synapse in a virtualenv, make sure to activate it before upgrading. For example, if Synapse is installed in a virtualenv in ~/synapse/env then run:

     source ~/synapse/env/bin/activate
     pip install --upgrade .

     Include any relevant extras between square brackets, e.g. pip install --upgrade ".[postgres,oidc]".

  3. If you're using poetry to manage a Synapse installation, run:

     poetry install

     Include any relevant extras with --extras, e.g. poetry install --extras postgres --extras oidc. It's probably easiest to run poetry install --extras all.

  4. Restart Synapse:

     synctl restart

To check whether your update was successful, you can check the running server version with:

# you may need to replace 'localhost:8008' if synapse is not configured
# to listen on port 8008.

curl http://localhost:8008/_synapse/admin/v1/server_version

Rolling back to older versions

Rolling back to previous releases can be difficult, due to database schema changes between releases. Where we have been able to test the rollback process, this will be noted below.

In general, you will need to undo any changes made during the upgrade process, for example:

• pip:

  source env/bin/activate
  # replace `1.3.0` accordingly:
  pip install matrix-synapse==1.3.0

• Debian:

  # replace `1.3.0` and `stretch` accordingly:
  wget https://packages.matrix.org/debian/pool/main/m/matrix-synapse-py3/matrix-synapse-py3_1.3.0+stretch1_amd64.deb
  dpkg -i matrix-synapse-py3_1.3.0+stretch1_amd64.deb

Upgrading to v1.67.0

Direct TCP replication is no longer supported: migrate to Redis

Redis support was added in v1.13.0 with it becoming the recommended method in v1.18.0. It replaced the old direct TCP connections (which was deprecated as of v1.18.0) to the main process. With Redis, rather than all the workers connecting to the main process, all the workers and the main process connect to Redis, which relays replication commands between processes. This can give a significant CPU saving on the main process and is a prerequisite for upcoming performance improvements.

To migrate to Redis add the redis config, and remove the TCP replication listener from config of the master and worker_replication_port from worker config. Note that a HTTP listener with a replication resource is still required.

Minimum version of Poetry is now v1.2.0

The minimum supported version of poetry is now 1.2. This should only affect those installing from a source checkout.

Upgrading to v1.66.0

Delegation of email validation no longer supported

As of this version, Synapse no longer allows the tasks of verifying email address ownership, and password reset confirmation, to be delegated to an identity server. This removal was previously planned for Synapse 1.64.0, but was delayed until now to give homeserver administrators more notice of the change.

To continue to allow users to add email addresses to their homeserver accounts, and perform password resets, make sure that Synapse is configured with a working email server in the email configuration section (including, at a minimum, a notif_from setting.)

Specifying an email setting under account_threepid_delegates will now cause an error at startup.

Upgrading to v1.64.0

Deprecation of the ability to delegate e-mail verification to identity servers

Synapse v1.66.0 will remove the ability to delegate the tasks of verifying email address ownership, and password reset confirmation, to an identity server.

If you require your homeserver to verify e-mail addresses or to support password resets via e-mail, please configure your homeserver with SMTP access so that it can send e-mails on its own behalf. Consult the configuration documentation for more information.

The option that will be removed is account_threepid_delegates.email.

Changes to the event replication streams

Synapse now includes a flag indicating if an event is an outlier when replicating it to other workers. This is a forwards- and backwards-incompatible change: v1.63 and workers cannot process events replicated by v1.64 workers, and vice versa.

Once all workers are upgraded to v1.64 (or downgraded to v1.63), event replication will resume as normal.

frozendict release

frozendict 2.3.3 has recently been released, which fixes a memory leak that occurs during /sync requests. We advise server administrators who installed Synapse via pip to upgrade frozendict with pip install --upgrade frozendict. The Docker image matrixdotorg/synapse and the Debian packages from packages.matrix.org already include the updated library.

Upgrading to v1.62.0

New signatures for spam checker callbacks

As a followup to changes in v1.60.0, the following spam-checker callbacks have changed signature:

• user_may_join_room
• user_may_invite
• user_may_send_3pid_invite
• user_may_create_room
• user_may_create_room_alias
• user_may_publish_room
• check_media_file_for_spam

For each of these methods, the previous callback signature has been deprecated.

Whereas callbacks used to return bool, they should now return Union["synapse.module_api.NOT_SPAM", "synapse.module_api.errors.Codes"].

For instance, if your module implements user_may_join_room as follows:

async def user_may_join_room(self, user_id: str, room_id: str, is_invited: bool)
    if ...:
        # Request is spam
        return False
    # Request is not spam
    return True

you should rewrite it as follows:

async def user_may_join_room(self, user_id: str, room_id: str, is_invited: bool)
    if ...:
        # Request is spam, mark it as forbidden (you may use some more precise error
        # code if it is useful).
        return synapse.module_api.errors.Codes.FORBIDDEN
    # Request is not spam, mark it as such.
    return synapse.module_api.NOT_SPAM

Upgrading to v1.61.0

Removal of deprecated community/groups

This release of Synapse will remove deprecated community/groups from codebase.

Worker endpoints

For those who have deployed workers, following worker endpoints will no longer exist and they can be removed from the reverse proxy configuration:

• ^/_matrix/federation/v1/get_groups_publicised$
• ^/_matrix/client/(r0|v3|unstable)/joined_groups$
• ^/_matrix/client/(r0|v3|unstable)/publicised_groups$
• ^/_matrix/client/(r0|v3|unstable)/publicised_groups/
• ^/_matrix/federation/v1/groups/
• ^/_matrix/client/(r0|v3|unstable)/groups/

Upgrading to v1.60.0

Adding a new unique index to state_group_edges could fail if your database is corrupted

This release of Synapse will add a unique index to the state_group_edges table, in order to prevent accidentally introducing duplicate information (for example, because a database backup was restored multiple times).

Duplicate rows being present in this table could cause drastic performance problems; see issue 11779 for more details.

If your Synapse database already has had duplicate rows introduced into this table, this could fail, with either of these errors:

On Postgres:

synapse.storage.background_updates - 623 - INFO - background_updates-0 - Adding index state_group_edges_unique_idx to state_group_edges
synapse.storage.background_updates - 282 - ERROR - background_updates-0 - Error doing update
...
psycopg2.errors.UniqueViolation: could not create unique index "state_group_edges_unique_idx"
DETAIL:  Key (state_group, prev_state_group)=(2, 1) is duplicated.

(The numbers may be different.)

On SQLite:

synapse.storage.background_updates - 623 - INFO - background_updates-0 - Adding index state_group_edges_unique_idx to state_group_edges
synapse.storage.background_updates - 282 - ERROR - background_updates-0 - Error doing update
...
sqlite3.IntegrityError: UNIQUE constraint failed: state_group_edges.state_group, state_group_edges.prev_state_group

Expand this section for steps to resolve this problem

On Postgres

Connect to your database with psql.

BEGIN;
DELETE FROM state_group_edges WHERE (ctid, state_group, prev_state_group) IN (
  SELECT row_id, state_group, prev_state_group
  FROM (
    SELECT
      ctid AS row_id,
      MIN(ctid) OVER (PARTITION BY state_group, prev_state_group) AS min_row_id,
      state_group,
      prev_state_group
    FROM state_group_edges
  ) AS t1
  WHERE row_id <> min_row_id
);
COMMIT;

On SQLite

At the command-line, use sqlite3 path/to/your-homeserver-database.db:

BEGIN;
DELETE FROM state_group_edges WHERE (rowid, state_group, prev_state_group) IN (
  SELECT row_id, state_group, prev_state_group
  FROM (
    SELECT
      rowid AS row_id,
      MIN(rowid) OVER (PARTITION BY state_group, prev_state_group) AS min_row_id,
      state_group,
      prev_state_group
    FROM state_group_edges
  )
  WHERE row_id <> min_row_id
);
COMMIT;

For more details

This comment on issue 11779 has queries that can be used to check a database for this problem in advance.

New signature for the spam checker callback check_event_for_spam

The previous signature has been deprecated.

Whereas check_event_for_spam callbacks used to return Union[str, bool], they should now return Union["synapse.module_api.NOT_SPAM", "synapse.module_api.errors.Codes"].