workers.md

Scaling synapse via workers

For small instances it recommended to run Synapse in monolith mode (the default). For larger instances where performance is a concern it can be helpful to split out functionality into multiple separate python processes. These processes are called 'workers', and are (eventually) intended to scale horizontally independently.

Synapse's worker support is under active development and subject to change as we attempt to rapidly scale ever larger Synapse instances. However we are documenting it here to help admins needing a highly scalable Synapse instance similar to the one running matrix.org.

All processes continue to share the same database instance, and as such, workers only work with PostgreSQL-based Synapse deployments. SQLite should only be used for demo purposes and any admin considering workers should already be running PostgreSQL.

Main process/worker communication

The processes communicate with each other via a Synapse-specific protocol called 'replication' (analogous to MySQL- or Postgres-style database replication) which feeds streams of newly written data between processes so they can be kept in sync with the database state.

Additionally, processes may make HTTP requests to each other. Typically this is used for operations which need to wait for a reply - such as sending an event.

As of Synapse v1.13.0, it is possible to configure Synapse to send replication via a Redis pub/sub channel, and is now the recommended way of configuring replication. This is an alternative to the old direct TCP connections to the main process: rather than all the workers connecting to the main process, all the workers and the main process connect to Redis, which relays replication commands between processes. This can give a significant cpu saving on the main process and will be a prerequisite for upcoming performance improvements.

(See the Architectural diagram section at the end for a visualisation of what this looks like)

Setting up workers

A Redis server is required to manage the communication between the processes. (The older direct TCP connections are now deprecated.) The Redis server should be installed following the normal procedure for your distribution (e.g. apt install redis-server on Debian). It is safe to use an existing Redis deployment if you have one.

Once installed, check that Redis is running and accessible from the host running Synapse, for example by executing echo PING | nc -q1 localhost 6379 and seeing a response of +PONG.

The appropriate dependencies must also be installed for Synapse. If using a virtualenv, these can be installed with:

pip install matrix-synapse[redis]

Note that these dependencies are included when synapse is installed with pip install matrix-synapse[all]. They are also included in the debian packages from matrix.org and in the docker images at https://hub.docker.com/r/matrixdotorg/synapse/.

To make effective use of the workers, you will need to configure an HTTP reverse-proxy such as nginx or haproxy, which will direct incoming requests to the correct worker, or to the main synapse instance. See reverse_proxy.md for information on setting up a reverse proxy.

To enable workers you should create a configuration file for each worker process. Each worker configuration file inherits the configuration of the shared homeserver configuration file. You can then override configuration specific to that worker, e.g. the HTTP listener that it provides (if any); logging configuration; etc. You should minimise the number of overrides though to maintain a usable config.

Next you need to add both a HTTP replication listener and redis config to the shared Synapse configuration file (homeserver.yaml). For example:

# extend the existing `listeners` section. This defines the ports that the
# main process will listen on.
listeners:
  # The HTTP replication port
  - port: 9093
    bind_address: '127.0.0.1'
    type: http
    resources:
     - names: [replication]

redis:
    enabled: true

See the sample config for the full documentation of each option.

Under no circumstances should the replication listener be exposed to the public internet; it has no authentication and is unencrypted.

In the config file for each worker, you must specify the type of worker application (worker_app), and you should specify a unqiue name for the worker (worker_name). The currently available worker applications are listed below. You must also specify the HTTP replication endpoint that it should talk to on the main synapse process. worker_replication_host should specify the host of the main synapse and worker_replication_http_port should point to the HTTP replication port. If the worker will handle HTTP requests then the worker_listeners option should be set with a http listener, in the same way as the listeners option in the shared config.

For example:

worker_app: synapse.app.generic_worker
worker_name: worker1

# The replication listener on the main synapse process.
worker_replication_host: 127.0.0.1
worker_replication_http_port: 9093

worker_listeners:
 - type: http
   port: 8083
   resources:
     - names:
       - client
       - federation

worker_log_config: /home/matrix/synapse/config/worker1_log_config.yaml

...is a full configuration for a generic worker instance, which will expose a plain HTTP endpoint on port 8083 separately serving various endpoints, e.g. /sync, which are listed below.

Obviously you should configure your reverse-proxy to route the relevant endpoints to the worker (localhost:8083 in the above example).

Finally, you need to start your worker processes. This can be done with either synctl or your distribution's preferred service manager such as systemd. We recommend the use of systemd where available: for information on setting up systemd to start synapse workers, see systemd-with-workers. To use synctl, see synctl_workers.md.

Available worker applications

synapse.app.generic_worker

This worker can handle API requests matching the following regular expressions: