specification.rst

propagation event to occur, informing likely-interested parties of the new
values. This change is conveyed using two separate mechanisms:

 - a ``m.room.member`` event is sent to every room the user is a member of,
   to update the ``displayname`` and ``avatar_url``.
 - a presence status update is sent, again containing the new values of the
   ``displayname`` and ``avatar_url`` keys, in addition to the required
   ``presence`` key containing the current presence state of the user.

Both of these should be done automatically by the home server when a user
successfully changes their displayname or avatar URL fields.

Additionally, when home servers emit room membership events for their own
users, they should include the displayname and avatar URL fields in these
events so that clients already have these details to hand, and do not have to
perform extra roundtrips to query it.


Identity
========
.. NOTE::
  This section is a work in progress.

.. TODO-doc Dave
  - 3PIDs and identity server, functions


Federation
==========

Federation is the term used to describe how to communicate between Matrix home
servers. Federation is a mechanism by which two home servers can exchange
Matrix event messages, both as a real-time push of current events, and as a
historic fetching mechanism to synchronise past history for clients to view. It
uses HTTPS connections between each pair of servers involved as the underlying
transport. Messages are exchanged between servers in real-time by active
pushing from each server's HTTP client into the server of the other. Queries to
fetch historic data for the purpose of back-filling scrollback buffers and the
like can also be performed. Currently routing of messages between homeservers
is full mesh (like email) - however, fan-out refinements to this design are
currently under consideration.

There are three main kinds of communication that occur between home servers:

:Queries:
   These are single request/response interactions between a given pair of
   servers, initiated by one side sending an HTTPS GET request to obtain some
   information, and responded by the other. They are not persisted and contain
   no long-term significant history. They simply request a snapshot state at
   the instant the query is made.

:Ephemeral Data Units (EDUs):
   These are notifications of events that are pushed from one home server to
   another. They are not persisted and contain no long-term significant
   history, nor does the receiving home server have to reply to them.

:Persisted Data Units (PDUs):
   These are notifications of events that are broadcast from one home server to
   any others that are interested in the same "context" (namely, a Room ID).
   They are persisted to long-term storage and form the record of history for
   that context.

EDUs and PDUs are further wrapped in an envelope called a Transaction, which is
transferred from the origin to the destination home server using an HTTP PUT
request.


Transactions
------------
.. WARNING::
  This section may be misleading or inaccurate.

The transfer of EDUs and PDUs between home servers is performed by an exchange
of Transaction messages, which are encoded as JSON objects, passed over an HTTP
PUT request. A Transaction is meaningful only to the pair of home servers that
exchanged it; they are not globally-meaningful.

Each transaction has:
 - An opaque transaction ID.
 - A timestamp (UNIX epoch time in milliseconds) generated by its origin
   server.
 - An origin and destination server name.
 - A list of "previous IDs".
 - A list of PDUs and EDUs - the actual message payload that the Transaction
   carries.
 
``origin``
  Type: 
    String
  Description:
    DNS name of homeserver making this transaction.
    
``ts``
  Type: 
    Integer
  Description:
    Timestamp in milliseconds on originating homeserver when this transaction 
    started.
    
``previous_ids``
  Type:
    List of strings
  Description:
    List of transactions that were sent immediately prior to this transaction.
    
``pdus``
  Type:
    List of Objects.
  Description:
    List of updates contained in this transaction.

::

 {
  "transaction_id":"916d630ea616342b42e98a3be0b74113",
  "ts":1404835423000,
  "origin":"red",
  "destination":"blue",
  "prev_ids":["e1da392e61898be4d2009b9fecce5325"],
  "pdus":[...],
  "edus":[...]
 }

The ``prev_ids`` field contains a list of previous transaction IDs that the
``origin`` server has sent to this ``destination``. Its purpose is to act as a
sequence checking mechanism - the destination server can check whether it has
successfully received that Transaction, or ask for a retransmission if not.

The ``pdus`` field of a transaction is a list, containing zero or more PDUs.[*]
Each PDU is itself a JSON object containing a number of keys, the exact details
of which will vary depending on the type of PDU. Similarly, the ``edus`` field
is another list containing the EDUs. This key may be entirely absent if there
are no EDUs to transfer.

(* Normally the PDU list will be non-empty, but the server should cope with
receiving an "empty" transaction.)

PDUs and EDUs
-------------
.. WARNING::
  This section may be misleading or inaccurate.

All PDUs have:
 - An ID
 - A context
 - A declaration of their type
 - A list of other PDU IDs that have been seen recently on that context
   (regardless of which origin sent them)

``context``
  Type:
    String
  Description:
    Event context identifier
    
``origin``
  Type:
    String
  Description:
    DNS name of homeserver that created this PDU.
    
``pdu_id``
  Type:
    String
  Description:
    Unique identifier for PDU within the context for the originating homeserver

``ts``
  Type:
    Integer
  Description:
    Timestamp in milliseconds on originating homeserver when this PDU was
    created.

``pdu_type``
  Type:
    String
  Description:
    PDU event type.

``prev_pdus``
  Type:
    List of pairs of strings
  Description:
    The originating homeserver and PDU ids of the most recent PDUs the
    homeserver was aware of for this context when it made this PDU.

``depth``
  Type:
    Integer
  Description:
    The maximum depth of the previous PDUs plus one.


.. TODO-spec paul
  - Update this structure so that 'pdu_id' is a two-element [origin,ref] pair
    like the prev_pdus are

For state updates:

``is_state``
  Type:
    Boolean
  Description:
    True if this PDU is updating state.
    
``state_key``
  Type:
    String
  Description:
    Optional key identifying the updated state within the context.
    
``power_level``
  Type:
    Integer
  Description:
    The asserted power level of the user performing the update.
    
``required_power_level``
  Type:
    Integer
  Description:
    The required power level needed to replace this update.

``prev_state_id``
  Type:
    String
  Description:
    PDU event type.
    
``prev_state_origin``
  Type:
    String
  Description:
    The PDU id of the update this replaces.
    
``user_id``
  Type:
    String
  Description:
    The user updating the state.

::

 {
  "pdu_id":"a4ecee13e2accdadf56c1025af232176",
  "context":"#example.green",
  "origin":"green",
  "ts":1404838188000,
  "pdu_type":"m.text",
  "prev_pdus":[["blue","99d16afbc857975916f1d73e49e52b65"]],
  "content":...
  "is_state":false
 }

In contrast to Transactions, it is important to note that the ``prev_pdus``
field of a PDU refers to PDUs that any origin server has sent, rather than
previous IDs that this ``origin`` has sent. This list may refer to other PDUs
sent by the same origin as the current one, or other origins.

Because of the distributed nature of participants in a Matrix conversation, it
is impossible to establish a globally-consistent total ordering on the events.
However, by annotating each outbound PDU at its origin with IDs of other PDUs
it has received, a partial ordering can be constructed allowing causality
relationships to be preserved. A client can then display these messages to the
end-user in some order consistent with their content and ensure that no message
that is semantically in reply of an earlier one is ever displayed before it.

PDUs fall into two main categories: those that deliver Events, and those that
synchronise State. For PDUs that relate to State synchronisation, additional
keys exist to support this:

::

 {...,
  "is_state":true,
  "state_key":TODO-doc
  "power_level":TODO-doc
  "prev_state_id":TODO-doc
  "prev_state_origin":TODO-doc}

EDUs, by comparison to PDUs, do not have an ID, a context, or a list of
"previous" IDs. The only mandatory fields for these are the type, origin and
destination home server names, and the actual nested content.

::

 {"edu_type":"m.presence",
  "origin":"blue",
  "destination":"orange",
  "content":...}
  
  
Protocol URLs
-------------
.. WARNING::
  This section may be misleading or inaccurate.

All these URLs are namespaced within a prefix of::

  /_matrix/federation/v1/...

For active pushing of messages representing live activity "as it happens"::

  PUT .../send/:transaction_id/
    Body: JSON encoding of a single Transaction
    Response: TODO-doc

The transaction_id path argument will override any ID given in the JSON body.
The destination name will be set to that of the receiving server itself. Each
embedded PDU in the transaction body will be processed.


To fetch a particular PDU::

  GET .../pdu/:origin/:pdu_id/
    Response: JSON encoding of a single Transaction containing one PDU

Retrieves a given PDU from the server. The response will contain a single new
Transaction, inside which will be the requested PDU.
  

To fetch all the state of a given context::

  GET .../state/:context/
    Response: JSON encoding of a single Transaction containing multiple PDUs

Retrieves a snapshot of the entire current state of the given context. The
response will contain a single Transaction, inside which will be a list of PDUs
that encode the state.

To backfill events on a given context::

  GET .../backfill/:context/
    Query args: v, limit
    Response: JSON encoding of a single Transaction containing multiple PDUs

Retrieves a sliding-window history of previous PDUs that occurred on the given
context. Starting from the PDU ID(s) given in the "v" argument, the PDUs that
preceeded it are retrieved, up to a total number given by the "limit" argument.
These are then returned in a new Transaction containing all of the PDUs.


To stream events all the events::

  GET .../pull/
    Query args: origin, v
    Response: JSON encoding of a single Transaction consisting of multiple PDUs

Retrieves all of the transactions later than any version given by the "v"
arguments.


To make a query::

  GET .../query/:query_type
    Query args: as specified by the individual query types
    Response: JSON encoding of a response object

Performs a single query request on the receiving home server. The Query Type
part of the path specifies the kind of query being made, and its query
arguments have a meaning specific to that kind of query. The response is a
JSON-encoded object whose meaning also depends on the kind of query.

Backfilling
-----------
.. NOTE::
  This section is a work in progress.

.. TODO-doc
  - What it is, when is it used, how is it done

SRV Records
-----------
.. NOTE::
  This section is a work in progress.

.. TODO-doc
  - Why it is needed

State Conflict Resolution
-------------------------
.. NOTE::
  This section is a work in progress.

.. TODO-doc
  - How do conflicts arise (diagrams?)
  - How are they resolved (incl tie breaks)
  - How does this work with deleting current state

Security
========

.. NOTE::
  This section is a work in progress.

Server-Server Authentication
----------------------------

.. TODO-doc
  - Why is this needed.
  - High level overview of process.
  - Transaction/PDU signing
  - How does this work with redactions? (eg hashing required keys only)

End-to-End Encryption
---------------------

.. TODO-doc
  - Why is this needed.
  - Overview of process
  - Implementation

Lawful Interception
-------------------

Key Escrow Servers
~~~~~~~~~~~~~~~~~~

Threat Model
------------

Denial of Service
~~~~~~~~~~~~~~~~~

The attacker could attempt to prevent delivery of messages to or from the
victim in order to:

* Disrupt service or marketing campaign of a commercial competitor.
* Censor a discussion or censor a participant in a discussion.
* Perform general vandalism.

Threat: Resource Exhaustion
+++++++++++++++++++++++++++

An attacker could cause the victims server to exhaust a particular resource
(e.g. open TCP connections, CPU, memory, disk storage)

Threat: Unrecoverable Consistency Violations
++++++++++++++++++++++++++++++++++++++++++++

An attacker could send messages which created an unrecoverable "split-brain"
state in the cluster such that the victim's servers could no longer dervive a
consistent view of the chatroom state.

Threat: Bad History
+++++++++++++++++++

An attacker could convince the victim to accept invalid messages which the
victim would then include in their view of the chatroom history. Other servers
in the chatroom would reject the invalid messages and potentially reject the
victims messages as well since they depended on the invalid messages.

.. TODO-spec
  Track trustworthiness of HS or users based on if they try to pretend they
  haven't seen recent events, and fake a splitbrain... --M

Threat: Block Network Traffic
+++++++++++++++++++++++++++++

An attacker could try to firewall traffic between the victim's server and some
or all of the other servers in the chatroom.

Threat: High Volume of Messages
+++++++++++++++++++++++++++++++

An attacker could send large volumes of messages to a chatroom with the victim
making the chatroom unusable.

Threat: Banning users without necessary authorisation
+++++++++++++++++++++++++++++++++++++++++++++++++++++

An attacker could attempt to ban a user from a chatroom with the necessary
authorisation.

Spoofing
~~~~~~~~

An attacker could try to send a message claiming to be from the victim without
the victim having sent the message in order to:

* Impersonate the victim while performing illict activity.
* Obtain privileges of the victim.

Threat: Altering Message Contents
+++++++++++++++++++++++++++++++++

An attacker could try to alter the contents of an existing message from the
victim.

Threat: Fake Message "origin" Field
+++++++++++++++++++++++++++++++++++

An attacker could try to send a new message purporting to be from the victim
with a phony "origin" field.

Spamming
~~~~~~~~

The attacker could try to send a high volume of solicicted or unsolicted
messages to the victim in order to:

* Find victims for scams.
* Market unwanted products.

Threat: Unsoliticted Messages
+++++++++++++++++++++++++++++

An attacker could try to send messages to victims who do not wish to receive
them.

Threat: Abusive Messages
++++++++++++++++++++++++

An attacker could send abusive or threatening messages to the victim

Spying
~~~~~~

The attacker could try to access message contents or metadata for messages sent
by the victim or to the victim that were not intended to reach the attacker in
order to:

* Gain sensitive personal or commercial information.
* Impersonate the victim using credentials contained in the messages.
  (e.g. password reset messages)
* Discover who the victim was talking to and when.

Threat: Disclosure during Transmission
++++++++++++++++++++++++++++++++++++++

An attacker could try to expose the message contents or metadata during
transmission between the servers.

Threat: Disclosure to Servers Outside Chatroom
++++++++++++++++++++++++++++++++++++++++++++++

An attacker could try to convince servers within a chatroom to send messages to
a server it controls that was not authorised to be within the chatroom.

Threat: Disclosure to Servers Within Chatroom
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

An attacker could take control of a server within a chatroom to expose message
contents or metadata for messages in that room.

Rate limiting
-------------
Home servers SHOULD implement rate limiting to reduce the risk of being
overloaded. If a request is refused due to rate limiting, it should return a
standard error response of the form::

  {
    "errcode": "M_LIMIT_EXCEEDED",
    "error": "string",
    "retry_after_ms": integer (optional)
  }

The ``retry_after_ms`` key SHOULD be included to tell the client how long they
have to wait in milliseconds before they can try again.

.. TODO-spec
  - Surely we should recommend an algorithm for the rate limiting, rather than letting every
    homeserver come up with their own idea, causing totally unpredictable performance over
    federated rooms?


Policy Servers
==============
.. NOTE::
  This section is a work in progress.

.. TODO-spec
  We should mention them in the Architecture section at least: how they fit
  into the picture.

Enforcing policies
------------------


Content repository
==================
.. NOTE::
  This section is a work in progress.

.. TODO-spec
  - path to upload
  - format for thumbnail paths, mention what it is protecting against.
  - content size limit and associated M_ERROR.


Address book repository
=======================
.. NOTE::
  This section is a work in progress.

.. TODO-spec
  - format: POST(?) wodges of json, some possible processing, then return wodges of json on GET.
  - processing may remove dupes, merge contacts, pepper with extra info (e.g. matrix-ability of
    contacts), etc.
  - Standard json format for contacts? Piggy back off vcards?


Glossary
========
.. NOTE::
  This section is a work in progress.

Backfilling:
  The process of synchronising historic state from one home server to another,
  to backfill the event storage so that scrollback can be presented to the
  client(s). Not to be confused with pagination.

Context:
  A single human-level entity of interest (currently, a chat room)

EDU (Ephemeral Data Unit):
  A message that relates directly to a given pair of home servers that are
  exchanging it. EDUs are short-lived messages that related only to one single
  pair of servers; they are not persisted for a long time and are not forwarded
  on to other servers. Because of this, they have no internal ID nor previous
  EDUs reference chain.

Event:
  A record of activity that records a single thing that happened on to a context
  (currently, a chat room). These are the "chat messages" that Synapse makes
  available.

PDU (Persistent Data Unit):
  A message that relates to a single context, irrespective of the server that
  is communicating it. PDUs either encode a single Event, or a single State
  change. A PDU is referred to by its PDU ID; the pair of its origin server
  and local reference from that server.

PDU ID:
  The pair of PDU Origin and PDU Reference, that together globally uniquely
  refers to a specific PDU.

PDU Origin:
  The name of the origin server that generated a given PDU. This may not be the
  server from which it has been received, due to the way they are copied around
  from server to server. The origin always records the original server that
  created it.

PDU Reference:
  A local ID used to refer to a specific PDU from a given origin server. These
  references are opaque at the protocol level, but may optionally have some
  structured meaning within a given origin server or implementation.

Presence:
  The concept of whether a user is currently online, how available they declare
  they are, and so on. See also: doc/model/presence

Profile:
  A set of metadata about a user, such as a display name, provided for the
  benefit of other users. See also: doc/model/profiles

Room ID:
  An opaque string (of as-yet undecided format) that identifies a particular
  room and used in PDUs referring to it.

Room Alias:
  A human-readable string of the form #name:some.domain that users can use as a
  pointer to identify a room; a Directory Server will map this to its Room ID

State:
  A set of metadata maintained about a Context, which is replicated among the
  servers in addition to the history of Events.

User ID:
  A string of the form @localpart:domain.name that identifies a user for
  wire-protocol purposes. The localpart is meaningless outside of a particular
  home server. This takes a human-readable form that end-users can use directly
  if they so wish, avoiding the 3PIDs.

Transaction:
  A message which relates to the communication between a given pair of servers.
  A transaction contains possibly-empty lists of PDUs and EDUs.

.. TODO
  This glossary contradicts the terms used above - especially on State Events v. "State"
  and Non-State Events v. "Events".  We need better consistent names.

.. Links through the external API docs are below
.. =============================================

.. |createRoom| replace:: ``/createRoom``
.. _createRoom: /docs/api/client-server/#!/-rooms/create_room

.. |initialSync| replace:: ``/initialSync``
.. _initialSync: /docs/api/client-server/#!/-events/initial_sync

.. |/rooms/<room_id>/initialSync| replace:: ``/rooms/<room_id>/initialSync``
.. _/rooms/<room_id>/initialSync: /docs/api/client-server/#!/-rooms/get_room_sync_data

.. |login| replace:: ``/login``
.. _login: /docs/api/client-server/#!/-login

.. |register| replace:: ``/register``
.. _register: /docs/api/client-server/#!/-registration

.. |/rooms/<room_id>/messages| replace:: ``/rooms/<room_id>/messages``
.. _/rooms/<room_id>/messages: /docs/api/client-server/#!/-rooms/get_messages

.. |/rooms/<room_id>/members| replace:: ``/rooms/<room_id>/members``
.. _/rooms/<room_id>/members: /docs/api/client-server/#!/-rooms/get_members

.. |/rooms/<room_id>/state| replace:: ``/rooms/<room_id>/state``
.. _/rooms/<room_id>/state: /docs/api/client-server/#!/-rooms/get_state_events

.. |/rooms/<room_id>/send/<event_type>| replace:: ``/rooms/<room_id>/send/<event_type>``
.. _/rooms/<room_id>/send/<event_type>: /docs/api/client-server/#!/-rooms/send_non_state_event

.. |/rooms/<room_id>/state/<event_type>/<state_key>| replace:: ``/rooms/<room_id>/state/<event_type>/<state_key>``
.. _/rooms/<room_id>/state/<event_type>/<state_key>: /docs/api/client-server/#!/-rooms/send_state_event

.. |/rooms/<room_id>/invite| replace:: ``/rooms/<room_id>/invite``
.. _/rooms/<room_id>/invite: /docs/api/client-server/#!/-rooms/invite

.. |/rooms/<room_id>/join| replace:: ``/rooms/<room_id>/join``
.. _/rooms/<room_id>/join: /docs/api/client-server/#!/-rooms/join_room

.. |/rooms/<room_id>/leave| replace:: ``/rooms/<room_id>/leave``
.. _/rooms/<room_id>/leave: /docs/api/client-server/#!/-rooms/leave

.. |/rooms/<room_id>/ban| replace:: ``/rooms/<room_id>/ban``
.. _/rooms/<room_id>/ban: /docs/api/client-server/#!/-rooms/ban

.. |/join/<room_alias_or_id>| replace:: ``/join/<room_alias_or_id>``
.. _/join/<room_alias_or_id>: /docs/api/client-server/#!/-rooms/join

.. _`Event Stream`: /docs/api/client-server/#!/-events/get_event_stream