Spock Protocol

spock_output defines a libpq subprotocol for streaming tuples, metadata, etc., from the decoding plugin to receivers.

This protocol is an inner layer in a stack:

• TCP or unix sockets
• libpq protocol
  • libpq replication subprotocol (COPY BOTH etc.)
  • spock output plugin => consumer protocol

so clients can simply use libpq's existing replication protocol support, directly or via their libpq-wrapper driver.

This is a binary protocol intended for compact representation.

spock_output also supports a json-based text protocol with json representations of the same changesets, supporting all the same hooks etc., intended mainly for tracing/debugging/diagnostics. That protocol is not discussed here.

Table of Contents

• Protocol Flow
• Protocol Messages
• Startup Message
• Startup Errors
• Arguments Client Supplies to Output Plugin
• Extensibility
• JSON Protocol

Protocol Flow

The protocol flow is primarily from upstream walsender/decoding plugin to the downstream receiver.

The only information that flows downstream-to-upstream is:

• The initial parameter list sent to START_REPLICATION; and
• replay progress messages

We can accept an arbitrary list of params to START_REPLICATION. After that we have no general-purpose channel for information to flow upstream. That means we can't do a multi-step negotiation or handshake for determining the replication options to use, binary protocol, etc.

The main form of negotiation is the client getting a "take it or leave it" set of settings from the server in an initial startup message sent before any replication data (see below) and, if it doesn't like them, reconnecting with different startup options.

Except for the negotiation via initial parameter list and then startup message the protocol flow is the same as any other walsender-based logical replication plugin. The data stream is sent in COPY BOTH mode as a series of CopyData messages encapsulating replication data, and ends when the client disconnects. There's no facility for ending the COPY BOTH mode and returning to the walsender command parser to issue new commands. This is a limitation of the walsender interface, not spock_output.

Protocol Messages

The individual protocol messages are discussed in the following sub-sections. Protocol flow and logic comes in the next major section.

Absolutely all top-level protocol messages begin with a message type byte. While represented in code as a character, this is a signed byte with no associated encoding.

Since the PostgreSQL libpq COPY protocol supplies a message length there's no need for top-level protocol messages to embed a length in their header.

BEGIN Message

A stream of rows starts with a BEGIN message. Rows may only be sent after a BEGIN and before a COMMIT.

Forwarded Transaction Origin Message

The message after the BEGIN may be a forwarded transaction origin message indicating what upstream node the transaction came from.

Sent if the immediately prior message was a BEGIN message, the upstream transaction was forwarded from another node, and replication origin forwarding is enabled, i.e. forward_changeset_origins is t in the startup reply message.

A "node" could be another host, another DB on the same host, or pretty much anything. Whatever origin name is found gets forwarded. The origin identifier is of arbitrary and application-defined format. Applications should prefix their origin identifier with a fixed application name part, like bdr_, myapp_, etc. It is application-defined what an application does with forwarded transactions from other applications.

An origin message with a zero-length origin name indicates that the origin could not be identified but was (probably) not the local node. It is client-defined what action is taken in this case.

It is a protocol error to send/receive a forwarded transaction origin message at any time other than immediately after a BEGIN message.

The origin identifier is typically closely related to replication slot names and replication origins' names in an application system.

For more detail see Changeset Forwarding in the README.

COMMIT Message

A stream of rows ends with a COMMIT message.

There is no ROLLBACK message because aborted transactions are not sent by the upstream.

INSERT, UPDATE or DELETE Message

After a BEGIN or metadata message, the downstream should expect to receive zero or more row change messages, composed of an insert/update/delete message with zero or more tuple fields, each of which has one or more tuple field values.

The row's relidentifier must match that of the most recently preceding metadata message. All consecutive row messages must currently have the same relidentifier. (Later extensions to add metadata caching will relax these requirements for clients that advertise caching support; see the documentation on metadata messages for more detail).

It is an error to decode rows using metadata received after the row was received, or using metadata that is not the most recently received metadata revision that still predates the row. I.e. in the sequence M1, R1, R2, M2, R3, M4: R1 and R2 must be decoded using M1, and R3 must be decoded using M2. It is an error to use M4 to decode any of the rows, to use M1 to decode R3, or to use M2 to decode R1 and R2.

Row messages may not arrive except during a transaction as delimited by BEGIN and COMMIT messages. It is an error to receive a row message outside a transaction.

Any unrecognised tuple type or tuple part type is an error on the downstream that must result in a client disconnect and error message. Downstreams are expected to negotiate compatibility, and upstreams must not add new tuple types or tuple field types without negotiation.

The downstream reads rows until the next non-row message is received. There is no other end marker or any indication of how many rows to expect in a sequence.

Row Message Header

One or more tuple-parts fields follow.

Tuple Fields

Tuple tupleformat compatibility

Unrecognised tupleformat kinds are a protocol error for the downstream.

Tuple Field Value Fields

These message parts describe individual fields within a tuple.

There are two kinds of tuple value fields, abbreviated and full. Which is being read is determined based on the first field, kind.

Abbreviated tuple value fields are nothing but the message kind:

Full tuple value fields have a length and datum:

Tuple Field Values Kind Compatibility

Unrecognised field kind values are a protocol error for the downstream. The downstream may not continue processing the protocol stream after this point.

The upstream may not send internal or binary format values to the downstream without the downstream negotiating acceptance of such values. The downstream will also generally negotiate to receive type information to use to decode the values. See the section on startup parameters and the startup message for details.

Table/Row Metadata Messages

Before sending changed rows for a relation, a metadata message for the relation must be sent so the downstream knows the namespace, table name, column names, optional column types, etc. A relidentifier field, an arbitrary numeric value unique for that relation on that upstream connection, maps the metadata to following rows.

A client should not assume that relation metadata will be followed immediately (or at all) by rows, since future changes may lead to metadata messages being delivered at other times. Metadata messages may arrive during or between transactions.

The upstream may not assume that the downstream retains more metadata than the one most recent table metadata message. This applies across all tables, so a client is permitted to discard metadata for table x when getting metadata for table y. The upstream must send a new metadata message before sending rows for a different table, even if that metadata was already sent in the same session or even same transaction. This requirement will later be weakened by the addition of client metadata caching, which will be advertised to the upstream with an output plugin parameter.

Columns in metadata messages are numbered from 0 to natts-1, reading consecutively from start to finish. The column numbers do not have to be a complete description of the columns in the upstream relation, so long as all columns that will later have row values sent are described. The upstream may choose to omit columns it doesn't expect to send changes for in any given series of rows. Column numbers are not necessarily stable across different sets of metadata for the same table, even if the table hasn't changed structurally.

A metadata message may not be used to decode rows received before that metadata message.

Table Metadata Header

Column Delimiter

Each column's metadata begins with a column metadata header. This comes immediately after the natts field in the table metadata header or after the last metadata block in the prior column.

It has the same char header as all the others, and the flags field is the same size as the length field in other blocks, so it's safe to read this as a column metadata block header.

Currently, the only defined flag is 0x1 indicating that the column is part of the relation's identity key.

Column Metadata Block Header

All column metadata blocks share the same header, which is the same length as a column delimiter:

Column Name Block

This block just carries the name of the column, nothing more. It begins with a column metadata block, and the rest of the message is the column name.

Column Type Block

T.B.D.

Not defined in first protocol revision.

Likely to send a type identifier (probably the upstream oid) as a reference to a "type info" protocol message to be delivered before. Then we can cache the type descriptions and avoid repeating long schemas and names, just using the oids.

Needs to have room to handle:

• built-in core types
• extension types (ext version may vary)
• enum types (CREATE TYPE ... AS ENUM)
• range types (CREATE TYPE ... AS RANGE)
• composite types (CREATE TYPE ... AS (...))
• custom types (CREATE TYPE ( input = x_in, output = x_out ))

... some of which can be nested

Startup Message

After processing output plugin arguments, the upstream output plugin must send a startup message as its first message on the wire. It is a trivial header followed by alternating key and value strings represented as null-terminated unsigned char strings.

This message specifies the capabilities the output plugin enabled and describes the upstream server and plugin. This may change how the client decodes the data stream, and/or permit the client to disconnect and report an error to the user if the result isn't acceptable.

If replication is rejected because the client is incompatible or the server is unable to satisfy required options, the startup message may be followed by a libpq protocol FATAL message that terminates the session. See Startup Errors below.

The parameter names and values are sent as alternating key/value pairs as null-terminated strings, e.g.

key1\0parameter1\0key2\0value2\0

Startup Message Parameters

Since all parameter values are sent as strings, the value types given below specify what the value must be reasonably interpretable as.

Startup Message 'binary' Parameters

Startup Errors

If the server rejects the client's connection - due to non-overlapping protocol support, unrecognised parameter formats, unsupported required parameters like hooks, etc - then it will follow the startup reply message with a normal libpq protocol error message. (Current versions send this before the startup message).

Arguments Client Supplies to Output Plugin

The one opportunity for the downstream client to send information (other than replay feedback) to the upstream is at connect-time, as an array of arguments to the output plugin supplied to START LOGICAL REPLICATION.

There is no back-and-forth, no handshake.

As a result, the client mainly announces capabilities and makes requests of the output plugin. The output plugin will ERROR if required parameters are unset, or where incompatibilities that cannot be resolved are found. Otherwise the output plugin reports what it could and could not honour in the startup message it sends as the first message on the wire down to the client. The client chooses whether to continue replay or to disconnect and report an error to the user, then possibly reconnect with different options.

Output Plugin Arguments

The output plugin's key/value arguments are specified in pairs, as key and value. They're what's passed to START_REPLICATION, etc.

All parameters are passed in text form. They should be limited to 7-bit ASCII, since the server's text encoding is not known, but may be normalized precomposed UTF-8. The types specified for parameters indicate what the output plugin should attempt to convert the text into. Clients should not send text values that are outside the range for that type.

Capabilities

Many values are capabilities flags for the client, indicating that it understands optional features like metadata caching, binary format transfers, etc. In general the output plugin may disregard capabilities the client advertises as supported and act as if they are not supported. If a capability is advertised as unsupported or is not advertised the output plugin must not enable the corresponding features.

In other words, don't send the client something it's not expecting.

Protocol Versioning

Two parameters max_proto_version and min_proto_version, which clients must always send, allow negotiation of the protocol version. The output plugin must ERROR if the client protocol support does not overlap its own protocol support range.

The protocol version is only incremented when there are major breaking changes that all or most clients must be modified to accommodate. Most changes are done by adding new optional messages and/or by having clients advertise capabilities to opt in to features.

The current protocol version range is 4 (min) to 5 (max). Both nodes must be running Spock version >= 50000 to use protocol versions 4 and 5.

The startup_params_format parameter identifies the format of the startup parameter set itself (currently always "1"). Parameters are looked up by name and may appear in any order.

Client Requirements and Capabilities

Spock-Specific Parameters

These parameters are specific to the Spock replication extension and control what data is replicated.

General Client Information

These keys tell the output plugin about the client. They're mainly for informational purposes. In particular, the versions must not be used to determine compatibility for binary or send/recv format, as non-PostgreSQL clients will simply not send them at all but may still understand binary or send/recv format fields.

Parameters Relating to Exchange of Binary Values

The downstream may specify to the upstream that it is capable of understanding binary (PostgreSQL internal binary datum format), and/or send/recv (PostgreSQL binary interchange) format data by setting the binary.want_binary_basetypes and/or binary.want_internal_basetypes options, or other yet-to-be-defined options.

An upstream output plugin that does not support one or both formats may ignore the downstream's binary support and send text format, in which case it may ignore all binary.* parameters. All downstreams must support text format. An upstream output plugin must not send binary or send/recv format unless the downstream has announced it can receive it. If both upstream and downstream support both formats an upstream should prefer binary format and fall back to send/recv, then to text, if compatibility requires.

Internal and binary format selection should be done on a type-by-type basis. It is quite normal to send 'text' format for extension types while sending binary for built-in types.

The downstream must specify its compatibility requirements for internal and binary data if it requests either or both formats. The upstream must honour these by falling back from binary to send/recv, and from send/recv to text, where the upstream and downstream are not compatible.

An unspecified compatibility field must be presumed to be unsupported by the downstream so that older clients that don't know about a change in a newer version don't receive unexpected data. For example, in the unlikely event that PostgreSQL 99.8 switched to 128-bit DPD (Densely Packed Decimal) representations of NUMERIC instead of the current arbitrary-length BCD (Binary Coded Decimal) format, a new binary.dpd_numerics parameter would be added. Clients that didn't know about the change wouldn't know to set it, so the upstream would presume it unsupported and send text format NUMERIC to those clients. This also means that clients that support the new format wouldn't be able to receive the old format in binary from older servers since they'd specify dpd_numerics = true in their compatibility parameters.

At this time a downstream may specify compatibility with only one value for a given option; i.e. a downstream cannot say it supports both 4-byte and 8-byte sizeof(int). Leaving it unspecified means the upstream must assume the downstream supports neither. (A future protocol extension may allow clients to specify alternative sets of supported formats).

The pg_version option must not be used to decide compatibility. Use binary.basetypes_major_version instead.

Note: binary.maxalign is not sent by the downstream client. It is only present in the upstream's startup message. The upstream uses its own MAXIMUM_ALIGNOF for binary compatibility decisions.

Extensibility

Because of the use of optional parameters in output plugin arguments, and the confirmation/response sent in the startup packet, a basic handshake is possible between upstream and downstream, allowing negotiation of capabilities.

The output plugin must never send non-optional data or change its wire format without confirmation from the client that it can understand the new data. It may send optional data without negotiation.

When extending the output plugin arguments, add-ons are expected to prefix all keys with the extension name, and should preferably use a single top level key with a json object value to carry their extension information. Additions to the startup message should follow the same pattern.

Hooks and plugins can be used to add functionality specific to a client.

JSON Protocol

If proto_format is set to json then the output plugin will emit JSON instead of the custom binary protocol. JSON support is intended mainly for debugging and diagnostics.

The JSON format supports all the same hooks.