setup_etcd

The setup_etcd role configures and starts etcd as a distributed consensus cluster for high availability Postgres deployments. The role creates the etcd configuration file with cluster membership and starts the etcd service on all nodes in a zone.

The role performs the following tasks on inventory hosts:

• Generate TLS certificates for etcd peer and client communication on nodes that do not yet have etcd data.
• Generate the etcd configuration file listing all zone nodes as cluster peers.
• Start the etcd systemd service on any node where it is not running.
• Restart running members one at a time when their configuration changes.

Role Dependencies

This role requires the following roles for normal operation:

• role_config provides shared configuration variables to the role.
• install_etcd installs the etcd package and systemd service file.

When to Use

Execute this role on pgedge hosts in high availability configurations after installing etcd. Only high availability deployments require this role.

In the following example, the playbook invokes the role after installing etcd:

- hosts: pgedge
  collections:
    - pgedge.platform
  roles:
    - install_etcd
    - setup_etcd

Configuration

This role uses the following parameters from the inventory file:

See the Configuration Reference for defaults.

How It Works

The role configures etcd for distributed consensus within each zone.

1. Check for an existing etcd data directory.
2. On a node without one, generate TLS certificates: a certificate authority (ca.crt), a peer key and certificate (peer.key, peer.crt), and a server key and certificate (server.key, server.crt). A node with etcd data keeps the certificates it has.
3. Generate the etcd configuration file at {{ etcd_config_dir }}/etcd.yaml with node identity, cluster membership for all nodes in the same zone, and network endpoints for client (port 2379) and peer (port 2380) communication.
4. Enable the etcd service and start it on every node where it is not running. A member started here reads the new configuration, and the role records it as the configuration that member runs.
5. Restart each running member whose configuration differs from the one it was last started with, one at a time. Before each restart the role checks that every member of the zone is healthy, and after it waits until they are again, then records the configuration the member now runs.

Configuration Details

The generated configuration file includes the following settings.

Cluster identification uses these values:

• initial-cluster-state uses new for initial bootstrap.
• initial-cluster-token identifies the cluster as pgedge_cluster.

Network configuration uses these values:

• listen-client-urls listens on the node IP and localhost on port 2379.
• listen-peer-urls listens on the node IP for peer communication on port 2380.

Timeouts are set to 20 seconds for dial, read, and write operations.

Keyspace maintenance uses these values:

• auto-compaction-mode and auto-compaction-retention compact key history older than one hour by default.
• quota-backend-bytes sets the database size at which etcd stops accepting writes, 2 GiB by default.

Usage Examples

In the following example, the playbook deploys a complete HA cluster with etcd, Patroni, and Postgres:

- hosts: pgedge
  collections:
    - pgedge.platform
  vars:
    is_ha_cluster: true
  roles:
    - init_server
    - install_repos
    - install_pgedge
    - install_etcd
    - install_patroni
    - setup_postgres
    - setup_etcd
    - setup_patroni

Artifacts

This role generates the following files on inventory hosts:

Idempotency

This role renders the etcd configuration on every run, so a changed parameter reaches a running cluster with the next deployment. etcd reads its configuration only at startup and cannot reload it, so the role restarts each member whose file differs from the configuration it was last started with. It restarts them one at a time, so the zone keeps its quorum and Patroni keeps its leader lock. A run that changes nothing restarts nothing.

The role restarts a member whose file is newer than the etcd process, which it reads from etcd's process_start_time_seconds metric, rather than only a member whose file the run changed. If a run writes a member's new file but never restarts the member, because the run was refused or stopped partway, the next run restarts it. A member restarted by hand after its file changed counts as running the new configuration.

The bootstrap settings in the file (initial-cluster, initial-cluster-state and initial-cluster-token) are read only when a member starts with an empty data directory, so re-rendering them on an existing member has no effect.

Certificates are issued only to a node without etcd data. An existing member keeps the certificates it was built with.