Adding a Node to a Cluster

Adding a node involves managing transaction consistency and synchronization across all nodes. The cluster module uses two powerful tools to handle these challenges: pgBackRest, and the Spock extension. pgBackRest is used to create a backup of the source node, which is then restored on the new node, ensuring it starts as a replica of the source node. Then, while adding the new node to the cluster, the Spock extension's read-only mode ensures that the cluster remains consistent and no transactions are lost. This read-only mode is enabled on all nodes during the critical phases of the process, allowing for the safe promotion of the new node and seamless integration into the cluster.

Several critical steps must be taken to ensure that all transactions are correctly replicated to the new node while it catches up with the other nodes in the cluster.

Best Practices

To make node addition quick and safe, we recommend:

• Adding a node during a time that the cluster is less busy.
• Ensure your application is not currently executing long running transactions during the node addition.
• Understand that the read-only mode will not save transactions to retry them. If a new transaction occurs during the time that the databases are in read-only mode, the application will have to retry them when read-only mode is disabled.

Using the cluster add-node Command

To help manage node addition, the cluster module now includes the add-node command. You can use the pgEdge cluster add-node command to gracefully add a node to your cluster, and update the cluster .json file to accurately reflect the cluster definition. The syntax is:

./pgedge cluster add-node <cluster_name> <node_name>

Prerequisites

Before invoking the cluster add-node command, you'll need to address the following prerequisite steps on the host of the new node:

1. Configure passwordless ssh and passwordless sudo.
2. Install and configure the pgEdge Platform CLI; if you are installing on localhost, the cluster can share one pgEdge Platform installation.
3. Have the necessary permissions and SSH keys to access the node.
4. Configure pgBackRest in ~./bashrc on the new node.
5. Create the new node configuration JSON file (target_node.json) in the pgedge directory. If you are installing on localhost, this will be the pgedge directory that resides above your cluster.
6. Remove any pgBackRest artifacts on the new node; you can use the command: rm -rf /etc/pgbackrest/

Configuring pgBackRest

If you're using a shared drive for your cluster, add add this line to the ~/.bashrc file on each node:

export PGBACKREST_REPO1_CIPHER_PASS=YourCipherPassHere

If you're using an AWS S3 bucket for your backups, add the following lines to the ~/.bashrc file on each node of the cluster:

export PGBACKREST_REPO1_CIPHER_PASS=YourCipherPassHere
export PGBACKREST_REPO1_S3_KEY=AIYFHTUJVLPPE
export PGBACKREST_REPO1_S3_BUCKET=bucket-876t3xpf
export PGBACKREST_REPO1_S3_KEY_SECRET=G9tlpTwj2+yTKLO3qMjeKG9a7GkR4mo
export PGBACKREST_REPO1_S3_ENDPOINT=s3.amazonaws.com
export PGBACKREST_REPO1_S3_REGION=eu-west-2

New Node Configuration (target_node.json)

The cluster configuration JSON file is named cluster_name.json, and is located in the cluster directory under your pgEdge installation. When you add a new node to your cluster, the cluster_name.json file will be updated with information from the new node JSON file as part of the process. No manual modifications are required to the main cluster's .json file, but the main cluster's .json file must include a version in the spock_version property:

{
   "json_version":1.0,
   "cluster_name":"cluster1",
   "log_level":"info",
   "update_date":"2024-07-11 08:56:55GMT",
   "pgedge":{
      "pg_version":"16",
      "auto_start":"off",
      "spock":{
         "spock_version":"4.0",
         "auto_ddl":"off"
      }

Before adding a new node to a cluster, you must create a .json file that defines the new node. You create the new_node.json file under the pgedge installation directory for the cluster. For example, to add a fourth node to a cluster on a localhost, you would create a file named n4.json in the cluster's pgedge directory that contains:

{
 "json_version": 1.0,
 "node_groups": [
      {
      "ssh": {
        "os_user": "ec2-user",
        "private_key": ""
      },
      "name": "n4",
      "is_active": "on",
      "public_ip": "127.0.0.1",
      "private_ip": "127.0.0.1",
      "port": "6435",
      "path": "/home/ec2-user/work/platform_test/nc/pgedge/cluster/demo/n4"
      }
  ]
}

Then, to add the new node to the cluster, navigate into the cluster's pgedge directory and invoke the command:

./pgedge cluster add-node cluster_name source_node target_node [script=[bash_script]]

Where:

• cluster_name is the name of the cluster to which you are adding the node.
• source_node is the name of the source node from the main cluster. A backup of this node will be used to populate the database on the new node.
• target_node is the name of the new node.
• bash_script is the path to an optional bash script that will be executed during the node addition process.

For example, to add a new node named n4 to a cluster named demo, use the command:

./pgedge cluster add-node demo n1 n4