.. meta::
   :description: Guide for configuring ThinLinc for High Availability
                 (HA) operations, including installation of a new HA
                 cluster, ensuring synchronization between nodes, and
                 reconfiguring existing installations into HA mode.

.. _HA-install:

Configuration of ThinLinc for HA operations
-------------------------------------------

In this section, we describe how ThinLinc is configured for high
availability.

Installation of a new HA cluster
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

In this section, we will describe how to set up a new HA cluster. In the
examples we will use a primary node with the hostname **tlha-primary**
and IP address **10.0.0.2**, a secondary node with the hostname
**tlha-secondary** and IP address **10.0.0.3**, and a resource IP
address of **10.0.0.4** with the DNS name **tlha**.

1. Begin by installing ThinLinc as described in :ref:`install_install`
   on both nodes.

2. Both nodes in the HA cluster must have the same SSH host key. Copy
   :file:`/etc/ssh/ssh_host_*` from the primary host to the secondary
   host, and restart :command:`sshd` on the secondary host.

3. Install and configure the system-level high-availability software,
   for example the software provided by the ClusterLabs project, which
   can be found at https://clusterlabs.org. This and other
   high-availability software may also be provided as part of your
   distribution, so check for the solution which best fits your
   requirements before proceeding.

4. Configure the system's high-availability software to watch the status
   of the other machine via the network, and to enable the resource IP
   address **10.0.0.4** on the active node. The machine with the
   hostname **tlha-primary** should normally be active.

5. Configure each agent server to allow privileged operations both from
   **tlha-primary** and **tlha-secondary**:

   .. code:: console

      $ sudo tl-config '/vsmagent/allowed_clients=tlha-primary tlha-secondary'

   Also, set the :servconf:`/vsmagent/master_hostname` to the DNS name
   of the HA interface:

   .. code:: console

      $ sudo tl-config /vsmagent/master_hostname=tlha

   Restart each agent service (:file:`vsmagent.service`) after changing
   the configuration values.

6. Verify operations of the master service (:file:`vsmserver.service`)
   on both nodes. Make sure you can start the service properly on both
   hosts, and connect to the respective hosts. It should be possible to
   connect, using :program:`tlclient`, to both **tlha-primary** and to
   **tlha-secondary**.

   Both nodes should be configured with the same subcluster
   configuration.

   .. warning::

      It is **very important** that 127.0.0.1 is not in the list of
      agent servers of any subcluster. If the master servers are also
      acting as agent servers, their unique hostnames or IP addresses
      must be added to :servconf:`/subclusters/<name>/agents` instead of
      127.0.0.1. The reason for this is that 127.0.0.1 will be a
      different server based on which master is currently active.

7. After verifying that normal ThinLinc connections work as intended
   when using both the primary and the secondary master's hostname, it
   is time to enable HA. This is done by setting :servconf:`/HA/enabled`
   to ``1``, and by specifying the nodes in the cluster in
   :servconf:`/HA/nodes`. For example:

   .. code:: console

      $ sudo tl-config /HA/enabled=1
      $ sudo tl-config '/HA/nodes=tlha-primary.example.com tlha-secondary.example.com'

   Configuration should be identical on both nodes. Restart the master
   service (:file:`vsmserver.service`) on both nodes after
   configuration.

8. If the master service can't safely determine which of the two nodes
   in :servconf:`/HA/nodes` is the remote node, and which is the
   local node, it will start without HA enabled, and log a message. If
   this happens, validate your hostname and DNS setup. One of the
   entries of :servconf:`/HA/nodes` must match the local
   machine. Either the resolved IP of one of the entries in
   :servconf:`/HA/nodes` must match the local IP, or one entry
   must exactly match the local hostname as returned by
   :command:`uname -n`.

9. Once HA has been configured, tests should be performed in order to
   confirm that the failover works as expected. This can normally be
   done by simply removing the network cable from the primary node, and
   ensuring that the secondary node then takes over. Check also that any
   active ThinLinc sessions have been synchronized from the primary to
   the secondary node, and that logging in to such a session on the
   secondary node succeeds once the primary node has been disabled.

Your ThinLinc HA cluster is now configured! When sessions are created,
changed or deleted on the currently active node, this information will
be synchronized to the other node. If the other node has to take over
service, its copy of the session data will be up-to-date, and it can
start serving new requests instantly. When the primary node comes up
again, the secondary node will resynchronize with the primary node.

Reconfiguring an existing ThinLinc installation into HA mode
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

If you have an existing ThinLinc installation and want to eliminate the
single point of failure (the single master server), the procedure is
very much like the procedure for installing a new HA cluster.
