Proxmox

Migrating an Existing Proxmox Host into a New Three-Node Cluster

For a long time, my Proxmox environment consisted of a single Dell Precision Tower hosting my virtual machines and LXC containers. That setup worked well, but it was limited and power consumption was a killer.

I recently added three HP EliteDesk 705 G4 Mini PCs to my homelab:

pve-hp01
pve-hp02
pve-hp03

See Blog Article PXE Deploy – placeholder –

The goal was to build a new three-node Proxmox cluster, move away from the older Dell hardware, and eventually remove the Dell server without rebuilding everything from scratch.

This article documents the cluster migration itself: how I used the existing Dell host to create the cluster, added the new HP nodes, verified quorum and storage access, and safely removed the original node.

The migration of individual VMs and LXC containers, as well as the TrueNAS backup configuration, is covered here – plaecholder-

Starting point

The initial environment looked like this:

Existing Dell server
└── proxmox
    ├── Existing VMs
    └── Existing LXC containers

New HP systems
├── pve-hp01
├── pve-hp02
└── pve-hp03

The Dell server already contained productive workloads. The three HP nodes were newly installed with Proxmox VE and did not yet contain any important guests.

The desired final state was:

Proxmox cluster
├── pve-hp01
├── pve-hp02
└── pve-hp03

Old Dell server
└── Removed from the cluster

The Dell server would initially create and host the cluster configuration. After all required data and services had been transferred, it could be removed.

Why the existing Dell server created the cluster

One of the most important Proxmox cluster rules is that two existing clusters cannot simply be merged.

Joining a Proxmox node to a cluster replaces parts of that node’s /etc/pve configuration. Because of this, a node that already contains important guests should not casually be joined to another cluster without planning and backups.

The safest approach in my situation was:

  1. Create the cluster on the existing Dell server.
  2. Join the three empty HP nodes.
  3. Transfer everything required from the old PVE on my Dell Server
  4. Verify the new nodes.
  5. Remove the Dell Node from the cluster.

The initial topology therefore looked like this:

proxmox
└── pve-cluster
    ├── proxmox
    ├── pve-hp01
    ├── pve-hp02
    └── pve-hp03

Despite sometimes being called the “master,” the first Proxmox node is not a permanent master in the traditional sense. Proxmox uses Corosync and quorum-based cluster membership. Once the cluster exists, the first node is simply another voting member.

That becomes important later: the node that originally created the cluster can be removed.

Preparing the new Proxmox nodes

Before joining the HP systems, each node needed:

  • A unique hostname
  • A static IP address
  • Correct name resolution
  • Matching time synchronization
  • An empty guest configuration
  • Network connectivity to the other nodes

My node addresses were:

proxmox   192.1.178.20
pve-hp01  192.1.178.31
pve-hp02  192.1.178.32
pve-hp03  192.1.178.33

Each node should be able to resolve all other nodes by hostname.

A typical /etc/hosts configuration could look like this:

192.1.178.20 proxmox
192.1.178.31 pve-hp01
192.1.178.32 pve-hp02
192.1.178.33 pve-hp03

Name resolution can be tested from every node:

getent hosts proxmox
getent hosts pve-hp01
getent hosts pve-hp02
getent hosts pve-hp03

It is also worth verifying the hostnames themselves:

hostname
hostname --fqdn

Cluster communication is sensitive to incorrect addresses, duplicate hostnames, and inconsistent resolution. Fixing these issues before creating the cluster is much easier than correcting Corosync configuration later.

Creating the cluster

On the Dell server, I created the new cluster:

pvecm create pve-cluster

The cluster name in this example is:

pve-cluster

The result can be checked with:

pvecm status

At this stage, the output should show a single-node cluster with quorum:

Nodes:            1
Expected votes:   1
Total votes:      1
Quorate:          Yes

A one-node cluster has quorum because one out of one vote is available.

The cluster configuration is stored in:

/etc/pve/corosync.conf

Because /etc/pve is managed by the Proxmox cluster filesystem, it should not be treated like a normal local configuration directory.

Joining the HP nodes

The three HP nodes could now join the cluster.

This can be done through the command line or through the Proxmox web interface.

A command-line join normally looks like this:

pvecm add 192.1.178.31

In the Proxmox web interface you can use the Join as follows:

Datacenter
└── Cluster
    └── Join Information

The join information from the existing cluster can be copied and used on the new node through:

Datacenter
└── Cluster
    └── Join Cluster

After each node joined, I verified the cluster membership:

pvecm status

With all four nodes online, the output showed:

Nodes:            4
Expected votes:   4
Total votes:      4
Quorum:           3
Quorate:          Yes

The membership section contained all four Corosync addresses:

192.1.178.20
192.1.178.31
192.1.178.32
192.1.178.33

Understanding quorum before removing a node

A Proxmox cluster uses quorum to prevent multiple parts of a disconnected cluster from making conflicting changes.

With four nodes, the cluster required three votes:

Expected votes: 4
Quorum:         3

After removing the Dell server, the expected state would be:

Expected votes: 3
Quorum:         2

This means the remaining three-node cluster can tolerate the loss of one node and still retain quorum.

Three nodes online:
3 votes available → quorate

One node offline:
2 votes available → still quorate

Two nodes offline:
1 vote available → no quorum

This is one reason why three nodes are a useful minimum for a normal Proxmox cluster.

However, quorum must not be confused with high availability. A three-node cluster does not automatically restart or recover all workloads when one node fails. HA requires additional configuration and suitable storage or replication.

Cluster-wide storage configuration

Proxmox stores cluster storage definitions in:

/etc/pve/storage.cfg

Because this file resides in /etc/pve, it is distributed across the cluster automatically.

My configuration included:

dir: local
        path /var/lib/vz
        content images,iso,backup,vztmpl
        shared 0

lvmthin: local-lvm
        thinpool data
        vgname pve
        content images,rootdir

nfs: TN-NFS
        export /mnt/WD16TB/proxmox
        path /mnt/pve/TN-NFS
        server 192.1.178.5
        content images
        nodes proxmox

nfs: TN-NFS-BU
        export /mnt/WD16TB/pve-backups
        path /mnt/pve/TN-NFS-BU
        server 192.1.178.5
        content backup

The following line restricted TN-NFS to the old Dell node:

nodes proxmox

Before removing the Dell, that restriction had to be removed or changed.

To make the storage available to every cluster node, I could remove the line completely:

nfs: TN-NFS
        export /mnt/WD16TB/proxmox
        path /mnt/pve/TN-NFS
        server 192.1.178.5
        content images

Alternatively, the new nodes could be listed explicitly:

nodes pve-hp01,pve-hp02,pve-hp03

The storage configuration can be modified online through:

Datacenter
└── Storage

No reboot is required.

Although Proxmox distributes the storage configuration automatically, every node still mounts an NFS share independently. The NFS server must therefore authorize all participating node addresses.

The storage status should be checked on the HP nodes:

pvesm status

The relevant shares should report:

active

It is also useful to inspect the mount directly:

ls -lah /mnt/pve/TN-NFS
ls -lah /mnt/pve/TN-NFS-BU

Preserving ISO images and container templates

The Dell server no longer hosted guests, but it still contained local Proxmox content.

The relevant directories were:

/var/lib/vz/template/iso
/var/lib/vz/template/cache
/var/lib/vz/dump
/var/lib/vz/images

I checked them with:

ls -lah /var/lib/vz/template/iso
ls -lah /var/lib/vz/template/cache
ls -lah /var/lib/vz/dump
ls -lah /var/lib/vz/images

In this case:

  • template/iso contained ISO images
  • template/cache contained LXC templates
  • dump was empty
  • images was empty

ISO images and container templates can either be downloaded again or copied to another storage location before removing the node.

For shared Proxmox directory storage, the expected paths are:

template/iso
template/cache

For example:

mkdir -p /mnt/pve/TN-NFS/template/iso
mkdir -p /mnt/pve/TN-NFS/template/cache

Then copy the existing files:

cp -av /var/lib/vz/template/iso/. \
  /mnt/pve/TN-NFS/template/iso/

cp -av /var/lib/vz/template/cache/. \
  /mnt/pve/TN-NFS/template/cache/

The corresponding content types must be enabled for the storage:

content images,iso,vztmpl

Using the trailing /. in the copy source also works when directories contain hidden entries and avoids shell wildcard problems when a directory is empty.

After copying, compare the source and destination:

du -sh /var/lib/vz/template/iso
du -sh /mnt/pve/TN-NFS/template/iso

du -sh /var/lib/vz/template/cache
du -sh /mnt/pve/TN-NFS/template/cache

For a stronger check, compare hashes:

find /var/lib/vz/template/iso -type f -exec sha256sum {} \; | sort
find /mnt/pve/TN-NFS/template/iso -type f -exec sha256sum {} \; | sort

Final checks before dumping my old Dell Precision

Before deleting the old node from the cluster, I ran a final set of checks directly on the Dell server.

Check for virtual machines

qm list

The result was empty.

Check for LXC containers

pct list

The result was also empty.

Check cluster status

pvecm status

The cluster was healthy and all four nodes were online:

Nodes:            4
Expected votes:   4
Total votes:      4
Quorum:           3
Quorate:          Yes

Check HA resources

ha-manager status

The result showed quorum but no configured HA resources:

quorum OK
fencing standby (CRM watchdog standby)

Check Ceph

pveceph status

Ceph was not installed:

binary not installed: /usr/bin/ceph-mon

This confirmed that no Ceph monitor, manager, OSD, or other cluster storage role had to be removed from the Dell node.

Check storage availability

pvesm status

All configured storage targets were active:

TN-NFS        nfs      active
TN-NFS-BU     nfs      active
local         dir      active
local-lvm     lvmthin  active

The local LVM-Thin storage showed no allocated guest data.

At this point, the Dell server no longer provided:

  • A VM
  • An LXC container
  • An HA resource
  • A Ceph service
  • Unique local guest storage
  • Required ISO or template files

It was ready to be removed.

Removing the Dell node

The node removal command must be executed from one of the remaining HP nodes, not from the Dell being removed.

On pve-hp01, I first verified the node name:

pvecm nodes

The Dell node was called:

proxmox

I then removed it:

pvecm delnode proxmox

A successful removal normally returns a message similar to:

Killing node 1

The cluster status should then be checked immediately:

pvecm status

The expected result is:

Nodes:            3
Expected votes:   3
Total votes:      3
Quorum:           2
Quorate:          Yes

Only the HP nodes should remain in the membership list:

pve-hp01
pve-hp02
pve-hp03

The web interface on the HP cluster should no longer show the Dell node.

Why the removed Dell still showed the HP nodes

After removing the Dell from the active cluster, its own Proxmox web interface still showed:

Datacenter
├── proxmox
├── pve-hp01
├── pve-hp02
└── pve-hp03

The HP nodes appeared inactive.

This does not mean that the Dell is still an active member of the HP cluster.

The active three-node cluster had already removed the Dell. However, the Dell still retained its old local Corosync and pmxcfs cluster state.

There are therefore two separate operations:

  1. Remove the Dell from the active cluster.
  2. Clean the old cluster configuration from the Dell itself.

The first operation updates the remaining cluster.

The second operation determines what happens to the removed machine.

Turning the Dell back into a standalone Proxmox server

Before cleaning the Dell, verify on one of the HP nodes that the new cluster is healthy:

pvecm status

Do not continue unless the HP cluster shows:

Nodes:          3
Expected votes: 3
Total votes:    3
Quorate:        Yes

Then execute the following commands on the removed Dell host.

Stop Corosync and the Proxmox cluster filesystem:

systemctl stop corosync
systemctl stop pve-cluster

Start pmxcfs in local mode:

pmxcfs -l

Remove the old Corosync configuration:

rm -f /etc/pve/corosync.conf
rm -rf /etc/corosync/*

Stop the local pmxcfs process:

killall pmxcfs

Start the normal Proxmox cluster filesystem again:

systemctl start pve-cluster

Reboot the Dell:

reboot

After the reboot, its web interface should show only:

Datacenter
└── proxmox

The machine is now a standalone Proxmox host again.

A reinstall is also a valid option, especially when the server will be repurposed or decommissioned. For an empty node, a clean installation is often simpler than retaining an old configuration.

Final architecture

The final Proxmox cluster consists of the three HP EliteDesk systems:

pve-cluster
├── pve-hp01
├── pve-hp02
└── pve-hp03

The old Dell server is no longer a voting member.

Cluster quorum now behaves as follows:

Three HP nodes online
└── Quorum available

One HP node offline
└── Quorum still available

Two HP nodes offline
└── Quorum lost

The original node creating the cluster was therefore only a temporary part of the final design. Once its workloads, local files, and storage dependencies were removed, it could be safely deleted like any other cluster node.

Lessons learned

The most important lesson is that the first Proxmox cluster node is not a permanent controller. It can be removed once the remaining nodes form a healthy quorum.

Other useful takeaways from this migration:

  • Build the cluster around the node that already contains important workloads.
  • Join only empty or fully backed-up nodes.
  • Verify hostname resolution before configuring Corosync.
  • Proxmox storage definitions are cluster-wide, but network storage access is still node-specific.
  • Remove obsolete nodes restrictions before retiring a host.
  • Check local ISO images and LXC templates before deleting a node.
  • Verify VMs, containers, HA, Ceph, storage, and quorum before removal.
  • Run pvecm delnode from another cluster member.
  • A removed node may retain stale cluster information until its local configuration is cleaned.
  • Cluster membership, shared storage, backup, replication, and high availability are separate concepts.

The resulting setup is cleaner, smaller, and better prepared for future maintenance than the original single-server environment.