Upgrading my Five-Node Proxmox VE homelab from 8 to 9: Failures, Recovery, and Lessons Learned

Upgrading one Proxmox VE server is straightforward when repositories, boot packages, and storage dependencies are already consistent. Upgrading five servers in a small datacenter exposes every inconsistency that accumulated over previous installations and updates.

My lab contains five Proxmox VE hosts. I upgraded them one at a time to reduce the blast radius. Before working on each node, I stopped its virtual machines and LXC containers, updated the existing Proxmox VE 8 installation, changed the Debian and Proxmox repositories, performed the distribution upgrade, and validated the node before moving to the next server.

 1                         +---------------------------+
 2                         |       Administrator       |
 3                         +-------------+-------------+
 4                                       |
 5                        Node-by-node migration
 6                       Proxmox VE 8 → PVE 9.2.6
 7                                       |
 8                                       v
 9                 +-------------------------------------------+
10                 |           Process for each node           |
11                 |                                           |
12                 |  1. Stop virtual machines and containers |
13                 |  2. Fully update Proxmox VE 8            |
14                 |  3. Run pve8to9 --full                   |
15                 |  4. Change repositories to Trixie        |
16                 |  5. Simulate apt full-upgrade            |
17                 |  6. Upgrade and reboot the node          |
18                 |  7. Validate services and workloads      |
19                 +---------------------+---------------------+
20                                       |
21                                       v
22                    +--------------------------------+
23                    |         CDMX - Primary         |
24                    |        172.16.10.100           |
25                    |     PVE 8 → PVE 9.2.6          |
26                    +---------------+----------------+
27                                    |
28             +----------------------+----------------------+
29             |                      |                      |
30             v                      v                      v
31 +-----------------------+ +-----------------------+ +-----------------------+
32 |       Mazatlán        | |        Mérida         | |      Guadalajara      |
33 |    172.16.10.101      | |    172.16.10.102      | |    172.16.10.103      |
34 | PVE 8 → PVE 9.2.6     | | PVE 8 → PVE 9.2.6     | | PVE 8 → PVE 9.2.6     |
35 +-----------+-----------+ +-----------+-----------+ +-----------+-----------+
36             |                         |                         |
37             +-------------------------+-------------------------+
38                                       |
39                                       v
40                           +-----------------------+
41                           |        Puebla         |
42                           |    172.16.10.104      |
43                           | PVE 8 → PVE 9.2.6     |
44                           +-----------+-----------+
45                                       |
46                                       v
47                 +-------------------------------------------+
48                 |       Mini datacenter upgraded           |
49                 |    5 nodes running Proxmox VE 9.2.6      |
50                 +-------------------------------------------+

That was the plan. In practice, I encountered four important problems:

1. A systemd-boot meta-package that the pve8to9 checker required me to remove.

2. A removable GRUB loader that was not configured to receive updates.

3. Duplicate and mixed Bookworm/Trixie repositories.

4. Ceph packages from an old Bookworm backport that caused APT to propose removing proxmox-ve.

On two nodes, the Proxmox management stack was ultimately absent after the package transition. The virtual machines were not necessarily terminated immediately, but commands such as pveversion and services such as pveproxy, pvedaemon, and pvestatd were no longer available. That produced the web-interface message:

1Connection error 595: Connection refused

This article documents the incident and, more importantly, the safeguards I added before upgrading the remaining nodes.

After repairing the affected hosts and applying those safeguards, I completed the upgrade across all five servers. The difficult nodes became canaries that improved the procedure for every node that followed.

This is a lab incident report, not a substitute for the official upgrade guide. Package versions change over time. Always use the current Proxmox VE 8 to 9 upgrade guide and evaluate every APT transaction before accepting it.

My upgrade strategy

I treated the five-server environment as a rolling maintenance operation:

1. Confirm access through SSH and an out-of-band or local console.

2. Back up critical guests and configuration.

3. Stop the VMs and LXC containers on the node being upgraded.

4. Fully update Proxmox VE 8 before changing repository suites.

5. Run pve8to9 --full and resolve every failure.

6. Change Debian and Proxmox repositories from Bookworm to Trixie.

7. Run a simulated full upgrade and inspect removals.

8. Run the real full upgrade only when the Proxmox stack remains installed.

9. Validate packages, services, storage, networking, boot files, and guests.

10. Reboot and observe the node before proceeding to the next server.

Stopping the guests was conservative, but it simplified recovery and eliminated guest workload changes while the host package set was transitioning.

Pre-upgrade baseline

Before changing repositories, I now collect a baseline on every host:

1hostname
2pveversion -v
3cat /etc/os-release
4uname -r
5pvesm status
6qm list
7pct list
8df -h / /boot /boot/efi 2>/dev/null
9pve8to9 --full

I also save the package and repository state:

1dpkg-query -W -f='${binary:Package}\t${Version}\n' > /root/packages-before-pve9.txt
2
3grep -RnsE \
4  '^[[:space:]]*deb|^[[:space:]]*(Types|URIs|Suites|Components|Enabled):' \
5  /etc/apt/sources.list /etc/apt/sources.list.d/ 2>/dev/null

The pve8to9 script should be run repeatedly. Its recommendations can change after a conflicting package is removed, a new package is installed, or the upgrade reaches another stage.

Issue 1: the systemd-boot meta-package warning

The first failure was:

1FAIL: systemd-boot meta-package installed. This will cause problems on upgrades
2of other boot-related packages. Remove 'systemd-boot'.

In Debian Trixie, systemd-boot was split into several packages. The systemd-boot meta-package contains hooks that can automatically manage the bootloader. Proxmox normally manages applicable EFI System Partitions through proxmox-boot-tool, so the meta-package can interfere with that workflow.

However, it must not be removed blindly. First, I identified the actual boot path:

1test -d /sys/firmware/efi \
2  && echo 'Boot mode: UEFI' \
3  || echo 'Boot mode: BIOS'
4
5bootctl status
6efibootmgr -v
7findmnt /boot /boot/efi
8dpkg -l | grep -E 'proxmox-boot-tool|systemd-boot|grub'

On my cdmx node, the important findings were:

1Boot mode: UEFI
2systemd-boot not installed in ESP.
3BootCurrent: 0000
4File: \EFI\proxmox\shimx64.efi

The node was booting through the Proxmox signed shim and GRUB, not through systemd-boot. /etc/kernel/proxmox-boot-uuids did not exist because this GRUB installation was not managed as a proxmox-boot-tool ESP set. I therefore simulated removal first:

1apt -s remove systemd-boot

APT proposed removing only systemd-boot, so I proceeded:

1apt remove systemd-boot

I deliberately kept systemd-boot-efi and did not run apt autoremove. The old kernels remained useful as rollback options.

The exception is a host on which systemd-boot was manually installed and is genuinely used as the bootloader. That configuration must be assessed separately. The official upgrade checker is the authority for whether the meta-package should be removed.

Issue 2: GRUB did not update the removable EFI loader

During initramfs generation, one node reported:

1Removable bootloader found at '/boot/efi/EFI/BOOT/BOOTX64.efi',
2but GRUB packages not set up to update it!

The hook supplied the required configuration:

1echo 'grub-efi-amd64 grub2/force_efi_extra_removable boolean true' \
2  | debconf-set-selections -v -u

It then instructed me to reinstall GRUB:

1apt install --reinstall grub-efi-amd64

My first reinstall attempt failed because the host contained GRUB packages from two releases:

1grub-efi-amd64 depends on grub-efi-amd64-bin (= 2.12-9+pmx2)
2but 2.06-13+pmx7 is to be installed

The correct lesson was not to force one GRUB package. All GRUB components must come from the same repository generation. I paused the bootloader repair, corrected the repositories, completed the package transition, and only then reinstalled GRUB.

After the package set was consistent, I used:

1apt install --reinstall grub-efi-amd64
2update-grub
3efibootmgr -v

I did not reboot while GRUB packages were mismatched. A stale fallback loader under EFI/BOOT/BOOTX64.EFI can load modules from a different GRUB version and leave the machine at a GRUB rescue prompt.

Issue 3: duplicate and mixed repositories

Several nodes had both traditional .list files and newer Deb822 .sources files. APT warned:

1Target Packages (pve-no-subscription/binary-amd64/Packages)
2is configured multiple times

On one node, the Proxmox VE repository still used Bookworm while another repository already used Trixie. At another point, no Debian base repository was active, so APT could see new Proxmox packages but could not obtain compatible Trixie versions of systemd, LVM, AppArmor, and other base dependencies.

I reviewed all active definitions rather than performing a global search-and-replace:

1grep -RnsE \
2  '^[[:space:]]*deb|^[[:space:]]*(Types|URIs|Suites|Components|Enabled):' \
3  /etc/apt/sources.list /etc/apt/sources.list.d/ 2>/dev/null

For PVE 9, the repository set must be internally consistent:

• Debian trixie

• Debian trixie-updates

• Debian trixie-security

• Proxmox VE trixie

• The correct Ceph repository, if Ceph packages are installed

• No active bookworm, bpo12, or duplicate definitions

I kept one definition for each repository and renamed obsolete files with a .disabled suffix so the change remained reversible.

Example Debian Deb822 configuration:

 1Types: deb
 2URIs: http://deb.debian.org/debian
 3Suites: trixie trixie-updates
 4Components: main contrib non-free-firmware
 5Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg
 6
 7Types: deb
 8URIs: http://security.debian.org/debian-security
 9Suites: trixie-security
10Components: main contrib non-free-firmware
11Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg

Example no-subscription PVE Deb822 configuration:

1Types: deb
2URIs: http://download.proxmox.com/debian/pve
3Suites: trixie
4Components: pve-no-subscription
5Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg

Repository definitions should be taken from the current Proxmox package repository documentation, especially when using an enterprise subscription or Ceph.

Issue 4: APT tried to remove proxmox-ve

The most serious warning appeared during the full upgrade:

1W: (pve-apt-hook) You are attempting to remove the meta-package 'proxmox-ve'!

The hook correctly stopped the transaction. The wrong response would have been to create /please-remove-proxmox-ve and continue. That file is an explicit authorization to remove the Proxmox meta-package; it is not an upgrade fix.

APT was proposing a large Debian transition, including old libraries being replaced by t64 variants. Those replacements are expected in Trixie. Removing proxmox-ve, pve-manager, pve-qemu-kvm, qemu-server, or libpve-storage-perl is not expected.

I diagnosed the dependency chain with:

1apt -s install proxmox-ve
2
3apt-cache policy \
4  proxmox-ve pve-manager pve-qemu-kvm \
5  ceph-common librados2 librbd1 libcephfs2

The real conflict was Ceph:

1ceph-common requires librbd1 (= 19.2.3-pve1)
2but 19.2.5-1~bpo12+2 is selected

The affected nodes contained Ceph Squid client packages built as Debian 12 backports (~bpo12+2), while the configured Proxmox PVE 9 Ceph repository offered a mutually consistent -pve1 package family. APT preferred the numerically newer backport, but its exact-version dependencies were incompatible with the rest of the Trixie/PVE package set.

The resulting dependency chain was:

1incompatible Ceph packages
2  -> libpve-storage-perl cannot be installed
3  -> pve-manager cannot be installed
4  -> proxmox-ve is selected for removal

Ceph safety boundary

Before changing Ceph packages, I checked whether a node actually provided Ceph services:

1ceph -s
2ceph version
3dpkg -l | grep -E '^ii[[:space:]]+ceph-(mon|osd|mgr)'

My affected hosts contained Ceph client libraries but no Ceph MON, OSD, or MGR packages. That allowed me to align the client packages to the versions offered by the configured Proxmox Ceph Squid repository.

This distinction is critical. If a node is part of an active Ceph cluster, follow Proxmox's coordinated Ceph upgrade procedure. The PVE 8-to-9 documentation requires eligible Ceph deployments to reach Squid before the host upgrade. Do not copy a client-library downgrade command onto a Ceph storage node.

Incident-specific client package alignment

For my client-only nodes, the complete Ceph family had to be selected in one APT transaction because many packages use exact-version dependencies:

 1apt -s install --allow-downgrades \
 2  ceph-common=19.2.3-pve1 \
 3  ceph-fuse=19.2.3-pve1 \
 4  libcephfs2=19.2.3-pve1 \
 5  librados2=19.2.3-pve1 \
 6  libradosstriper1=19.2.3-pve1 \
 7  librbd1=19.2.3-pve1 \
 8  librgw2=19.2.3-pve1 \
 9  python3-ceph-common=19.2.3-pve1 \
10  python3-ceph-argparse=19.2.3-pve1 \
11  python3-cephfs=19.2.3-pve1 \
12  python3-rados=19.2.3-pve1 \
13  python3-rbd=19.2.3-pve1 \
14  python3-rgw=19.2.3-pve1 \
15  liblttng-ust1t64 \
16  proxmox-ve

The version above records my incident and will become outdated. Before using an equivalent command, inspect the current candidates:

1apt-cache policy \
2  ceph-common ceph-fuse libcephfs2 \
3  librados2 libradosstriper1 librbd1 librgw2 \
4  python3-ceph-common python3-ceph-argparse \
5  python3-cephfs python3-rados python3-rbd python3-rgw

I only removed -s after verifying that the simulation installed or preserved the complete Proxmox stack and did not propose additional destructive removals.

Recovery when the Proxmox management stack is missing

On affected nodes, this command failed:

1pveversion: command not found

The service units were also absent:

1Unit pvedaemon.service not found.
2Unit pvestatd.service not found.
3Unit pveproxy.service not found.

That confirmed that this was not merely a stopped proxy. pve-manager and related packages had to be restored.

Before recovery, I avoided rebooting and checked whether guest processes were still running:

1ps -eo pid,cmd | grep -E '[k]vm|[l]xc-start'

After aligning the Ceph clients and reinstalling proxmox-ve, I completed package configuration and restarted the management plane:

1dpkg --configure -a
2apt-get check
3systemctl daemon-reload
4systemctl enable --now pve-cluster
5systemctl restart pvedaemon pvestatd pveproxy

I tested the local API directly:

1curl -k -sS -o /dev/null -w 'HTTP %{http_code}\n' \
2  https://127.0.0.1:8006/api2/json/version

An HTTP 200 or 401 response proves that the API endpoint is responding. I then verified the package stack with pveversion -v and tested the web interface on port 8006.

A safer procedure for the remaining nodes

The incident changed my workflow. I added explicit gates rather than assuming that a successful apt update meant the repositories were correct.

1. Finish PVE 8 updates first

While repositories still point to Bookworm/PVE 8:

1apt update
2apt full-upgrade
3pve8to9 --full

Resolve all failures before changing suites.

2. Inspect Ceph before changing repositories

1ceph version 2>/dev/null
2dpkg -l | grep -E '^ii[[:space:]]+ceph-(mon|osd|mgr)'
3
4dpkg-query -W -f='${binary:Package}\t${Version}\n' 2>/dev/null \
5  | grep -E 'ceph|librados|librbd|librgw'

An actual Ceph cluster requires its own supported upgrade sequence. Client-only packages still need a repository that supplies a consistent package family.

3. Change repositories and verify every suite

After editing repository definitions:

1apt update
2
3grep -RnsE \
4  '^[[:space:]]*deb|^[[:space:]]*(Types|URIs|Suites|Components|Enabled):' \
5  /etc/apt/sources.list /etc/apt/sources.list.d/ 2>/dev/null

No active Bookworm or duplicate entries should remain.

4. Require APT to preserve Proxmox

1apt -s install proxmox-ve
2apt -s full-upgrade

Read the complete REMOVED section. Stop immediately if it contains any of these:

1proxmox-ve
2pve-manager
3pve-qemu-kvm
4qemu-server
5pve-container
6pve-ha-manager
7libpve-storage-perl

Do not treat the Proxmox APT hook as an obstacle to bypass. It is the final guardrail preventing a repository or dependency mistake from dismantling the management stack.

5. Use a full upgrade, not a plain upgrade

A major Debian release transition needs to install replacement dependencies and remove obsolete libraries. A plain apt upgrade kept hundreds of packages back in my environment. Once the simulation was clean, I used:

1apt full-upgrade

6. Validate before reboot

 1dpkg --audit
 2apt-get check
 3pveversion -v
 4
 5systemctl --no-pager --full status \
 6  pve-cluster pvedaemon pvestatd pveproxy
 7
 8pvesm status
 9qm list
10pct list
11update-grub
12efibootmgr -v
13ls -1 /boot/vmlinuz-*-pve

I retained at least one known-good PVE 8 kernel until the node completed its first successful PVE 9 boot. I did not run apt autoremove during the upgrade window.

What I learned

APT's removal plan matters more than the upgrade count

Messages such as “569 upgraded, 131 newly installed, 64 to remove” are not automatically wrong during a major Debian transition. The identity of the removed packages is what matters. Old libraries may legitimately be replaced; the Proxmox management stack may not.

Repository consistency includes storage clients

I initially focused on Debian and PVE repositories. The decisive conflict came from Ceph client packages with exact dependencies. Storage-related repositories deserve the same preflight scrutiny as the base operating system.

A meta-package is operationally important

proxmox-ve contains little software itself, but it anchors the supported package set. Its proposed removal signals that APT can no longer satisfy that set. The correct action is to repair the dependency graph, not authorize removal.

Bootloader repair must wait for package consistency

Reinstalling GRUB while grub-common, grub2-common, and grub-efi-amd64-bin came from different releases would have created a boot risk. Repository correction and package alignment came first.

One node at a time limited the damage

Because I upgraded serially, each failure became a new preflight test for the next server. The five-node project reinforced a simple infrastructure principle: canary the procedure, validate every gate, and only then repeat it.

Final checklist

• Current backups exist and restore paths are known.

• SSH and console access work.

• Guests on the target node are stopped or migrated.

• PVE 8 is fully updated before repository changes.

• pve8to9 --full reports no unresolved failures.

• The active bootloader is identified.

• Ceph server roles and client package versions are known.

• All repositories use the intended suite and none are duplicated.

• apt -s install proxmox-ve succeeds.

• apt -s full-upgrade preserves the Proxmox stack.

• No /please-remove-proxmox-ve bypass file exists.

• Package, service, storage, network, and boot checks pass before reboot.

• The node is validated after reboot before upgrading the next server.

Conclusion

My Proxmox VE 8-to-9 upgrade was not a single command; it was a dependency, repository, storage-client, and bootloader migration across five different installation histories.

The systemd-boot warning was safe to resolve only after identifying the active GRUB path. The GRUB fallback loader could only be updated safely after all GRUB packages came from the same release. Duplicate and mixed repositories explained the packages kept back. Finally, incompatible Ceph client builds explained why APT wanted to remove proxmox-ve.

The most valuable command in the entire project was not apt full-upgrade. It was the simulation immediately before it:

1apt -s full-upgrade

When a major infrastructure upgrade proposes removing its own management platform, stop. The warning is not the problem—it is evidence of the problem.