<\/p>\n\n\n\n
We have been luddites running plain Xen 4.x hypervisor atop of Debian for way too long. Manually launching\/stopping virtual machines, no live migration, no integrated backups, etc… So, ProxMox, here come.<\/p>\n\n\n\n
One problem.
Most all of our Linux-based instances were converted long ago to use the Xen-PVGrub boot subsystem to allow the vm’s Admin to internally maintain their own kernel. ProxMox doesn’t really have a facility to import those systems and launch\/them as-is. The SeaBIOS booting system in ProxMox really wants to have a partition map to grab onto. PVGrub based systems have no such partition maps on their boot disk devices.<\/p>\n\n\n\n
The solution: slipstream a partition map onto the front<\/em> of a virtual machines boot disk container. Assimilation can be done, but it requires precision of execution and patience.<\/p>\n\n\n\n Preflight check: 2. Restructure your virtual disks\/content within each VM so that the boot drive is as small as possible. This is to minimize the downtime by reducing the amount of data the block shifting process has to churn through when we go to drop in the partition map in that first 1MB of the block device. Once the mirroring process has sync’ed up the extents onto both PVs, shutdown the VM and then split the mirror:<\/p>\n\n\n\n 4. Start the VM back up and then begin the process of preparing the boot disk with the following:<\/p>\n\n\n\n 5. Update fstab to target mounts via UUID instead of Xen Virtual Disk device names. then edit fstab:<\/p>\n\n\n\n any xvd<\/em> mount needs to be changed to a UUID<\/em> mount. Resulting fstab:<\/p>\n\n\n\n 6. Now rebuild initrd and update grub:<\/p>\n\n\n\n 7. Shutdown the VM<\/strong> and attach the boot disk to the MBRFixer virtual machine:<\/p>\n\n\n\n you can verify it attached properly with a block-list:<\/p>\n\n\n\n Vdev\u00a0 BE\u00a0 handle state evt-ch ring-ref BE-path <\/p>\n\n\n\n While this currently PVGrub-configured boot disk is attached to the MBRFixer, the following tasks will be performed inside<\/strong> that VM: (you can check it attached properly with lsblk) 8b. shrink the file system slightly so there is unused\/disposable space at the end of the device: The device only needs to be shrunk by a single megabyte, but in practice its easier to just subtract a GB from the device size, so if it’s a 40GB boot drive, resize down to 39G:<\/p>\n\n\n\n If you are working with a file system type that cannot be shrunk, you’ll need to go the opposite route and expand<\/em> the disk. This doesn’t change the requirement for shifting the file system forward. Simply tacking another GB onto the end of the existing LV\/disk in order to have space to shift forward into works just as well.<\/p>\n\n\n\n 8c. Now that there should be nothing related to the containers file system within the last GB of the block device, I use a simple python script to shift the entire block device forward 1MB, so that we have a space at the beginning to manually insert a partition map. The script starts at the N-1 end of the device (ignoring the last 1MB block), reads each 1MB block then writes it to the block 1 MB forward in the block device. Once the process has completed, whatever was on the block device has now been shifted for by 1MB. The script completes the process by zeroing out the first 1MB block in preparation for the partition map. sha384: 8d. While that block shifting is going on, take this time to go into the ProxMox admin interface and create the new VM. I set the same vcpus and memory settings and copy over the MAC addresses into the Network Device(s) from the xen.cfg file:<\/p>\n\n\n\n 8e. Once the block shifting has completed, we are ready to do the partition\/grub magic. I have two scripts setup with the appropriate commands and they both assume<\/strong> the target device is attached as xvdb<\/em><\/p>\n\n\n\n # .\/mbrfixer-step1.sh<\/p>\n\n\n\n Make Partition Map 8f: grab the last line of output from the terminal and paste it in to execute the grub-install from within the chroot onto the mount of the partition created:<\/p>\n\n\n\n \/# grub-install \/dev\/xvdb;update-grub;exit; 8g: run step2 to detach the partition mapper: <\/p>\n\n\n\n del devmap : xvdb1<\/p>\n<\/blockquote>\n\n\n\n 9. Back to the Xen dom0 environment and detach from MBRFixer the now block-shifted, partition-map-containing, grub-installed, slightly smaller file-system, boot drive:<\/p>\n\n\n\n Now the boot disk is ready to move over into the ProxMox environment.<\/p>\n\n\n\n 10. On the target server, run a netcat listen process that will pipe into the device setup to hold the boot drive contents. (I use LVM):<\/p>\n\n\n\n On the source Xen server, start streaming the data from the newly assimilated boot drive to that listening socket:<\/p>\n\n\n\n Other secondary disk containers for the virtual system can be simply copied over without any manipulation, so a set of nc\/pv operations for those will suffice.<\/p>\n\n\n\n 11. On the ProxMox server, use the Console to launch the VM. <\/p>\n","protected":false},"excerpt":{"rendered":" Background We have been luddites running plain Xen 4.x hypervisor atop of Debian for way too long. Manually launching\/stopping virtual machines, no live migration, no…<\/p>\n
\n\n\n\n
<\/strong>1. Create a new tiny VM on the source Xen server called MBRFixer. Its entire purpose will be to run the partition map installation and GRUB updating commands on the target boot disk within a container that is cleanly separated from the dom0 environment. Explicitly install kpartx, fdisk, e2fsprogs,
parted, python3. Manually download and install the grub-pc-bin package.<\/p>\n\n\n\n
I’ve done conversions from a VM that started out with a single 400GB disk. A secondary drive was added to move all the “data” to (symlinks are your friend) and then the boot disk is trimmed down to 30-40G in size, with subsequent resize2fs and lvresize operations to finally shrink down that boot container. This is not a requirement, just an optimization to reduce downtime.
3. Make a backup of the boot disk. If you utilize LVM, simply use the LVM mirroring feature to have a copy made and prepared while the VM is sill running before you shut it down to begin the assimilation process.<\/p>\n\n\n\n# Assuming the main extents are on \/dev\/md0 and \n# \/dev\/md1 is another PV in the vm1vg1 Volume Group:\n\nlvconvert -y -m 1 vm1vg1\/wwwserver1-boot \/dev\/md1<\/code><\/pre>\n\n\n\nlvconvert -y --name=wwwserver1-boot-GX --splitmirrors 1 vm1vg1\/wwwserver1-boot<\/code><\/pre>\n\n\n\n# shift from grub-xen to grub-pc and add in the QGA.\n# wait until after the vm is assimilated before doing any other upgrades.\n\napt install grub-pc qemu-guest-agent\n\n# explicitly add in virtio modules to the initrd when it gets rebuilt\necho -e \"virtio_pci\\nvirtio_blk\\nvirtio_scsi\\nvirtio_net\" \\\n >> \/etc\/initramfs-tools\/modules<\/code><\/pre>\n\n\n\n
First drop them into fstab with blkid:<\/p>\n\n\n\nblkid >> \/etc\/fstab<\/code><\/pre>\n\n\n\n# \/etc\/fstab: static file system information. \n#\n# <file system> <mount point> \u00a0 <type>\u00a0 <options> \u00a0 \u00a0 \u00a0 <dump>\u00a0 <pass>\n\nproc\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \/proc \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 proc\u00a0 \u00a0 defaults\u00a0 \u00a0 \u00a0 \u00a0 0 \u00a0 \u00a0 \u00a0 0\ndevpts\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \/dev\/pts\u00a0 \u00a0 \u00a0 \u00a0 devpts\u00a0 rw,noexec,nosuid,gid=5,mode=620 0\u00a0 0\n\n\/dev\/xvda1\u00a0 \/\u00a0 ext4\u00a0 noatime,nodiratime,discard,errors=remount-ro 0 1\n\n\/swapfile \u00a0 none \u00a0 swap \u00a0 sw \u00a0 0 \u00a0 0\n\n\/dev\/xvda1: UUID=\"94477222-7621-4ea9-a0a5-cc31ca9feef3<\/strong>\" BLOCK_SIZE=\"4096\" TYPE=\"ext4\"<\/code><\/pre>\n\n\n\n# \/etc\/fstab: static file system information.\n#\n# <file system> <mount point> \u00a0 <type>\u00a0 <options> \u00a0 \u00a0 \u00a0 <dump>\u00a0 <pass>\n\nproc\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \/proc \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 proc\u00a0 \u00a0 defaults\u00a0 \u00a0 \u00a0 \u00a0 0 \u00a0 \u00a0 \u00a0 0\ndevpts\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \/dev\/pts\u00a0 \u00a0 \u00a0 \u00a0 devpts\u00a0 rw,noexec,nosuid,gid=5,mode=620 0\u00a0 0\n\nUUID=94477222-7621-4ea9-a0a5-cc31ca9feef3\u00a0 \/\u00a0 ext4\u00a0 noatime,nodiratime,discard,errors=remount-ro 0 1\n\n\/swapfile \u00a0 none \u00a0 swap \u00a0 sw \u00a0 0 \u00a0 0<\/code><\/pre>\n\n\n\nupdate-initramfs -u -k all\nupdate-grub<\/code><\/pre>\n\n\n\nxl block-attach mbrfixerX2 phy:\/dev\/vm1vg1\/wwwserver1-boot xvdb w<\/code><\/pre>\n\n\n\nxl block-list mbrfixerX2<\/code><\/pre>\n\n\n\n\n
51712 0 \u00a0 76 \u00a0 \u00a0 4 \u00a0 \u00a0 -1 \u00a0 \u00a0 -1 \u00a0 \u00a0 \u00a0 \/local\/domain\/0\/backend\/vbd\/76\/51712
51728<\/strong> 0 \u00a0 76 \u00a0 \u00a0 4 \u00a0 \u00a0 -1 \u00a0 \u00a0 -1 \u00a0 \u00a0 \u00a0 \/local\/domain\/0\/backend\/vbd\/76\/51728<\/p>\n<\/blockquote>\n\n\n\n
\n\n\n\n
8a. fsck the file system:<\/p>\n\n\n\n# fsck.ext4 -f \/dev\/xvdb<\/code><\/pre>\n\n\n\n# resize2fs -p \/dev\/xvdb 39G<\/code><\/pre>\n\n\n\n
block_shifter.py<\/a><\/p>\n\n\n\n
Etb88BHnMd0HsWddhNL2bMvLaQVaTaSLl9zYm7fU8ZdTLydxNNhJnL7rYyp40dYk<\/p>\n\n\n\n# python3 block_shifter.py \/dev\/xvdb\nDevice : \/dev\/xvdb\nSize : 6,442,450,944 bytes (6144 x 1024 KiB chunks)\nShifting chunks 0..6142 ? 1..6143\n\n [100.0%] chunk 0 ? 1 \nZeroing chunk 0 \u2026\nDone.\n<\/code><\/pre>\n\n\n\n#\n# Kernel + memory size\n#\nkernel = \"\/usr\/lib\/grub-xen\/grub-x86_64-xen.bin\"\n\nvcpus = '2'\nmemory = '2048'\npool = 'Pool-U3'\ncpu_weight = 384\n\n#\n# Disk device(s).\ndisk = [ \n 'phy:\/dev\/vm1vg1\/wwwserver1-boot,xvda1,w',\n 'phy:\/dev\/vm1vg1\/wwwserver1-data,xvdb,w',\n ]\n\n# Hostname\nname = 'wwwserver1'\n\n#\n# Networking\n#\nvif = [\n 'mac=00:16:3e:9d:a8:fa,bridge=xenbr10,type=vif,vifname=wwws1-dmz',\n 'mac=00:16:3e:fd:4c:77,bridge=xenbr99,type=vif,vifname=wwws1-pvt',\n ]\n\n\n#\n# Behaviour\n#\non_poweroff = 'destroy'\non_reboot = 'restart'\non_crash = 'restart'\n\n<\/code><\/pre>\n\n\n\n# mbrfixer-step1.sh\n\n#!\/usr\/bin\/env bash\n\n# Install GRUB into the partition\/bootloader space\n\necho \"Make Partition Map\"\n\n## Create partition\/map\nparted \/dev\/xvdb --script \\\n mklabel msdos \\\n mkpart primary ext4 1MiB 100% \\\n set 1 boot on\n\n\n# Expose the partition as a device\nkpartx -av \/dev\/xvdb\n\nPART=\/dev\/mapper\/xvdb1\n\n# Mount and chroot to install GRUB to the MBR\nmount \"$PART\" \/mnt\nmount --bind \/dev \/mnt\/dev\nmount --bind \/proc \/mnt\/proc\nmount --bind \/sys \/mnt\/sys\n\n# dump this into the command line so it can easily\n# be copied\/pasted\/executed in your terminal within the chroot.\necho \"grub-install \/dev\/xvdb;update-grub;exit;\"\n\nchroot \/mnt \/bin\/bash\n<\/code><\/pre>\n\n\n\n\n
add map xvdb1 (254:0): 0 20969472 linear 202:16 2048
grub-install \/dev\/xvdb;update-grub;exit;<\/p>\n<\/blockquote>\n\n\n\ngrub-install \/dev\/xvdb;update-grub;exit;<\/code><\/pre>\n\n\n\n\n
Installing for i386-pc platform.
Installation finished. No error reported.
Generating grub configuration file \u2026
Found linux image: \/boot\/vmlinuz-4.19.0-25-amd64
Found initrd image: \/boot\/initrd.img-4.19.0-25-amd64
Found linux image: \/boot\/vmlinuz-4.19.0-18-amd64
Found initrd image: \/boot\/initrd.img-4.19.0-18-amd64
Found linux image: \/boot\/vmlinuz-4.9.0-17-amd64
Found initrd image: \/boot\/initrd.img-4.9.0-17-amd64
Warning: os-prober will be executed to detect other bootable partitions.
Its output will be used to detect bootable binaries on them and create new boot entries.
Found Debian GNU\/Linux 10 (buster) on \/dev\/xvdb1
done
exit<\/p>\n<\/blockquote>\n\n\n\n# .\/mbrfixer-step2.sh<\/code><\/pre>\n\n\n\n\n
# xl block-detach mbrfixerX2 51728<\/code><\/pre>\n\n\n\n# \/dev\/vm3vg1\/vm-156-disk-0 should point to the disk created as part of ProxMox's Create VM wizard.\n\nnc -l -p 19001|dd bs=4M iflag=fullblock oflag=direct of=\/dev\/vm3vg1\/vm-156-disk-0\n\n# Make sure the logical volume is actually marked active.<\/code><\/pre>\n\n\n\npv -B4M \/dev\/vm1vg1\/wwwserver1-boot |nc -q2 10.0.0.3 19001<\/code><\/pre>\n\n\n\n
\n\n\n\n
It may be necessary to correct network interface device names in the network configuration: eth0<\/strong> becomes ens18<\/strong>, that sort of thing. Correct and reboot.<\/p>\n\n\n\n