Rootstrap configuration guide
- 1 Tester image creation, configuration guide
- 1.1 Overview
- 1.2 How to start kvm session manually
- 1.3 HDA and HDB image files
- 1.4 Steps to prepare HDB image for tester home
- 1.5 Steps to prepare a HDA image for a tester target VM
- 1.5.1 Create empty 6 GB disk image for system
- 1.5.2 Start KVM machine for installing new distro from DVD image to "hda"
- 1.5.3 From desktop, connect to VNC display with VNC viewer
- 1.5.4 After install, reboot, and start KVM without DVD
- 1.5.5 Before wiping repos config: Install utils that are used inside tester VM
- 1.5.6 Clear repositories config
- 1.5.7 Common to all: Add user build without homedir creation, uid=777 needs to be same as jenkins in Tester
- 1.5.8 Enable automatic start of /home/build/run
- 1.5.9 Add second image mount as /home in /etc/fstab
- 1.5.10 Shutdown VM, boot with two disks to verify /home gets mounted from sdb1
- 1.5.11 In hdb seed image (only needed once, hdb is re-used)
- 1.5.12 Shut down KVM, rename hda as seed
- 1.5.13 Useful offsets of rootfs in HDA image for loop mounting
- 1.5.14 Additional config and fixes in KVM Tester images
- 18.104.22.168 Remove other mailers, install sendmail (old)
- 22.214.171.124 Ubuntu: log.c assert panic in
- 126.96.36.199 OpenSUSE 12.1 and 12.2: loop module added to /etc for loading on boot
- 188.8.131.52 Fedora-19,20: increase loop devices max count
- 184.108.40.206 Fedora 19,20; remove some systemd-started services and packages we do not use (optional)
- 220.127.116.11 Fedora 19,20 remove fingerprint packages as the service always fails to start:
Tester image creation, configuration guide
Testing is performed in a KVM Virtual machine. A running VM has two disk images mounted:
- HDA contains system. There is one HDA per target system.
- HDB contains /home with project code and run script. All targets share one image.
To prepare a tester session, the Jenkins job loop-mounts HDB, copies tester scripts to HDB, un-mounts HDB, starts kvm-qemu. In the VM session, a script /home/build/run is started after bootup using a crontab entry. Testin script output redirects to tty of VM, comes to Jenkins console. Tester script performs shutdown after testing is complete. Control returns to Jenkins job which loop-mounts HDB image again and retrieves test reports. At the end of session log there will be tail of dmesg and syslog.
How to start kvm session manually
Sometimes it is needed to run/update seed images manually. For that, use similar cmd what automated tester uses, but remember to replace MAC address and VNC display number with some unique values, otherwise manual session may conflict with automated session if running in same host (I usually run manual sessions in debug worker tworker-179). Note about VNC number: starting with version 1.22.1 of tester scripts, vnc number is picked initially as Jenkins slot number, but if such a socket (5900+vncnum) is already in use, another number +20 is tried. That ensures vnc socket does not overlap and existing qemu process will not block a new one.
cp kvm-seed-hda-Fedora_19-x86_64-debug hda2 chmod 644 hda2 tar SxfO - < kvm-seed-hdb.1G.tar > hdb2 qemu-kvm -name f20-64 -cpu core2duo -M pc -m 2048 -hda hda2 -hdb hdb2 -net nic,macaddr=52:54:78:87:86:85 -net user -vnc :43
HDA and HDB image files
HDA and HDB are sparse files. User should take care of not losing sparseness in copy and transfer. cp preserves sparseness, but rsync by default loses sparseness unless -S option is used.
For new target, always start with creation of new sparse 6G image. Attempt to re-use of previous images will make spareseness worse.
Steps to prepare HDB image for tester home
Note: Only needed once, as all targets share same hdb.
qemu-img create -f raw hdb 20G
Initialize a new hdb
In KVM session: create partition (can use parted), format eft4 FS on it: (assuming the HDB disk is /dev/sdb)
parted /dev/sdb mklabel msdos mkpart primary 1 -1 q mkfs.ext4 /dev/sdb1 mount /dev/sdb1 /home
Steps to prepare a HDA image for a tester target VM
Create empty 6 GB disk image for system
qemu-img create -f raw hda 6G
Start KVM machine for installing new distro from DVD image to "hda"
Note: install system tries to guess needed swap size based on RAM size (at least Ubuntu does this) so do not use large RAM for installation, because we dont want to spend too big part of 6GB for swap.
qemu-kvm -name Fedora16 -M pc -m 1024 -boot d -hda hda -drive file=Fedora-16-x86_64-DVD.iso,media=cdrom -vnc :22
From desktop, connect to VNC display with VNC viewer
examples: - xfvnc, using Jenkins worker IP and display as :22 - gvncviewer 10.237.71.179:22
Notes for DVD and distro type selection:
- Fedora: select Base Environment=Minimal Install, Add-Ons=Standard, review/modify disk partitions, partition scheme:Standard
- Fedora 22/32bit: can not start X server, I used text based install
- Ubuntu: server-ARCH.iso (not desktop, it has no option for small system); use F4: "minimal VM". Later in install: components: Base system, OpenSSH server.
- Fedora-18,19, CentOS_6: resize partitions, root:4616 MB swap:1024MB (or default will make big swap and small rootfs)
- openSUSE: set screen resolution to 800x600, using F3; Desktop Selection: Other-Minimal Server Selection, Installation Settings: revert Firewall (disabled) and ssh (enabled)
After install, reboot, and start KVM without DVD
- Ubuntu: login as "ubuntu" (was created duing installation), sudo bash, passwd -- to set root passwd and being able to login as root
- Ubuntu 12.xx,13.xx: service whoopsie stop; apt-get remove whoopsie libwhoopsie0
Before wiping repos config: Install utils that are used inside tester VM
Tester VM will not have repos configured, but after installation, repos config is needed to perform 1st update and to install utils. Point to local mirror, or config proxy, etc.
yum update yum install which python-nose python-coverage python-mock pylint make Fedora 22 note: yum is replaced with dnf: dnf install which python-nose python-coverage python-mock pylint make
yum update Centos_6: yum install openssh-clients net-tools Centos_7: yum install net-tools yum install python-nose python-coverage python-mock pylint make
apt-get update apt-get upgrade apt-get install debianutils python-nose python-coverage python-mock pylint make
zypper ref zypper up #Note: zypper proposes rm of base-conflicts, so no need to type it zypper rm patterns-openSUSE-minimal_base-conflicts # (otherwise does not let install python-pylint) zypper in util-linux python-nose python-coverage python-mock pylint make openSUSE 13.2: zypper in util-linux python-nose python-coverage python-mock python3-pylint make e2fsprogs
openSUSE: can switch graphical boot screen off: yast -- System-Boot Loader -- Options
Clear repositories config
Starting with version 1.35.0 of tools-testing package, tester VMs do not have any repo config in them. Repo config is created dynamically by tester scripts, based on /etc/tools-tester.d/base-repos-ENV.conf which is installed on all worker hosts from package tools-testing-settings-ENV Jobs that use HDA in snapshot mode (regular tester jobs), can leave created repo config there, it will not be saved. Job that should leave usable state behind, (like packages update job), needs to clear repo config before performing shutdown.
Cleaning of repo config after creation of new VM from distro:
- Fedora, CentOS
rm -f /etc/yum.repos.d/*
Same as Fedora. Note that CentOS repo config uses EPEL from Fedora repos.
CentOS_6 wants to rename ethX with every next unique MAC we use. To stop it: /etc/udev/rules.d/70-persistent-net.rules: modify eth0 line to be:
SUBSYSTEM="net", ACTION="add", DRIVERS=="?*", KERNELS=="0000:00:03.0", NAME="eth0"
Note: KERNELS value comes from pci address, check: ls -l /sys/class/net
remove /lib/udev/rules.d/75-persistent-net-generator.rules which would overwrite above 70-persistent-net.rules file
CentOS_7: no need to change udev stuff
- Debian, Ubuntu
rm -f /etc/apt/sources.list
rm -f $/etc/zypp/repos.d/*
Common to all: Add user build without homedir creation, uid=777 needs to be same as jenkins in Tester
useradd -M -u 777 build
Enable automatic start of /home/build/run
During history of otctools testing, several methods of starting testing have been tried. It is problematic to keep multiple methods (systemd vs init-scripts vs ubuntu config style). Under heavy load, problems arise because of startup race and order of events:
- slow startup of DHCP iface causes testing fail if tester starts too early;
if-up hooks were tried, but slow mount of /home directory still caused failures;
Finally, current unified method seems to work on all targets. There is entry in root crontab:
@reboot /root/runtester > /dev/ttyS0 2>&1
/root/runtester checks for tester start conditions:
- iface up and usable i.e. default route is set to 10.0.2.2 - /home is mounted
If these do not come true during 300s, we reboot for retry, and there is limit set to reboot count. Reboot in kvm session will restart VM without returning control to Jenkins.
/root/runtester: This script is now part of tools-testing git repo.
Add second image mount as /home in /etc/fstab
/dev/sdb1 /home ext4 defaults 1 1
Shutdown VM, boot with two disks to verify /home gets mounted from sdb1
qemu-kvm -name "Ubuntu-11" -M pc -m 2048 -hda hda -hdb hdb -vnc :0
In hdb seed image (only needed once, hdb is re-used)
- Create git identity for user build: /home/build/.gitconfig:
[user] email = firstname.lastname@example.org name = Jenkins Worker
- Create /home/build/reports directory owned by user build.
Shut down KVM, rename hda as seed
Note that images mgmt happens on Tester-179 and seed images are named as -debug. Deploy scripts will handle renaming and replicating to other workers.
mv hda kvm-seed-hda-$REPO-$ARCH-debug
Example name: kvm-seed-hda-Centos_6-x86_64-debug
hdb is shared by all, so this is only needed for first time:
mv hdb kvm-seed-hdb
For trials, and any sessions run manually, should copy kvm-hdb-seed to hdb and discard it later.
Useful offsets of rootfs in HDA image for loop mounting
In many cases, start of VM is not needed for changes in rootfs, loop mount can be used instead. Here are notes for loop mounting.
# CentOS_6: cant mount, uses LVM # Debian 7, Ubuntu 12.04, 12.10, 13.10 : rootfs starts 2048, *512=1048576 # mount -o loop,offset=1048576 -t ext4 -v hda tmpmount/ # Fedora 19: rootfs partition starts 3074048, *512=1573912576 # mount -o loop,offset=1573912576 -t ext4 -v hda tmpmount/ # Fedora 20: rootfs partition starts 2283520, *512=1169162240 # mount -o loop,offset=1169162240 -t ext4 -v hda tmpmount/ # openSUSE 12.1-x86_64, 12.2-x86_64, 12.3 : rootfs partition starts 1044480, *512=534773760 # mount -o loop,offset=534773760 -t ext4 -v hda tmpmount/ # openSUSE 12.1-i586: rootfs partition starts 2072576, *512=1061158912 # mount -o loop,offset=1061158912 -t ext4 -v hda tmpmount/ # openSUSE 12.2-i586 : rootfs partition starts 1814528, *512=929038336 # mount -o loop,offset=929038336 -t ext4 -v hda tmpmount/ # openSUSE 13.1-i586: rootfs partition starts 2072576, *512=1061158912 # mount -o loop,offset=1061158912 -t ext4 -v hda tmpmount/ # openSUSE 13.1-x86_64: rootfs partition starts 2056192, *512=1052770304 # mount -o loop,offset=1052770304 -t ext4 -v hda tmpmount/
Additional config and fixes in KVM Tester images
Remove other mailers, install sendmail (old)
In several kvm seed images we had to remove postfix pkg, because cats-* packages require sendmail and this installation conflict is not automatically handled. Check that no other mailers like postfix, exim are installed, safer is to install sendmail in seed image.
Ubuntu: log.c assert panic in
August 2013: Ubuntu images kernel cmdline got appended "--no-log" because sometimes there was panic when booting up:
- assertion failed in log_clear_unflushed: log->io
vi /etc/default/grub # locate line: GRUB_CMDLINE_LINUX_DEFAULT and add --no-log to its value update-grub
OpenSUSE 12.1 and 12.2: loop module added to /etc for loading on boot
2013-11-19: mic testing failed with GPT parted error, reason seemed to be that kernel loop module was not loaded after boot. Solution: add file /etc/module-load.d/loop.conf with single line "loop" in it.
Fedora-19,20: increase loop devices max count
Added /etc/modprobe.d/loop.conf with single line:
options loop max_loop=32
Fedora 19,20; remove some systemd-started services and packages we do not use (optional)
yum remove mdmonitor nfs remote-fs smartd mcelog rngd
Fedora 19,20 remove fingerprint packages as the service always fails to start:
yum remove fprintd fprintd-pam libfprint