User:Az990tony/edublog-beta-sw: Difference between revisions

From OLPC
Jump to navigation Jump to search
 
(109 intermediate revisions by 3 users not shown)
Line 7: Line 7:
Based on the hardware, the following devices are available:
Based on the hardware, the following devices are available:


'''CD/DVD Burner''' /dev/hda (/dev/cdrom, /dev/dvd, /dev/dvdrw, and /dev/sr0)
; CD/DVD Burner :
/dev/hda (it is in the Primary/Master position on the first EIDE cable)


'''SATA drive 1''' /dev/sda (first drive it finds)
/dev/cdrom (many operating systems use this)


'''SATA drive 2''' /dev/sdb (second drive, if connected)
/dev/dvd (and this)


'''USB keys or external drives''' /dev/sdc1 (/dev/sdd1, /dev/sde1 or /dev/sdf1)
/dev/dvdrw (and this)


'''Ethernet NIC (motherboard)''' eth0 (external WAN)
; SATA disks :


'''Ethernet NIC (PCI cards)''' eth1, eth2 (internal LAN)
/dev/sda (this is drive 1)

/dev/sdb (this is drive 2)

Note that /dev/sda1, sda2, sda3 and sda4 can be bootable primary partitions. Typically,
sda4 is configured as an "extended partition" which allows subsets to be defined as logical partitions sda5, sda6, and so on.

; USB keys or external drives :

/dev/sdf for the drive itself

/dev/sdf1 (for the VFAT file system usually on it)

; Ethernet Network Interface cards :

eth0, eth1 and eth2 seemed to be assigned to the one motherboard connector and two
PCI cards differently between operating systems. I will need to track down the MAC
addresses to keep them straight.


= RAID and Logical Volume Manager =
= RAID and Logical Volume Manager =
Line 45: Line 27:
== Redundant Array of Inexpensive Disks (RAID) ==
== Redundant Array of Inexpensive Disks (RAID) ==


We will focus exclusively on RAID1 mirroring between two identical size disks:
Configurations include:
; RAID 0 :
Stripes data across for better performance. Two 160GB drives would look like a fast 320GB drive, but if you lose any one drive, you have lost everything
; RAID 1 :
; RAID 1 :
Mirrors the data across drives. Multiple copies means you have better protection, so if you lose any one drive, the rest of the system can still run. Two 160GB drives would look like a single 160GB drive. Performance is the same or slightly slower than a single drive.
Mirrors the data across drives. Multiple copies means you have better protection, so if you lose any one drive, the rest of the system can still run. Two 160GB drives would look like a single 160GB drive. Performance is the same or slightly slower than a single drive.

; RAID 5 :
Other configurations are explained in:
Stripes with parity for added protection. One drive's worth is set aside for parity, so three 160GB drives would hold 320GB of data; four 160GB drives would hold 480GB of data. Since we have only two drives, this option does not apply to us. Performance is faster for read, but can be slower for writes, depending on how the parity methods are implemented.
[[http://tldp.org/HOWTO/Software-RAID-0.4x-HOWTO-2.html What is RAID?]]
; RAID 10 :
Striped and Mirrored. Sometimes referred to as RAID 0+1 or RAID 1+0, this combines RAID 1 with RAID 0. Data is striped across disk 1 and disk 2, disk 3 is a mirror of disk 1 stripe, and disk 4 is mirror of disk 2 stripe. Thus four 160GB drives holds 320GB of data. If you lose any one drive, the mirrored copy is still available, so the system will be able to continue to run. Performance is same or slightly slower than a 2-drive RAID 0 configuration.
; JBOD :
JBOD stands for "Just a Bunch of Disks", a term meaning there is no RAID for protection or performance improvement. This refers to deployments where each disk drive is independent.


RAID-1 Considerations:
RAID-1 Considerations:
* Can mirror the entire drive ( /dev/sda <--> /dev/sdb ) or individual same-sized partitions ( /dev/sda1 <--> /dev/sdb1 )
* Can mirror the entire drive ( /dev/sda <--> /dev/sdb ) or individual same-sized partitions ( /dev/sda1 <--> /dev/sdb1 )
* Can mirror swap space. This appears to be preferred best practice over just having swap space on separate disks.
* Can mirror swap space. This appears to be preferred best practice over just having swap space on separate disks.
* Mirrored drives appear as (/dev/md0, /dev/md1) or (/dev/dm-0, /dev/dm1)
* Mirrored drives appear as:
** Software RAID (/dev/md0, /dev/md1)
** or, Hardware-assisted BIOS "fakeraid" (/dev/dm-0, /dev/dm1) which is not supported on the hardware I have for this project
* In general, a mirrored partition can be used wherever a native partition can be used
* In general, a mirrored partition can be used wherever a native partition can be used
* Can be used as ''physical volumes'' for Logical Volume Manager
* Can be used as ''physical volumes'' for Logical Volume Manager
* /boot directory can be on RAID1, but not be on RAID0, RAID5, RAID10 or any other RAID configuration.


RAID rebuild:
RAID rebuild:
Line 69: Line 49:
can be used to rebuild.
can be used to rebuild.


Place the surviving drive as "drive 1", remove the bad drive from the system.
Place the surviving drive as "drive 1", disconnect the bad drive from the system.
* boot up
* boot up
* indicate that drive 2 has failed
* indicate that drive 2 has failed
Line 78: Line 58:
* copy over the MBR from "drive 1" using "dd" utility
* copy over the MBR from "drive 1" using "dd" utility
* add the new partitions to /dev/md0, /dev/md1, etc. (the data will be copied over)
* add the new partitions to /dev/md0, /dev/md1, etc. (the data will be copied over)

For more information, see:
[[http://tldp.org/HOWTO/Software-RAID-HOWTO.html Linux Software RAID HOWTO]].


== Logical Volume Manager (LVM2) ==
== Logical Volume Manager (LVM2) ==


Logical Volume Manager allows a level of storage virtualization between the physical hardware and the operating system and applications.
Logical Volume Manager allows a level of storage virtualization between the physical hardware and the operating system and applications. There are two formats, LVM (older) and LVM2 (newer). We will use LVM2.


; Physical Volumes :
; Physical Volumes :
Line 100: Line 83:
* A volume group can contain physical volumes of different sizes
* A volume group can contain physical volumes of different sizes
* Linux cannot boot from a Logical Volume, this can be solved by having a separate /boot directory that is not part of the Logical Volume, and the rest of root (/) in the logical volume.
* Linux cannot boot from a Logical Volume, this can be solved by having a separate /boot directory that is not part of the Logical Volume, and the rest of root (/) in the logical volume.
* Logical volumes cannot be resized using the "gparted" tool
* Physical volumes cannot be resized using the "gparted" tool
* Linux swap space can be defined onto logical volume
* A physical volume can be RAID1 mirror pair
* The "partimage" program can backup logical volumes, but can only restore these to non-logical volumes, such as a primary partition, extended partition, or RAID1 mirror pair.

For more information, HOWTOforge.com has a nice
[[http://www.howtoforge.com/linux_lvm Beginner's Guide To LVM]].


= Disk Layout =
= Disk Layout =
Line 116: Line 105:
RAID+LVM configuration.
RAID+LVM configuration.


1. Simple JBOD (no RAID, no LVM)
1. Simple JBOD (no RAID, no LVM)
<tt>
<tt>
/dev/sda /dev/sdb
/dev/sda /dev/sdb
Line 200: Line 189:


The following sizes can be used:
The following sizes can be used:
; rescue -- 4GB :
This would be enough to have a basic operating system with
software utilities, and include manuals, ISO images of CDs, and so on. This could
fit on a single DVD or several CDs for recovery purposes.

; boot -- 680 MB :
; boot -- 680 MB :
This could fit on a single CD for recovery
This could fit on a single CD for recovery
Line 210: Line 194:
kernels to select from during the boot process. Most OS only need 100MB, so this
kernels to select from during the boot process. Most OS only need 100MB, so this
should be plenty.
should be plenty.

; rescue -- 4GB :
This would be enough to have a basic operating system with
software utilities, and include manuals, ISO images of CDs, and so on. This could
fit on a single DVD or several CDs for recovery purposes.


; scratch -- 30GB each :
; scratch -- 30GB each :
Line 219: Line 208:
This can be swap during the initial installs.
This can be swap during the initial installs.


; base1, base2 -- rest of the disk (60GB of so each) :
; base1, base2, base3 -- rest of the disk (40GB of so each) :
This will be the /dev/sda4 extended partition with /dev/sda6 and /dev/sda7
This will be the /dev/sda4 extended partition carved into smaller logical partitions
as the extended partition as physical volumes for LVM.
that can be defined as physical volumes or RAID-1 mirrored pairs.


== Initial Drive Format ==
== Creating the Disk Partition table ==


For some reason, with fresh disks that have no format or partition, most 32-bit LiveCD
For some reason, with fresh disks that have no format or partition, most 32-bit LiveCD
were unable to boot. To solve this, I booted a 64-bit AMD64-based LiveCD (Fedora 8 in my case), created the partitions and then could use a 32-bit LiveCD for the rest of the install. SystemRescueCD is also capable of booting in either 64-bit or 32-bit mode, use "boot: rescue64" to boot into 64-bit mode.
were unable to boot. To solve this, I booted a 64-bit AMD64-based LiveCD (Fedora 8 in my case), created the partitions and then could use a 32-bit LiveCD for the rest of the install.

[http://sysresccd.org/Main_Page SysRescCD 1.0.3] is also capable of booting in either 64-bit or 32-bit mode, use "boot: rescue64" to boot into 64-bit mode. If you received the disk drive from someone else, and are concerned about remnants of
any past data or operating systems, you can use the following command to clobber all data with random data. At 35 MB/sec, this would take about 90 minutes for each 160GB drive.

<tt>
dd if=/dev/urandom of=/dev/sda
</tt>


Boot from the "SysRescCD" and use "fdisk" to configure /dev/sda. The final layout
Boot from the "SysRescCD" and use "fdisk" to configure /dev/sda. The final layout
Line 240: Line 236:
/dev/sda1 * 1 80 642568+ 83 Linux
/dev/sda1 * 1 80 642568+ 83 Linux
/dev/sda2 81 581 4024282+ 83 Linux
/dev/sda2 81 581 4024282+ 83 Linux
/dev/sda3 582 4229 29302560 83 Linux
/dev/sda4 4230 19457 122318910 5 Extended
/dev/sda4 4230 19457 122318910 5 Extended
/dev/sda5 4230 4497 2152678+ 82 Linux swap / Solaris
/dev/sda5 4230 4497 2152678+ 82 Linux swap / Solaris
/dev/sda6 4498 19457 120246493+ 83 Linux
</tt>
</tt>


Note that all of the partitions are of type "83" (native Linux) except
Note that all of the partitions are of type "83" (native Linux) except
/dev/sda5 which is type "82" (Linux swap). You will need to "reboot" for
/dev/sda5 which is type "82" (Linux swap).
these changes to the partition table to take effect.

To save time, you can configure one drive /dev/sda then copy over the configuration
to the second drive as follows:

<tt>
sfdisk -d /dev/sda | sfdisk /dev/sdb --force
</tt>


= Operating System =
= Operating System =
Line 261: Line 247:
The team wants to develop and test the EduBlog software on a variety of
The team wants to develop and test the EduBlog software on a variety of
platforms. The operating systems considered were:
platforms. The operating systems considered were:

; XS School Server :
We're using the XS-163 (the latest level at this time).
This is a customized version of Fedora 7
with special settings to run as a school server. The CD runs unattended, works
when there is only a single SATA drive installed, and wipes away any previous
partition table to form a new partition table.


; Fedora 7 :
; Fedora 7 :
Line 267: Line 260:
plan was to install Fedora 7 natively. Fedora 7 could also be used graphically
plan was to install Fedora 7 natively. Fedora 7 could also be used graphically
by Glen on premises to help with problem determination.
by Glen on premises to help with problem determination.

; XS School Server :
The XS-163 is the latest level. This is a customized version of Fedora 7
with special settings to run as a school server. Getting this installed into
a multi-boot environment will require "squashfs surgery" to change the
anaconda-ks.cfg kickstart configuration file.


; Debian 4.0-r3 Etch :
; Debian 4.0-r3 Etch :
Line 288: Line 275:
disk drive.
disk drive.


[[ http://sysresccd.org/Main_Page | SysRescCD 1.0.3 ]] supports ''sshd'' so that with the right network setup, someone could effect software repairs remotely.
[http://sysresccd.org/Main_Page SysRescCD 1.0.3] supports ''sshd'' so that with the right network setup, someone could effect software repairs remotely.
It has a variety of utilities, including "partimage" that can be used to backup
It has a variety of utilities, including "partimage" that can be used to backup
individual partitions or logical volumes. Installing on hard disk is a matter of
individual partitions or logical volumes. Installing on hard disk is a matter of
Line 300: Line 287:
borrowing from the concept of building a large house, apartment or office structure.
borrowing from the concept of building a large house, apartment or office structure.


There are two ways to get to LVM+RAID:
Have only a single SATA disk connected. This will eliminate the OS expecting to use
# standard &rarr; RAID &rarr; LVM+RAID
RAID and complicating the configuration. With a single drive, all of the OS should
# standard &rarr; LVM &rarr; LVM+RAID
install nice and clean.


Having tested out both ways, I have found the second method simpler.
=== Preparation ===

Here is the basic sequence:
# Prepare the first OS (XS-163 School Server)
# Prepare the second OS (Fedora 7)
# Prepare the third OS (Debian 4 Etch)
# Re-define partition tables for LVM
# Devine LVM2 pv, vg and lv and move data into the appropriate logical volume
# Assemble the individual OS images into a "multi-boot" environment
# Connect the second SATA drive
# Create RAID pairs of each partition you want protected

== Preparation ==

Linux installers make a few general assumptions
* You are a novice user willing to accept a standard disk layour (true for most people)
* You are installing only a single OS on the system (true for most)
* It will take up all the disk space available (true for most)
* That you won't mind that some partitions are LVM2 Logical Volumes (true for most)
* If you have two identical-size drives, then you might want Software RAID capability, creating crazy devices like /dev/mapper/pdc_bbbhp1 instead of /dev/sda1. Unless you specify RAID during the initial interactive setup, it will create these crazy devices, but not actually put them into RAID mode for you. The default is "No RAID, but define crazy device names anyways" (what ?)

If you have a normal Linux distribution, you can go through an interactive menu and choose which options you like. However, the XS-163.iso for XS School Server is non-interactive (batch, unattended, hands off) installation that assumes "standard" device names.

To get around these assumptions, we will detach the red SATA signal cable from the motherboard of the second disk drive (you can leave the power cable attached to the drive itself is fine).
We will wipe the system clean three times. Each time we will install a
fresh operating system image as if it were the only operating system we are going to use, and then backup the appropriate files to external media using "partimage" utility.

Step 1. Wipe the System clean


Step 1.
For each OS, wipe the system clean. The following command will zero out the first
For each OS, wipe the system clean. The following command will zero out the first
cylinder of your hard disk eliminating the master boot record and partition table.
cylinder of your hard disk eliminating the master boot record and partition table.


<tt>
<tt>
[root ~]# dd if=/dev/zero of=/dev/sda bs=512 count=1
[root ~]# dd if=/dev/zero of=/dev/sda bs=512 count=16
</tt>
</tt>


Step 2.
Step 2. Install from CD

Use "fdisk" to create a simple partition layout, something like:
Run the installation from CD. The XS-163 will carve up the disk as follows:


<tt>
<tt>
/dev/sda
/dev/sda1 /boot -- 100MB
MBR (boot loader / disk layout )
/dev/sda2 / -- (root) 8GB (8192MB)
/dev/sda1 as /boot (choose 500MB)
/dev/sda5 pv -- physical volume for LVM2, rest of disk
/dev/sda2 as / (choose 30GB)
VolGroup00 -- volume group on physical volume
/dev/sda3 as swap (choose 2GB)
LogVol00 /library
LogVol01 swap
</tt>
</tt>


If you are installing Fedora 7, Debian 4, etc. wipe out all partitions, and choose
Step 3.
"create custom setup" and define the following simple setup. The rest of the drive capacity will be unused, but it is important to make the initial /(root) directory as small as possible.
Format each partition. In this case "-L" sets the file system label,
and "-j" indicates "ext3 file system" format. I will call the boot partition
"BOOT", the root partition "FEDORA7" and the swap space "SWAP1".


<tt>
<tt>
[root ~]# mke2fs -L BOOT -j /dev/sda1
/dev/sda1 /boot -- 200MB
/dev/sda2 / -- (root) 8GB (8192MB)
[root ~]# mke2fs -L FEDORA7 -j /dev/sda2
/dev/sda3 swap
[root ~]# mkswap -L SWAP1 /dev/sda3
</tt>
</tt>


Step 3. Capture initial list of application packages
Step 4.
Run the installation from CD. Typically, these have interactive options that
allow you to keep the existing partitions and use them "as is". Sometimes,
an installation CD will blow away the entire drive and build its own,
or blow away the entire drive and let you define new partitions. If this happens,
try to use partitions similar to the above.


Before you change anything or install any packages, do the following, which will
Installing XS School Server, you may need to perform [[ User:Az990tony/squashfs-surgery | "squashfs surgery" ]] to create a modified installation CD.
document all of the base RPM packages on your XS163 or Fedora 7 system. That way you
can compare any new packages that were installed afterwards and see the differences
using ''diff'' command. For Debian 4, use the ''apt-get'' utility instead.
<tt>
[root ~]# cd /root
[root ~]# mkdir /boot/details
[root ~]# rpm -qa | sort > /boot/details/rpm-xs163.txt
</tt>


Step 4. Create partition images
Step 5.

After you update the RPMs and customize it however you like, we will capture some
We will now backup this OS image. These ''partition images'' will then
information that will be helpful later.
be used to re-assemble the final system.
We can use this procedure for [[Backup-and-Restore-Using-SysRescueCD]].

Here is what you end up with:

For each OS you will have:
* master.mbr -- this is the Master Boot Record (first 446 bytes of each disk)
* sfdisk-d.txt -- this is the disk layout, human-and-computer readable
* grubmenu.txt -- this has the initial boot parameters (/boot/grub/menu.lst --> grub.conf)
* etcfstab.txt -- this is the list of mount points (/etc/fstab file)
* sda1boot.img.000 -- this is partition image of the /boot directory
* sda2base.img.000 -- this is first piece of the partition image of the / (root) directory
* sda2base.img.001 -- (the files are 692MB each, small enough so they can be burned onto CDrom, uploaded to an FTP site, sent over using "yousendit.com", or fit on a FAT32 file system.)

== Re-Define the Partion Tables ==

We'll choose a configuration similar to the left half of "6. RAID and LVM with rescue and scratch partitions" list above in the section
[[Possible_Disk_Layout_configurations]].


Capture all of the RPM packages installed. This can only be done from
a running system.
<tt>
<tt>
/dev/sda
[root ~]# cd /root
MBR1 (boot loader/layout)
[root ~]# rpm -qa | sort > rpm-qa.prep
/dev/sda1 as /boot
/dev/sda2 as /rescue (/ for our rescue version of Fedora7)
/dev/sda3 as /scratch1
/dev/sda4 as extended partition holding sda5..sda8
_
/dev/sda5 as swap
/dev/sda6 as physical vol
/dev/sda7 as / (for debian 4) and later physical vol
/dev/sda8 as / (for XS 163) and later physical vol
</tt>
</tt>


The /boot will contain the kernel images of all the operating systems, as well as the
Capture the "menu.lst" in /boot/grub. We'll copy it over to the /root directory. This version has ''timeout'' set to 30 seconds, commented out ''hiddenmenu'' and changed root=/dev/mapper/pdc_bbbhp2 to root=LABEL=FEDORA7. You will want to get rid of every /dev/mapper you find as they will get in the way in later steps. Use ''e2label'' to change the label of the / directory if you need to. Each OS root system should have a unique name.
multi-boot /boot/grub directory. This will allow you to choose the different OS from
a menu.

I chose 4GB to hold a set of rescue files, and then later decided to install Debian 4,
it fits in 4GB! I create three equal-sized 10GB partitions, one for scratch (temp) space,
one for the root directory of Fedora 7, and one for root directory of XS 163 School Server.
Finally, I divided the rest of the volume into three 40GB pieces which can be used for
physical volumes of an LVM Volume Group.

Here is what the start/end cylinder counts look like using "sfdisk -l /dev/sda":


<tt>
<tt>
[root ~]# cd /root
[root ~] sfdisk -l /dev/sda
Disk /dev/sda: 19457 cylinders, 255 heads, 63 sectors/track
[root ~]# cp /boot/grub/menu.lst boot-grub-menu.prep
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0
[root ~]# cat boot-grub-menu.prep

# grub.conf generated by anaconda
Device Boot Start End #cyls #blocks Id System
/dev/sda1 * 0+ 79 80- 642568+ 83 Linux
/dev/sda2 * 80 638 559 4490167+ 83 Linux
/dev/sda3 639 4241 3603 28941097+ 83 Linux
/dev/sda4 4242 19456 15215 122214487+ 5 Extended
/dev/sda5 4242+ 4504 263- 2112516 83 Linux
/dev/sda6 4505+ 9488 4984- 40033948+ 8e Linux LVM
/dev/sda7 9489+ 14472 4984- 40033948+ 83 Linux
/dev/sda8 14473+ 19456 4984- 40033948+ 83 Linux
</tt>

Since there is so much work setting this up, it might be good to save it to a file.
This can be done using the following command:

<tt>
[root ~]# sfdisk -d /dev/sda > sfdisk-d.txt
</tt>

This is what the file looks like. Not that "size" is counting in terms of 512 byte
sectors. So /dev/sda1 is 512 * 1285137 or 657990144 bytes (658 MB).

<tt>
[root ~] sfdisk -d /dev/sda
# partition table of /dev/sda
unit: sectors
/dev/sda1 : start= 63, size= 1285137, Id=83, bootable
/dev/sda2 : start= 1285200, size= 8980335, Id=83, bootable
/dev/sda3 : start= 10265535, size= 57882195, Id=83
/dev/sda4 : start= 68147730, size=244428975, Id= 5
/dev/sda5 : start= 68147793, size= 4225032, Id=83
/dev/sda6 : start= 72372888, size= 80067897, Id=8e
/dev/sda7 : start=152440848, size= 80067897, Id=83
/dev/sda8 : start=232508808, size= 80067897, Id=83
</tt>

To restore, use "cat sfdisk-d | sfdisk /dev/sda" command. You can
cut and paste the above and try it yourself. Note, this will clobber any previous
partition table and all data may be lost.

== Configure LVM2 for Logical Volumes ==

Now that we have multi-boot and RAID1 mirroring in place, we can create Logical Volumes.
This section was inspired by the following tutorials:
* [http://www.linuxdevcenter.com/pub/a/linux/2006/04/27/managing-disk-space-with-lvm.html Managing disk space with LVM]
* [http://www.howtoforge.com/setting-up-lvm-on-top-of-software-raid1-rhel-fedora Setting up LVM on top of Software RAID - Fedora]


=== Step 1. Create VolGroup00 for Library and Swap ===

Let's create some Logical Volumes for the XS school server.
First, we'll define three mirrored devices to serve as physical space.
Use "fdisk /dev/sda" Make sure that /dev/sda8 is type "8e" (LVM). Note that this Volume
Group will not be RAID1-mirrored until a later step.

<tt>
[root ~]# fdisk /dev/sda
</tt>

Define the volume group. We use "pvcreate" to format the disk, then we'll call our new volume group "VolGroup00".

<tt>
[root ~]# pvcreate /dev/sda8
[root ~]# vgcreate VolGroup00 /dev/sda8
[root ~]# vgdisplay
</tt>

Now let's create the logical volumes. The "--extents" parameter sets the size in number of 4MB pieces, so here "library" is a tiny volume (we can always expand it later!!!) The "--size" allows you to specify the volume group in M(MB) or G(GB). We make the swap space 2G to match the amount of memory we have installed on our system.

<tt>
[root ~]# lvcreate VolGroup00 --name library --extents 1
[root ~]# lvcreate VolGroup00 --name swap --size 2G
[root ~]# lvdisplay
</tt>

Now, to activate these into the system. For Library, the initial XS CD created it empty
except for a lone "users" directory. We can recreate that here. The echo with ">>" will append this one line to the end of your existing /etc/fstab file.

<tt>
[root ~]# mkfs.ext3 -L LIBRARY /dev/VolGroup00/LogVol00
[root ~]# mkdir /library
[root ~]# mount /dev/VolGroup00/LogVol00 /library
[root ~]# mkdir /library/users
[root ~]# cp /etc/fstab /etc/fstab.bak
[root ~]# echo "LABEL=LIBRARY /library ext3 defaults 0 0" >> /etc/fstab
</tt>

For swap space, we can leave SWAP1 alone, and create SWAP2, activate SWAP2, and de-activate SWAP1. Update the /etc/fstab file so that we always use SWAP2 from now on. You can use "vi" editor to change SWAP1 to SWAP2 or use the "sed" command to do it for you.

<tt>
[root ~]# mkswap -L SWAP2 /dev/VolGroup00/LogVol01
[root ~]# swapon LABEL=SWAP2
[root ~]# swapoff LABEL=SWAP1
[root ~]# cp /etc/fstab /etc/fstab.bak
[root ~]# sed 's/LABEL=SWAP1/LABEL=SWAP2/g' /etc/fstab.bak > /etc/fstab
</tt>

=== Step 2. Create logical volumes for each root ===

Now we will create 3GB logical volumes to hold the root directory of
each OS image.

<tt>
[root ~]# lvcreate VolGroup00 --name xs163 --size 3G
[root ~]# lvcreate VolGroup00 --name fedora7 --size 3G
[root ~]# lvcreate VolGroup00 --name debian4 --size 3G
[root ~]# lvdisplay
</tt>

=== Step 3. Shared HOME directory ===

With LVM, it is now easy to create new logical volumes for whatever you need, and they
can be accessed from all three Operating Systems. Create a logical volume "home" with
LABEL "HOME".

<tt>
[root ~]# lvcreate VolGroup00 --name home --size 10G
[root ~]# lvdisplay
[root ~]# mkfs.ext3 -L HOME /dev/VolGroup00/home
</tt>

You will need to add this to your /etc/fstab so that it gets mounted everytime the system boots up. For three operating systems, you will need this added to all three files.

<tt>
[root ~]# echo "LABEL=HOME /home ext3 defaults 0 0" >> /etc/fstab
</tt>

Login as root and create a group and userid. Note, Debian starts its ids at 1000, and Fedora at 500, so to be safe, we specify four-digit (nnnn) for both. Here we define user "wally" to group "myfriends". Use the same group and user id on all three operating systems. Each Operating System has its own way of crypting the password, so you have to enter the password on each of the three OS images.

<tt>
[root ~]# groupadd -g 1234 myfriends
[root ~]# useradd -g 1234 -u 4567 wally
[root ~]# passwd wally
</tt>

Now users can create files when running under any of these three OS images.

== Assemble the Images into multi-boot ==

All three of the Linux operating systems support the "GRUB" boot loader. Of these,
I liked the graphics on Fedora 7, so I will choose to use that as the base.

Let's boot from the [http://sysresccd.org/ SysRescueCD 1.0.3] CD. I have all
my partition images on an external USB drive, but this can work with CD or DVD as well.

===Step 1. Restore Fedora 7 ===

We will use sda1 for /boot, and sda2 for / (root). Use partimage to restore each partition. Note that if you have several in a series,
you just need to specify the *.000 version, and it will pick up the others automatically.

<tt>
[root ~]# mount /dev/sdc1 /mnt/backup
[root ~]# cd /mnt/backup/fedora7
[root fedora7]# dd if=master.mbr of=/dev/sda bs=446 count=1
[root fedora7]# partimage -e -b restore /dev/sda1 sda1_boot.img.000
[root fedora7]# partimage -e -b restore /dev/sda2 sda2_base.img.000
</tt>

We will need to edit a few files. As a precaution, I always make a bacukp copy (*.bak)
of any file that I am editing with "vi", in case I type something wrong and accidently
save it.

<tt>
[root ~]# cd /mnt/backup/fedora7
[root fedora7]# mkdir /mnt/fedora
[root fedora7]# mount /dev/sda2 /mnt/fedora
[root fedora7]# mount /dev/sda1 /mnt/fedora/boot
[root fedora7]# cd /mnt/fedora/boot/grub
[root grub]# cp menu.lst menu.bak
[root grub]# vi menu.lst
[root grub]# cd /mnt/fedora/etc
[root etc]# cp fstab fstab.bak
[root etc]# vi /mnt/fedora/etc/fstab
[root etc]# cp mtab mtab.bak
[root etc]# vi /mnt/fedora/etc/mtab
</tt>

For menu.lst (which is just a symbolic link to grub.conf, by the way), I changed the following:
* timeout value to "15" seconds
* put a "#" in front of "hiddenmenu" (so that I could see the menu)
* changed to "root=/dev/sda5"
* added "panic=30" (optional)
** If you get a kernel panic, it probably means there is a mismatch between menu.lst, fstab or mtab files. Make sure you change all three before rebooting your system. The "panic=30" allows the system to reboot after 30 seconds of a kernel panic, giving you enough time to insert your rescue CD to fix the problem.
** Without "panic=30" the kernel panic will just sit there, and you then have to press the power button for 5 seconds to shut the system down, then hit the power button again, and then on
my system I have to hit "f1" to confirm that my system was turned off in that manner, or
"Del" to go into the BIOS setup program.
* Added "# 0-Fedora" as a comment to remind me that "default=0" will select it.(optional)

Note that the default=0 means that the first stanza will be selected if you do nothing for
the timeout value (15 seconds). On the menu, you will see the 15 seconds count down to 0,
and the default highlighted. Your file will look something like this:

<tt>
# /boot/grub/menu.lst --> grub.conf generated by anaconda
#boot=/dev/sda
#
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE: You have a /boot partition. This means that
# all kernel and initrd paths are relative to /boot/, eg.
# root (hd0,0)
# kernel /vmlinuz-version ro root=/dev/mapper/pdc_bbbhp2
# initrd /initrd-version.img
#boot=/dev/mapper/pdc_bbbh
default=0
default=0
timeout=30
timeout=15
splashimage=(hd0,0)/grub/splash.xpm.gz
splashimage=(hd0,0)/grub/splash.xpm.gz
#hiddenmenu
#hiddenmenu
title Fedora7 (2.6.23.17-88.fc7)
# 0-Fedora7
title Fedora 7 Core (2.6.21-1.3194.fc7)
root (hd0,0)
root (hd0,0)
kernel /vmlinuz-2.6.23.17-88.fc7 ro root=LABEL=FEDORA7
kernel /vmlinuz-2.6.21-1.3194.fc7 ro root=/dev/sda2 panic=30
initrd /initrd-2.6.23.17-88.fc7.img
initrd /initrd-2.6.21-1.3194.fc7.img
</tt>
</tt>


The /etc/fstab file will look something like this. sda2, sda1, and sda5 are
Capture the file system table (fstab). These are the file systems mounted at boot time.
the root, boot, and swap locations. If you have anything that says LABEL=
Notice that I use LABEL=FEDORA7 for the root directory, LABEL=BOOT for the boot directory,
change it to the appropriate /dev/sda* device.
and LABEL=SWAP1 for the swap space. Using LABEL= will make the process easier when we re-assemble
all of the pieces into the final system.


<tt>
<tt>
# /etc/fstab
[root ~]# cd /root
/dev/sda5 / ext3 defaults 1 1
[root ~]# cp /etc/fstab etc-fstab.prep
/dev/sda1 /boot ext3 defaults 1 2
[root ~]# cat etc-fstab.prep
LABEL=FEDORA7 / ext3 defaults 1 1
LABEL=BOOT /boot ext3 defaults 1 2
tmpfs /dev/shm tmpfs defaults 0 0
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
proc /proc proc defaults 0 0
LABEL=SWAP1 none swap defaults 0 0
/dev/sda5 swap swap defaults 0 0
</tt>
</tt>


The /etc/mtab might also need to be updated to match the fstab.
Capture the disk layout. We will be making a new disk layout, but this information

could be helpful during re-assembly.
Since we moved the root directory location, we may want to re-install grub.
This will refresh the appropriate files into /mnt/fedora/boot/grub (which is actually on
/dev/sda1), but will not over-write any menu.lst, grub.conf, or spash.xpm.gz files
you already have.


<tt>
<tt>
[root ~]# cd /root
[root fedora7]# mkdir /mnt/fedora
[root ~]# sfdisk -d /dev/sda > sfdisk-d.prep
[root fedora7]# mount /dev/sda2 /mnt/fedora
[root ~]# cat sfdisk-d.prep
[root fedora7]# mount /dev/sda1 /mnt/fedora/boot
[root fedora7]# grub-install --root-directory=/mnt/fedora --no-floppy
# partition table of /dev/sda
unit: sectors
/dev/sda1 : start= 63, size= 1285137, Id=83, bootable
/dev/sda2 : start= 1285200, size= 64581300, Id=83
/dev/sda3 : start= 0, size= 0, Id= 0
/dev/sda4 : start= 65866500, size= 4305420, Id= 5
/dev/sda5 : start= 65866563, size= 4305357, Id=82
</tt>
</tt>


We can now unmount the USB drive, use "sync" to make sure the USB drive can be detached, and reboot with no CD in the CD tray.
Step 6.
We will now backup this OS image. These backups will then
be used to re-assemble the final system. For this example, I will use /dev/sdb3 (ext3)
as temporary space to hold the backup images, and then burn onto DVD. If you have a large external USB
drive, you could use that instead. To get a clean backup, I will boot from
[[ http://sysresccd.org/Main_Page | SysRescCD 1.0.3 ]] and perform all my work from there. We see that /mnt/boot has only
19MB of data, and /mnt/base has 3.3GB of data. Both could fit on a single DVD. The "mount -l" shows that everything is mounted successfully.


<tt>
<tt>
[root ~]# cd /root
[root fedora7]# cd /root
[root ~]# mke2fs -L BACKUP -j /dev/sdb2
[root ~]# umount /mnt/fedora
[root ~]# mkdir /mnt/boot /mnt/base /mnt/backup
[root ~]# sync
[root ~]# mount /dev/sda1 /mnt/boot
[root ~]# eject
[root ~]# mount /dev/sda2 /mnt/base
[root ~]# reboot
</tt>
[root ~]# mount /dev/sdb2 /mnt/backup

[root ~]# du -sh /mnt/boot /mnt/base
If all goes well, you should have a running Fedora system again.
19M /mnt/boot

3.3G /mnt/base
<h4>Step 2. Restore Debian</h4>
[root ~]# mount -l

tmpfs on / type tmpfs (rw)
For the second OS, we cannot restore "/boot" over the existing one, so we redirect this to
/dev/loop0 on /mnt/livecd type squashfs (ro,relatime)
/dev/sda3 which is our scratch space.
tmpfs on /mnt/cdrom type tmpfs (rw,relatime,size=204800k)

<tt>
[root ~]# mount /dev/sdc1 /mnt/backup
[root debian4]# cd /mnt/backup/debian4
[root debian4]# partimage -e -b restore /dev/sda3 sda1_boot.img.000
[root debian4]# partimage -e -b restore /dev/sda2 sda2_base.img.000
</tt>

Now we need to copy parts of the sratch space over into the /boot directory. The easiest
way I have found to do this is to use "vi" on two files, position your cursor on the
stanza you want to carry over from scratch, "5yy" will put those five lines in memory,
"Esc:n" will switch to the next file, and then position the cursor to where you want,
and use "p" to paste from memory.

<tt>
[root ~]# cd /mnt/backup/debian4
[root debian4]# mkdir /mnt/debian /mnt/scratch
[root debian4]# mount /dev/sda5 /mnt/debian
[root debian4]# mount /dev/sda3 /mnt/debian/boot
[root debian4]# mount /dev/sda1 /mnt/scratch
[root debian4]# cd /mnt/debian/boot/grub
[root grub]# cp menu.lst menu.bak
[root grub]# vi /mnt/scratch/menu.lst menu.lst
</tt>

You're merged menu.lst will look something like this. Note that the kernel levels are different, and that the Debian stanza has "root=/dev/sda2" instead of what Fedora has.

<tt>
# /boot/grub/menu.lst --> grub.conf generated by anaconda
#boot=/dev/sda
#
default=0
timeout=15
splashimage=(hd0,0)/grub/splash.xpm.gz
#hiddenmenu
# 0-Fedora7
title Fedora 7 Core (2.6.21-1.3194.fc7)
root (hd0,0)
kernel /vmlinuz-2.6.21-1.3194.fc7 ro root=/dev/sda5 panic=30
initrd /initrd-2.6.21-1.3194.fc7.img
# 1-Debian4
title Debian GNU/Linux, kernel 2.6.18-6-486
root (hd0,0)
kernel /vmlinuz-2.6.18-6-486 root=/dev/sda2 ro panic=30
initrd /initrd.img-2.6.18-6-486
</tt>

You will also have to edit the ''fstab'' and ''mtab'' files, as we did for Fedora.
The /boot will be /dev/sda1, the / (root) will be /dev/sda2, and the swap is /dev/sda7.
Reboot and make sure that you can launch either operating system successfully.

=== Step 2. Migrate XS163 root directory into Logical volume ===

Since we are dealing with our / (root) directory, we will perform some of these steps from a
different OS that has mount access to both /dev/md6 and VolGroup space.
Boot up one of the other OS (Fedora, Debian) or from
[http:://sysresccd.org/ SysRescCD 1.03] LiveCD. Make sure that you have access to
both /dev/md6 and VolGroup00.

<tt>
[root ~]# mdadm --examine --scan
[root ~]# mdadm --query /dev/md6
[root ~]# vgscan
[root ~]# vgdisplay
</tt>

Let's see how big the original root directory is. We will give it a "OLDXS" label so that we can make sure it is different than the new one. We will mount this as /mnt/old-root.

<tt>
[root ~]# mkdir /mnt/old-root
[root ~]# e2label /dev/md6 OLDXS
[root ~]# mount /dev/md6 /mnt/old-root
[root ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/md5 3.0G 2.2G 665M 77% /
/dev/md1 618M 265M 322M 46% /boot
tmpfs 950M 0 950M 0% /dev/shm
/dev/md6 9.1G 1.5G 7.2G 17% /mnt/old-root
</tt>

The root directory is 9.1GB allocated, but only using 1.5GB of actual space! Let's create 3GB logical volumes. In an effort to keep things simple, I set the LABEL (upper case) same as the logical volume name (lower case). We can display the what is mounted to confirm, notice the labels help identify which is which.

<tt>
[root ~]# lvcreate VolGroup00 --name xs163 --size 3G
[root ~]# lvdisplay
[root ~]# mkfs.ext3 -L XS163 /dev/VolGroup00/xs163
[root ~]# mkdir /mnt/new-root
[root ~]# mount /dev/VolGroup00/xs163 /mnt/new-root
[root ~]# mount -l
/dev/md5 on / type ext3 (rw) [FEDORA7]
...
...
/dev/sda1 on /mnt/boot type ext3 (rw) [BOOT]
/dev/md6 on /mnt/old-root type ext3 (rw) [OLDXS]
/dev/sda2 on /mnt/base type ext3 (rw) [FEDORA7]
/dev/mapper/VolGroup00-xs163 on /mnt/new-root type ext3 (rw) [XS163]
/dev/sdb3 on /mnt/backup type ext3 (rw) [BACKUP]
</tt>
</tt>


Now we are ready to copy. This can take several minutes. After the copy, we use the "touch" command to create two unique files, "+lvm" on the new, and "+md6" on the old. This is to help us keep straight which is which. Show the top level directory of each to show they are nearly identical. We will now reboot into the XS163 operating system image (still on /dev/md6).
We will copy the information we captured in step 5 over to the backup directory.


<tt>
<tt>
[root ~]# cp /mnt/base/root/*.prep /mnt/backup
[root ~]# cp -ax /mnt/old-root/. /mnt/new-root
[root ~]# touch /mnt/old-root/+md6
[root ~]# touch /mnt/new-root/+lvm
[root ~]# ls /mnt/new-root /mnt/old-root
/mnt/new-root:
bin etc halt library media proc selinux tmp
boot fedora7 home lost+found mnt root srv usr
dev fsroot.olpc lib +lvm opt sbin sys var
_
/mnt/old-root:
bin etc halt library media proc selinux tmp
boot fedora7 home lost+found mnt root srv usr
dev fsroot.olpc lib +md6 opt sbin sys var
[root ~]# reboot
</tt>
</tt>


We now need to update /etc/mtab and /etc/fstab (on /dev/md6). We will save the originals, just in case anything goes wrong. On both files, change /dev/md6 to /dev/mapper/VolGroup00-xs163. Also, you can change SWAP1 over to SWAP2.
We will now invoke ''partimage'' utility to perform the backups. ''partimage'' works
with partitions that contain file systems, including logical volumes. Only the actual
data is backed up, and compression options are available. The partitions must not be
mounted during the partimage processing. For BOOT, -z0 indicates no compression is
required, as it is only a few MB in size. For the root partition, we use -z1 for ''gzip''
compression, and use "-V660" to indicate that the image should be split into pieces
small enough to be burned onto a CDrom. (660 MiB is approximately 692 MB, and most CDrom
can hold 700MB. Check your media for capacity.


<tt>
<tt>
[root ~]# cd /mnt/backup
[root ~]# mkdir /mnt/new-root
[root /mnt/backup]# umount /mnt/boot
[root ~]# mount /dev/VolGroup00/xs163 /mnt/new-root
[root /mnt/backup]# umount /mnt/base
[root ~]# cd /etc
[root /mnt/backup]# sync
[root etc]# cp mtab mtab.md6
[root etc]# cp fstab fstab.md6
[root /mnt/backup]# partimage -d -b -z0 save /dev/sda1 sda1-BOOT.img
[root etc]# cat mtab
[root /mnt/backup]# partimage -d -b -z1 -V660 save /dev/sda1 sda1-BOOT.img
/dev/md6 / ext3 rw 0 0
Volume size: 692060160 bytes (660 MiB)
...
/dev/md1 /boot ext3 rw 0 0
/dev/mapper/VolGroup00-xs163 /mnt/new-root ext3 rw 0 0
[root etc]#
[root etc]# cat fstab
/dev/md6 / ext3 defaults 1 1
/dev/md1 /boot ext3 defaults 1 2
...
LABEL=SWAP1 none swap defaults 0 0
[root etc]#
[root etc]# vi mtab fstab
</tt>
</tt>


We now edit /boot/grub/menu.lst (this is on /dev/md1). We duplicate the stanza. On
Thanks to compression, the entire backup directory is only 1.1GB in size. This could
the first stanza, we will change "root=" to /dev/mapper/VolGroup00-xs163. On the
be burned onto two CDs or one DVD. In this case, we use ''growisofs'' to burn the list
second stanza, we will leave "root=" alone, but change initrd-*.img to initrd-*.md6
of files onto a single DVD.
instead. The default=0 and fallback=1 means that it will try the new one first, and
if it fails, will try the second one. Just in case, we put "panic=30" on each, which
means in the event of a kernel panic, it will reboot automatically after 30 seconds.
To help remember, we add "lvm" and "md6" at the end of each title.


<tt>
<tt>
[root /mnt/backup]# ls
[root ~]# cd /boot/grub
[root grub]# cp menu.lst menu.md6
boot-grub-menu.prep rpm-qa.prep sda2-FEDORA7.img.000 sfdisk-d.prep
[root grub]# vi menu.lst
etc-fstab.prep sda1-BOOT.img.000 sda2-FEDORA7.img.001
# /boot/grub/menu.lst --> grub.conf generated by anaconda
[root /mnt/backup]# du -sh
#
1.1G .
#boot=/dev/sda
[root /mnt/backup]# growisofs -Z /dev/dvd -R -J /mnt/backup/*
default=0
fallback=1
timeout=15
splashimage=(hd0,0)/grub/splash.xpm.gz
#hiddenmenu
# 0-XS163
title XS163 School Server (2.6.23.1-21.fc7) lvm
root (hd0,0)
kernel /vmlinuz-2.6.23.1-21.fc7 ro root=/dev/mapper/VolGroup00-xs163 panic=30
initrd /initrd-2.6.23.1-21.fc7.img
# 1-XS163
title XS163 School Server (2.6.23.1-21.fc7) md6
root (hd0,0)
kernel /vmlinuz-2.6.23.1-21.fc7 ro root=/dev/md6 panic=30
initrd /initrd-2.6.23.1-21.fc7.md6
</tt>

Now we have to rebuild the "initrd". This is located in the /boot directory. We are going to rename the existing one from *.img to *.md6. The "mkinitrd" program reads the grub file,
locates the stanza that has the *.img file, figures out what the root= should be, reads the /etc/mtab and /etc/fstab, and builds a packed file that is used during boot. Seeing that it needs both RAID and LVM drivers, the resulting packed file. Note that we don't have to specify "preload=raid1" because the system already knows there are RAID volumes already mounted and will include those drivers as well. The result is that the new initrd is 466KB bigger than before.

<tt>
[root grub]# cd /boot
[root boot]# uname -r
2.6.23.1-21.fc7
[root boot]# mv initrd-`uname -r`.img initrd-`uname -r`.md6
[root boot]# mkinitrd initrd-`uname -r`.img `uname -r`
[root boot]# ls *1-21* -l
-rw-r--r-- 1 root root 78275 2008-06-18 20:16 config-2.6.23.1-21.fc7
-rw------- 1 root root 3542513 2008-06-22 14:30 initrd-2.6.23.1-21.fc7.img
-rw------- 1 root root 3075780 2008-06-20 22:24 initrd-2.6.23.1-21.fc7.md6
-rw-r--r-- 1 root root 844454 2008-06-18 20:16 System.map-2.6.23.1-21.fc7
-rw-r--r-- 1 root root 1968672 2008-06-18 20:16 vmlinuz-2.6.23.1-21.fc7
</tt>

Remember to update both Master Boot Records, the one on /dev/sda, and the one on /dev/sdb.
This is done with five simple lines on the interactive grub screen. The third line, "device (hd0) /dev/sdb" tells grub to treat the second drive as if it were the first drive. In the event the first drive fails, the second drive (survivor) will become the new first drive.
Ensure that your /boot/grub/menu.lst does not have any references to (hd1) or this will cause problems in the event of a failure of either drive.

<tt>
[root etc]# grub
GNU GRUB version 0.97 (640K lower / 3072K upper memory)
_
[ Minimal BASH-like line editing is supported. For the first word, TAB
lists possible command completions. Anywhere else TAB lists the possible
completions of a device/filename.]
_
grub> root (hd0,0)
Filesystem type is ext2fs, partition type 0xfd
_
grub> setup (hd0)
Checking if "/boot/grub/stage1" exists... no
Checking if "/grub/stage1" exists... yes
Checking if "/grub/stage2" exists... yes
Checking if "/grub/e2fs_stage1_5" exists... yes
Running "embed /grub/e2fs_stage1_5 (hd0)"... 16 sectors are embedded. succeeded
Running "install /grub/stage1 (hd0) (hd0)1+16 p (hd0,0)/grub/stage2 /grub/grub.conf"...succeeded
Done.
_
grub> device (hd0) /dev/sdb
_
grub> root (hd0,0)
Filesystem type is ext2fs, partition type 0xfd
_
grub> setup (hd0)
Checking if "/boot/grub/stage1" exists... no
Checking if "/grub/stage1" exists... yes
Checking if "/grub/stage2" exists... yes
Checking if "/grub/e2fs_stage1_5" exists... yes
Running "embed /grub/e2fs_stage1_5 (hd0)"... 16 sectors are embedded. succeeded
Running "install /grub/stage1 (hd0) (hd0)1+16 p (hd0,0)/grub/stage2 /grub/grub.conf"... succeeded
Done.
_
grub> quit
</tt>

Now that we have all that updated, we need to update the new /etc/mtab and /etc/fstab over to the new logical volume version. We can restore the old versions on /dev/md6 so that we have that as our fallback in case things go wrong.
This time when we reboot, we will select the "lvm" stanza (which is the default). We will display both the new (LVM) versions of mtab and fstab, then the old (/dev/md6) versions, to confirm they are different.

<tt>
[root ~]# cd /etc
[root etc]# cp mtab /mnt/new-root/etc
[root etc]# cp fstab /mnt/new-root/etc
[root etc]# cp mtab.md6 mtab
[root etc]# cp fstab.md6 fstab
[root etc]# cd /mnt/new-root/etc
[root etc]# cat mtab fstab
[root etc]# cd /etc
[root etc]# cat mtab fstab
[root etc]# umount /mnt/new-root
[root etc]# sync
[root etc]# reboot
</tt>

The moment of truth. Reboot and select the "lvm" stanza from Grub. If all goes well, you can confirm that you are indeed running on the new logical volume with a few commands. Comment "df -h" shows that root is 48 percent full (1.5GB out of 3GB); "ls /" shows the "+lvm" that we touched on the previous step; and "mount -l" lists all the current mounts. The label is correctly shown as "XS163".

<tt>
[root ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup00-xs163
3.0G 1.4G 1.5G 48% /
/dev/md1 618M 269M 319M 46% /boot
tmpfs 950M 0 950M 0% /dev/shm
/dev/mapper/VolGroup00-fedora7
3.0G 2.2G 688M 77% /mnt/fedora7
[root@schoolserver ~]# ls /
bin etc home library media proc selinux tmp
boot fedora7 ifcfg-eth0-77 lost+found mnt root srv usr
dev fsroot.olpc lib +lvm opt sbin sys var
[root@schoolserver ~]# mount -l
/dev/mapper/VolGroup00-xs163 on / type ext3 (rw) [XS163]
...
...
/dev/md1 on /boot type ext3 (rw) [BOOT]
99.05% done, estimate finish Tue Jun 17 15:56:46 2008
</tt>
Total translation table size: 0

Total rockridge attributes bytes: 874
This same process can be used for Fedora 7 and Debian 4 to move their root directories into the same VolGroup00. On Debian, use "update-initramfs" instead of "mkinitrd".
Total directory bytes: 0

Path table size(bytes): 10

Max brk space used 0
<h4>Step 3. Restore XS School Server</h4>
535076 extents written (1045 MB)

builtin_dd: 535088*2KB out @ average 7.9x1352KBps
We can add as many operating systems as we like using this approach. For XS system,
/dev/dvd: flushing cache
we will only restore the /boot and /(root) directories. We will reuse /dev/sda3 as scratch space, and restore the root directory into /dev/sda6. We can restore or re-create
/dev/dvd: updating RMA
the /library partition later.
/dev/dvd: closing session

[root /mnt/backup]# cd /root
<tt>
[root /mnt/backup]# umount /mnt/backup
[root /mnt/backup]# sync
[root ~]# mount /dev/sdc1 /mnt/backup
[root /mnt/backup]# eject
[root ~]# cd /mnt/backup/xs163
[root xs163]# partimage -e -b restore /dev/sda3 sda1_boot.img.000
[root xs163]# partimage -e -b restore /dev/sda6 sda2_base.img.000
</tt>

Carry over the stanza to the existing /boot/grub/menu.lst, so that you now will have three stanzas in your menu.list, #0-Fedora, #1-Debian, and #2-XS163.
Here is the stanza for the XS school server (XS-163 level).

<tt>
# 2-XS163
title XS163 School Server (2.6.23.1-21.fc7)
root (hd0,0)
kernel /vmlinuz-2.6.23.1-21.fc7 ro root=/dev/sda6 panic=30
initrd /initrd-2.6.23.1-21.fc7.img
</tt>

Edit /etc/fstab, and /etc/mtab as we did with fedora and debian above. If you just want to merge /library in the / (root) directory, use the following commands:

<tt>
[root ~]# mkdir /library
[root ~]# mkdir /library/users
</tt>

<h4>Take Fresh Backup</h4>

Before we continue, now is a good time to
backup your multi-boot system using the [[Backup-and-Restore-Using-SysRescueCD]] procedure.
Put these in a different folder, marked "multi-boot" or something, so that you can
remember to keep them separate from the previous partition images. Take a copy of
every /etc/fstab from /dev/sda2, /dev/sda5 and /dev/sda

== Connect second disk drive and configure RAID ==

We installed all the OS images with a single drive, took a fresh backup, so now it is time to reconnect the SATA signal cable to the motherboard.

I found while testing out these procedures that it was easier just to leave the left
side panel off the computer, so that I could connect or detach the red SATA signal cable as needed. For safety, I always did a full shutdown, disconnected the 115 volt power source from
the back of the machine, and then attached/detached the red SATA signal cable.

For this, I was inspired by the excellent resources on HOWTOforge.com, especially these
long, but extremely well-written tutorials by Falko Timme.

* [[http://www.howtoforge.com/software-raid1-grub-boot-debian-etch How to Setup Software RAID on a running Debian system]]
* [[http://www.howtoforge.com/software-raid1-grub-boot-fedora-8 How to Setup Software RAID on a running Fedora system]]

Each tutorial follows these basic steps (the sequence is slightly different between Fedora and Debian)
# Use "sfdisk -d /dev/sda | sfdisk /dev/sdb" to copy over the partition layout to the second drive
# install "initramfs/mkinitrd" and "mdadm" (Mirrored Disk Administration) program
# Use "modprobe" to load "raid1" module, you can load "linear", "raid0" and others, but I only did "raid1". For debian, you must also load "md" module.
# Use "fdisk" to change sdbx from type "83"(Linux) to type "fd"(RAID autodetect)
# Use "mdadm --zero-superblock /dev/sdbx" to clear any previous RAID values
# Use "mdadm --create /dev/mdx --level=1 raid-disks=2 missing /dev/sdbx" to create a "deprecated" half-mirror, which I will call "(missing+sdbx)" shown as "_U".
# Format the half-mirror with "mkfs.ext3 -L MYLABEL /dev/mdx" (putting unique labels on each one greatly helped me keep them straight!)
# Update "/etc/mdadm.conf" with this new RAID setting.
# Mount "/dev/mdx" as "/mnt/mdx" and update the /etc/mtab, /etc/fstab. Add a new stanza in your /boot/grub/menu.lst files to indicate "root (hd1,0)" and/or "ro root=/dev/mdx". The first points to the /boot directory, and the second to the / (root) directory.
# Update the "initrd" file based on the values in /etc/mtab and /etc/fstab, and the modules you have loaded.
# copy the files over from /dev/sdax to /dev/mdx with "cp -dpRX . /mnt/mdx"
# run "grub" against both master boot records (hd0 is /dev/sda, and hd1 is /dev/sdb, according to "/boot/grub/device.map" file)
# reboot from the new stanza using the new /boot and new /root.
# Use "fdisk /dev/sda" to change the partition type of /dev/sdax from "83" to "fd" (RAID autodetect)
# Use "mdadm /dev/mdx --add /dev/sdax" to make the half-mirror whole. Now it is (sdax+sdbx). This will copy what is on /dev/sdbx over to /dev/sdax block for block. Let this finish! Do not reboot until you get the "UU" on "cat /proc/mdstat" command.
# Update /etc/mdadm.conf from "mdadm --examine --scan >> /etc/mdadm.conf" and edit out the duplicate lines with "vi". If there are two lines for /dev/mdx, delete the first one and keep the second.
# Update /boot/grub/menu.lst so that the stanza boots from root=(hd0,0) again.
# Upadte the "initrd" again, reboot and confirm everything works

Here were a few things to watch out for.

* Fedora and XS163 use "yum install", but Debian uses "apt-get install"
* I found it confusing to have /dev/md0 for (sda1+sdb1), /dev/md2 for (sda5+sdb5), and so on, so I named mine all to match /dev/md1=(sda1+sdb1), /dev/md5=(sda5+sdb5), etc.
* Fedora and XS163 expect the "mdadm.conf" to be in the /etc directory, but Debian expects it to be in the /etc/mdadm/mdadm.conf directory. (See my symbolic link workaround below.)
* Fedora uses "mkinitrd" and Debian uses "update-initramfs -u" (see my preload=raid1 workaround for XS163 below)
* The tutorial assumes you will convert over the /boot and / (root) over all at the same time, but then you have problems with the other two OS images. (See my conversion sequence below.)

<h4>/etc/mdadm.conf</h4>

I was following the Debian method, and put my config options and RAID settings in /etc/mdadm/mdadm.conf on Fedora systems, and failed miserably. I wanted all the OS
images to have them in the same location, so I used a symbolic link on Debian. Now,
all three OS images have their values in /etc/mdadm.conf location.

<tt>
[root ~]# cd /etc
[root etc]# ln -s /etc/mdadm/mdadm.conf mdadm.conf
[root etc]# ls -l mdadm.conf
</tt>

<h4>initrd Intial Ram file system images</h4>

The tutorial updates the initrd twice for each operating system. At first I skipped the step, thinking "I already have one" or "I already did that".

* The tutorial creates a unique backup each time, which is probably not a bad idea, but I used the "mkinitrd -f" option that overwrites the previous file so I wouldn't have that many copies.
* It is important to update your initrd images based on the values in the /etc/fstab, /etc/mtab and modprobe loaded modules so that your system boots properly. If you don't do this right,
you may get "kernel panic" or "file system integrity checks". This can only be done from the running system, otherwise "uname -r" returns the wrong value and you have it linked to the wrong kernel.
* I was having problems with the XS163 mkinitrd, so used "mkinitrd --preload=radi1" parameter and that seemed to solve the problem. This is needed if your /boot or / (root) directory is RAID1 mirrored from a different operating system and you did not update your /etc/mdadm.conf correctly.
* It is important to only reboot when all the right changes have been made as a group. Rebooting too soon or in between steps is a bad idea.

<h4>RAID conversion sequence</h4>

The tutorial works fine for single OS image systems. However, if you convert one
OS /boot and / (root) directories, then when you try to boot into the second OS,
it can't make sense of the /boot directory, or worse, changes one drive but not the
other. The trick is to convert the /boot drive last.

I will refer to the tutorials steps as "R1" through "R18" above.

Step 1.
You only need to do "R1" once, when the second drive is connected. You have a fresh
backup that includes "sfdisk-d.txt", so you can always re-create it if you had to.

Step 2.
Convert Debian (/dev/sda2) root partition only, using steps R2-11, R13-R16, and R18.
For Step R9 change "ro root=/dev/md2" but leave "root (hd0,0)" alone.
You can skip steps R12 and R17, since that only applies for /boot conversion.

Step 3.
Convert Fedora (/dev/sda5) root partition only, using steps R2-11, R13-R16, and R18.
Step R8, copy over the "etc/mdadm.conf" from Debian, and use "mdadm --assemble --scan" to repopulate the list of active pairs. The debian file has a lot more extra
configuration option descriptions.
For Step R9 change "ro root=/dev/md5" but leave "root (hd0,0)" alone.
Again, you can skip steps R12 and R17, since that only applies for /boot conversion.

Step 4.
Convert XS School Server (/dev/sda6) root partition only, using steps R2-11, R13-R16, and R18.
For Step R9 change "ro root=/dev/md6" but leave "root (hd0,0)" alone.
Step R8, copy over the "etc/mdadm.conf" from Fedora, which now has all of the Debian and Fedora settings combined. Again, you can skip steps R12 and R17, since that only applies for /boot conversion.

Step 5.
Convert the common /boot directory, from any of the above operating systems, using steps R4-R18.
You only have to convert this once, and it doesn't matter which system you do it from.

Step 6.
Take the /etc/mdadm.conf and copy the final version with all of the devices (md1, md2, md5, and md6) to /etc of each operating system. This can all be done from one OS to update the others. For example, from the XS163 image:

<tt>
[root ~]# cd /etc
[root etc]# mkdir /debian
[root etc]# mount /dev/md2 /debian
[root etc]# cp mdadm,conf /debian/etc/
[root etc]# mkdir /fedora
[root etc]# mount /dev/md5 /fedora
[root etc]# cp mdadm,conf /fedora/etc/
</tt>

Step 7.
We can either migrate our "swap" space over to a RAID pair now, or migrate to a Logical Volume where it will be part of a RAID pair as well. If you are not going to use LVM, here is the process for mirroring your swap space LABEL=SWAP1.

<tt>
[root ~]# swapoff /dev/sda7
[root ~]# mdadm --create /dev/md7 --level=1 --raid-disks=2 /dev/sda7 /dev/sdb7
[root ~]# mkswap -L SWAP1 /dev/md7
[root ~]# swapoff /dev/md7
</tt>

Why not just have two SWAP spaces, one on each drive? In the event of a disk failure, you
want swap to be mirrored, so that the system keeps running.

<h4>Take Fresh Backup </h4>

Before we continue, now is a good time to
backup your multi-boot system using the [[Backup-and-Restore-Using-SysRescueCD]] procedure.
Put these in a different folder, marked "with RAID" or something, so that you can
remember to keep them separate from the previous partition images. Instead of
sda1, sda2, etc. you will be using md1, md2, etc.

Take a copy of every /etc/fstab from /dev/md2, /dev/md5 and /dev/md6.

Your backup.sh might contain additional lines to capture the RAID configuration settings.

<tt>
cat /proc/mdstat > mdstat.txt
mdadm --assemble --scan
mdadm --examine --scan > md_conf.txt
mdadm --detail /dev/md* > mdadm.txt
</tt>

=== Step 4. Move LVM to RAID mirror pair ===

So now that we have LVM working, we can move this into a RAID1 mirror configuration. We create the second part of the pair, dev/sdb8, and extend the VolGroup00 over to it. Then we move all of the data from /dev/sda8 over to /dev/sdb8, and then remove /dev/sda8 from the Volume Group. The pvmove process can take a while to complete.

<tt>
[root ~]# pvcreate /dev/sdb8
[root ~]# pvdisplay
[root ~]# vgmknodes VolGroup00
[root ~]# vgextend VolGroup00 /dev/sdb8
[root ~]# pvmove /dev/sda8 /dev/sdb8
[root ~]# vgreduce VolGroup00 /dev/sda8
[root ~]# pvremove /dev/sda8
</tt>

Use "fdisk" to change /dev/sda8 type from "8e"(LVM) over to "fd"(raid Autodetect).

<tt>
[root ~]# fdisk /dev/sda
[root ~]# partprobe
</tt>

We will use "mdadm" utility to define a degraded half-mirror (sda8+missing). This will allow us to create /dev/md8, which we will add to our VolGroup00 and move the data into it.

<tt>
[root ~]# mdadm --create /dev/md8 --level=1 --raid-disks=2 /dev/sda8 missing
[root ~]# pvcreate /dev/md8
[root ~]# vgextend VolGroup00 /dev/md8
[root ~]# pvmove /dev/sdb8 /dev/md8
[root ~]# pvdisplay
[root ~]# vgreduce VolGroup00 /dev/sdb8
[root ~]# pvremove /dev/sdb8
</tt>

Use "fdisk" to change /dev/sdb8 type from "8e"(LVM) over to "fd"(raid Autodetect).

<tt>
[root ~]# fdisk /dev/sdb
[root ~]# partprobe
</tt>

Now we can make the raid a full pair. When we add the second volume, it will copy over all of the blocks. Use the "watch" command to see it during this process. Once complete we can grow the volume pair. The "pvresize" and "mdadm --grow" means that instead of having 38.19GB with 3.25MB unusable, we now have 38.19GB with 3.06MB unusable. In effect, we gained 190KB.

<tt>
[root ~]# mdadm --manage /dev/md8 --add /dev/sdb8
[root ~]# watch cat /proc/mdstat
Personalities : [raid1]
md8 : active raid1 sdb8[2] sda8[0]
40049920 blocks [2/1] [U_]
[===========>.........] recovery = 56.9% (22792000/40049920)
finish=5.7min speed=50032K/sec
[root ~]# pvresize /dev/md8
[root ~]# mdadm --grow /dev/md0 -z max
[root ~]# pvdisplay
[root ~]# lvdisplay
</tt>
</tt>


Latest revision as of 16:26, 23 February 2010

This software stack is based on This Beta Server hardware built for the Educational Blogger Project .

Devices Available

Based on the hardware, the following devices are available:

CD/DVD Burner /dev/hda (/dev/cdrom, /dev/dvd, /dev/dvdrw, and /dev/sr0)

SATA drive 1 /dev/sda (first drive it finds)

SATA drive 2 /dev/sdb (second drive, if connected)

USB keys or external drives /dev/sdc1 (/dev/sdd1, /dev/sde1 or /dev/sdf1)

Ethernet NIC (motherboard) eth0 (external WAN)

Ethernet NIC (PCI cards) eth1, eth2 (internal LAN)

RAID and Logical Volume Manager

RAID and LVM are supported on most recent Linux distributions and provide some key advantages. However, they can also be difficult to work with and require knowledge in particular commands to handle properly.

Redundant Array of Inexpensive Disks (RAID)

We will focus exclusively on RAID1 mirroring between two identical size disks:

RAID 1

Mirrors the data across drives. Multiple copies means you have better protection, so if you lose any one drive, the rest of the system can still run. Two 160GB drives would look like a single 160GB drive. Performance is the same or slightly slower than a single drive.

Other configurations are explained in: [What is RAID?]

RAID-1 Considerations:

  • Can mirror the entire drive ( /dev/sda <--> /dev/sdb ) or individual same-sized partitions ( /dev/sda1 <--> /dev/sdb1 )
  • Can mirror swap space. This appears to be preferred best practice over just having swap space on separate disks.
  • Mirrored drives appear as:
    • Software RAID (/dev/md0, /dev/md1)
    • or, Hardware-assisted BIOS "fakeraid" (/dev/dm-0, /dev/dm1) which is not supported on the hardware I have for this project
  • In general, a mirrored partition can be used wherever a native partition can be used
  • Can be used as physical volumes for Logical Volume Manager
  • /boot directory can be on RAID1, but not be on RAID0, RAID5, RAID10 or any other RAID configuration.

RAID rebuild: In the event a disk fails in RAID-1 mode, the single surviving drive can be used to continue operations. When a new replacement drive is available, the following procedures can be used to rebuild.

Place the surviving drive as "drive 1", disconnect the bad drive from the system.

  • boot up
  • indicate that drive 2 has failed

Place the new drive as "drive 2"

  • boot up
  • copy over the disk layout from "drive 1" using "sfdisk" utility
  • copy over the MBR from "drive 1" using "dd" utility
  • add the new partitions to /dev/md0, /dev/md1, etc. (the data will be copied over)

For more information, see: [Linux Software RAID HOWTO].

Logical Volume Manager (LVM2)

Logical Volume Manager allows a level of storage virtualization between the physical hardware and the operating system and applications. There are two formats, LVM (older) and LVM2 (newer). We will use LVM2.

Physical Volumes

A physical volume (pv) can be an entire disk drive (/dev/sda), partition (/dev/sda1), or RAID mirror (/dev/md0)

Volume Groups

A volume group (vg) combines the space capacity of one or more physical volumes. A physical volume can only be part of a single volume group. A volume group can host one or more logical volumes.

Logical Extents

The volume group is divided into fixed-size pieces called "logical extents" or LE. These are typically 32MB in size.

Logical Volume

A logical volume is a "virtual drive" that appears like a device or partition to the operating system. A logical volume can be the entire space of a single volume group, or a subset of a single volume group. Logical volumes can be increased or decreased in size, and can be moved as needed onto or away from specific physical volumes. A logical volume is one or more logical extents, so the smallest Logical Volume is one LE (typicallly 32MB). A logical extent can only be part of one logical volume.

LVM2 Considerations:

  • A volume group can contain physical volumes of different sizes
  • Linux cannot boot from a Logical Volume, this can be solved by having a separate /boot directory that is not part of the Logical Volume, and the rest of root (/) in the logical volume.
  • Physical volumes cannot be resized using the "gparted" tool
  • Linux swap space can be defined onto logical volume
  • A physical volume can be RAID1 mirror pair
  • The "partimage" program can backup logical volumes, but can only restore these to non-logical volumes, such as a primary partition, extended partition, or RAID1 mirror pair.

For more information, HOWTOforge.com has a nice [Beginner's Guide To LVM].

Disk Layout

Most Linux systems only require the following:

  • A Master Boot Record (MBR) that contains the boot loader and disk layout
  • A root directory (/)
    • The root directory can be split up into subdirectories for /boot, /var, /home on separate partitions
  • swap space (typically as big or bigger than installed RAM)

Possible Disk Layout configurations

Several disk layouts are possible. Depending on the Operating System installation procedure, you may have to install in JBOD, RAID or LVM, and then migrate to be a full RAID+LVM configuration.

1. Simple JBOD (no RAID, no LVM) 

/dev/sda                                 /dev/sdb
MBR (boot loader / disk layout )         MBR (disk layout only)
  /dev/sda1 as /boot                         /dev/sdb1 for extra space
  /dev/sda5 as /
  /dev/sda6 as swap

2. RAID (no LVM) 

/dev/sda                                                 /dev/sdb
MBR1 (boot loader/layout)  <-independent->                  MBR2 (boot loader/layout)
  /dev/sda1 as /boot       <- /dev/md0  ->                  /dev/sdb1
  /dev/sda5 as /           <- /dev/md1  ->                  /dev/sdb5
  /dev/sda6 as swap        <- /dev/md2  ->                  /dev/sdb6

2. LVM (no RAID) 

/dev/sda                                 /dev/sdb
MBR (boot loader / disk layout )         MBR (disk layout only)                        
  /dev/sda1 as /boot                         /dev/sdb1 as physical volume
  /dev/sda5 as physical volume
where:
VolGroup00 --- combines /dev/sda5 and /dev/sdb1
LogVol00 -- as /
LogVol01 -- as swap

3. RAID (no LVM) 

/dev/sda                                            /dev/sdb
MBR1 (boot loader/layout)      <-independent->      MBR2 (boot loader/layout)
  /dev/sda1 as /boot           <- /dev/md0  ->          /dev/sdb1
  /dev/sda5 as /               <- /dev/md1  ->          /dev/sdb5
  /dev/sda6 as swap            <- /dev/md2  ->          /dev/sdb6

4. RAID and LVM 

/dev/sda                                            /dev/sdb
MBR1 (boot loader/layout)      <-independent->      MBR2 (boot loader/layout)
  /dev/sda1 as /boot           <- /dev/md0  ->          /dev/sdb1
  /dev/sda5 as physical vol    <- /dev/md1  ->          /dev/sdb5   
where:
VolGroup00 --- /dev/md1
LogVol00 -- as /
LogVol01 -- as swap

5. RAID and LVM with rescue partition 

/dev/sda                                            /dev/sdb
MBR1 (boot loader/layout)      <-independent->      MBR2 (boot loader/layout)
  /dev/sda1 as /boot           <- /dev/md0  ->          /dev/sdb1
  /dev/sda2 as /rescue         <- /dev/md1  ->          /dev/sdb2
  /dev/sda5 as physical vol    <- /dev/md2  ->          /dev/sdb5   
where:
VolGroup00 --- /dev/md2
LogVol00 -- as /
LogVol01 -- as swap


6. RAID and LVM with rescue and scratch partitions 

/dev/sda                                            /dev/sdb
MBR1 (boot loader/layout)      <-independent->      MBR2 (boot loader/layout)
  /dev/sda1 as /boot           <- /dev/md0  ->          /dev/sdb1
  /dev/sda1 as /rescue         <- /dev/md1  ->          /dev/sdb2
  /dev/sda3 as /scratch1       <-independent->          /dev/sdb3 as /scratch2
  /dev/sda5 as swap            <-independent->          /dev/sdb5   
  /dev/sda6 as physical vol    <- /dev/md2  ->          /dev/sdb6  
  /dev/sda7 as physical vol    <- /dev/md3  ->          /dev/sdb6 
where:
VolGroup00 --- combine /dev/md2 and /dev/md3
LogVol00 -- as /
LogVol01 -- as swap

Partition Sizes

The following sizes can be used:

boot -- 680 MB

This could fit on a single CD for recovery purposes. /boot will contain /boot/grub boot loader, and can contain alternative kernels to select from during the boot process. Most OS only need 100MB, so this should be plenty.

rescue -- 4GB

This would be enough to have a basic operating system with software utilities, and include manuals, ISO images of CDs, and so on. This could fit on a single DVD or several CDs for recovery purposes.

scratch -- 30GB each

This can be used to download ISO files, store "partimage" backups, and so on. These are not RAID protected, so if you lose a disk drive, these files will not be mirrored on the other drive.

swap -- 2GB

This can be swap during the initial installs.

base1, base2, base3 -- rest of the disk (40GB of so each)

This will be the /dev/sda4 extended partition carved into smaller logical partitions that can be defined as physical volumes or RAID-1 mirrored pairs.

Initial Drive Format

For some reason, with fresh disks that have no format or partition, most 32-bit LiveCD were unable to boot. To solve this, I booted a 64-bit AMD64-based LiveCD (Fedora 8 in my case), created the partitions and then could use a 32-bit LiveCD for the rest of the install.

SysRescCD 1.0.3 is also capable of booting in either 64-bit or 32-bit mode, use "boot: rescue64" to boot into 64-bit mode. If you received the disk drive from someone else, and are concerned about remnants of any past data or operating systems, you can use the following command to clobber all data with random data. At 35 MB/sec, this would take about 90 minutes for each 160GB drive. 

dd if=/dev/urandom of=/dev/sda

Boot from the "SysRescCD" and use "fdisk" to configure /dev/sda. The final layout should look something like this: 

Disk /dev/sda: 160.0 GB, 160041885696 bytes
255 heads, 63 sectors/track, 19457 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x000a0a65
_                                                           _
  Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          80      642568+  83  Linux
/dev/sda2              81         581     4024282+  83  Linux
/dev/sda4            4230       19457   122318910    5  Extended
/dev/sda5            4230        4497     2152678+  82  Linux swap / Solaris

Note that all of the partitions are of type "83" (native Linux) except /dev/sda5 which is type "82" (Linux swap).

Operating System

The team wants to develop and test the EduBlog software on a variety of platforms. The operating systems considered were:

XS School Server

We're using the XS-163 (the latest level at this time). This is a customized version of Fedora 7 with special settings to run as a school server. The CD runs unattended, works when there is only a single SATA drive installed, and wipes away any previous partition table to form a new partition table.

Fedora 7

The XS School Server is based on Fedora 7, so if we are unable to get the XS installation kickstart file to support the disk layout, we felt a backup plan was to install Fedora 7 natively. Fedora 7 could also be used graphically by Glen on premises to help with problem determination.

Debian 4.0-r3 Etch

Ceibal (OLPC Uruguay) has chosen to use Debian 4.0-r3 Etch instead for their projects. It was a stretch goal to see if we could set up a "dual-boot" or "multi-boot" environment that would support XS-Debian, Fedora-Debian, or XS-Fedora-Debian configurations.

System Rescue

While not a complete Operating System to run EduBlog or perform development work, we might need to have an independent OS for administration or maintenance. Many utilities work best on data that is not currently active in use. This could be done with a LiveCD or as a separate partition on the disk drive.

SysRescCD 1.0.3 supports sshd so that with the right network setup, someone could effect software repairs remotely. It has a variety of utilities, including "partimage" that can be used to backup individual partitions or logical volumes. Installing on hard disk is a matter of copying just seven files over to /sysrcd directory on a primary or mirrored partition, but not in a Logical Volume. "GRUB" as the boot loader can be updated to run this at reboot.

Scaffolding

A simple technique to build a complicated configuration is known as "scaffolding", borrowing from the concept of building a large house, apartment or office structure.

There are two ways to get to LVM+RAID:

  1. standard → RAID → LVM+RAID
  2. standard → LVM → LVM+RAID

Having tested out both ways, I have found the second method simpler.

Here is the basic sequence:

  1. Prepare the first OS (XS-163 School Server)
  2. Prepare the second OS (Fedora 7)
  3. Prepare the third OS (Debian 4 Etch)
  4. Re-define partition tables for LVM
  5. Devine LVM2 pv, vg and lv and move data into the appropriate logical volume
  6. Assemble the individual OS images into a "multi-boot" environment
  7. Connect the second SATA drive
  8. Create RAID pairs of each partition you want protected

Preparation

Linux installers make a few general assumptions

  • You are a novice user willing to accept a standard disk layour (true for most people)
  • You are installing only a single OS on the system (true for most)
  • It will take up all the disk space available (true for most)
  • That you won't mind that some partitions are LVM2 Logical Volumes (true for most)
  • If you have two identical-size drives, then you might want Software RAID capability, creating crazy devices like /dev/mapper/pdc_bbbhp1 instead of /dev/sda1. Unless you specify RAID during the initial interactive setup, it will create these crazy devices, but not actually put them into RAID mode for you. The default is "No RAID, but define crazy device names anyways" (what ?)

If you have a normal Linux distribution, you can go through an interactive menu and choose which options you like. However, the XS-163.iso for XS School Server is non-interactive (batch, unattended, hands off) installation that assumes "standard" device names.

To get around these assumptions, we will detach the red SATA signal cable from the motherboard of the second disk drive (you can leave the power cable attached to the drive itself is fine). We will wipe the system clean three times. Each time we will install a fresh operating system image as if it were the only operating system we are going to use, and then backup the appropriate files to external media using "partimage" utility.

Step 1. Wipe the System clean

For each OS, wipe the system clean. The following command will zero out the first cylinder of your hard disk eliminating the master boot record and partition table. 

[root ~]#  dd if=/dev/zero of=/dev/sda bs=512 count=16

Step 2. Install from CD

Run the installation from CD. The XS-163 will carve up the disk as follows: 

/dev/sda1     /boot -- 100MB
/dev/sda2     /     -- (root) 8GB (8192MB)
/dev/sda5     pv    -- physical volume for LVM2, rest of disk

VolGroup00  -- volume group on physical volume
LogVol00    /library
LogVol01    swap

If you are installing Fedora 7, Debian 4, etc. wipe out all partitions, and choose "create custom setup" and define the following simple setup. The rest of the drive capacity will be unused, but it is important to make the initial /(root) directory as small as possible. 

/dev/sda1     /boot -- 200MB
/dev/sda2     /     -- (root) 8GB (8192MB)
/dev/sda3     swap

Step 3. Capture initial list of application packages

Before you change anything or install any packages, do the following, which will document all of the base RPM packages on your XS163 or Fedora 7 system. That way you can compare any new packages that were installed afterwards and see the differences using diff command. For Debian 4, use the apt-get utility instead. 

[root ~]# cd /root
[root ~]# mkdir /boot/details
[root ~]# rpm -qa | sort > /boot/details/rpm-xs163.txt

Step 4. Create partition images

We will now backup this OS image. These partition images will then be used to re-assemble the final system. We can use this procedure for Backup-and-Restore-Using-SysRescueCD.

Here is what you end up with:

For each OS you will have:

  • master.mbr -- this is the Master Boot Record (first 446 bytes of each disk)
  • sfdisk-d.txt -- this is the disk layout, human-and-computer readable
  • grubmenu.txt -- this has the initial boot parameters (/boot/grub/menu.lst --> grub.conf)
  • etcfstab.txt -- this is the list of mount points (/etc/fstab file)
  • sda1boot.img.000 -- this is partition image of the /boot directory
  • sda2base.img.000 -- this is first piece of the partition image of the / (root) directory
  • sda2base.img.001 -- (the files are 692MB each, small enough so they can be burned onto CDrom, uploaded to an FTP site, sent over using "yousendit.com", or fit on a FAT32 file system.)

Re-Define the Partion Tables

We'll choose a configuration similar to the left half of "6. RAID and LVM with rescue and scratch partitions" list above in the section Possible_Disk_Layout_configurations. 

/dev/sda                      
MBR1 (boot loader/layout)      
  /dev/sda1 as /boot           
  /dev/sda2 as /rescue (/ for our rescue version of Fedora7)  
  /dev/sda3 as /scratch1
  /dev/sda4 as extended partition holding sda5..sda8
  _
  /dev/sda5  as swap            
  /dev/sda6  as physical vol    
  /dev/sda7  as / (for debian 4) and later physical vol
  /dev/sda8  as / (for XS 163) and later physical vol

The /boot will contain the kernel images of all the operating systems, as well as the multi-boot /boot/grub directory. This will allow you to choose the different OS from a menu.

I chose 4GB to hold a set of rescue files, and then later decided to install Debian 4, it fits in 4GB! I create three equal-sized 10GB partitions, one for scratch (temp) space, one for the root directory of Fedora 7, and one for root directory of XS 163 School Server. Finally, I divided the rest of the volume into three 40GB pieces which can be used for physical volumes of an LVM Volume Group.

Here is what the start/end cylinder counts look like using "sfdisk -l /dev/sda": 

[root ~] sfdisk -l /dev/sda
Disk /dev/sda: 19457 cylinders, 255 heads, 63 sectors/track
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

  Device Boot Start     End   #cyls    #blocks   Id  System
/dev/sda1   *      0+     79      80-    642568+  83  Linux
/dev/sda2   *     80     638     559    4490167+  83  Linux
/dev/sda3        639    4241    3603   28941097+  83  Linux
/dev/sda4       4242   19456   15215  122214487+   5  Extended
/dev/sda5       4242+   4504     263-   2112516   83  Linux
/dev/sda6       4505+   9488    4984-  40033948+  8e  Linux LVM
/dev/sda7       9489+  14472    4984-  40033948+  83  Linux
/dev/sda8      14473+  19456    4984-  40033948+  83  Linux

Since there is so much work setting this up, it might be good to save it to a file. This can be done using the following command: 

[root ~]#   sfdisk -d /dev/sda  > sfdisk-d.txt

This is what the file looks like. Not that "size" is counting in terms of 512 byte sectors. So /dev/sda1 is 512 * 1285137 or 657990144 bytes (658 MB). 

[root ~] sfdisk -d /dev/sda
# partition table of /dev/sda
unit: sectors
/dev/sda1 : start=       63, size=  1285137, Id=83, bootable
/dev/sda2 : start=  1285200, size=  8980335, Id=83, bootable
/dev/sda3 : start= 10265535, size= 57882195, Id=83
/dev/sda4 : start= 68147730, size=244428975, Id= 5
/dev/sda5 : start= 68147793, size=  4225032, Id=83
/dev/sda6 : start= 72372888, size= 80067897, Id=8e
/dev/sda7 : start=152440848, size= 80067897, Id=83
/dev/sda8 : start=232508808, size= 80067897, Id=83

To restore, use "cat sfdisk-d | sfdisk /dev/sda" command. You can cut and paste the above and try it yourself. Note, this will clobber any previous partition table and all data may be lost.

Configure LVM2 for Logical Volumes

Now that we have multi-boot and RAID1 mirroring in place, we can create Logical Volumes. This section was inspired by the following tutorials:


Step 1. Create VolGroup00 for Library and Swap

Let's create some Logical Volumes for the XS school server. First, we'll define three mirrored devices to serve as physical space. Use "fdisk /dev/sda" Make sure that /dev/sda8 is type "8e" (LVM). Note that this Volume Group will not be RAID1-mirrored until a later step. 

[root ~]#  fdisk /dev/sda

Define the volume group. We use "pvcreate" to format the disk, then we'll call our new volume group "VolGroup00". 

[root ~]#  pvcreate /dev/sda8
[root ~]#  vgcreate VolGroup00 /dev/sda8
[root ~]#  vgdisplay

Now let's create the logical volumes. The "--extents" parameter sets the size in number of 4MB pieces, so here "library" is a tiny volume (we can always expand it later!!!) The "--size" allows you to specify the volume group in M(MB) or G(GB). We make the swap space 2G to match the amount of memory we have installed on our system. 

[root ~]#  lvcreate VolGroup00  --name library --extents 1
[root ~]#  lvcreate VolGroup00  --name swap --size 2G
[root ~]#  lvdisplay

Now, to activate these into the system. For Library, the initial XS CD created it empty except for a lone "users" directory. We can recreate that here. The echo with ">>" will append this one line to the end of your existing /etc/fstab file. 

[root ~]#  mkfs.ext3 -L LIBRARY /dev/VolGroup00/LogVol00
[root ~]#  mkdir /library
[root ~]#  mount /dev/VolGroup00/LogVol00 /library
[root ~]#  mkdir /library/users
[root ~]#  cp /etc/fstab /etc/fstab.bak
[root ~]#  echo "LABEL=LIBRARY /library ext3 defaults 0 0" >> /etc/fstab

For swap space, we can leave SWAP1 alone, and create SWAP2, activate SWAP2, and de-activate SWAP1. Update the /etc/fstab file so that we always use SWAP2 from now on. You can use "vi" editor to change SWAP1 to SWAP2 or use the "sed" command to do it for you. 

[root ~]#  mkswap -L SWAP2 /dev/VolGroup00/LogVol01
[root ~]#  swapon LABEL=SWAP2
[root ~]#  swapoff LABEL=SWAP1
[root ~]#  cp /etc/fstab /etc/fstab.bak
[root ~]#  sed 's/LABEL=SWAP1/LABEL=SWAP2/g' /etc/fstab.bak > /etc/fstab

Step 2. Create logical volumes for each root

Now we will create 3GB logical volumes to hold the root directory of each OS image. 

[root ~]#  lvcreate VolGroup00  --name xs163 --size 3G
[root ~]#  lvcreate VolGroup00  --name fedora7 --size 3G
[root ~]#  lvcreate VolGroup00  --name debian4 --size 3G
[root ~]#  lvdisplay

Step 3. Shared HOME directory

With LVM, it is now easy to create new logical volumes for whatever you need, and they can be accessed from all three Operating Systems. Create a logical volume "home" with LABEL "HOME". 

[root ~]#  lvcreate VolGroup00 --name home --size 10G
[root ~]#  lvdisplay
[root ~]#  mkfs.ext3 -L HOME /dev/VolGroup00/home

You will need to add this to your /etc/fstab so that it gets mounted everytime the system boots up. For three operating systems, you will need this added to all three files. 

[root ~]#  echo "LABEL=HOME /home ext3 defaults 0 0" >> /etc/fstab

Login as root and create a group and userid. Note, Debian starts its ids at 1000, and Fedora at 500, so to be safe, we specify four-digit (nnnn) for both. Here we define user "wally" to group "myfriends". Use the same group and user id on all three operating systems. Each Operating System has its own way of crypting the password, so you have to enter the password on each of the three OS images. 

[root ~]#  groupadd -g 1234 myfriends
[root ~]#  useradd -g 1234 -u 4567 wally
[root ~]#  passwd wally

Now users can create files when running under any of these three OS images.

Assemble the Images into multi-boot

All three of the Linux operating systems support the "GRUB" boot loader. Of these, I liked the graphics on Fedora 7, so I will choose to use that as the base.

Let's boot from the SysRescueCD 1.0.3 CD. I have all my partition images on an external USB drive, but this can work with CD or DVD as well.

Step 1. Restore Fedora 7

We will use sda1 for /boot, and sda2 for / (root). Use partimage to restore each partition. Note that if you have several in a series, you just need to specify the *.000 version, and it will pick up the others automatically. 

[root ~]# mount /dev/sdc1 /mnt/backup
[root ~]#  cd /mnt/backup/fedora7
[root fedora7]#  dd if=master.mbr of=/dev/sda bs=446 count=1
[root fedora7]#  partimage -e -b restore /dev/sda1 sda1_boot.img.000
[root fedora7]#  partimage -e -b restore /dev/sda2 sda2_base.img.000

We will need to edit a few files. As a precaution, I always make a bacukp copy (*.bak) of any file that I am editing with "vi", in case I type something wrong and accidently save it. 

[root ~]#  cd /mnt/backup/fedora7
[root fedora7]#  mkdir /mnt/fedora
[root fedora7]#  mount /dev/sda2 /mnt/fedora
[root fedora7]#  mount /dev/sda1 /mnt/fedora/boot
[root fedora7]#  cd /mnt/fedora/boot/grub
[root grub]#  cp menu.lst menu.bak
[root grub]#  vi menu.lst
[root grub]#  cd /mnt/fedora/etc
[root etc]#  cp fstab fstab.bak
[root etc]#  vi /mnt/fedora/etc/fstab
[root etc]#  cp mtab mtab.bak
[root etc]#  vi /mnt/fedora/etc/mtab

For menu.lst (which is just a symbolic link to grub.conf, by the way), I changed the following:

  • timeout value to "15" seconds
  • put a "#" in front of "hiddenmenu" (so that I could see the menu)
  • changed to "root=/dev/sda5"
  • added "panic=30" (optional)
    • If you get a kernel panic, it probably means there is a mismatch between menu.lst, fstab or mtab files. Make sure you change all three before rebooting your system. The "panic=30" allows the system to reboot after 30 seconds of a kernel panic, giving you enough time to insert your rescue CD to fix the problem.
    • Without "panic=30" the kernel panic will just sit there, and you then have to press the power button for 5 seconds to shut the system down, then hit the power button again, and then on

my system I have to hit "f1" to confirm that my system was turned off in that manner, or "Del" to go into the BIOS setup program.

  • Added "# 0-Fedora" as a comment to remind me that "default=0" will select it.(optional)

Note that the default=0 means that the first stanza will be selected if you do nothing for the timeout value (15 seconds). On the menu, you will see the 15 seconds count down to 0, and the default highlighted. Your file will look something like this: 

# /boot/grub/menu.lst --> grub.conf generated by anaconda
#boot=/dev/sda
#
default=0
timeout=15
splashimage=(hd0,0)/grub/splash.xpm.gz
#hiddenmenu
# 0-Fedora7
title Fedora 7 Core (2.6.21-1.3194.fc7)
       root (hd0,0)
       kernel /vmlinuz-2.6.21-1.3194.fc7 ro root=/dev/sda2 panic=30
       initrd /initrd-2.6.21-1.3194.fc7.img

The /etc/fstab file will look something like this. sda2, sda1, and sda5 are the root, boot, and swap locations. If you have anything that says LABEL= change it to the appropriate /dev/sda* device. 

# /etc/fstab
/dev/sda5               /                       ext3    defaults        1 1
/dev/sda1               /boot                   ext3    defaults        1 2
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
/dev/sda5               swap                    swap    defaults        0 0

The /etc/mtab might also need to be updated to match the fstab.

Since we moved the root directory location, we may want to re-install grub. This will refresh the appropriate files into /mnt/fedora/boot/grub (which is actually on /dev/sda1), but will not over-write any menu.lst, grub.conf, or spash.xpm.gz files you already have. 

[root fedora7]#  mkdir /mnt/fedora
[root fedora7]#  mount /dev/sda2 /mnt/fedora
[root fedora7]#  mount /dev/sda1 /mnt/fedora/boot
[root fedora7]#  grub-install --root-directory=/mnt/fedora --no-floppy

We can now unmount the USB drive, use "sync" to make sure the USB drive can be detached, and reboot with no CD in the CD tray. 

[root fedora7]#  cd /root
[root ~]#  umount /mnt/fedora
[root ~]#  sync
[root ~]#  eject
[root ~]#  reboot

If all goes well, you should have a running Fedora system again.

Step 2. Restore Debian

For the second OS, we cannot restore "/boot" over the existing one, so we redirect this to /dev/sda3 which is our scratch space. 

[root ~]# mount /dev/sdc1 /mnt/backup
[root debian4]#  cd /mnt/backup/debian4
[root debian4]#  partimage -e -b restore /dev/sda3 sda1_boot.img.000
[root debian4]#  partimage -e -b restore /dev/sda2 sda2_base.img.000

Now we need to copy parts of the sratch space over into the /boot directory. The easiest way I have found to do this is to use "vi" on two files, position your cursor on the stanza you want to carry over from scratch, "5yy" will put those five lines in memory, "Esc:n" will switch to the next file, and then position the cursor to where you want, and use "p" to paste from memory. 

[root ~]#  cd /mnt/backup/debian4
[root debian4]#  mkdir /mnt/debian /mnt/scratch
[root debian4]#  mount /dev/sda5 /mnt/debian
[root debian4]#  mount /dev/sda3 /mnt/debian/boot
[root debian4]#  mount /dev/sda1 /mnt/scratch
[root debian4]#  cd /mnt/debian/boot/grub
[root grub]#  cp menu.lst menu.bak
[root grub]#  vi /mnt/scratch/menu.lst menu.lst

You're merged menu.lst will look something like this. Note that the kernel levels are different, and that the Debian stanza has "root=/dev/sda2" instead of what Fedora has. 

# /boot/grub/menu.lst --> grub.conf generated by anaconda
#boot=/dev/sda
#
default=0
timeout=15
splashimage=(hd0,0)/grub/splash.xpm.gz
#hiddenmenu
# 0-Fedora7
title Fedora 7 Core (2.6.21-1.3194.fc7)
       root (hd0,0)
       kernel /vmlinuz-2.6.21-1.3194.fc7 ro root=/dev/sda5 panic=30
       initrd /initrd-2.6.21-1.3194.fc7.img
# 1-Debian4
title Debian GNU/Linux, kernel 2.6.18-6-486
       root    (hd0,0)
       kernel    /vmlinuz-2.6.18-6-486 root=/dev/sda2 ro panic=30 
       initrd    /initrd.img-2.6.18-6-486

You will also have to edit the fstab and mtab files, as we did for Fedora. The /boot will be /dev/sda1, the / (root) will be /dev/sda2, and the swap is /dev/sda7. Reboot and make sure that you can launch either operating system successfully.

Step 2. Migrate XS163 root directory into Logical volume

Since we are dealing with our / (root) directory, we will perform some of these steps from a different OS that has mount access to both /dev/md6 and VolGroup space. Boot up one of the other OS (Fedora, Debian) or from [http:://sysresccd.org/ SysRescCD 1.03] LiveCD. Make sure that you have access to both /dev/md6 and VolGroup00. 

[root ~]#  mdadm --examine --scan
[root ~]#  mdadm --query /dev/md6
[root ~]#  vgscan
[root ~]#  vgdisplay

Let's see how big the original root directory is. We will give it a "OLDXS" label so that we can make sure it is different than the new one. We will mount this as /mnt/old-root. 

[root ~]#  mkdir /mnt/old-root
[root ~]#  e2label /dev/md6 OLDXS
[root ~]#  mount /dev/md6 /mnt/old-root
[root ~]#  df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/md5              3.0G  2.2G  665M  77% /
/dev/md1              618M  265M  322M  46% /boot
tmpfs                 950M     0  950M   0% /dev/shm
/dev/md6              9.1G  1.5G  7.2G  17% /mnt/old-root

The root directory is 9.1GB allocated, but only using 1.5GB of actual space! Let's create 3GB logical volumes. In an effort to keep things simple, I set the LABEL (upper case) same as the logical volume name (lower case). We can display the what is mounted to confirm, notice the labels help identify which is which. 

[root ~]#  lvcreate VolGroup00 --name xs163 --size 3G
[root ~]#  lvdisplay
[root ~]#  mkfs.ext3 -L XS163 /dev/VolGroup00/xs163
[root ~]#  mkdir /mnt/new-root
[root ~]#  mount /dev/VolGroup00/xs163 /mnt/new-root
[root ~]#  mount -l
/dev/md5 on / type ext3 (rw) [FEDORA7]
...
/dev/md6 on /mnt/old-root type ext3 (rw) [OLDXS]
/dev/mapper/VolGroup00-xs163 on /mnt/new-root type ext3 (rw) [XS163]

Now we are ready to copy. This can take several minutes. After the copy, we use the "touch" command to create two unique files, "+lvm" on the new, and "+md6" on the old. This is to help us keep straight which is which. Show the top level directory of each to show they are nearly identical. We will now reboot into the XS163 operating system image (still on /dev/md6). 

[root ~]#  cp -ax /mnt/old-root/. /mnt/new-root
[root ~]#  touch /mnt/old-root/+md6
[root ~]#  touch /mnt/new-root/+lvm
[root ~]# ls /mnt/new-root /mnt/old-root
/mnt/new-root:
bin   etc          halt  library     media  proc  selinux  tmp
boot  fedora7      home  lost+found  mnt    root  srv      usr
dev   fsroot.olpc  lib   +lvm        opt    sbin  sys      var
_
/mnt/old-root:
bin   etc          halt  library     media  proc  selinux  tmp
boot  fedora7      home  lost+found  mnt    root  srv      usr
dev   fsroot.olpc  lib   +md6        opt    sbin  sys      var
[root ~]# reboot

We now need to update /etc/mtab and /etc/fstab (on /dev/md6). We will save the originals, just in case anything goes wrong. On both files, change /dev/md6 to /dev/mapper/VolGroup00-xs163. Also, you can change SWAP1 over to SWAP2. 

[root ~]# mkdir /mnt/new-root
[root ~]# mount /dev/VolGroup00/xs163 /mnt/new-root
[root ~]# cd /etc
[root etc]# cp mtab mtab.md6
[root etc]# cp fstab fstab.md6
[root etc]# cat mtab
/dev/md6 / ext3 rw 0 0
...
/dev/md1 /boot ext3 rw 0 0
/dev/mapper/VolGroup00-xs163 /mnt/new-root ext3 rw 0 0
[root etc]#
[root etc]# cat fstab
/dev/md6                /                       ext3    defaults        1 1
/dev/md1                /boot                   ext3    defaults        1 2
...
LABEL=SWAP1             none                    swap    defaults        0 0
[root etc]#
[root etc]# vi mtab fstab

We now edit /boot/grub/menu.lst (this is on /dev/md1). We duplicate the stanza. On the first stanza, we will change "root=" to /dev/mapper/VolGroup00-xs163. On the second stanza, we will leave "root=" alone, but change initrd-*.img to initrd-*.md6 instead. The default=0 and fallback=1 means that it will try the new one first, and if it fails, will try the second one. Just in case, we put "panic=30" on each, which means in the event of a kernel panic, it will reboot automatically after 30 seconds. To help remember, we add "lvm" and "md6" at the end of each title. 

[root ~]# cd /boot/grub
[root grub]# cp menu.lst menu.md6
[root grub]# vi menu.lst
# /boot/grub/menu.lst --> grub.conf generated by anaconda
#
#boot=/dev/sda
default=0
fallback=1
timeout=15
splashimage=(hd0,0)/grub/splash.xpm.gz
#hiddenmenu
# 0-XS163
title XS163 School Server (2.6.23.1-21.fc7) lvm
       root (hd0,0)
       kernel /vmlinuz-2.6.23.1-21.fc7 ro root=/dev/mapper/VolGroup00-xs163 panic=30
       initrd /initrd-2.6.23.1-21.fc7.img
# 1-XS163
title XS163 School Server (2.6.23.1-21.fc7) md6
       root (hd0,0)
       kernel /vmlinuz-2.6.23.1-21.fc7 ro root=/dev/md6 panic=30
       initrd /initrd-2.6.23.1-21.fc7.md6

Now we have to rebuild the "initrd". This is located in the /boot directory. We are going to rename the existing one from *.img to *.md6. The "mkinitrd" program reads the grub file, locates the stanza that has the *.img file, figures out what the root= should be, reads the /etc/mtab and /etc/fstab, and builds a packed file that is used during boot. Seeing that it needs both RAID and LVM drivers, the resulting packed file. Note that we don't have to specify "preload=raid1" because the system already knows there are RAID volumes already mounted and will include those drivers as well. The result is that the new initrd is 466KB bigger than before. 

[root grub]# cd /boot
[root boot]# uname -r
2.6.23.1-21.fc7
[root boot]# mv initrd-`uname -r`.img initrd-`uname -r`.md6
[root boot]# mkinitrd initrd-`uname -r`.img `uname -r`
[root boot]# ls *1-21* -l
-rw-r--r-- 1 root root   78275 2008-06-18 20:16 config-2.6.23.1-21.fc7
-rw------- 1 root root 3542513 2008-06-22 14:30 initrd-2.6.23.1-21.fc7.img
-rw------- 1 root root 3075780 2008-06-20 22:24 initrd-2.6.23.1-21.fc7.md6
-rw-r--r-- 1 root root  844454 2008-06-18 20:16 System.map-2.6.23.1-21.fc7
-rw-r--r-- 1 root root 1968672 2008-06-18 20:16 vmlinuz-2.6.23.1-21.fc7

Remember to update both Master Boot Records, the one on /dev/sda, and the one on /dev/sdb. This is done with five simple lines on the interactive grub screen. The third line, "device (hd0) /dev/sdb" tells grub to treat the second drive as if it were the first drive. In the event the first drive fails, the second drive (survivor) will become the new first drive. Ensure that your /boot/grub/menu.lst does not have any references to (hd1) or this will cause problems in the event of a failure of either drive. 

[root etc]# grub
   GNU GRUB  version 0.97  (640K lower / 3072K upper memory)
_
[ Minimal BASH-like line editing is supported.  For the first word, TAB
  lists possible command completions.  Anywhere else TAB lists the possible
  completions of a device/filename.]
_
grub> root (hd0,0)
Filesystem type is ext2fs, partition type 0xfd
_
grub> setup (hd0)
Checking if "/boot/grub/stage1" exists... no
Checking if "/grub/stage1" exists... yes
Checking if "/grub/stage2" exists... yes
Checking if "/grub/e2fs_stage1_5" exists... yes
Running "embed /grub/e2fs_stage1_5 (hd0)"...  16 sectors are embedded. succeeded
Running "install /grub/stage1 (hd0) (hd0)1+16 p (hd0,0)/grub/stage2  /grub/grub.conf"...succeeded
Done.
_
grub> device (hd0) /dev/sdb
_
grub> root (hd0,0)
Filesystem type is ext2fs, partition type 0xfd
_
grub> setup (hd0)
Checking if "/boot/grub/stage1" exists... no
Checking if "/grub/stage1" exists... yes
Checking if "/grub/stage2" exists... yes
Checking if "/grub/e2fs_stage1_5" exists... yes
Running "embed /grub/e2fs_stage1_5 (hd0)"...  16 sectors are embedded. succeeded
Running "install /grub/stage1 (hd0) (hd0)1+16 p (hd0,0)/grub/stage2 /grub/grub.conf"... succeeded
Done.
_
grub> quit

Now that we have all that updated, we need to update the new /etc/mtab and /etc/fstab over to the new logical volume version. We can restore the old versions on /dev/md6 so that we have that as our fallback in case things go wrong. This time when we reboot, we will select the "lvm" stanza (which is the default). We will display both the new (LVM) versions of mtab and fstab, then the old (/dev/md6) versions, to confirm they are different. 

[root ~]# cd /etc
[root etc]# cp mtab /mnt/new-root/etc
[root etc]# cp fstab /mnt/new-root/etc
[root etc]# cp mtab.md6 mtab
[root etc]# cp fstab.md6 fstab
[root etc]# cd /mnt/new-root/etc
[root etc]# cat mtab fstab
[root etc]# cd /etc
[root etc]# cat mtab fstab
[root etc]# umount /mnt/new-root
[root etc]# sync
[root etc]# reboot

The moment of truth. Reboot and select the "lvm" stanza from Grub. If all goes well, you can confirm that you are indeed running on the new logical volume with a few commands. Comment "df -h" shows that root is 48 percent full (1.5GB out of 3GB); "ls /" shows the "+lvm" that we touched on the previous step; and "mount -l" lists all the current mounts. The label is correctly shown as "XS163". 

[root ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-xs163
                     3.0G  1.4G  1.5G  48% /
/dev/md1              618M  269M  319M  46% /boot
tmpfs                 950M     0  950M   0% /dev/shm
/dev/mapper/VolGroup00-fedora7
                      3.0G  2.2G  688M  77% /mnt/fedora7
[root@schoolserver ~]# ls /
bin   etc          home           library     media  proc  selinux  tmp
boot  fedora7      ifcfg-eth0-77  lost+found  mnt    root  srv      usr
dev   fsroot.olpc  lib            +lvm        opt    sbin  sys      var
[root@schoolserver ~]# mount -l
/dev/mapper/VolGroup00-xs163 on / type ext3 (rw) [XS163]
...
/dev/md1 on /boot type ext3 (rw) [BOOT]

This same process can be used for Fedora 7 and Debian 4 to move their root directories into the same VolGroup00. On Debian, use "update-initramfs" instead of "mkinitrd".


Step 3. Restore XS School Server

We can add as many operating systems as we like using this approach. For XS system, we will only restore the /boot and /(root) directories. We will reuse /dev/sda3 as scratch space, and restore the root directory into /dev/sda6. We can restore or re-create the /library partition later. 

[root ~]# mount /dev/sdc1 /mnt/backup
[root ~]#  cd /mnt/backup/xs163
[root xs163]#  partimage -e -b restore /dev/sda3 sda1_boot.img.000
[root xs163]#  partimage -e -b restore /dev/sda6 sda2_base.img.000

Carry over the stanza to the existing /boot/grub/menu.lst, so that you now will have three stanzas in your menu.list, #0-Fedora, #1-Debian, and #2-XS163. Here is the stanza for the XS school server (XS-163 level). 

# 2-XS163
title XS163 School Server (2.6.23.1-21.fc7)
      root (hd0,0)
      kernel /vmlinuz-2.6.23.1-21.fc7 ro root=/dev/sda6 panic=30
      initrd /initrd-2.6.23.1-21.fc7.img

Edit /etc/fstab, and /etc/mtab as we did with fedora and debian above. If you just want to merge /library in the / (root) directory, use the following commands: 

[root ~]# mkdir /library
[root ~]# mkdir /library/users

Take Fresh Backup

Before we continue, now is a good time to backup your multi-boot system using the Backup-and-Restore-Using-SysRescueCD procedure. Put these in a different folder, marked "multi-boot" or something, so that you can remember to keep them separate from the previous partition images. Take a copy of every /etc/fstab from /dev/sda2, /dev/sda5 and /dev/sda

Connect second disk drive and configure RAID

We installed all the OS images with a single drive, took a fresh backup, so now it is time to reconnect the SATA signal cable to the motherboard.

I found while testing out these procedures that it was easier just to leave the left side panel off the computer, so that I could connect or detach the red SATA signal cable as needed. For safety, I always did a full shutdown, disconnected the 115 volt power source from the back of the machine, and then attached/detached the red SATA signal cable.

For this, I was inspired by the excellent resources on HOWTOforge.com, especially these long, but extremely well-written tutorials by Falko Timme.

Each tutorial follows these basic steps (the sequence is slightly different between Fedora and Debian)

  1. Use "sfdisk -d /dev/sda | sfdisk /dev/sdb" to copy over the partition layout to the second drive
  2. install "initramfs/mkinitrd" and "mdadm" (Mirrored Disk Administration) program
  3. Use "modprobe" to load "raid1" module, you can load "linear", "raid0" and others, but I only did "raid1". For debian, you must also load "md" module.
  4. Use "fdisk" to change sdbx from type "83"(Linux) to type "fd"(RAID autodetect)
  5. Use "mdadm --zero-superblock /dev/sdbx" to clear any previous RAID values
  6. Use "mdadm --create /dev/mdx --level=1 raid-disks=2 missing /dev/sdbx" to create a "deprecated" half-mirror, which I will call "(missing+sdbx)" shown as "_U".
  7. Format the half-mirror with "mkfs.ext3 -L MYLABEL /dev/mdx" (putting unique labels on each one greatly helped me keep them straight!)
  8. Update "/etc/mdadm.conf" with this new RAID setting.
  9. Mount "/dev/mdx" as "/mnt/mdx" and update the /etc/mtab, /etc/fstab. Add a new stanza in your /boot/grub/menu.lst files to indicate "root (hd1,0)" and/or "ro root=/dev/mdx". The first points to the /boot directory, and the second to the / (root) directory.
  10. Update the "initrd" file based on the values in /etc/mtab and /etc/fstab, and the modules you have loaded.
  11. copy the files over from /dev/sdax to /dev/mdx with "cp -dpRX . /mnt/mdx"
  12. run "grub" against both master boot records (hd0 is /dev/sda, and hd1 is /dev/sdb, according to "/boot/grub/device.map" file)
  13. reboot from the new stanza using the new /boot and new /root.
  14. Use "fdisk /dev/sda" to change the partition type of /dev/sdax from "83" to "fd" (RAID autodetect)
  15. Use "mdadm /dev/mdx --add /dev/sdax" to make the half-mirror whole. Now it is (sdax+sdbx). This will copy what is on /dev/sdbx over to /dev/sdax block for block. Let this finish! Do not reboot until you get the "UU" on "cat /proc/mdstat" command.
  16. Update /etc/mdadm.conf from "mdadm --examine --scan >> /etc/mdadm.conf" and edit out the duplicate lines with "vi". If there are two lines for /dev/mdx, delete the first one and keep the second.
  17. Update /boot/grub/menu.lst so that the stanza boots from root=(hd0,0) again.
  18. Upadte the "initrd" again, reboot and confirm everything works

Here were a few things to watch out for.

  • Fedora and XS163 use "yum install", but Debian uses "apt-get install"
  • I found it confusing to have /dev/md0 for (sda1+sdb1), /dev/md2 for (sda5+sdb5), and so on, so I named mine all to match /dev/md1=(sda1+sdb1), /dev/md5=(sda5+sdb5), etc.
  • Fedora and XS163 expect the "mdadm.conf" to be in the /etc directory, but Debian expects it to be in the /etc/mdadm/mdadm.conf directory. (See my symbolic link workaround below.)
  • Fedora uses "mkinitrd" and Debian uses "update-initramfs -u" (see my preload=raid1 workaround for XS163 below)
  • The tutorial assumes you will convert over the /boot and / (root) over all at the same time, but then you have problems with the other two OS images. (See my conversion sequence below.)

/etc/mdadm.conf

I was following the Debian method, and put my config options and RAID settings in /etc/mdadm/mdadm.conf on Fedora systems, and failed miserably. I wanted all the OS images to have them in the same location, so I used a symbolic link on Debian. Now, all three OS images have their values in /etc/mdadm.conf location. 

[root ~]#  cd /etc
[root etc]#  ln -s /etc/mdadm/mdadm.conf mdadm.conf
[root etc]#  ls -l mdadm.conf

initrd Intial Ram file system images

The tutorial updates the initrd twice for each operating system. At first I skipped the step, thinking "I already have one" or "I already did that".

  • The tutorial creates a unique backup each time, which is probably not a bad idea, but I used the "mkinitrd -f" option that overwrites the previous file so I wouldn't have that many copies.
  • It is important to update your initrd images based on the values in the /etc/fstab, /etc/mtab and modprobe loaded modules so that your system boots properly. If you don't do this right,

you may get "kernel panic" or "file system integrity checks". This can only be done from the running system, otherwise "uname -r" returns the wrong value and you have it linked to the wrong kernel.

  • I was having problems with the XS163 mkinitrd, so used "mkinitrd --preload=radi1" parameter and that seemed to solve the problem. This is needed if your /boot or / (root) directory is RAID1 mirrored from a different operating system and you did not update your /etc/mdadm.conf correctly.
  • It is important to only reboot when all the right changes have been made as a group. Rebooting too soon or in between steps is a bad idea.

RAID conversion sequence

The tutorial works fine for single OS image systems. However, if you convert one OS /boot and / (root) directories, then when you try to boot into the second OS, it can't make sense of the /boot directory, or worse, changes one drive but not the other. The trick is to convert the /boot drive last.

I will refer to the tutorials steps as "R1" through "R18" above.

Step 1. You only need to do "R1" once, when the second drive is connected. You have a fresh backup that includes "sfdisk-d.txt", so you can always re-create it if you had to.

Step 2. Convert Debian (/dev/sda2) root partition only, using steps R2-11, R13-R16, and R18. For Step R9 change "ro root=/dev/md2" but leave "root (hd0,0)" alone. You can skip steps R12 and R17, since that only applies for /boot conversion.

Step 3. Convert Fedora (/dev/sda5) root partition only, using steps R2-11, R13-R16, and R18. Step R8, copy over the "etc/mdadm.conf" from Debian, and use "mdadm --assemble --scan" to repopulate the list of active pairs. The debian file has a lot more extra configuration option descriptions. For Step R9 change "ro root=/dev/md5" but leave "root (hd0,0)" alone. Again, you can skip steps R12 and R17, since that only applies for /boot conversion.

Step 4. Convert XS School Server (/dev/sda6) root partition only, using steps R2-11, R13-R16, and R18. For Step R9 change "ro root=/dev/md6" but leave "root (hd0,0)" alone. Step R8, copy over the "etc/mdadm.conf" from Fedora, which now has all of the Debian and Fedora settings combined. Again, you can skip steps R12 and R17, since that only applies for /boot conversion.

Step 5. Convert the common /boot directory, from any of the above operating systems, using steps R4-R18. You only have to convert this once, and it doesn't matter which system you do it from.

Step 6. Take the /etc/mdadm.conf and copy the final version with all of the devices (md1, md2, md5, and md6) to /etc of each operating system. This can all be done from one OS to update the others. For example, from the XS163 image: 

[root ~]#  cd /etc
[root etc]#  mkdir /debian
[root etc]#  mount /dev/md2 /debian
[root etc]#  cp mdadm,conf /debian/etc/
[root etc]#  mkdir /fedora
[root etc]#  mount /dev/md5 /fedora
[root etc]#  cp mdadm,conf /fedora/etc/

Step 7. We can either migrate our "swap" space over to a RAID pair now, or migrate to a Logical Volume where it will be part of a RAID pair as well. If you are not going to use LVM, here is the process for mirroring your swap space LABEL=SWAP1. 

[root ~]#  swapoff /dev/sda7
[root ~]#  mdadm --create /dev/md7 --level=1 --raid-disks=2 /dev/sda7 /dev/sdb7
[root ~]#  mkswap -L SWAP1 /dev/md7
[root ~]#  swapoff /dev/md7

Why not just have two SWAP spaces, one on each drive? In the event of a disk failure, you want swap to be mirrored, so that the system keeps running.

Take Fresh Backup

Before we continue, now is a good time to backup your multi-boot system using the Backup-and-Restore-Using-SysRescueCD procedure. Put these in a different folder, marked "with RAID" or something, so that you can remember to keep them separate from the previous partition images. Instead of sda1, sda2, etc. you will be using md1, md2, etc.

Take a copy of every /etc/fstab from /dev/md2, /dev/md5 and /dev/md6.

Your backup.sh might contain additional lines to capture the RAID configuration settings. 

cat /proc/mdstat > mdstat.txt
mdadm --assemble --scan 
mdadm --examine --scan  > md_conf.txt
mdadm --detail /dev/md* > mdadm.txt

Step 4. Move LVM to RAID mirror pair

So now that we have LVM working, we can move this into a RAID1 mirror configuration. We create the second part of the pair, dev/sdb8, and extend the VolGroup00 over to it. Then we move all of the data from /dev/sda8 over to /dev/sdb8, and then remove /dev/sda8 from the Volume Group. The pvmove process can take a while to complete. 

[root ~]#  pvcreate /dev/sdb8
[root ~]#  pvdisplay
[root ~]#  vgmknodes VolGroup00
[root ~]#  vgextend VolGroup00 /dev/sdb8 
[root ~]#  pvmove /dev/sda8 /dev/sdb8
[root ~]#  vgreduce VolGroup00 /dev/sda8
[root ~]#  pvremove /dev/sda8

Use "fdisk" to change /dev/sda8 type from "8e"(LVM) over to "fd"(raid Autodetect). 

[root ~]#  fdisk /dev/sda
[root ~]#  partprobe

We will use "mdadm" utility to define a degraded half-mirror (sda8+missing). This will allow us to create /dev/md8, which we will add to our VolGroup00 and move the data into it. 

[root ~]#  mdadm --create /dev/md8 --level=1 --raid-disks=2 /dev/sda8 missing
[root ~]#  pvcreate /dev/md8
[root ~]#  vgextend VolGroup00 /dev/md8
[root ~]#  pvmove /dev/sdb8 /dev/md8
[root ~]#  pvdisplay
[root ~]#  vgreduce VolGroup00 /dev/sdb8
[root ~]#  pvremove /dev/sdb8

Use "fdisk" to change /dev/sdb8 type from "8e"(LVM) over to "fd"(raid Autodetect). 

[root ~]#  fdisk /dev/sdb
[root ~]#  partprobe

Now we can make the raid a full pair. When we add the second volume, it will copy over all of the blocks. Use the "watch" command to see it during this process. Once complete we can grow the volume pair. The "pvresize" and "mdadm --grow" means that instead of having 38.19GB with 3.25MB unusable, we now have 38.19GB with 3.06MB unusable. In effect, we gained 190KB. 

[root ~]#  mdadm --manage /dev/md8 --add /dev/sdb8
[root ~]#  watch cat /proc/mdstat
Personalities : [raid1]
md8 : active raid1 sdb8[2] sda8[0]
     40049920 blocks [2/1] [U_]
     [===========>.........]  recovery = 56.9% (22792000/40049920) 
                              finish=5.7min speed=50032K/sec
[root ~]#  pvresize /dev/md8
[root ~]#  mdadm --grow /dev/md0 -z max
[root ~]#  pvdisplay
[root ~]#  lvdisplay

Middleware

Additional Utilities

Retrieved from "http://wiki.laptop.org/mediawiki/index.php?title=User:Az990tony/edublog-beta-sw&oldid=232080"