User:Az990tony/edublog-beta-sw

This software stack is based on This Beta Server hardware built for the Educational Blogger Project .

Devices Available

Based on the hardware, the following devices are available:

CD/DVD Burner

/dev/hda (it is in the Primary/Master position on the first EIDE cable, but can be represented on some systems as /dev/cdrom, /dev/dvd, /dev/dvdrw, and /dev/sr0)

SATA disks

/dev/sda (this is drive 1)

/dev/sdb (this is drive 2)

Note that /dev/sda1, sda2, sda3 and sda4 can be bootable primary partitions. Typically, sda4 is configured as an "extended partition" which allows subsets to be defined as logical partitions sda5, sda6, and so on.

USB keys or external drives

/dev/sdc for the drive itself /dev/sdc1 (for the VFAT file system usually on it)

Note: depending on when the USB was attached, this could be /dev/sdd1, /dev/sde1 or /dev/sdf1.

Ethernet Network Interface cards

eth0, eth1 and eth2 seemed to be assigned to the one motherboard connector and two PCI cards differently between operating systems. I will need to track down the MAC addresses to keep them straight.

RAID and Logical Volume Manager

RAID and LVM are supported on most recent Linux distributions and provide some key advantages. However, they can also be difficult to work with and require knowledge in particular commands to handle properly.

Redundant Array of Inexpensive Disks (RAID)

Configurations include:

RAID 0

Stripes data across for better performance. Two 160GB drives would look like a fast 320GB drive, but if you lose any one drive, you have lost everything

RAID 1

Mirrors the data across drives. Multiple copies means you have better protection, so if you lose any one drive, the rest of the system can still run. Two 160GB drives would look like a single 160GB drive. Performance is the same or slightly slower than a single drive.

RAID 5

Stripes with parity for added protection. One drive's worth is set aside for parity, so three 160GB drives would hold 320GB of data; four 160GB drives would hold 480GB of data. Since we have only two drives, this option does not apply to us. Performance is faster for read, but can be slower for writes, depending on how the parity methods are implemented.

RAID 10

Striped and Mirrored. Sometimes referred to as RAID 0+1 or RAID 1+0, this combines RAID 1 with RAID 0. Data is striped across disk 1 and disk 2, disk 3 is a mirror of disk 1 stripe, and disk 4 is mirror of disk 2 stripe. Thus four 160GB drives holds 320GB of data. If you lose any one drive, the mirrored copy is still available, so the system will be able to continue to run. Performance is same or slightly slower than a 2-drive RAID 0 configuration.

LINEAR

This just strings together a bunch of different sized disks sequentially. A 160GB drive, 320GB and 500GB drive could be combined into 980GB.

JBOD

JBOD stands for "Just a Bunch of Disks", a term meaning there is no RAID for protection or performance improvement. This refers to deployments where each disk drive is independent.

RAID-1 Considerations:

• Can mirror the entire drive ( /dev/sda <--> /dev/sdb ) or individual same-sized partitions ( /dev/sda1 <--> /dev/sdb1 )
• Can mirror swap space. This appears to be preferred best practice over just having swap space on separate disks.
• Mirrored drives appear as:
  • Software RAID (/dev/md0, /dev/md1)
  • or, Hardware-assisted BIOS "fakeraid" (/dev/dm-0, /dev/dm1) which is not supported on the hardware I have for this project
• In general, a mirrored partition can be used wherever a native partition can be used
• Can be used as physical volumes for Logical Volume Manager
• /boot directory can be on RAID1, but not be on RAID0, RAID5, RAID10 or any other RAID configuration.

RAID rebuild: In the event a disk fails in RAID-1 mode, the single surviving drive can be used to continue operations. When a new replacement drive is available, the following procedures can be used to rebuild.

Place the surviving drive as "drive 1", remove the bad drive from the system.

• boot up
• indicate that drive 2 has failed

Place the new drive as "drive 2"

• boot up
• copy over the disk layout from "drive 1" using "sfdisk" utility
• copy over the MBR from "drive 1" using "dd" utility
• add the new partitions to /dev/md0, /dev/md1, etc. (the data will be copied over)

Logical Volume Manager (LVM2)

Logical Volume Manager allows a level of storage virtualization between the physical hardware and the operating system and applications.

Physical Volumes

A physical volume (pv) can be an entire disk drive (/dev/sda), partition (/dev/sda1), or RAID mirror (/dev/md0)

Volume Groups

A volume group (vg) combines the space capacity of one or more physical volumes. A physical volume can only be part of a single volume group. A volume group can host one or more logical volumes.

Logical Extents

The volume group is divided into fixed-size pieces called "logical extents" or LE. These are typically 32MB in size.

Logical Volume

A logical volume is a "virtual drive" that appears like a device or partition to the operating system. A logical volume can be the entire space of a single volume group, or a subset of a single volume group. Logical volumes can be increased or decreased in size, and can be moved as needed onto or away from specific physical volumes. A logical volume is one or more logical extents, so the smallest Logical Volume is one LE (typicallly 32MB). A logical extent can only be part of one logical volume.

LVM2 Considerations:

• A volume group can contain physical volumes of different sizes
• Linux cannot boot from a Logical Volume, this can be solved by having a separate /boot directory that is not part of the Logical Volume, and the rest of root (/) in the logical volume.
• Logical volumes cannot be resized using the "gparted" tool

Disk Layout

Most Linux systems only require the following:

• A Master Boot Record (MBR) that contains the boot loader and disk layout
• A root directory (/)
  • The root directory can be split up into subdirectories for /boot, /var, /home on separate partitions
• swap space (typically as big or bigger than installed RAM)

Possible Disk Layout configurations

Several disk layouts are possible. Depending on the Operating System installation procedure, you may have to install in JBOD, RAID or LVM, and then migrate to be a full RAID+LVM configuration.

1. Simple JBOD (no RAID, no LVM)

/dev/sda                                 /dev/sdb
MBR (boot loader / disk layout )         MBR (disk layout only)
  /dev/sda1 as /boot                         /dev/sdb1 for extra space
  /dev/sda5 as /
  /dev/sda6 as swap

2. RAID (no LVM)

/dev/sda                                                 /dev/sdb
MBR1 (boot loader/layout)  <-independent->                  MBR2 (boot loader/layout)
  /dev/sda1 as /boot       <- /dev/md0  ->                  /dev/sdb1
  /dev/sda5 as /           <- /dev/md1  ->                  /dev/sdb5
  /dev/sda6 as swap        <- /dev/md2  ->                  /dev/sdb6

2. LVM (no RAID)

/dev/sda                                 /dev/sdb
MBR (boot loader / disk layout )         MBR (disk layout only)                        
  /dev/sda1 as /boot                         /dev/sdb1 as physical volume
  /dev/sda5 as physical volume
where:
VolGroup00 --- combines /dev/sda5 and /dev/sdb1
LogVol00 -- as /
LogVol01 -- as swap

3. RAID (no LVM)

/dev/sda                                            /dev/sdb
MBR1 (boot loader/layout)      <-independent->      MBR2 (boot loader/layout)
  /dev/sda1 as /boot           <- /dev/md0  ->          /dev/sdb1
  /dev/sda5 as /               <- /dev/md1  ->          /dev/sdb5
  /dev/sda6 as swap            <- /dev/md2  ->          /dev/sdb6

4. RAID and LVM

/dev/sda                                            /dev/sdb
MBR1 (boot loader/layout)      <-independent->      MBR2 (boot loader/layout)
  /dev/sda1 as /boot           <- /dev/md0  ->          /dev/sdb1
  /dev/sda5 as physical vol    <- /dev/md1  ->          /dev/sdb5   
where:
VolGroup00 --- /dev/md1
LogVol00 -- as /
LogVol01 -- as swap

5. RAID and LVM with rescue partition

/dev/sda                                            /dev/sdb
MBR1 (boot loader/layout)      <-independent->      MBR2 (boot loader/layout)
  /dev/sda1 as /boot           <- /dev/md0  ->          /dev/sdb1
  /dev/sda2 as /rescue         <- /dev/md1  ->          /dev/sdb2
  /dev/sda5 as physical vol    <- /dev/md2  ->          /dev/sdb5   
where:
VolGroup00 --- /dev/md2
LogVol00 -- as /
LogVol01 -- as swap

6. RAID and LVM with rescue and scratch partitions

/dev/sda                                            /dev/sdb
MBR1 (boot loader/layout)      <-independent->      MBR2 (boot loader/layout)
  /dev/sda1 as /boot           <- /dev/md0  ->          /dev/sdb1
  /dev/sda1 as /rescue         <- /dev/md1  ->          /dev/sdb2
  /dev/sda3 as /scratch1       <-independent->          /dev/sdb3 as /scratch2
  /dev/sda5 as swap            <-independent->          /dev/sdb5   
  /dev/sda6 as physical vol    <- /dev/md2  ->          /dev/sdb6  
  /dev/sda7 as physical vol    <- /dev/md3  ->          /dev/sdb6 
where:
VolGroup00 --- combine /dev/md2 and /dev/md3
LogVol00 -- as /
LogVol01 -- as swap

Partition Sizes

The following sizes can be used:

rescue -- 4GB

This would be enough to have a basic operating system with software utilities, and include manuals, ISO images of CDs, and so on. This could fit on a single DVD or several CDs for recovery purposes.

boot -- 680 MB

This could fit on a single CD for recovery purposes. /boot will contain /boot/grub boot loader, and can contain alternative kernels to select from during the boot process. Most OS only need 100MB, so this should be plenty.

scratch -- 30GB each

This can be used to download ISO files, store "partimage" backups, and so on. These are not RAID protected, so if you lose a disk drive, these files will not be mirrored on the other drive.

swap -- 2GB

This can be swap during the initial installs.

base1, base2 -- rest of the disk (60GB of so each)

This will be the /dev/sda4 extended partition with /dev/sda6 and /dev/sda7 as the extended partition as physical volumes for LVM.

Creating the Disk Partition table

For some reason, with fresh disks that have no format or partition, most 32-bit LiveCD were unable to boot. To solve this, I booted a 64-bit AMD64-based LiveCD (Fedora 8 in my case), created the partitions and then could use a 32-bit LiveCD for the rest of the install. SystemRescueCD is also capable of booting in either 64-bit or 32-bit mode, use "boot: rescue64" to boot into 64-bit mode.

Boot from the "SysRescCD" and use "fdisk" to configure /dev/sda. The final layout should look something like this:

Disk /dev/sda: 160.0 GB, 160041885696 bytes
255 heads, 63 sectors/track, 19457 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x000a0a65
_                                                           _
  Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          80      642568+  83  Linux
/dev/sda2              81         581     4024282+  83  Linux
/dev/sda3             582        4229    29302560   83  Linux
/dev/sda4            4230       19457   122318910    5  Extended
/dev/sda5            4230        4497     2152678+  82  Linux swap / Solaris
/dev/sda6            4498       19457   120246493+  83  Linux

Note that all of the partitions are of type "83" (native Linux) except /dev/sda5 which is type "82" (Linux swap). You will need to "reboot" for these changes to the partition table to take effect.

To save time, you can configure one drive /dev/sda then copy over the configuration to the second drive as follows:

sfdisk -d /dev/sda | sfdisk /dev/sdb --force

Operating System

The team wants to develop and test the EduBlog software on a variety of platforms. The operating systems considered were:

Fedora 7

The XS School Server is based on Fedora 7, so if we are unable to get the XS installation kickstart file to support the disk layout, we felt a backup plan was to install Fedora 7 natively. Fedora 7 could also be used graphically by Glen on premises to help with problem determination.

XS School Server

The XS-163 is the latest level. This is a customized version of Fedora 7 with special settings to run as a school server. The CD runs unattended, works when there is only a single SATA drive installed, and wipes away any previous partition table to form a new partition table.

Debian 4.0-r3 Etch

Ceibal (OLPC Uruguay) has chosen to use Debian 4.0-r3 Etch instead for their projects. It was a stretch goal to see if we could set up a "dual-boot" or "multi-boot" environment that would support XS-Debian, Fedora-Debian, or XS-Fedora-Debian configurations.

System Rescue

While not a complete Operating System to run EduBlog or perform development work, we might need to have an independent OS for administration or maintenance. Many utilities work best on data that is not currently active in use. This could be done with a LiveCD or as a separate partition on the disk drive.

[[ http://sysresccd.org/Main_Page | SysRescCD 1.0.3 ]] supports sshd so that with the right network setup, someone could effect software repairs remotely. It has a variety of utilities, including "partimage" that can be used to backup individual partitions or logical volumes. Installing on hard disk is a matter of copying just seven files over to /sysrcd directory on a primary or mirrored partition, but not in a Logical Volume. "GRUB" as the boot loader can be updated to run this at reboot.

Scaffolding

A simple technique to build a complicated configuration is known as "scaffolding", borrowing from the concept of building a large house, apartment or office structure.

Here is the basic sequence:

1. Install a single OS on a single SATA disk drive (make sure the other drive is disconnected, such as by pulling the SATA signal cable off the motherboard)
2. backup that installation onto CD or DVD media
3. Wipe the system clean and install the second OS and back this up to CD or DVD media
4. Re-define partition tables on both with no LVM2 or RAID
5. Restore the individual OS backups into each place on the new system
6. Connect the second SATA drive
7. Create RAID pairs of each partition you want protected
8. Devine LVM2 pv, vg and lv and move data into the appropriate logical volume

Preparation

Step 1. For each OS, wipe the system clean. The following command will zero out the first cylinder of your hard disk eliminating the master boot record and partition table.

[root ~]#  dd if=/dev/zero of=/dev/sda bs=512 count=1

Step 2. Use "fdisk" to create a simple partition layout, something like:

/dev/sda                                
MBR (boot loader / disk layout )        
 /dev/sda1 as /boot   (choose 500MB)                      
 /dev/sda2 as /       (choose 30GB)
 /dev/sda3 as swap    (choose 2GB)

Step 3. Format each partition. In this case "-L" sets the file system label, and "-j" indicates "ext3 file system" format. I will call the boot partition "BOOT", the root partition "FEDORA7" and the swap space "SWAP1".

[root ~]#   mke2fs -L BOOT -j /dev/sda1     
[root ~]#   mke2fs -L FEDORA7 -j /dev/sda2  
[root ~]#   mkswap -L SWAP1 /dev/sda3

Step 4. Run the installation from CD. Typically, these have interactive options that allow you to keep the existing partitions and use them "as is". Sometimes, an installation CD will blow away the entire drive and build its own, or blow away the entire drive and let you define new partitions. If this happens, try to use partitions similar to the above.

Step 5. After you update the RPMs and customize it however you like, we will capture some information that will be helpful later.

Capture all of the RPM packages installed. This can only be done from a running system.

[root ~]# cd /root
[root ~]# rpm -qa | sort > rpm-qa.prep

Capture the "menu.lst" in /boot/grub. We'll copy it over to the /root directory. This version has timeout set to 30 seconds, commented out hiddenmenu and changed root=/dev/mapper/pdc_bbbhp2 to root=LABEL=FEDORA7. You will want to get rid of every /dev/mapper you find as they will get in the way in later steps. Use e2label to change the label of the / directory if you need to. Each OS root system should have a unique name.

[root ~]# cd /root
[root ~]# cp /boot/grub/menu.lst boot-grub-menu.prep
[root ~]# cat boot-grub-menu.prep
# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE:  You have a /boot partition.  This means that
#          all kernel and initrd paths are relative to /boot/, eg.
#          root (hd0,0)
#          kernel /vmlinuz-version ro root=/dev/mapper/pdc_bbbhp2
#          initrd /initrd-version.img
#boot=/dev/mapper/pdc_bbbh
default=0
timeout=30
splashimage=(hd0,0)/grub/splash.xpm.gz
#hiddenmenu
title Fedora7 (2.6.23.17-88.fc7)
       root (hd0,0)
       kernel /vmlinuz-2.6.23.17-88.fc7 ro root=LABEL=FEDORA7
       initrd /initrd-2.6.23.17-88.fc7.img

Capture the file system table (fstab). These are the file systems mounted at boot time. Notice that I use LABEL=FEDORA7 for the root directory, LABEL=BOOT for the boot directory, and LABEL=SWAP1 for the swap space. Using LABEL= will make the process easier when we re-assemble all of the pieces into the final system.

[root ~]# cd /root
[root ~]# cp /etc/fstab etc-fstab.prep
[root ~]# cat etc-fstab.prep
LABEL=FEDORA7           /                       ext3    defaults        1 1
LABEL=BOOT              /boot                   ext3    defaults        1 2
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
LABEL=SWAP1             none                    swap    defaults        0 0

Capture the disk layout. We will be making a new disk layout, but this information could be helpful during re-assembly.

[root ~]# cd /root
[root ~]# sfdisk -d /dev/sda > sfdisk-d.prep
[root ~]# cat sfdisk-d.prep
# partition table of /dev/sda
unit: sectors
/dev/sda1 : start=       63, size=  1285137, Id=83, bootable
/dev/sda2 : start=  1285200, size= 64581300, Id=83
/dev/sda3 : start=        0, size=        0, Id= 0
/dev/sda4 : start= 65866500, size=  4305420, Id= 5
/dev/sda5 : start= 65866563, size=  4305357, Id=82

Step 6. We will now backup this OS image. These backups will then be used to re-assemble the final system. For this example, I will use /dev/sdb3 (ext3) as temporary space to hold the backup images, and then burn onto DVD. If you have a large external USB drive, you could use that instead. To get a clean backup, I will boot from [[ http://sysresccd.org/Main_Page | SysRescCD 1.0.3 ]] and perform all my work from there. We see that /mnt/boot has only 19MB of data, and /mnt/base has 3.3GB of data. Both could fit on a single DVD. The "mount -l" shows that everything is mounted successfully.

[root ~]#   cd /root
[root ~]#   mke2fs -L BACKUP -j /dev/sdb2
[root ~]#   mkdir /mnt/boot /mnt/base /mnt/backup
[root ~]#   mount /dev/sda1 /mnt/boot
[root ~]#   mount /dev/sda2 /mnt/base
[root ~]#   mount /dev/sdb2 /mnt/backup
[root ~]#   du -sh /mnt/boot /mnt/base
19M     /mnt/boot
3.3G    /mnt/base
[root ~]#   mount -l
tmpfs on / type tmpfs (rw)
/dev/loop0 on /mnt/livecd type squashfs (ro,relatime)
tmpfs on /mnt/cdrom type tmpfs (rw,relatime,size=204800k)
...
/dev/sda1 on /mnt/boot type ext3 (rw) [BOOT]
/dev/sda2 on /mnt/base type ext3 (rw) [FEDORA7]
/dev/sdb3 on /mnt/backup type ext3 (rw) [BACKUP]

We will copy the information we captured in step 5 over to the backup directory.

[root ~]#  cp /mnt/base/root/*.prep /mnt/backup

We will now invoke partimage utility to perform the backups. partimage works with partitions that contain file systems, including logical volumes. Only the actual data is backed up, and compression options are available. The partitions must not be mounted during the partimage processing. For BOOT, -z0 indicates no compression is required, as it is only a few MB in size. For the root partition, we use -z1 for gzip compression, and use "-V660" to indicate that the image should be split into pieces small enough to be burned onto a CDrom. (660 MiB is approximately 692 MB, and most CDrom can hold 700MB. Check your media for capacity.

[root ~]#  cd /mnt/backup
[root /mnt/backup]#  umount /mnt/boot
[root /mnt/backup]#  umount /mnt/base
[root /mnt/backup]#  sync
[root /mnt/backup]#  partimage -d -b -z0 save /dev/sda1 sda1-BOOT.img
[root /mnt/backup]#  partimage -d -b -z1 -V660 save /dev/sda1 sda1-BOOT.img
Volume size: 692060160 bytes (660 MiB)

Thanks to compression, the entire backup directory is only 1.1GB in size. This could be burned onto two CDs or one DVD. In this case, we use growisofs to burn the list of files onto a single DVD.

[root /mnt/backup]#  ls
boot-grub-menu.prep  rpm-qa.prep        sda2-FEDORA7.img.000  sfdisk-d.prep
etc-fstab.prep       sda1-BOOT.img.000  sda2-FEDORA7.img.001
[root /mnt/backup]#  du -sh
1.1G    .
[root /mnt/backup]#  growisofs -Z /dev/dvd -R -J /mnt/backup/*
...
99.05% done, estimate finish Tue Jun 17 15:56:46 2008
Total translation table size: 0
Total rockridge attributes bytes: 874
Total directory bytes: 0
Path table size(bytes): 10
Max brk space used 0
535076 extents written (1045 MB)
builtin_dd: 535088*2KB out @ average 7.9x1352KBps
/dev/dvd: flushing cache
/dev/dvd: updating RMA
/dev/dvd: closing session
[root /mnt/backup]#  cd /root
[root /mnt/backup]#  umount /mnt/backup
[root /mnt/backup]#  sync
[root /mnt/backup]#  eject