User:Az990tony/backup

From OLPC
< User:Az990tony
Revision as of 15:01, 28 February 2008 by Az990tony (talk | contribs) (→‎Overview)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

Overview

In our target design environment, there are three machines. 

+------------+          +--------------+        +-------------------+
| Internet   |--------- | XF (squid)   |---+----| XS (schoolserver) |
| Modem      |          | Dansguardian |   |    | LAMP+Moodle       |
+------------+          +--------------+   |    +-------------------+          
                                           |
                                    +------+-----+
                                    | XB (backup)|
                                    |            |       
                                    +------------+

In this scheme, XB would automatically backup XF and XS, and scripts could be used to convert XB into either an XF or XS as needed to call it "into service" as one or the other.

In my test environment, I have only two machines. 

+------------+          +------------+        +-------------------+
| Internet   |--------- | XF (squid) |--------| XS (schoolserver) |
| Modem      |          |            |        | LAMP+Moodle       |
+------------+          +------------+        +-------------------+

In this backup scheme:

  • XF will backup its own files and files from XS
  • XS will backup its own files and files from XF

This backup scheme is based on rsnapshot. See http://rsnapshot.org for details.

Under /library/.snapshots/ there will be the following directories

  • hourly.0 # most recent backup
  • hourly.1 # previous backup
  • hourly.2 # backup before that
  • daily.0 # yesterday's hourly.0
  • daily.1 # hourly.0 of the day before that
  • daily.2
  • daily.3
  • daily.4
  • weekly.0 # hourly.0 of last Monday's backup
  • weekly.1 # hourly.0 of previous Monday
  • weekly.2
  • weekly.3
  • monthly.0 # first backup of this month
  • monthly.1 # first backup of last month
  • monthly.2 # first backup of month before that

The backups use hard links, so if not much changes, the total amount of space is actually quite small.

To take a fresh backup, the following command can be used: 

[root@schoolserver ~]  rsnapshot hourly

This will delete hourly.2, move all hourly.1 files to hourly.2, move all files from hourly.0 to hourly.1 and then do an rsync to determine what has changed since the last backup.

Backups can be fully automated using a crontab schedule

This backup scheme is flexible enough to work on a single machine that combines squid/moodle, or a larger deployment with multiple machines, where every machine keeps the backups of two other machines.

XF (squid/filter/cache)

Here are the files and directories involved on XF

  • /root/.ssh/xfkey # this is a key generated on XF with no passphrase
  • /root/.ssh/xskey.pub # this is a key generated on XS with no passphrase
  • /root/bin/remote-xs.sh # this script will invoke backup-xs.sh script on XS
  • /root/bin/backup-xf.sh # XS remote-xf.sh will invoke this script
  • /etc/rsnapshot.conf # rsnapshot performs rsync over ssh connection
  • /library/.snapshots/ # all of the backups will be kept here
  • /library/.snapshots/hourly.0/local-xf/ # backups of local XF taken here
  • /library/.snapshots/hourly.0/remote-xs/ # backups of remote XS taken here over SSH

rsnapshot.conf

This is the /etc/rsnapshot.conf I used. 

#################################################
# rsnapshot.conf - rsnapshot configuration file #
#################################################
#                                               #
# PLEASE BE AWARE OF THE FOLLOWING RULES:       #
#                                               #
# This file requires tabs between elements      #
#                                               #
# Directories require a trailing slash:         #
#   right: /home/                               #
#   wrong: /home                                #
#                                               #
# updated by Tony Pearson 02/26/2008 xf         #
#################################################

#######################
# CONFIG FILE VERSION #
#######################

config_version	1.2

###########################
# SNAPSHOT ROOT DIRECTORY #
###########################

# All snapshots will be stored under this root directory.
#
# /library is LVM with most space 
#
snapshot_root	/library/.snapshots/

# If no_create_root is enabled, rsnapshot will not automatically create the
# snapshot_root directory. This is particularly useful if you are backing
# up to removable media, such as a FireWire or USB drive.
#
#no_create_root	1

#################################
# EXTERNAL PROGRAM DEPENDENCIES #
#################################

# LINUX USERS:   Be sure to uncomment "cmd_cp". This gives you extra features.
# EVERYONE ELSE: Leave "cmd_cp" commented out for compatibility.
#
# See the README file or the man page for more details.
#
cmd_cp		/bin/cp

# uncomment this to use the rm program instead of the built-in perl routine.
#
cmd_rm		/bin/rm

# rsync must be enabled for anything to work. This is the only command that
# must be enabled.
#
cmd_rsync	/usr/bin/rsync

# Uncomment this to enable remote ssh backups over rsync.
#
cmd_ssh		/usr/bin/ssh

# Comment this out to disable syslog support.
#
cmd_logger	/usr/bin/logger

# Uncomment this to specify the path to "du" for disk usage checks.
# If you have an older version of "du", you may also want to check the
# "du_args" parameter below.
#
cmd_du		/usr/bin/du

# Uncomment this to specify the path to rsnapshot-diff.
#
# yum install rsnapshot put everything under /usr/bin
#
cmd_rsnapshot_diff	/usr/bin/rsnapshot-diff

# Specify the path to a script (and any optional arguments) to run right
# before rsnapshot syncs files
#
#cmd_preexec	/path/to/preexec/script

# Specify the path to a script (and any optional arguments) to run right
# after rsnapshot syncs files
#
#cmd_postexec	/path/to/postexec/script

#########################################
#           BACKUP INTERVALS            #
# Must be unique and in ascending order #
# i.e. hourly, daily, weekly, etc.      #
#########################################

interval	hourly	3
interval	daily	5
interval	weekly	4
interval	monthly	3

############################################
#              GLOBAL OPTIONS              #
# All are optional, with sensible defaults #
############################################

# Verbose level, 1 through 5.
# 1     Quiet           Print fatal errors only
# 2     Default         Print errors and warnings only
# 3     Verbose         Show equivalent shell commands being executed
# 4     Extra Verbose   Show extra verbose information
# 5     Debug mode      Everything
#
verbose		2

# Same as "verbose" above, but controls the amount of data sent to the
# logfile, if one is being used. The default is 3.
#
loglevel	3

# If you enable this, data will be written to the file you specify. The
# amount of data written is controlled by the "loglevel" parameter.
#
logfile	/var/log/rsnapshot

# If enabled, rsnapshot will write a lockfile to prevent two instances
# from running simultaneously (and messing up the snapshot_root).
# If you enable this, make sure the lockfile directory is not world
# writable. Otherwise anyone can prevent the program from running.
#
lockfile	/var/run/rsnapshot.pid

# Default rsync args. All rsync commands have at least these options set.
#
# -a 		archive mode
# -r 		recursive
# -x 		do not span LVM boundaries
# --delete      delete files that don't exist on source
# --relative    relative directory names
# --numeric-ids don't map uid/gid from user/group names
#
rsync_short_args	-arx
rsync_long_args	--delete --numeric-ids --relative 

# ssh has no args passed by default, but you can specify some here.
#
# xfkey must have been generated on this system with no passphrase
ssh_args	-i /root/.ssh/xfkey

# Default arguments for the "du" program (for disk space reporting).
# The GNU version of "du" is preferred. See the man page for more details.
# If your version of "du" doesn't support the -h flag, try -k flag instead.
#
#du_args	-csh

# If this is enabled, rsync won't span filesystem partitions within a
# backup point. This essentially passes the -x option to rsync.
# The default is 0 (off).
#
#one_fs		0

# The include and exclude parameters, if enabled, simply get passed directly
# to rsync. If you have multiple include/exclude patterns, put each one on a
# separate line. Please look up the --include and --exclude options in the
# rsync man page for more details on how to specify file name patterns. 
# 
#include	???
#include	???
#exclude	???
#exclude	???

# The include_file and exclude_file parameters, if enabled, simply get
# passed directly to rsync. Please look up the --include-from and
# --exclude-from options in the rsync man page for more details.
#
#include_file	/path/to/include/file
#exclude_file	/path/to/exclude/file

# If your version of rsync supports --link-dest, consider enable this.
# This is the best way to support special files (FIFOs, etc) cross-platform.
# The default is 0 (off).
#
link_dest	1	

# When sync_first is enabled, it changes the default behaviour of rsnapshot.
# Normally, when rsnapshot is called with its lowest interval
# (i.e.: "rsnapshot hourly"), it will sync files AND rotate the lowest
# intervals. With sync_first enabled, "rsnapshot sync" handles the file sync,
# and all interval calls simply rotate files. See the man page for more
# details. The default is 0 (off).
#
#sync_first	0

# If enabled, rsnapshot will move the oldest directory for each interval
# to [interval_name].delete, then it will remove the lockfile and delete
# that directory just before it exits. The default is 0 (off).
#
#use_lazy_deletes	0

###############################
### BACKUP POINTS / SCRIPTS ###
###############################

# LOCALHOST
backup_script		/root/bin/backup-xf.sh	unused0
backup	/home/		local-xf/
backup	/root/		local-xf/
backup	/etc/		local-xf/
#backup	/usr/		local-xf/
backup	/var/log/	local-xf/

# SCHOOLSERVER
backup_script	/bin/date "+ backup of schoolserver started at %c"	unused1
backup_script	/root/bin/remote-xs.sh		unused2
backup	root@schoolserver:/home/	remote-xs/
backup	root@schoolserver:/root/	remote-xs/
backup	root@schoolserver:/etc/		remote-xs/
backup	root@schoolserver:/var/log	remote-xs/
backup_script	/bin/date "+ backup of schoolserver ended  at %c"	unused3

Here is my remote-xs.sh script (MIT license) 

#!/bin/sh
#
# Copyright (c) 2008 Tony Pearson.
#
# Licensed under the MIT license for contribution to the 
# One Laptop per Child (OLPC) foundation.
# 
# Permission is hereby granted, free of charge, to any person 
# obtaining a copy of this software and associated documentation 
# files (the "Software"), to deal in the Software without 
# restriction, including without limitation the rights to use, 
# copy, modify, merge, publish, distribute, sublicense, and/or sell 
# copies of the Software, and to permit persons to whom the 
# Software is furnished to do so, subject to the following 
# conditions: 
# 
# The above copyright notice and this permission notice shall be 
# included in all copies or substantial portions of the Software. 
# 
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 
# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES 
# OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND 
# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT 
# HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, 
# WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING 
# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR 
# OTHER DEALINGS IN THE SOFTWARE. 
# 
# For more information see the Open Source Initiative:
# http://www.opensource.org/licenses/mit-license.php
#
KEY="/root/.ssh/xfkey"
DEST="root@schoolserver"
ssh -i $KEY $DEST "/root/bin/backup-xs.sh"
Retrieved from "http://wiki.laptop.org/mediawiki/index.php?title=User:Az990tony/backup&oldid=112986"