Why Replication?

• Backups. As part of a backup strategy, replication avoids shutting down the master if backups are made from the slave.
• Scale-out. Replication can allow the distribution of reads over multiple replication slaves.
• Reliability. Replication can provide a ready backup in the event of a failure.

This post will only address the use of replication in a backup strategy, but the setup steps are valid for other uses as well.

Assumptions:

• The master database has existing data.
• The master and slave have different IP addresses.
• The master database to be replicated is named exampledb.
• The database exampledb exists on the master but not on the slave.
• The database uses MyISAM tables.
• The master and slave run on Fedora. This should only affect the directory structure.
• The master and slave use port 3306. This affects firewall setup and Master configuration. To use a different port, set --master-port in the CHANGE MASTER TO command.

Preliminary steps.

Configure the firewall to allow the master and slave to communicate.

You have a firewall, right? Assuming the master server is 2xx.xx.xx.01 and the slave is 2xx.xx.xx.02 corresponding to server_id=1 for the master and server_id=2 for the slave.

On the master add the following to /etc/sysconfig/iptables

# Allow incoming and outgoing traffic on port 3306 for MySQL slave server
-A INPUT -p tcp -s 20x.xx.xx.02 --sport 1024:65535 -d 2xx.xx.xx.01 --dport 3306 -m state --state NEW,ESTABLISHED -j ACCEPT
-A OUTPUT -p tcp -s 2xx.xx.xx.01 --sport 3306 -d 2xx.xx.xx.02 --dport 1024:65535 -m state --state ESTABLISHED -j ACCEPT

On the slave add the following to /etc/sysconfig/iptables

# Allow incoming and outgoing traffic on port 3306 for MySQL slave server
-A INPUT -p tcp -s 2xx.xx.xx.01 --sport 1024:65535 -d 2xx.xx.xx.02 --dport 3306 -m state --state NEW,ESTABLISHED -j ACCEPT
-A OUTPUT -p tcp -s 2xx.xx.xx.02 --sport 3306 -d 2xx.xx.xx.01 --dport 1024:65535 -m state --state ESTABLISHED -j ACCEPT

Restart the firewall

# /sbin/service iptables restart

Create a replication user, repl on the slave server.

# /usr/sbin/useradd -r repl
# /usb/bin/passwd repl

Grant privileges to the replication user on the master server.

mysql> GRANT FILE, REPLICATION SLAVE ON *.*
    -> TO 'repl'@'%' IDENTIFIED BY 'password';

Prepare the Master

In order to prepare the master, it will have to be locked for a short time. While it’s locked we will create a dump file, and edit the configuration in /etc/my.cnf.

You must enable binary logging so we’ll start by configuring the master server binary logging and restarting the master.

Each server in the replication group must have a unique id. The master server is typically assigned an id of 1.

The master server must allow networking, so ensure that --skip-networking and --bind-address are commented out if present.

While your at it, check that the pemissions of the file my.cnf are 0400 or 0600.

Edit the master database’s configuration in the file /etc/my.conf. Add the lines under the [mysqld] section.

[mysqld]
# enable networking and listen on all IP addresses
# by commenting out the following two lines if they exist
#skip-networking
#bind-address            = 127.0.0.1
# The following line will create the log file
# /var/lib/mysql/mysql-bin.00000x
log-bin=mysql-bin
# set expire_logs_days no lower than the number of days the slave is behind
expire_logs_days=3
server-id=1
# substitue 'hostname' with the actual name of your host
relay-log='hostname'-relay-bin
# duplicate the following line for every database that needs to be replicated
# substitute 'exampledb' with actual name of the database to be replicated
binlog-do-db='exampledb'

Restart the master

# /etc/init.d/mysqld --report-host restart

The --report-host option will make checking the replication status easier.

Check the error log at /var/log/mysqld.log after restarting mysqld.

Next we will lock the master database, record the binary log file’s position, and create a database dump that we’ll use to initialize the slave database.

Lock the database

# mysql -p -u root
mysql> use exampledb;
mysql> FLUSH TABLES WITH READ LOCK;

Obtain status of the binary log on the master.

mysql > SHOW MASTER STATUS;
+------------------+----------+--------------+------------------+
| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+------------------+----------+--------------+------------------+
| mysql-bin.000001 |       98 | exampledb    |                  |
+------------------+----------+--------------+------------------+
1 row in set (0.00 sec)

Record File and Position, you’ll need it to synchronize the slave.

Leave this client running while the dump file is created.

Create a dump backup file

# mysqldump -u root -p exampledb > exampledb.sql

On master, unlock the tables

mysql > UNLOCK TABLES;

Setting up the slave

Edit the slave’s configuration in the file /etc/my.cnf

[mysqld]
log-bin=mysql-bin
master-host=master-hostname
master-user=slave-user
master-password=slave-password
server-id=2
# duplicate the following line for every database that needs to be replicated
# substitute 'exampledb' with actual name of the database to be replicated
replicate-do-db='exampledb'
report-host='slave-hostname'

Since the master.info file overides settings in my.cnf, you may prefer to set up the slave using the CHANGE MASTER TO command. As a minimum, you need to use the CHANGE MASTER TO command to set the values of MASTER_LOG_FILE and MASTER_LOG_POS

Change the option values to the actual values for your servers

mysql> CHANGE MASTER TO
    ->     MASTER_HOST='master_host_name',
    ->     MASTER_USER='replication_user_name',
    ->     MASTER_PASSWORD='replication_password',
    ->     MASTER_LOG_FILE='recorded_log_file_name',
    ->     MASTER_LOG_POS='recorded_log_position';

Start the slave

$ /etc/init.d/mysqld --skip-slave start

The --skip-slave option tells the server not to start the slave threads. We’re using this option because we aren’t quite ready to replicate.

Create the database

# mysql -u root -p
mysql> create database exampledb;
mysql> grant all on exampledb.* to 'repl'@'localhost';

Import the dump file

# mysql -u root -p exampledb < full_dump_file.sql

Start the slave threads

$ mysql> START SLAVE;

Checking the status of the master and slave

On slave

mysql> SHOW SLAVE STATUS\G;

*************************** 1. row ***************************
             Slave_IO_State: Waiting for master to send event
                Master_Host: fluidrails.com
                Master_User: repl
                Master_Port: 3306
              Connect_Retry: 60
            Master_Log_File: mysql-bin.000004
        Read_Master_Log_Pos: 98
             Relay_Log_File: hakota-relay-bin.000073
              Relay_Log_Pos: 235
      Relay_Master_Log_File: mysql-bin.000004
           Slave_IO_Running: Yes
          Slave_SQL_Running: Yes
            Replicate_Do_DB: garden
        Replicate_Ignore_DB:
         Replicate_Do_Table:
     Replicate_Ignore_Table:
    Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
                 Last_Errno: 0
                 Last_Error:
               Skip_Counter: 0
        Exec_Master_Log_Pos: 98
            Relay_Log_Space: 235
            Until_Condition: None
             Until_Log_File:
              Until_Log_Pos: 0
         Master_SSL_Allowed: No
         Master_SSL_CA_File:
         Master_SSL_CA_Path:
            Master_SSL_Cert:
          Master_SSL_Cipher:
             Master_SSL_Key:
      Seconds_Behind_Master: 0
1 row in set (0.00 sec)

ERROR:
No query specified

On master

mysql> SHOW MASTER STATUS\G;
*************************** 1. row ***************************
File: mysql-bin.000004
Position: 98
Binlog_Do_DB: garden
Binlog_Ignore_DB:
1 row in set (0.00 sec)

ERROR:
No query specified

mysql> SHOW PROCESSLIST\G;
+------+------+-----------------------+------+-------------+------+----------------------------------------------------------------+------------------+
| Id   | User | Host                  | db   | Command     | Time | State                                                    | Info             |
+------+------+-----------------------+------+-------------+------+----------------------------------------------------------------+------------------+
| 6235 | repl | www.hakota.com:43033  | NULL | Binlog Dump | 3250 | Has sent all binlog to slave; waiting for binlog to be updated | NULL             |
+------+------+-----------------------+------+-------------+------+----------------------------------------------------------------+------------------+

mysql> SHOW SLAVE HOSTS;
+-----------+------------+------+-------------------+-----------+
| Server_id | Host       | Port | Rpl_recovery_rank | Master_id |
+-----------+------------+------+-------------------+-----------+
|         2 | hakota.com | 3306 |                 0 |         1 |
+-----------+------------+------+-------------------+-----------+
1 row in set (0.00 sec)

Logs and Backups

Now that we’ve enabled binary logging we need to implement a process to manage the logs. One approach is to use SHOW SLAVE STATUS (on slave) and SHOW BINARY LOGS (on master) to determine which logs are in use. Suppose the slave is using Master_Log_File: master-bin.000004 Make sure your backups contain the logs about to be deleted, those prior to the target log master-bin.000004, then

mysql> PURGE MASTER LOGS TO 'master-bin.000004';

will delete all logs prior to master-bin.000004

In addition, now that we’ve implemented replication, our backup strategy needs to include the master.info and relay-log.info status files as well as the binary logs.

Before executing mysqldump on the slave, replication should be temporarily stopped.

# mysqladmin -u root -p  stop-slave
# mysqldump -u root -p exampledb > exampledb.sql
# mysqladmin -u root -p start-slave

References

MySQL 5.0 Reference Manual, Chapter 15 Replication

You will need to use a version of vim with GUI support, gvim, on Fedora, to ensure the colors are correct. You can install gvim on Fedora with the following command

$ sudu yum install gvim*

On Fedora, syntax scripts are in the directory /usr/share/vim/vim70/syntax/ and color schemes are in the directory /usr/share/vim/vim70/colors/. You can get other scripts from vim online.

Launch gvim from a terminal or the menu Applications->Programming->Vi Improved.

If you want embedded style in the HTML output then issue the colon command

~
:let html_use_css = 1

If you like the vividchalk color scheme, issue the command

~
:colorscheme vividchalk

To create an HTML version of the file, issue the command

~
:TOhtml

and save the HTML file by issuing the “ZZ” command, and quit the session to leave the original file unchanged with the command

~
:q!

The :TOhtml command produces a complete HTML page. To embed the highlighted syntax within a post, simply change the <body> tag to an appropriate CSS tag for your theme and delete the <html> and <head> sections. Here is a snippet as an example of embedding the highlighted source within a blog post. The HTML was created with the commands

~
:let html_use_css = 0

~
:TOhtml

Without CSS, the generated HTML needs to be enclosed in a tag that can enclose the <font> tag, for example, the <code> tag.

class Zipcode < ActiveRecord::Base
  belongs_to :postoffice
  has_many :locations

  def self.find_by_postoffice_name(postoffice_name)
    postoffice_id = Postoffice.find_by_name(postoffice_name).id
    find(:all, :conditions => [“postoffice_id = ?“, postoffice_id])
  end

  def self.find_by_postoffice_id(postoffice_id)
      find(:all, :conditions => [“postoffice_id = ?“, postoffice_id])
  end
end

As an added bonus, the output from vim using css validates to HTML 4.01 Strict.

See the vim documentation for a complete rundown on syntax highlighting in vim.

You won’t be seeing a “How-to install WordPress” post on this blog in the style of Installing Typo on Fedora 6 slice from Slicehost because installing WordPress is just plain simple, and it works.

And there you have it, WordPress vs. Typo in one sentence. There are other reasons of course, but I should have seen the other difficulties to come right from the start. More often than we want to admit, first impressions do matter.

For backing up the database we won’t benefit from the snapback technique because the data will change often enough that a compressed archive will be more efficient then hard links (see analysis in Website Backup Strategies). Because the script archives and compresses the dump files, rsync will treat them as changed even if the data in the database hasn’t changed. Even if you don’t archive and compress the dump files, you must still take care to ensure that the dump files are not considered different by rsync due to creation time (use --size-only), and that the dump files are not trivially different due to metadata like timestamps embedded in the file mysqldump (use --skip-comments). You can use also use snapback on /var/lib/mysql, but its not guaranteed to work.

The strategy will be to set up a cron job to run a script on the local backup server that will invoke a script on the remote server. The script on the remote server will execute mysqldump. The reason for invoking a script on the remote sever is that we don’t want to allow databases connections except from localhost.

I’ve chosen to use mysqldump for the flexibility to edit SQL if needed. The downside is that we are only capturing the individual databases, not the user tables or grants. Additionally, I’m going to dump the table schema separately from the data so that I don’t waste space and bandwidth backing up tables like cache.

Remote Script

The remote script called dbbackup.sh (download) has the following usage

Usage: dbbackup.sh [OPTIONS] "

Where [OPTIONS] are :

<-d database name> is the name of the MySQL database
<-u database user> is database user
<-p password> is the database user's password 

destination is the local directory where backups will be stored

The general design of the remote script is as follows:

• process command line arguments
• change directories to the destination
• get a list of tables in the database
• create a list of tables for which data will be backed up
• execute mysqldump –no-data on all tables
• archive and compress the dump
• test the compressed archive
• execute mysqldump –no-create-info on selected tables
• archive and compress the dump
• test the compressed archive

Local Script

The local script named dbsnapshot.sh (download) has the following usage:

Usage: dbsnapshot.sh [OPTIONS] "

Where [OPTIONS] are :

[-t type] one of hourly, daily, weekly, monthly
[-P port] is the ssh port number
<-s server> is the remote server's as identified in known_hosts
<-d database names> database names should be comma separated
                                     or quoted white space
<-u database user> is database user
<-p password> is the database user's password
<-H [n]> where n is the number of hourly backups,default is 6
<-D [n]> where n is the number of daily backups, default is 7
<-W [n]> where n is the number of weekly backups, default is 4
<-M [n]> where n is the number of monthly backups, default is 12"

source is the directory where backups are on the remote server
destination is the local directory where backups will be stored

The general design of the local script is as follows:

• process command line arguments
• set the max number of backups based on type, -t argument
• rotate the existing backups
• create new backup by invoking remote script
• copy the new backup to local
• delete oldest backup that has rotated off the stack

Crontab

The script is meant to be run as a cron job. Your crontab will typically look something like this:

0 */4 * * * dbsnapback.sh [OPTIONS] -t hourly src dst >>  dev/null >2&1
59 3 * * * dbsnapback.sh [OPTIONS] -t daily scr dst  >> dev/null >2&1
58 3 * * 0 dbsnapback.sh [OPTIONS] -t weekly scr dst  >> dev/null >2&1
57 3 1 * * dbsnapback.sh [OPTIONS] -t monthly scr dst  >> dev/null >2&1

To Do List

Like any script, these two scripts could use some improvements. I’ll point out the weaknesses that I see as fair warning.

• The remote script name and path is hard-coded in the local script
• The tables for which data is not backed up are Drupal specific. Probably should be a command line option to take a list of tables that are to be excluded.
• The directory that stores the backups should be mounted read only after the script runs.
• The compressed archives should be removed from the remote server after being copied to the backup server
• rsync is not preserving original permissions and ownership, need to look into –link-dest
• the script should use a configuration file because there are too many command line arguments and most are required

That wraps up the first part of the series for website backup strategies. In the next parts we’ll cover how a site is restored and how a site is upgraded.

Part I Part II Part III

In Part II Revision Control I mentioned my preference for git over other revision control systems. One reason is that git is guaranteed to give you back what you put into it because it computes sha1 and notices things like disk corruption. Recall, I mentioned in the section on svn that you might want to mirror your svn repository with svnsync. That only addresses corruption that may occur after a commit to the respository, it doesn’t do anything for you if the corruption occurs during he commit. The other nice aspect of git is that it is distributed so that every repository is in a sense a backup if we can match the signature. For these reasons, the next part of our backup strategy will backup up the entire set of files accessible to the web, including those under revision control.

As I mentioned in part I Website Backup Strategies, the snapback technique is ideally suited for website backups because the files are not expected to change very often. The snapback technique is well documented on Mike Rubel’s site. It takes advantage of hard links to reduce space and rsync to reduce bandwidth for files that don’t change often over the backup time horizon. The method depends on having a /usr/bin/cp that can make hard links, like GNU cp -al does.

As I mentioned in part I, there is an excellent implementation called snapback2 from Perusion.

In this post, I’m going to walk through an example where a local machine is used as a backup server to access the remote web server via ssh that hosts the site we want to backup. If you don’t have ssh setup, see my post Securing a new Fedora 6 Slice. The local server needs to have Perl installed and snapback2 depends on Config::ApacheFormat. The snapback2 README covers installation, and the Perusion site also has online documentation. The documentation is also installed as a manpage.

$ cd /usr/local/src

$ /usr/bin/wget \
> http://search.cpan.org/CPAN/authors/id/M/MI/MIKEH/Snapback2-0.913.tar.gz

$ /bin/tar xvf Snapback2-0.913.tar.gz

$ cd Snapback2-0.913

$ /usr/bin/perl Makefile.PL

$ /usr/bin/make

$ /usr/bin/make test

$ /usr/bin/make install

On Fedora, you can set up Perl to get modules from CPAN

$ /usr/bin/yum install perl-libwww-perl
$ /usr/bin/perl -MCPAN -e shell
> get Bundle::CPAN

Then installing a new module can be as simple as typing

$ /usr/bin/perl -MCPAN -e 'install Config::ApacheFormat'

To configue snapback2, you use an apache-like configuration file, /etc/snapback/snapback.conf

    Hourlies    6
    Dailies     7
    Weeklies    4
    Monthlies  12
    AutoTime   Yes

    AdminEmail webmaster@mysite.com
    LogFile    /home/myuser/var/log/snapback.log
    Exclude *debug
    Exclude core.*
    SnapbackRoot /etc/snapback
    RsyncShell '/usr/bin/ssh -i /home/myuser/.ssh/id_dsa -p 2222'

    Destination /home/myuser/mnt/backup1

    <Backup mysite.com>
      Directory /var/www/html/mysite/
    </Backup>

Note the RsyncShell directive used to set up an ssh connection with a key and port number.

Next we need to setup a cron job. See this intro if you unfamiliar with cron.

    $ /usr/bin/crontab -e

Add the line

10 */4 * * * /usr/bin/snapback2

Note that the configuration file specified Hourlies 6 and the cron is set up to run every 4 hours, covering 24 (6 * 4) hours with six backups every four hours.

Because cron is not run under myuser, the ssh keys should not have a passphrase unless you are using something like keychain. If you use passphrase, be aware that the process will not survive a reboot automatically, you will have to reenter the passphrase after a reboot. Obviously, the keys without a passphrase must be safeguarded.

In the next part, I’ll detail how to set up something very similar to snapback to backup a MySQL database using bash scripts on the local and remote servers.

Part I Part II Part IV

CVS

Drupal is maintained in cvs. So cvs seems like a natural option. Pro Drupal Development covers this quite well in Chapter 21 “Development Best Practices”. There is a very short paragraph in the chapter about mixing SVN and CVS to deal with custom source.

Assuming cvs is installed, using CVS is as easy as

$ cd /var/www/html
$ /usr/bin/cvs \
> -d:pserver:anonymous:anonymous@cvs.drupal.org:/cvs/drupal checkout \
> -r DRUPAL-5 mysite

Which will create /var/www/html/drupal

And updates are as simple as

$ cd /var/www/html/drupal
$ /usr/bin/cvs update -dP -r DRUPAL-5

But if you customize anything within the core source, then you’ll start to run into some difficulty. For starters, you will need to store the custom code in a different repository. Remember the short paragraph in Pro Drupal Development about mixing CVS and SVN? If custom source is not stored in another repository, the custom changes will not survive the update.

I’m pretty sure that I’ll be customizing something, headers, logos, maybe some javascript to get transparent images to work properly in IE. So, for my purposes, I’m going to look for another approach.

SVN

Using a svn repository solves the problem of keeping core source and custom source in a single repository at the cost of a little additional work. In the case of Drupal, the source is not in SVN so we cannot simply checkout the source from the development team repository. If you are new to svn, a good resource is Mike Mason’s excellent book Pragmatic Version Control: Using Subversion(2nd Edition). Another good resource is Version Control with Subversion

The following workflow assumes everything is being done on a local machine, and the user has appropriate permissions.

Create a repository

$ mkdir /var/svn/repos
$ svnadmin create /var/svn/repos

Get the source

$ cd /usr/local/src
$ wget http://ftp.drupal.org/files/projects/drupal-5.1.tar.gz

Uncompress the archive

$ tar xvf drupal-5.1.tar.gz

Import the source into the repository

$ svn import --no-auto-props -m"initial import of drupal 5.1" \
> drupal-5.1 file:///var/svn/repos/vendorsrc/drupal/current

Create a tag

$ svn copy -m"Tag 5.1 vendor drop" \
> file:///var/svn/repos/vendorsrc/drupal/current \
> file:///var/svn/repos/vendorsrc/drupal/5.1

Create project

$ svn mkdir -m"Create project myproject" file:///var/svn/repos/myproject

Copy into main development branch

$ svn copy file:///var/svn/repos/vendorsrc/drupal/5.1 \
> file:///var/svn/repos/myproject/trunk

Check out the project to create a working directory

$ svn co file:///var/svn/svnrepos/myproject/trunk \
> /var/www/html/myproject

Ignore files/ directory

$ svn mkdir files files/images files/images/temp files/css files/color

$ svn propset svn:ignore "*" files/ files/images \
> files/images/temp files/css files/color

$ svn commit -m "ignore files/ and subdirectories content from now"

If you don’t set ignore on the files/ directory and its subdirectories then svn status will get pretty busy. You need to set the ignore property on files/ and each subdiretory in files/, for example, files/css, files/images, files/images/temp, etc.

Any custom changes we make can go into the same project respository.

When it’s time to upgrade Drupal to a new version, the basic update workflow is as follows:

• checkout the current version of the core source in a working directory

• make the working directory look like a pristine copy of the new version by copying, and svn adding or deleting as required. To make your life easier, use svn.load.dirs.pl

• commit the new version and add a tag

• merge the old and new versions in your project’s working directory make sure you use the tag because the branch moves, the tag is fixed

• resolve any conflicts and commit the changes to your project

More details can be found at Vendor Branches

Finally, you may want to consider using svnsync to mirror your repository.

GIT

In part one, I mentioned the excellent article on Verion Control Blog. This is my preferred process for source control of Drupal powered sites. The article is a very comprehensive how-to, so I don’t need to duplicate it here.

The basic workflow involves creating lines of development for

• drupal - contains the core source distribution
• drupal-and-modules is a clone of drupal plus contributed modules and themes
• drupal-production is a clone of drupal-and-modules with project customizations

The lines of development are chained together with cloning and changes propogate from drupal/ through drupal-production/.

I will point out one area of concern with the approach as detailed in the article. Exploding the latest version of Drupal in the drupal/ line of development does not deal with files or directories that may have been deleted in the latest version.

For more info on git, you should start with the documentation.

So there you have it, three approaches to keeping the Drupal source under revision control. I’ve stated my preference but everyone’s situation is different and you can choose.

In the next part, I’ll detail how to backup the files that are not under revision control, for example, the files/ directory. The final segment will address backing up the MySQL database.

Part I Part III Part IV

In previous posts, I’ve covered installation of Rails and Drupal, and also some rudimentary security. But running a successful site requires a strategy for upgrades and backups. In this post I’m going to introduce some initial concepts for a web site backup strategy that will also touch on upgrades. I’ll be using a Drupal CMS powered site as an example. The principles should apply for other LAMP based web sites as well. In subsequent posts, I’ll be getting into the how-to, to implement the strategy. I’ll start by stating the desired goals for the backup strategy.

Goals

• Simple; Use existing tools
• Automatic; Set up as a cron job
• Secure; Don’t introduce new security risks
• Efficient; Minimize space, bandwidth, and restoration time

I find that it’s helpful to categorize the types of files that make up a typical Drupal web site.

• Drupal source files
• Contributed modules and themes
• Custom versions of the above
• Files uploaded to the site
• The database

Source, Contributed source, and custom source

Drupal source files, contributed modules and themes, and custom versions of the latter will not change often and it makes sense to maintain them in a revision control repository.

You have lots of choices for a revision control system. I’ve decided to use git based on an excellent article on Version Control Blog. I like git because the separation of core Drupal from the contributed modules and the custom code seems more intuitive then branching in other systems.

Uploaded Files

Files uploaded to the site, also will not often change once uploaded. However, they may be deleted and there will always be new files to deal with. Uploaded files will be added and deleted by users using the CMS, so they will be outside of revision control system. Because of the administrative burden of adding and deleting uploaded content to revision control, it doesn’t make sense to try to keep them in a repository. If you have a novel solution that solves this problem I’d love to hear from you.

In Drupal, uploaded files are stored in a directory /files making it easy to backup just that directory. The /files directory is an excellent candidate for a novel rotataing backup system sometimes known as snapback. Snapback is based on the work of Mike Rubel and others. The basic idea is that for files that haven’t changed over the backup horizon, hard links instead of copies are maintained. The hardl links significantly reduce the space and bandwidth required for the backups.

Luckily, there is an excellent Perl implementation of the snapaback strategy, called snapback2 from Perusian

Database

The database consists of structure and data. The structure of the database may not change very frequently however the data in several tables probably will. So it makes sense to backup up the database structure and data separately. By doing so, we can also avoid backing up data in cache, sessions, and watchdog tables. Like uploaded files, the database will be backed up periodically in a rotating system.

MySQL is a popular choice for LAMP based systems and Drupal, so we’ll assume a MySQL database and use the mysqldump program to create the backups.

Unfortunately the database backups will probably not benefit from the “snapback” technique. To see why, lets do some back-of-the-envelope calculations.

If g is the tar gzip compression factor, and p is the percentage of the change in the backup files, s is the total size of the files, k is the number of backups, then for snapback to use less space than compressed archives, the following must hold

sgk > spk + s(1-p)

Where s(1-p) is the size of the unchanged files of which there is only one copy due to hard links

Simplifying by dividing by s,

gk > pk + (1-p)

If we ignore (1-p) which is small when k is large, then it is clear that for the method to have benefit, the percent of changed files must be less then the compression factor.

g > p

In my testing, I’ve been seeing compression factors between .15 and .20 for mysqldump files. Its hard to imagine a database that changes less than 15-20%

On a directory where the files don’t change much, and g > p holds, then the break even number of backups, k is given by

k = ceil((1-p)/(g-p))

For example, assume the percent of change, p, is 5% and the compression factor, g, is 15%, then the break even number of backup copies, k, is 10. A typical backup scheme has 6 hourly, 7 daily, 4 weekly, and 12 monthly = 29 copies, call it thirty, so in this example there is about a 3X savings over compressed archives. This is why I like the snapback technique for the uploaded files.

Therefore, to implement the database backup, we’ll being using some simple bash scripts on the local backup server and the remote web server that execute mysqldump, archive and compress the output, and move it to the remote server securely.

Well, that wraps up the outline of our strategy. In my next series of posts, I’ll cover in detail what it takes to implement each part of our three-tiered backup strategy.

Part II Part III Part IV

Secure the Slice

The first steps are exactly the same as my previous post, Securing a new Fedora 6 Slice. If you followed the instructions in that post you have accomplished the following

• changed root’s password
• yum updated the base installation
• yum installed sudo
• created a user with sudo privileges
• created a public key on your local machine and copied it to your slice
• disabled password authentication via ssh
• diabled challenge response authentication via ssh
• changed the ssh port
• disabled root login via ssh
• installed denyhosts (optional)
• installed and configured a firewall using iptables

Install Required Software

Now that the slice is more secure, we can install the software required by Drupal.

• Apache Web Server, httpd
• MySQL Server
• PHP
• GD
• Sendmail

Login into the slice and yum install the following packages

$ /usr/bin/sudo /usr/bin/yum -y install \
> wget \
> tar \
> gzip \
> make \
> gcc \
> openssh-clients \
> mysql \
> mysql-server \
> php \
> php-mysql \
> php-devel \
> php-gd \
> gd \
> gd-devel \
> httpd  \
> sendmail \
> sendmail-mc \
> sendmail-cf

Start the mysqld server

$ /usr/bin/sudo /etc/init.d/mysqld restart

Ensure MySQL starts at boot

$ /usr/bin/sudo /sbin/chkconfig --add mysqld
$ /usr/bin/sudo /sbin/chkconfig --level 345 mysqld on

Secure Initial MySQL Accounts

see Securing the initial MySQL accounts

$  /bin/su
# /usr/bin/mysql -u root
mysql> SET PASSWORD FOR 'root'@'localhost' = PASSWORD('newpwd');
mysql> exit
# exit
$

Create a Drupal system user

$ /usr/bin/sudo /usr/sbin/useradd -r drupal

Create MySQL user account

see CREATE USER Sytnax

$ /bin/su
# mysql -p -u root
mysql> create user 'drupal'@'localhost';

Create the database for your Drupal site

mysql> create database mysite;
mysql> grant all on mysite.* to 'drupal'@'localhost';
mysql> exit;
# exit;
$

Download the Drupal software

see Download

Change directories to /usr/local/src

$ cd /usr/local/src

$ /usr/bin/sudo /usr/bin/wget \
> http://ftp.drupal.org/files/projects/drupal-5.2.tar.gz

$ /usr/bin/sudo /bin/tar xvf drupal-5.2.tar.gz

$ usr/bin/sudo /bin/cp -r drupal-5.2 /var/www/html/mysite

Make settings.php writeable by the web server user

$ cd /var/www/html/mysite

$ /usr/bin/sudo /bin/chown root.apache \
> /var/www/html/mysite/sites/default/settings.php

$ /usr/bin/sudo /bin/chmod g+w \
> /var/www/html/mysite/sites/default/settings.php

Make the files/ directory and its subdirectories writable by the web server user

$ /usr/bin/sudo /bin/mkdir files files/color files/css \
> files/images files/images/temp

$ /usr/bin/sudo /bin/chown root.apache files files/color files/css \
> files/images files/images/temp

$ usr/bin/sudo /bin/chmod g+w files files/color files/css \
> files/images files/images/temp

Set up cron

Add a shell script to /etc/cron.hourly

# !/bin/sh
# $Id: cron-curl.sh,v 1.3 2006/08/22 07:38:24 dries Exp $
curl  --silent --compressed http://mysite.com/cron.php

Get optional modules and themes

Setup Apache Web Server

edit /etc/httpd/conf/httpd.conf

#ServerName :80
ServerName www.mysite.com:80
#Listen 12.34.56.78:80
Listen your.slice.ip.addr:80

Add

<Files *.inc>
    Deny From All
</Files>
<Files *.class>
    Deny From All
</Files>
<Files MANIFEST>
    Deny From All
</Files>

See this article for tuning Apache for performance

Create a virtual host

Edit /etc/httpd/conf/httpd.conf

<VirtualHost hostname:80>
ServerAdmin webmaster@mysite.com
DocumentRoot /var/www/html/mysite
ServerName www.mysite.com

Options -Indexes +FollowSymLinks
ErrorLog logs/mysite-error_log
CustomLog logs/mysite-access_log combined
DirectoryIndex index.html index.html.var index.php

<Directory "/var/www/html/mysite">
  AllowOverride all
</Directory>
</VirtualHost>

Start the Web Server

$ /usr/sbin/apachectl configtest
$ /sbin/chkconfig --add httpd
$ /sbin/chkconfig --level 345 httpd on
$ /usr/sbin/apachectl start

Configure PHP

See Description of core php.ini directives

You may need to adjust uploadmaxfilesize postmaxsize There is a good security article on the Gallery2 site that is worth reading.

Install Drupal

Navigate to http://mysite.com/install.php and follow along with the online install. When finished, change permission on /var/www/html/mysite/sites/default/settings.php

$ /usr/bin/sudo /bin/chmod g-w /var/www/html/mysite/sites/default/settings.php

Setup Sendmail

Configure Linux Mail Servers is a comprehensive article, I’ll just hit the highlights.

Configure DNS correctly

Add the following records

• mail pointing to your slice’s IP,
• MX pointing to mail
• TXT pointing to v=spf1 a mx -all

Configure /etc/resolv.conf

Add the following line above the line nameserver

    domain mysite.com

Configure /etc/hosts

127.0.0.1       mysite.com localhost.localdomain localhost

Configure /etc/sendmail.mc

Make sure sendmail is listening on all interfaces (0.0.0.0)

$ /bin/netstat -an | grep :25 | grep tcp
tcp 0 0 0.0.0.0:25 0.0.0.0:* LISTEN

Comment out DAEMON_OPTIONS in /etc/mail/sendmail.mc if it is only listening on loopback

dnl DAEMON_OPTIONS(\`Port=smtp,Addr=127.0.0.1, Name=MTA')

Make sure these lines are commented out to avoid having your server used to forward spam

dnl FEATURE(`accept_unresolvable_domains')dnl
dnl FEATURE(`relay_based_on_MX')dnl

Configure /etc/mail/access

Add your domain

# by default we allow relaying from localhost...
Connect:localhost.localdomain           RELAY
Connect:localhost                       RELAY
Connect:127.0.0.1                       RELAY
Connect:mysite.com                      RELAY

Configure /etc/mail/local-host-names

Add all aliases for your server

    mysite.com

Configure /etc/mail/virtusertable

Add email address/user pairs

    root@mysite.com myuser
    webmaster@mysite.com myuser
    postmaster@mysite.com myuser
    info@mysite.com myuser
    abuse@mysite.com myuser
    apache@mysite.com myuser

Configure /etc/aliases

Edit user that receives root’s email

    # Person who should get root's mail
    #root:           marc
    root             webmaster@mysite.com

Update /etc/sysconfig/iptables by adding

#Allow mail
-A INPUT -p tcp --dport 25 -j ACCEPT
-A OUTPUT -p tcp --dport 25 -j ACCEPT
-A INPUT -p tcp --dport 110 -j ACCEPT
-A OUTPUT -p tcp --dport 110 -j ACCEPT

Optionally Configure spam tools

see Configure Linux Mail Servers

Optionally set up POP3

If you want to read your mail using a client on your PC, you need to set up POP3.

$ /usr/bin/sudo /usr/bin/yum -y install dovecot
$ /usr/bin/sudo /sbin/chkconfig --add dovecot
$ /usr/bin/sudo /sbin/chkconfig --level 345 dovecot on
$ /usr/bin/sudo /etc/init.d/dovecot start

Edit /etc/dovecot.conf

    #protocols = imap imaps pop3 pop3s
    protocols = pop3

Configure your client to receive mail from mail.mysite.com

I don’t recommend that try to configure your slice to relay mail from your PC’s client software. Just use your ISP’s SMTP server to send mail. But if you insist, read this guide first.

Optionally install a mail client

Pine is a simple lightweight mail reader that you can use to read mail from a terminal session when you are logged on to your slice.

$ usr/bin/sudo /bin/rpm -ivh http://rpm.livna.org/livna-release-6.rpm

ensure enable=1 is set to enable=0 in the following files

    /etc/yum.repos.d/livna.repo
    /etc/yum.repos.d/livna-devel.repo
    /etc/yum.repos.d/livna-testing.repo

This will disable the livna repository for regular yum updates

Then, you can install Pine with:

$ /usr/bin/sudo /usr/bin/yum --enablerepo=livna install pine

Here are two of the best ones …

• Marc Andreesen

• Joel Spolsky

In this post, I suggest that if you are trying to hire great workers, the process may be more critical to your success then your interviewing technique.

Your success in hiring great workers depends on how many great workers you have the opportunity to interview and your ability to interview.

Suppose you can identify the top 10% of any group 90% of the time. You get it wrong only 10% of the time. After you interview ten candidates, you’ve narrowed the field to one from the top 10, and one from the bottom 9. If you pick one now, you’re looking at a 50% chance that you pick correctly. You don’t like symmetry? Ok, you only get it wrong 5% of the time. Your odds of picking correctly have only improved to 2/3. Clearly, its best not to choose after a single evaluation.

What can be done to improve the odds of choosing correctly? You can either increase the number of great hires within the population of candidates you interview, or use a process of sequential multiple evaluations. This post will focus on the latter approach, improving the process. Unfortunately, everything we do to make our company attractive to great workers will make it attractive to everyone.

One way to improve your results is to make a serial sequence of decisions that narrows the field such that the number of qualified candidates remaining after each decision increases as a percentage of the total remaining candidates. In other words, the process has the effect of increasing the probability that any of the remaining candidates meets the criteria. A single elimination process is the easiest to illustrate.

In this post, I won’t address the merits of any particular evaluation technique. Pick any you like that is more effective than a coin toss. That’s not a flippant comment; you have to do better than 50/50 for the following process to work. Lou Adler has some interesting things to say about how to improve your interview techniques.

What does this look like in practice? It looks like American Idol. The candidate pool is evaluated and only candidates that pass the first evaluation continue. In successive rounds the evaluation focuses on different criteria that are increasingly more challenging and more relevant.

To illustrate, I’ll use another numerical example. Suppose the goal is to hire someone who is in the top 20% of qualified candidates. For the sake of argument assume all interviewers can pick winners 80% of the time and they mistakenly pick unqualified candidates only 20% of the time.

Because I like easy numbers, suppose 100 candidates are to be evaluated, and we need to pick one from the top 20.

After the first interview, 16 from the top 20% make it to the next round and 16 from the bottom 80% go forward as well. Sixty-eight are sent home. If a choice is made after the first interview, again, we only have a 50/50 chance of getting it right. After the second interview, 13 from the top 20% survive and 3 from the bottom 80% survive. One more round, and we’re done with over 94% probability that we pick a top 20% candidate. Pick any numbers you’d like, apply Bayes theorem, the results will point to the same process. In each iteration we are increasing the prior probability that a surviving candidate meets our criteria.

Most interviewing processes don’t work this way. Usually, a group interviews every candidate and a group decision is made. Invariably, the group is dominated by one or more influential members. Effectively, there is a single decision maker. But we’ve seen that even if an interviewer has a very good technique for selecting qualified candidates, she doesn’t have a very good chance of getting it right when the prior probability of a candidate meeting the criteria is low.

In the single elimination process I’ve described, it’s important that the interviewers do not discuss candidates prior to interviewing the candidate, or prior to making their decision. It’s equally important that they understand the math and don’t rely on the fact that the candidates have survived previous interviews.

What are the downsides to a single elimination process? No matter how you go about it, qualified candidates will not be selected. Therefore, it’s important that candidates understand the process and are treated respectfully.

Thanks to Greg Yut of Supply Beyond for reading and commenting on a draft of this post.

• Get ready
• Clear Drupal cache
• Dump current Drupal database to a backup file
• Create new database and grant privileges
• Run stream editor against db backup file to fix paths
• Load new database
• Ensure current Drupal site directory is updated in svn
• Export current site from repository to new directory
• Edit files/default/settings.php to point to new db
• Configure Apache to use new directory
• Enter a new A record in DNS
• Test

To get ready, turn off css cache and make sure $base_url is commented out in your files/default/settings.php file. I also disabled clean URLs, not sure if it’s necessary. Your Drupal directory should already be under subversion control. Put up your maintenance page since we are going to clear cache.

The next step is to clear Drupal cache and dump the database to a backup file.

Put the following code in a page, and set up a menu to access it. Make sure you restrict access to admin for this menu. Don’t be lazy and do it from the command line - it’ll be faster this time and slower next.

db_query("DELETE FROM {cache} WHERE 1");
db_query("DELETE FROM {cache_filter} WHERE 1");
db_query("DELETE FROM {cache_menu} WHERE 1");
db_query("DELETE FROM {cache_page} WHERE 1");

Ok, for the command line purists, it’s like this

mysql> DELETE FROM cache WHERE 1;

Then dump the database to a backup file

$ /usr/bin/mysqldump dbname > mysite-backup.sql

Make the current site accessible again by taking down your maintenance page.

Create a new database and grant privileges

mysql> create database newdbname;
mysql> grant all on newdbname.* to 'drupaluser'@'localhost';

Before loading the new database we need to run a stream editor against the backup file and change any hard coded paths.

Check the backup file by grep’ing for the previous subdomain and path of the current install directory

$ /bin/grep -r mysite *

$ /bin/grep -r www *

Then edit something like this

$ /usr/bin/perl -pi.bak -e's/http:\/\/www.example/http:\/\/subdomain.example/g;
> s/\/html\/mysite/\/html\/mynewsite/g;' mysite-backup.sql

And load the new database

$ /usr/bin/mysql newdbname < mysite-backup.sql

My Drupal installation was in /var/www/html/mysite under subversion control. After cleaning up the working directory and checking in all changes, I exported mysite into a new working directory.

$ cd /var/www/html
$ /usr/bin/svn export file:///var/svn/repos/mysite mynewsite

Make sure the files/ directory is writable by the web server process owner.

Edit files/default/settings.php and change the name of the database, and make sure $base_url is commented out.

Configure a new virtual host in /etc/httpd/conf/httpd.conf for subdomain.example.com. Copy the <virtualhost> entry for www.example.com and change the ServerName, DocumentRoot, and log file name. If you have any Directory entries, change the path to your new directory. Check your configurations changes

$ /usr/sbin/apachectl configtest

And restart

$ /usr/sbin/apachectl graceful

Make sure you have an A record in DNS for subdomain.example.com

Finally test every page.