Notes from the Rackspace Breakfix Competition

Here is a special guest post from Robert Taylor, a Rackspace Senior Systems Engineer.

Rackspace Break-fix Competition
LinuxWorld 2008
San Francisco

Rackspace Hosting held a break-fix competition at LinuxWorld 2008 in San Francisco. The competition pitted System Administrators comfortable with the LAMP (Linux, Apache, MySQL, and PHP or Perl) platform against one another in a 4 question, 20 minute contest. The top three competitors who answered the most questions in the shortest time won one of the following prizes:

1st Place: Apple iPod Touch
2nd Place: Nintendo Wii
3rd Place: Garmin GPS

A break-fix competition presents problems with an otherwise working system that must be resolved. System administrators call upon their knowledge and troubleshooting abilities in a break-fix competition and is a good gauge of the depth of those skills and abilities. The Rackspace break-fix is tailored toward the Hosting industry as that is our forte.

Contestants were given root access to a remotely-hosted virtual machine running Red Hat Enterprise Linux 5.2. The questions for this break-fix were as follows:

1. Using the ‘ping’ utility, successfully ping ‘’ at

2. Successfully run the command:
mysql world < /root/insert.sql

(Valid solution will leave the insert.sql file as is).

3. Successfully execute the command:
echo “this file is secret” >/tmp/secret.txt

(Valid solution will not affect functionality of the /tmp directory.)

4. Successfully start the Apache web server, without losing any configured functionality

Many of our contestants asked for the solution to the problems we presented. Here is an explanation of the break and a way to resolve the issue. Also noted are non-solutions; this was a break-fix competition in the context of resolving problems on a LAMP system belonging to a customer. Solutions that technically allowed the specified operation to succeed yet without resolving the underlying issue by damaging data, web content or other system functionality did not measure up to the spirit of the competition and were disqualified.

1. Ping
# ping
PING ( 56(84) bytes of data.
ping: sendmsg: Operation not permitted
ping: sendmsg: Operation not permitted

The problem was that iptables was configured to drop out-bound connections to
# iptables -L
DROP all — anywhere

The solution was to remove the rule or stop iptables. We preferred seeing a lasting solution such as:
# iptables -D OUTPUT -d -j DROP
# service iptables save

# ping -c 1
PING ( 56(84) bytes of data.
64 bytes from ( icmp_seq=0 ttl=119 time=11.1 ms

2. Run a MySQL insert
The /root/insert.sql file contained the following code (that was not to be modified):
[root@lw3 ~]# cat /root/insert.sql
use world;

INSERT INTO `City` (`Name`, `CountryCode`, `District`, `Population`)
VALUES (‘San Antonio’, ‘USR’, ‘Manitoba’, 1250000);

There is nothing wrong with these SQL statements. Editing the file to something different that would not use the index on word.City would not be a solution in the spirit of a break-fix. Here’s the problem:
[root@lw3 ~]# mysql world < /root/insert.sql
ERROR 126 (HY000) at line 3: Incorrect key file for table ‘./world/City.MYI’; try to repair it

The problem is presented via STDERR: the index for the City table in the world database is broken. We used “dd” to overwrite the first 512 bytes with garbage to make the break. The solution? Also presented via STDERR: try to repair it. Here’s the most popular valid solution:

# mysql -e “repair table City” world


# service mysqld stop && myisamchk -r /var/lib/mysql/world/City && service mysqld start

3. Alter contents of /tmp/secret.txt
The file currently reads:
[root@lw3 ~]# cat /tmp/secret.txt
This text is to be changed.

But, this file can’t be modified:
[root@lw3 ~]# echo “this file is secret” > /tmp/secret.txt
-bash: /tmp/secret.txt: Permission denied

Here is the permissions of the file (not that it matters; we’re root):
[root@lw3 ~]# ll /tmp/secret.txt
-r-xr-xr-x 1 root root 28 Aug 5 08:05 /tmp/secret.txt

Here’s the permissions for the /tmp dir. Note that it is properly set with the sticky bit:
[root@lw3 ~]# ls -ld /tmp
drwxrwxrwt 5 root root 4096 Aug 5 14:00 /tmp

The problem is not SELinux:
[root@lw3 ~]# getenforce

Nor facls:
[root@lw3 ~]# getfacl /tmp/secret.txt
getfacl: Removing leading ‘/’ from absolute path names
# file: tmp/secret.txt
# owner: root
# group: root

The reason /tmp/secret.txt cannot be modified is that the immutable attribute is set:
[root@lw3 ~]# lsattr /tmp/secret.txt
—-i——– /tmp/secret.txt

Use chattr to remove this attribute that blocks any modification:
[root@lw3 ~]# chattr -i /tmp/secret.txt

Re-run the command:
echo “this file is secret” > /tmp/secret.txt

[root@lw3 ~]# cat /tmp/secret.txt
this file is secret

4. Start Apache
Using the standard Red Hat service command, or /etc/init.d/httpd directly, or your favorite method of choice, try to start Apache. It fails.
[root@lw3 ~]# service httpd start
Starting httpd: httpd: Could not reliably determine the server’s fully qualified domain name, using for ServerName

Most everyone – even our 1st Place winner! — solved the ServerName mis-configuration. However, the lack of a reliably determined FQDN for the server is not why Apache will not start. Instead, check the log files:
[root@lw3 ~]# tail -3 /var/log/httpd/error_log
[Tue Aug 05 07:08:29 2008] [notice] caught SIGTERM, shutting down
(2)No such file or directory: httpd: could not open error log file /etc/httpd/logs/nick/
Unable to open logs

The problem is that Apache cannot open the log files defined in its configuration settings. Specifically, Apache cannot open the error log file /etc/httpd/logs/nick/ The problem is that the directory specified, /etc/httpd/logs/nick does not exist.

[root@lw3 ~]# find /etc/httpd/logs -type d | wc -l

However, there are a number of “nick.es5vm*” files:
[root@lw3 ~]# ls /etc/httpd/logs/nick.es5vm*
/etc/httpd/logs/ …

The culprit is in Apache’s httpd.conf file that specifies
[root@lw3 ~]# grep -n nick/ /etc/httpd/conf/httpd.conf
990: ErrorLog logs/nick/

We broke Apache by altering line 990 from to nick/ . This “fat finger” mistake can be resolved by either correcting the ErrorLog directive to use “nick.” instead of “nick/” or creating the nick directory where it is expected:

[root@lw3 ~]# mkdir /var/log/httpd/nick
[root@lw3 ~]# service httpd start
Starting httpd: httpd: Could not reliably determine the server’s fully qualified domain name, using for ServerName
[ OK ]

Confirm that Apache is running:

[root@lw3 ~]# curl -I localhost
HTTP/1.1 200 OK
Date: Tue, 05 Aug 2008 15:30:39 GMT
Server: Apache/2.2.3 (Red Hat)
X-Powered-By: PHP/5.1.6
Connection: close
Content-Type: text/html; charset=UTF-8

The final break is fixed.

We had 52 competitors at LinuxWorld this year. Of this number, 25 answered at least one problem successfully. Here’s the break down of the numbers:

Correct number of solved breaks:
1 question: 8 contestants
2 questions: 5 contestants
3 questions: 4 contestants
4 questions: 8 contestants

By problem, here are the number of correct solutions:
Ping problem (iptables): 22 contestants
MySQL (repair table): 13 contestants
Echo (chattr): 13 contestants
Apache (ErrorLog directive): 10 contestants

The winners solved all four breaks in the following times:
1st Place – 8 minutes
2nd Place – 14 minutes
3rd Place – 15 minutes (plus a tie-break question)

We had a tie break round for the 3rd place winner. Three contestants finished the break-fix in 15 minutes each. In such cases we require a run-off for the prize. All three 3rd place finishers in the main break-fix competed in the tie break question and the winner was able to successfully solve the question first.

Rack Blogger is our catchall blog byline, subbed in when a Racker author moves on, or used when we publish a guest post. You can email Rack Blogger at



Please enter your comment!
Please enter your name here