Code Like Zell

Setting up Rails, MySQL, PHP, Apache, and Git on EC2

Posted 19 February 2011.

This is a step by step tutorial on setting up an Amazon Linux EC2 instance with Rails, PHP, MySQL, Apache, and Git. You will have three EBS volumes mounted, one for each of the following:

MySQL data and logs
httpd virtual hosts and logs
Git repositories

You will bundle your instance into it’s own AMI, launch an instance from that AMI, and recreate the state of your first instance by moving EBS volumes.

Launching and Accessing Your First Instance

The first thing we need is a set of command line tools for accessing AWS. For this tutorial we will be using Tim Kay’s AWS. To install:

  $ cd /usr/bin
  $ sudo curl https://github.com/timkay/aws/raw/master/aws -o aws
  $ sudo chmod +x aws

Next, get your AWS access keys by going to your AWS Account page and clicking on Security Credentials. Under Access Credentials will be two keys: an Access Key and a Secret Access Key. Substitute your keys into the following:

  $ echo "<your-access-key>" >> ~/.awssecret
  $ echo "<your-secret-key>" >> ~/.awssecret
  $ chmod 600 ~/.awssecret

Now we can see what regions are available for our EC2 instance:

  $ aws describe-regions

Let’s tell aws what region to use. For pricing on the various regions, go here. I am using us-east-1:

  $ echo "--region=us-east-1" >> ~/.awsrc

And let’s see what zones are available within the region we selected:

  $ aws describe-availability-zones

Before we launch an instance, we need to setup a key pair and security group. The key pair will be used to access the instance for the first time, we will name the key pair ec2temp:

  $ aws add-keypair ec2temp > ~/.ssh/ec2temp.key
  $ chmod 400 ~/.ssh/ec2temp.key

The security group specifies what ports to open on the instance. We will open ports 22, 80, and 443; you can leave off 443 if you do not need HTTPS. Let’s name the security group Web1:

  $ aws add-group Web1 -d "Ports 22 (ssh), 80 (http), and 443 (https) are open"
  $ aws auth Web1 -P tcp -p 22 -s 0.0.0.0/0
  $ aws auth Web1 -P tcp -p 80 -s 0.0.0.0/0
  $ aws auth Web1 -P tcp -p 443 -s 0.0.0.0/0

We are now ready to launch an instance. We are going to start with an Amazon Linux S3-Backed 32-bit AMI, found here. The AMI ID is ami-d59d6bbc. Here I am launching a small instance in the us-east-1a zone with the ec2temp key pair and the Web1 security group:

  $ aws run ami-d59d6bbc -k ec2temp -g Web1 -z us-east-1a

Your instance will take a minute to finish launching. You can check the status of your instance with aws din (short for describe-instances):

  $ aws din | awk -F '[+|]' '{print $2,$4,$6}'

Repeat the command above until the instance state changes to running. At that point, copy the public DNS name of your instance. Now you can access your instance using ec2-user:

  $ echo "export EC2_HOST=<your-public-dns>" >> ~/.bash_profile
  $ source ~/.bash_profile
  $ ssh -i ~/.ssh/ec2temp.key ec2-user@$EC2_HOST

Setting up User Accounts

You are now logged in as ec2-user, which has sudo access by default. Now create your user accounts, in my case Lou and Todd will be using this server.

  $ sudo useradd lou
  $ sudo useradd todd

In another window, use ssh-keygen to create a keypair called id_rsa_ec2, then scp it to your instance and append it to your authorized_keys file:

  local$ ssh-keygen
  local$ cd ~/.ssh
  local$ scp -i ec2temp.key id_rsa_ec2.pub ec2-user@$EC2_HOST:~

Now, back on the instance:

  ec2-user$ sudo su lou 
       lou$ mkdir -m 700 ~/.ssh
       lou$ touch ~/.ssh/authorized_keys
       lou$ chmod 600 ~/.ssh/authorized_keys
       lou$ exit
  ec2-user$ cat ~/id_rsa_ec2.pub | sudo tee -a /home/lou/.ssh/authorized_keys

Locally, verify that you can ssh in using your new user account:

local$ ssh -i ~/.ssh/id_rsa_ec2 lou@$EC2_HOST
  lou$ exit

If you are getting denied, tail /var/log/secure for clues. Next, as ec2-user, let’s set some passwords:

  $ sudo passwd root
  $ sudo passwd lou
  $ sudo passwd todd

And set colorful prompts so we can tell who we are:

  $ echo 'export PS1="\e[1;31m[\u@\h \w]\\$ \e[m"' | sudo tee -a /root/.bashrc
  $ echo 'export PS1="\e[0;35m[\u@\h \w]\$ \e[m"' | sudo tee -a /home/lou/.bashrc
  $ echo 'export PS1="\e[0;32m[\u@\h \w]\$ \e[m"' | sudo tee -a /home/todd/.bashrc
  $ source ~/.bash_profile

Now let’s remove ec2-user. But first, we need to edit sudoers:

  $ sudo vim /etc/sudoers

  Change this line of the file:
    ec2-user ALL = NOPASSWD: ALL
  
  To this (in this case lou is the admin): 
    lou ALL=(ALL) ALL

And then remove the ec2-user:

  ec2-user$ exit
     local$ ssh -i ~/.ssh/id_rsa_ec2 lou@$EC2_HOST
       lou$ sudo userdel -r ec2-user

While we are tightening things up, let’s wipe out the authorized_key file for root:

  lou$ sudo su root -c '> ~/.ssh/authorized_keys'
  lou$ exit

Modifying Your SSH Config File

To make it easier to access your instance, add the following to your local ~/.ssh/config:

  Host=*compute-1.amazonaws.com
  User=<your-user-name>
  IdentityFile=~/.ssh/id_rsa_ec2

Now you can ssh into your instance with:

  $ ssh $EC2_HOST

Setting up Git

The git repositories will be on an EBS. On the filesystem, they will be at /home/git/repos. Let’s create a git user and add our other users to the git group. I am going to use root for the remaining commands; if you aren’t comfortable doing this, then prefix each command with sudo.

  $ sudo su
  # yum install -y git
  # useradd git
  # usermod -a -G git lou
  # usermod -a -G git todd

And make the git directory group read and writeable.

  # chmod -R g+rwX /home/git

Also, we want all files and directories created under /home/git to retain the git group:

  # chmod g+s /home/git

Let’s allocate an EBS to hold your repositories. Do this from your local machine, and be sure to create your volume in the same zone as your instance:

  local$ aws create-volume --size 5 --zone us-east-1a

A volume ID will be returned. We need that ID and the ID of your instance to attach the EBS. To get the instance ID:

  local$ aws din | cut -b -15

And then to attach:

  local$ aws attach-volume -i <instance-id> -d /dev/sdf <volume-id>

When I attach the EBS as /dev/sdf it shows up on my instance as /dev/xvdf, your mileage may vary. Also, as far as I can tell there is no way to tag the volume using aws. We are going to create three volumes by the time this tutorial is over, and we need to be able to distinguish between them. So, go to the EC2 Console, select Volumes, right click your volume, and tag it with the name “GIT”.

While you are in the Console, create two additional Volumes. Attach them as /dev/sdg and /dev/sdh; label them MySQL and Apache, respectively.

Back on the instance, format your volume for Git:

  # mkfs -t ext3 /dev/xvdf

And mount it:

  # echo "/dev/xvdf /mnt/vol1 ext3 noatime 0 0" >> /etc/fstab
  # mkdir /mnt/vol1
  # mount /mnt/vol1

Now we will bind the /mnt/vol1/repos directory to /home/git/repos:

  # echo "/mnt/vol1/repos /home/git/repos none bind" >> /etc/fstab
  # mkdir -m 770 /mnt/vol1/repos
  # su git -c 'mkdir -m 770 ~/repos'
  # mount /home/git/repos
  # chown git /home/git/repos
  # chgrp git /home/git/repos

Let’s make our first git repo:

  # su git
  git$ mkdir -m 770 ~/repos/testing.git
  git$ cd ~/repos/testing.git
  git$ git init --bare --shared=group
  git$ exit

From your local machine, verify that you can push to the new repo:

  local$ mkdir testing
  local$ cd testing
  local$ git init 
  local$ git remote add origin ssh://<user>@$EC2_HOST/home/git/repos/testing.git
  local$ echo "nothing here" > README
  local$ git add .
  local$ git commit -m "initial commit"
  local$ git push origin head

One final note regarding git: if you are moving repositories from an old location to your instance, clone them with the --bare option, e.g.

  $ git clone --bare ssh://user@an.old.host...

Setting up MySQL

First, install MySQL:

  # yum install -y mysql mysql-server
  # mysql_install_db
  # service mysqld start
  # mysql_secure_installation

Next, let’s put the data and log directories on an EBS. The majority of these steps are from Eric Hammond’s excellent tutorial.

We already allocated an EBS and attached it as /dev/xvdg in a previous step. Next, format and mount it:

  #  yum install -y xfsprogs
  #  grep -q xfs /proc/filesystems || modprobe xfs
  #  mkfs -t xfs /dev/xvdg
  #  echo "/dev/xvdg /mnt/vol2 xfs noatime 0 0" >> /etc/fstab
  #  mkdir /mnt/vol2
  #  mount /mnt/vol2

We have an EBS volume formatted with xfs at /mnt/vol2. We can now move
our MySQL data and log directories to the volume:

  # service mysqld stop

  # mkdir /mnt/vol2/lib
  # mv /var/lib/mysql /mnt/vol2/lib/

  # mkdir -p /mnt/vol2/log/mysql
  # mkdir /var/log/mysql /var/lib/mysql

  # cp /etc/my.cnf /etc/my.cnf.bak
  # sed -i 's|/var/log/|/var/log/mysql/|g' /etc/my.cnf

  # echo "/mnt/vol2/lib/mysql /var/lib/mysql none bind" >> /etc/fstab
  # mount /var/lib/mysql
  # echo "/mnt/vol2/log/mysql /var/log/mysql none bind" >> /etc/fstab
  # mount /var/log/mysql

  # chown -R mysql /var/lib/mysql /var/log/mysql
  # chgrp -R mysql /var/lib/mysql /var/log/mysql

Start up mysqld

  # service mysqld start

Setting up Apache

We are going to have our httpd logs and virtual hosts on a third EBS. The process is very similar to what we just went through with MySQL. Our virtual hosts will be at /var/www/vhosts, and the httpd logs at /var/log/httpd.

First install Apache:

  # yum install -y httpd

Our final EBS was attached to our instance as /dev/xvdh. Format and mount:

  # mkfs -t xfs /dev/xvdh
  # echo "/dev/xvdh /mnt/vol3 xfs noatime 0 0" >> /etc/fstab
  # mkdir /mnt/vol3
  # mount /mnt/vol3

Now let’s bind /var/www/vhosts to /mnt/vol3/vhosts and /var/log/httpd to /mnt/vol3/log/httpd:

  # mkdir /mnt/vol3/log
  # mv /var/log/httpd /mnt/vol3/log/

  # mkdir -p /mnt/vol3/www/vhosts
  # mkdir /var/www/vhosts /var/log/httpd

  # echo "/mnt/vol3/www/vhosts /var/www/vhosts none bind" >> /etc/fstab
  # echo "/mnt/vol3/log/httpd /var/log/httpd none bind" >> /etc/fstab

  # mount /var/www/vhosts
  # mount /var/log/httpd

  # service httpd start

You can convince yourself that the httpd log is on the EBS by tailing the access_log at /mnt/vol3/log/httpd/access_log and pointing your browser to the public DNS of your instance. You should see the Apache test page.

Testing PHP, Apache, and MySQL

First let’s get php and verify that Apache serves a simple php file:

  # yum install -y php
  # echo '<?php echo("hello from php"); ?>' >> /var/www/html/index.php
  # chgrp apache /var/www/html/index.php

Restart Apache:

  # service httpd restart

And point your browser to the public DNS of your instance. You should see “hello from php”. Next, let’s test the php mysql connection. To do this, we will manually enter a single row using the mysql client, then fetch it with php:

  # yum install -y php-mysql
  # mysql -u root -p
  
  mysql> CREATE DATABASE phptest;
  mysql> GRANT ALL PRIVILEGES ON phptest.* TO foo@localhost IDENTIFIED BY 'nachos';
  mysql> USE phptest;
  mysql> CREATE TABLE posts(id int(11) NOT NULL AUTO_INCREMENT, title varchar(255),
                            PRIMARY KEY (id)) ENGINE=InnoDB;
  mysql> INSERT INTO posts (title) VALUES ('hello from mysql');
  mysql> exit;

Now, to retrieve that row, edit /var/www/html/index.php to look like this:

<?php 
  mysql_connect("localhost", "foo", "nachos");
  mysql_select_db("phptest"); 
  $result=mysql_query("SELECT * FROM posts LIMIT 1");
  mysql_close();
  echo(mysql_result($result,0,"title")); 
?>

Restart Apache, then refresh your browser and verify that it says “hello from mysql”. If it does not, tail the default apache error log for clues:

  # tail -f /var/log/httpd/error_log

Once your test works, do a little cleanup:

  # rm /var/www/html/index.php
  # mysql -u root -p
  mysql> drop user foo@localhost;
  mysql> drop database phptest;
  mysql> exit;

On to Ruby and Rubygems!

First we need some ruby packages and gcc-c++ for building native extensions:

  # yum install -y gcc-c++ ruby ruby-libs ruby-devel ruby-irb rubygems
  # gem --version

I prefer not to install rdoc and ri files when installing gems on a server (feel free to skip this):

  # echo "gem: --no-ri --no-rdoc" >> ~/.gemrc

Installing Passenger (mod_rails)

  # yum install -y curl-devel httpd-devel openssl-devel zlib-devel
  # gem install passenger 
  # passenger-install-apache2-module

Follow the instructions that are spit out by the passenger install (by editing /etc/httpd/conf/httpd.conf).

Testing Rails

Let’s test out a rails app. We will need the mysql gem:

  # yum install -y mysql-devel
  # gem install mysql

Next, install Rails (I still use 2.3.8):

  # gem install rails -v=2.3.8

Then, create a test app:

  # cd /var/www/vhosts
  # rails -d mysql news

And edit httpd.conf to add a virtual host:

<VirtualHost *:80>
    # ServerName codelikezell.com
    DocumentRoot /var/www/vhosts/news/public
    <Directory /var/www/vhosts/news/public>
      AllowOverride all 
      Options -MultiViews
    </Directory>
</VirtualHost>

Restart Apache and refresh your browser, you should see the Riding Rails page. Let’s get the database involved to test the Rails MySQL connection:

  # cd /var/www/vhosts/news
  # rm public/index.html
  # mysqladmin -u root -p create news_production
  # vim config/database.yml
  
  Edit the production username and password: 
    username: news
    password: nachos

Grant privileges to the db user:

  # mysql -u root -p
  mysql> grant all privileges on news_production.* to news@localhost identified by 'nachos';
  mysql> exit

And generate some scaffolding:

  # script/generate scaffold Post title:string
  # RAILS_ENV=production rake db:migrate
  # vim config/routes.rb

Edit routes.rb to look like this:

  ActionController::Routing::Routes.draw do |map|
    map.resources :posts
    map.root :controller => "posts"
  end

Restart Apache, reload your browser, and (hopefully) behold a functioning Rails site! Put a few entries into the app so when we launch a second instance we can test whether the data stays intact.

Get Ready to Bundle

Let’s bundle up what we have into a new AMI. We need an X.509 certificate from Amazon to create a bundle. Go to your AWS account page and select Security Credentials. Under Access Credentials there is a tab for X.509 Certificates, select it. Create a new certificate and download the Private Key and X.509 pair to your ~/.ec2 directory.

While you are on the Security Credentials page, make note of your AWS Account ID, Access Key, and Secret Key—you will need all of these.

Next, scp the key and cert files to your instance. From the EC2 User Guide: “It is important to upload the key and cert files into /mnt to prevent them from being bundled with the new AMI.” So, let’s do that:

  local$ cd ~/.ec2
  local$ scp pk-xyz.pem cert-xyz.pem lou@$EC2_HOST:~
  local$ ssh lou@$EC2_HOST
    lou$ sudo mv cert-xyz.pem /mnt
    lou$ sudo mv pk-xyz.pem /mnt

Stop mysqld and httpd, and unmount the EBS volumes:

  # service httpd stop
  # service mysqld stop
  
  # umount /var/www/vhosts
  # umount /var/log/httpd
  # umount /var/lib/mysql
  # umount /var/log/mysql
  # umount /home/git/repos
  # umount /mnt/vol*

Verify that your EBS volumes are all unmounted by running the mount command with no arguments.

From your local machine, detach the EBS volumes. We will attach these again on an instance created from a new AMI. Get the volume ids, detach them, and then verify they are detached:

  $ aws describe-volumes | cut -b -150
  $ aws detach-volume <vol1-id>
  $ aws detach-volume <vol2-id>
  $ aws detach-volume <vol3-id>
  $ aws describe-volumes | cut -b -150

Now we can bundle:

   lou$ sudo su
  root# ec2-bundle-vol -k /mnt/pk-xyz.pem -c /mnt/cert-xyz.pem -u <your-aws-id-WITHOUT-dashes>

Next, let’s upload the bundle to S3. The bucket and subfolder will be created automatically if they do not already exist. As an example, in the command below I would use -b zell-amis/web1-image to upload my bundle to the web1-image subfolder of the zell-amis bucket:

  # cd /tmp
  # ec2-upload-bundle -b <bucket>/<subfolder> -m image.manifest.xml -a <aws-access-key> -s <aws-secret-key>

Note: if you get an error such as “Error talking to S3: Server.AccessDenied(403): Access Denied”, it could be because the S3 bucket name you chose is already taken—S3 buckets have a global namespace.

Finally, register your image from your local machine:

  $ aws register-image <bucket>/<subfolder>/image.manifest.xml -n <name-your-image>

Creating an instance of your new AMI

The last command should have returned an AMI id in the response. Let’s launch an instance of it:

  $ aws run <your-new-ami-id> -g Web1 -z us-east-1a

Now, just like last time, monitor the status of your launching instance:

  $ aws din | awk -F '[+|]' '{print $2,$3,$4,$6}'

Once it is running, note the public DNS name. Before we login let’s attach the three EBS volumes. We need their volume IDs:

  $ aws describe-volumes

Examine the tagSet column, then:

  $ aws attach-volume -i <instance-id> -d /dev/sdf <GIT-volume-id>
  $ aws attach-volume -i <instance-id> -d /dev/sdg <MySQL-volume-id>
  $ aws attach-volume -i <instance-id> -d /dev/sdh <Apache-volume-id>

Now ssh into the new instance as your admin user:

  $ ssh <public-DNS>

And go mount crazy:

  $ sudo mkdir /mnt/vol1 /mnt/vol2 /mnt/vol3
  
  $ sudo mount /mnt/vol1
  $ sudo mount /mnt/vol2
  $ sudo mount /mnt/vol3

  $ sudo mount /home/git/repos
  $ sudo mount /var/log/mysql
  $ sudo mount /var/lib/mysql
  $ sudo mount /var/log/httpd
  $ sudo mount /var/www/vhosts

Next, start mysqld and httpd and hold on to your privates:

  $ sudo service mysqld start
  $ sudo service httpd start

Browse to the public DNS of this instance and you should be staring at the Rails app with any entries you previously put into it.

A couple final things:

Starting httpd and mysqld at Boot

  $ sudo chkconfig --levels 235 mysqld on
  $ sudo chkconfig --levels 235 httpd on

Test it!

  $ sudo reboot

When your instance comes back, all the volumes should be mounted and mysqld and httpd should be running.

Cleanup

  $ sudo rm -rf /home/git/repos/testing.git
  $ sudo rm -rf /var/www/vhosts/news
  $ mysql -u root -p 
  mysql> drop user news@localhost;
  mysql> drop database news_production;
  mysql> exit;

Attaching an Elastic IP

Through the AWS Management console, allocate an Elastic IP address and attach it to your instance by right clicking on it. Now, locally, add the following to ~/.ssh/config:

  Host ec2
  Hostname=<your-elastic-ip-address>
  User=<your-user-name>
  IdentityFile=~/.ssh/id_rsa_ec2