Amazon AWS EC2 Automatic Snapshot Daily Cron Job

There seems to be very poor documentation about setting up an automatic backup, daily snapshot with Amazon AWS EC2. I am currently running Ubuntu 14.04 with ISPConfig. Here is how to setup a daily cron job for backup.

First, you must get your Access Key ID and Secret Access Key from Amazon. There is plenty of documentation about this. You may want to setup an IAM user so this access key has limited permissions, as Amazon recommends. I’ve linked to the Amazon support page if you are not familiar with this. This is so that Amazon knows you are authorized to make this request.

Second, log in to your AWS console and click on ‘Volumes’ under the EBS (elastic block storage) section. Make note of your Volume-ID, which you will need so that Amazon knows which volume to backup.

Next, you must install the Amazon command line tools. All of the sources should already be setup for you, if you used an Amazon image. Just execute the following code from the command shell and install.

sudo apt-get install ec2-api-tools

Here is the command used for your cron job:

/usr/bin/ec2-create-snapshot -O aws-secret-key -W aws-access-key-id  volume-id -d "Daily Backup $(date +'%m/%d/%y')"

For a daily backup, use the hour and minutes that you want the job to run. Use a * for days-of-month, months, and days-of-week. This means it will run at a specific time of day, but every day.

Breakdown:

  • -O AWS Secret Access Key
  • -W AWS Access Key ID
  • -d Snapshot Name (in our case, it will be Daily Backup mm/dd/yy). Of course, you can change this as needed.

ISPConfig Configuration:

First, check to make sure your permissions are setup for Reseller and Client to allow for you to create cron jobs. Also, you will want to setup the maximum number of cron jobs (-1 is unlimited) and the type of cron job to ‘Full Cron’. To setup a cron job with ISPConfig, you must link to a .sh file. Below, replace ‘clientXX’ and ‘webXX’ with your actual client number.

Create a directory in your client folder called ‘cron’. This should be located at: /var/www/clients/clientXX/cron/

cd /var/www/clients/clientXX
mkdir cron

Inside, create a file called daily-snapshot.sh.

vi cron/daily-snapshot.sh

This file should include the following:

#!/bin/bash

# content of your script

/usr/bin/ec2-create-snapshot -O aws-secret-key -W aws-access-key-id  volume-id -d "Daily Backup $(date +'%m/%d/%y')"

Also, you much execute the following commands to give the correct permissions to this file:

chown -R webXX:clientXX /var/www/clients/clientXX/cron/
chmod 755 /var/www/clients/clientXX/cron/daily-snapshot.sh

Finally, you are now ready to create your cron job in ISPConfig. Go to the ‘Cron Jobs’ section under the sites tab. Create a new cron job. Select your site. For a daily backup, use the hour and minutes that you want the job to run. Use a * for days-of-month, months, and days-of-week. This means it will run at a specific time of day, but every day. Enter the following for the command to run:

/var/www/clients/clientXX/cron/daily-snapshot.sh

Links:

SSI on Apache 2.4+, Conditional Statements – new format

I moved my site to a new server, and all of my conditional SSI statements stopped working. There is plenty of documentation for using SSILegacyExprParser, but very little documentation on the new method.

I used SSI to deliver a different html file depending on the domain visited. Originally, this was to get past my web host restrictions for total domains. However, when a particular client has many domains with very simple websites, I prefer to have all files in the same folder.

After the update, I was getting the following errors in my error.log folder:
[Tue Feb 17 01:58:08.865059 2015] [include:error] [pid 4855] [client xxx.xxx.xxx.xxx:yyyyy]: Could not parse expr "$HTTP_HOST = "mysite.com"" in /var/www/mysite.com/web/index.shtml: Parse error near '$'
and
[Tue Feb 17 01:52:11.580714 2015] [include:error] [pid 4294] [client xxx.xxx.xxx.xxx:yyyyy]: Could not parse expr "%{HTTP_HOST}=mysite.com" in /var/www/mysite.com/web/index.shtml: syntax error, unexpected T_OP_CONCAT, expecting '('

This is a result of the new syntax requirements for conditional expressions in Apache 2.4 and newer. Here is the code that worked for me:


<!--#if expr='%{HTTP_HOST} = "site1.com" || %{HTTP_HOST} = "www.site1.com"' -->
<!--#include virtual="page1.html" -->
<!--#elif expr='%{HTTP_HOST} = "site2.com" || %{HTTP_HOST} = "www.site2.com"' -->
<!--#include virtual="page2.html" -->
<!--#elif expr='%{HTTP_HOST} = "site3.com" || %{HTTP_HOST} = "www.site3.com"' -->
<!--#include virtual="page3.html" -->
<!--#else-->
<!--#include virtual="others.html" -->
<!--#endif-->

Fix “HTTP request length 134926 (so far) exceeds MaxRequestLen” error on Linux Ubuntu

If you get a 500 error in a webpage with apache webserver and fastcgi, take a look into the apache error.log file. This can either be the global error.log or the error.log of the website where you got the error. If you find a error similar to this one:

[Fri Apr 9 02:38:12 2014] [warn] [client 100.200.100.10] mod_fcgid: HTTP request length 134926 (so far) exceeds MaxRequestLen (131072), referer: http://www.example.tld/administrator/index.php?option=com_installer

then the MaxRequestLen setting of mod_fccgid is too low. To fix that, edit the file /etc/apache2/mods-available/fcgid.conf

# vi /etc/apache2/mods-available/fcgid.conf

and add or edit the line “MaxRequestLen 15728640″ to set the Request Limit to 15MB. The resulting file should contain these settings:

AddHandler fcgid-script .fcgi
FcgidConnectTimeout 20
MaxRequestLen 15728640

Save the changes and restart apache:

# /etc/init.d/apache2 restart

Deleting/viewing mail in postfix queue (mailq)

Postfix stores mails in a queue before sending it. Postfix has a command line called ‘postsuper’ which can be used to delete unsent mail from the queue. Before removing mail from the queue it is good idea to list all mail in the queue.

List all of the mail which is in the queue.
Code:

mailq

View the contents of a specific message.
Code:

postcat -q MESSAGE_ID

Remove a message from the “mailq”.
Code:

postsuper -d mailID

where mailID is the ID of the mail in the mail queue.

Delete all queued mail from the mailq.
Code:

postsuper -d ALL

Upload files with ssh / scp in Linux

The following code will transfer myfile.zip to the folder ‘myfolder’ on the server:

Code:
scp /home/username/documents/myfile.zip username@myserver.com:/home/username/myfolder/

scp = Linux command to transfer files ‘Secure Copy’
/home/username/documents/ = the directory where myfile.zip resides
myfile.zip = the file to transfer
username@myserer.com = username on myserver.com
/home/username/myfolder/ = the directory on myserver.com where myfile.zip should be transferred

You can also do the same in reverse:

Code:
scp username@myserver.com:/home/username/myfolder/myfile.zip /home/username/documents/

If you want to transfer a directory, just leave the trailing ‘/’ and add the ‘-r’ flag:

Code:
scp -r username@myserver.com:/home/username/myfolder/ /home/username/documents/

If you are connecting to Amazon hosting or need to use an identity file, try the following:

Code:
scp -v -r  -i ~/.ssh/identity.pem ~/Documents/myfolder/ username@myserver.com:/home/username/

-v Verbose – see the file transfer progress
-r Recursive – transfer a directory and all files inside
-i ~/.ssh/identity.pem Identity – use the identity file located at ~/.ssh/identity.pem
username@myserer.com = username on myserver.com
/home/username/ = the directory on myserver.com where myfolder should be transferred

Enable mounting of remotedisk remote CD/DVD on any unsupported mac

If you are like me, and your CD/DVD has stopped working on an old mac, you can use CD/DVD sharing to access the drive on another computer. Unfortunately, if your old computer originally had a CD/DVD drive then this feature is disabled by default. You can execute the following commands in terminal to enable Remote Disk.


$ defaults write com.apple.NetworkBrowser EnableODiskBrowsing -bool true
$ defaults write com.apple.NetworkBrowser ODSSupported -bool true

Restart your computer and then you’ll see the Remote CD/DVD under devices in the Finder’s sidebar.

Use terminal to delete all hidden files on a mac

I recently came across a client’s computer that had a tremendous number of hidden files that were causing problems. I tried many different ways to delete these, unsuccessfully. Unfortunately, there were about 80,000 hidden files nested within hundreds of directories. Here is the code that I used to delete them:

First cd to the root directory where you want to start deleting. Then execute the following code:
find . -name "\._*" -type f -delete

Most of the hidden files will start with ‘._’ but some may still remain. For me it was .DS_Store and .FBCLockFolder. To delete these, I executed just searched using the finder by selecting ‘other’ in the search criteria and scrolling to visibility. Then select invisible. Make sure the search scope is only in the folder you want to search. Then just select all and delete.

Convert Pixles to REM: WordPress calculator for responsive design

Instructions

Just type the value in pixels and hit Enter. You will get two results – one to be used for font-size, width, height, margin, padding and the other result for line-height.

REM SIZE CALCULATOR

Rem
Line-height

Tip: A single click then selects the whole result, after that you can copy and paste it to your CSS.

Use rm to delete a directory

To delete a file or directory in terminal (command line) use the rm command.

Here is how to delete an entire directory:
rm -dfr directory

OPTIONS
-d, –directory
unlink FILE, even if it is a non-empty directory (super-user only)
-f, –force
ignore nonexistent files, never prompt
-i, –interactive
prompt before any removal
-r, -R, –recursive
remove the contents of directories recursively
-v, –verbose
explain what is being done
–help
display this help and exit
–version
output version information and exit

To remove a file whose name starts with a ‘-‘, for example `-foo’, use one of these commands:
rm -- -foo
rm ./-foo

To remove a file whose name contains a ‘*’ (or other special character), you will need to escape it. For example, I wanted to delete all directories starting with ‘~*w’:
rm -dfr ~\*w*

rsync via SSH

To copy files from one server to another via ssh:

First login to the destination server and cd to the directory that you want to copy files into.
Then type rsync -ave ssh username@sourceserver:directory/path/to/copy/ .
NOTE: don’t forget the ‘.’at the end. This is important.

On TextDrive (old FreeBSD servers), the exact path is:
rsync -ave ssh username@server.textdrive.com:domains/mydomain.com/public_html/ .