Synching folder with Amazon S3

Synching folder with Amazon S3

The other day I set out to finally start synching one of my backup folders on one of our Ubuntu Servers with Amazon S3. The reasons for this, are obvious. Amazon S3 is very cheap and reliable and is a good fit for keeping “smaller” chunks of files as a backup storage. So, apart from writing a script that runs trough crontab, which was easy to write, I’ve spent far more time to get the S3 script running.┬áSo, here are the steps to sync any folder with Amazon S3.

First off, I created a additional bucket on S3. Let’s call this backup “mybackup”.

Since, I did not wanted to install anything additionally to run the S3 scripts (there is a very popular S3Rsync for Ruby out there) I wanted to get it running with the “bash” shell itself. Why install another language, if you got everything already, right?

So, I went ahead and downloaded the S3-bash scripts. As the name implies, these are scripts that work in the bash shell and nothing else is needed for it. Unfortunately, the documentation lacks big time, so I had to run all over the net to find some. Since they are very sparse, here are the steps to get it running.

After you have unpacked the scripts you get three scripts called;

  • s3-get – for getting files from S3
  • s3-put – for putting files on S3
  • s3-delete – for deleting files on S3

Since we want to put files on S3, we are going to focus on the “s3-put” script. Here is a example how the commands for the s3-put script would look like:

s3-put -k {yourkey} -s awssecretfile -T /backup/ /s3-backup/

The explanation of the params is as follows:

  • -k
    The “-k” parameter is your Amazon S3 key
  • -s
    The “-s” parameter is a path to a file (which you have to write) which contains your Amazon S3 secret key. Please read further down, in order to avoid any errors with this.
  • -T
    The “-T” parameter should be the absolute path to your file that you want to out on Amazon S3
  • file bucket and path on S3
    Last but not least, you have to pass the path to your S3 bucket and the file name to the command string.

Now, that wasn’t so hard right? Well, there is one small thing that drove me crazy during my initial setup. That is that the file with my Amazon S3 secret key kept on throwing an error. Somehow the length was not matching and some other errors. After some digging around I figured that one has to write to the file again in order to get rid of the 41 bytes error message. To do that issue the following command:

cat awssecretfile | tr -d 'n' >> awssecretfile-new

Right, so I’m hoping this helps anyone out there. Have fun.

Trusted by over 2,000 organizations with millions of emails

Helpmonks helps us answer customer emails at lighting speed
We couldn't do our job without Helpmonks!
Helpmonks, the best thing since sliced bread
Excellent option for collaborating email amongst your team
A good platform that is here to grow and help you grow
Helpmonks is a great tool that our company utilizes and depends on
Great shared inbox for the team
Fantastic email support service for any business
Easy to navigate and catalog data for specific users
Solid Helpdesk software w/ responsive customer support
A really good solution to manage customer support mailbox
Good support, useful features

One more thing...

Helpmonks gives you the ability tocollaborate as a groupin your shared mailbox, features aTeam To-Do App, powerfulCustomer Relationship Management (CRM)capabilities, a completeLive-Chat tool, a mighty Email Marketing Platform that includes sending ofemail campaignsandemail automation, collision detection, email tracking, reminders, mentions, custom fields, custom applications, customer satisfaction ratings, Single-Sign-On (SSO) & SAML authentication, Trello, Zapier, Slack, an API, and much more.

Check outthe complete feature listor sign up for afree 30-day shared inbox trial now