R & hadoop: How to access Amazon S3 cloud storage from command line in Linux

Last updated on June 14, 2013 Authored by Dan Nanni 1 Comment

Amazon S3 is a cloud storage provided by Amazon Web Services (AWS). Amazon S3 publishes a set of web services interfaces, upon which many third-party commercial services or client software are developed.
In this tutorial, I will describe how to access Amazon S3 cloud storage from the command line in Linux.
One of the most popular Amazon S3 command line clients is s3cmd, which is written in python. As a simple AWS S3 command line tool, s3cmd is ideal to use when you want to run scripted cron jobs such as daily backups.

Install s3cmd on Linux

To install s3cmd on Ubuntu or Debian:

$ sudo apt-get install s3cmd

To install s3cmd on Fedora:

$ sudo yum install s3cmd

To install s3cmd on CentOS or RHEL, download .rpm package from its official site, and install it manually. For 64-bit CentOS or RHEL 6:

$ sudo rpm -ivh s3cmd-1.0.0-4.1.x86_64.rpm

Configure s3cmd

When you run s3cmd for the first time, you need to configure it by running the following command.

$ s3cmd --configure

It will ask you for a series of questions:

access key and secret key for AWS S3
encryption password for encrypted data transfer to and from AWS S3.
path to GPG program used to encrypt data (e.g., /usr/bin/gpg)
whether to use HTTPS protocol
name and port of HTTP proxy if used

Configuration will then be saved as a plain text in ~/.s3cfg.

Basic Usage of s3cmd

To list all existing buckets in your AWS S3 account:

$ s3cmd ls

2011-05-28 22:30  s3://mybucket1
2011-05-29 00:14  s3://mybucket2

To create a new bucket:

$ s3cmd mb s3://dev99

Bucket 's3://dev99/' created

To upload files to an existing bucket:

$ s3cmd put 1.png 2.png 3.png s3://dev99

1.png -> s3://dev99/1.png  [1 of 3]
 26261 of 26261   100% in    5s     4.33 kB/s  done
2.png -> s3://dev99/2.png  [2 of 3]
 201430 of 201430   100% in    2s    98.05 kB/s  done
3.png -> s3://dev99/3.png  [3 of 3]
 46630 of 46630   100% in    0s    56.62 kB/s  done

The default access permission for the uploaded files is "private", meaning that only you, with correct access and secret keys, will be able to access the files.
To upload files to an existing bucket with public access permission:

$ s3cmd put --acl-public 4.png s3://dev99

4.png -> s3://dev99/4.png  [1 of 1]
 30778 of 30778   100% in    8s     3.34 kB/s  done
Public URL of the object is: http://dev99.s3.amazonaws.com/4.png

With public access permission granted, the uploaded file can be accessed by anyone by going to http://dev99.s3.amazonaws.com/4.png on any web browser.
To view the content of an existing bucket:

$ s3cmd ls s3://dev99

2013-06-02 02:52     26261   s3://dev99/1.png
2013-06-02 02:52    201430   s3://dev99/2.png
2013-06-02 02:52     46630   s3://dev99/3.png
2013-06-02 02:56     30778   s3://dev99/4.png

To download files (e.g., all .png files) contained in an existing bucket:

$ s3cmd get s3://dev99/*.png

s3://dev99/1.png -> ./1.png  [1 of 4]
 26261 of 26261   100% in    0s    39.39 kB/s  done
s3://dev99/2.png -> ./2.png  [2 of 4]
 201430 of 201430   100% in    7s    24.64 kB/s  done
s3://dev99/3.png -> ./3.png  [3 of 4]
 46630 of 46630   100% in    1s    39.34 kB/s  done
s3://dev99/4.png -> ./4.png  [4 of 4]
 30778 of 30778   100% in    0s    97.01 kB/s  done

To remove files in an existing bucket:

$ s3cmd del s3://dev99/*.png

File s3://dev99/1.png deleted
File s3://dev99/2.png deleted
File s3://dev99/3.png deleted
File s3://dev99/4.png deleted

To get information about an existing bucket, including its storage location and access control lists:

$ s3cmd info s3://dev99

s3://dev99/ (bucket):
   Location:  us-east-1
   ACL:       dan.nanni: READ
   ACL:       dan.nanni: WRITE
   ACL:       dan.nanni: READ_ACP
   ACL:       dan.nanni: WRITE_ACP

To encrypt a file before uploading it to an existing bucket:

$ s3cmd -e put encrypt.png s3://dev99

/tmp/tmpfile-pzT1zV3kLZlxDwqA0kwy -> s3://dev99/encrypt.png  [1 of 1]
 196890 of 196890   100% in    1s    99.51 kB/s  done

When downloading an encrypted file with s3cmd, it will automatically detect the encryption, and decrypt the file on the fly upon downloading it. Thus, to download and access an encrypted file, simply run as usual:

$ s3cmd get s3://dev99/encrypt.png

s3://dev99/encrypt.png -> ./encrypt.png  [1 of 1]
 196890 of 196890   100% in    1s   131.29 kB/s  done

To remove an existing bucket:

$ s3cmd rb s3://dev99

Bucket 's3://dev99/' removed

Note that you cannot remove a bucket if it is not empty.

Subscribe to Xmodulo

Do you want to receive Linux FAQs, detailed tutorials and tips published at Xmodulo? Enter your email address below, and we will deliver our Linux posts straight to your email box, for free. Delivery powered by Google Feedburner.

Dan Nanni

Dan Nanni is the founder and also a regular contributor of Xmodulo.com. He is a Linux/FOSS enthusiast who loves to get his hands dirty with his Linux box. He likes to procrastinate when he is supposed to be busy and productive. When he is otherwise free, he likes to watch movies and shop for the coolest gadgets.

Your name can also be listed here. Write for us as a freelancer.

One thought on “How to access Amazon S3 cloud storage from command line in Linux”

Hi,
Thanks for the helpful article on how to install & use s3 tool for command line access of s3 Bucket.
just wondering how can we use s3cmd to download wiki dumps on s3 Bucket.
Or is there any way to directly download wiki dumps on s3 Bucket.
Thanks in advance
Pushkar S

R & hadoop

How to access Amazon S3 cloud storage from command line in Linux