Amazon S3 (Simple Storage Service) is an online storage web service offered by Amazon Web Services. Amazon S3 provides unlimited storage through a simple web services interface. Data can be easily stored and retrieved at any time, from anywhere on the web. Amazon charges in proportion to the amount of data stored and applies charges for sending and receiving data. Online storage is generally when a company, service, or application allows you to virtually store any computer file on the internet. ... Web services architecture The W3C defines a Web service (many sources also capitalize the second word, as in Web Services) as a software system designed to support interoperable Machine to Machine interaction over a network. ... Amazon Web Services logo The Amazon Web Services (AWS) are a collection of remote computing services (also called web services) offered over the Internet by Amazon. ... Web services architecture The W3C defines a Web service (many sources also capitalize the second word, as in Web Services) as a software system designed to support interoperable Machine to Machine interaction over a network. ...

Amazon S3 uses the same scalable storage infrastructure that Amazon.com uses to run its own global e-commerce network. Amazon S3 is reported to store more than 10 billion objects as of November 2007.

Launched in March 2006^[1], Amazon S3 is currently being used by small start-ups and enterprise clients as a web hosting service, image hosting service, back-up system, and more.

Design

S3's design aims to provide scalability, high availability, and low latency at commodity costs. It has been suggested that this article or section be merged with Scale (computing). ... A protocol and associated execution that ensures a certain relative degree of computing-system operational continuity in any downtime event. ... Low latency allows human-unnoticeable delays between an input being processed and the corresponding output providing real time characteristics. ... Commodity computing is computing done on commodity computers as opposed to supermicrocomputers or boutique computers. ...

S3 stores arbitrary objects up to 5 gigabytes in size, each accompanied by up to 2 kilobytes of metadata. Objects are organized into buckets (each owned by an AWS account), and identified within each bucket by a unique, user-assigned key. This article is about computer files and file systems in general terms. ... This article is about the unit of measurement. ... Depending on the context in which it is used, the word kilobyte may mean either 1,000 or 1,024 bytes. ... Metadata is data about data. ... Amazon Web Services logo The Amazon Web Services (AWS) are a collection of remote computing services (also called web services) offered over the Internet by Amazon. ...

Buckets and objects can be created, listed, and retrieved using either a REST-style HTTP interface or a SOAP interface. Additionally, objects can be downloaded using the HTTP GET interface and the BitTorrent protocol. “REST” redirects here. ... Hypertext Transfer Protocol (HTTP) is a communications protocol used to transfer or convey information on intranets and the World Wide Web. ... For other uses, see Soap (disambiguation). ...

Requests are authorized using an access control list associated with each bucket and object. In computer security, an access control list (ACL) is a list of permissions attached to an object. ...

Bucket names and keys are chosen so that objects are addressable using HTTP URLs: // Uniform Resource Locator (URL) formerly known as Universal Resource Locator, is a technical, Web-related term used in two distinct meanings: In popular usage and many technical documents, it is a synonym for Uniform Resource Identifier (URI); Strictly, the idea of a uniform syntax for global identifiers of network-retrievable...

• http://s3.amazonaws.com/bucket/key
• http://bucket.s3.amazonaws.com/key
• http://bucket/key (where bucket is a DNS CNAME record pointing to s3.amazonaws.com)

Because objects are accessible by unmodified HTTP clients, S3 can be used to replace significant existing web hosting infrastructure. The Amazon AWS Authentication mechanism allows the bucket owner to create an authenticated URL with time-bounded validity. That is, someone can construct a URL that can be handed off to a third-party for access for a period such as the next thirty minutes, or the next twenty-four hours. This may be useful in some circumstances. The Domain Name System (DNS) associates various sorts of information with so-called domain names; most importantly, it serves as the phone book for the Internet by translating human-readable computer hostnames, e. ... An example of rack mounted servers. ...

Every item in a bucket can also be served up as a BitTorrent feed, so the S3 store can act as a seed host for a torrent, and any BitTorrent client can retrieve the file.

A bucket can be configured to save HTTP log information to a sibling bucket; this can be used in later data mining operations. Data mining is the principle of sorting through large amounts of data and picking out relevant information. ...

Implementation

The implementation details of S3 are not documented, but it is possible to infer aspects of it from the behaviour (and pricing) of the service.

• There is an S3 repository in the US; there is another in Europe. Their locations are not public.
• From an EC2 server, read access can initially be slow. This could imply that S3 data is not always in the same physical location as all the EC2 servers.
• Amazon do not bill for data transferred between the US EC2 server farm and the US S3 store, but they do for access between the servers and the European S3 store. While the S3 store and the EC2 servers may be in separate locations, they must be close enough together that cost of transferring data is neglible. Both services are presumably hosted on the same Metropolitan Area Ethernet.
• Connectivity between the S3 datastores and the rest of the Internet is good. This implies they are close to one of the main Internet exchanges.
• After the initial slow access, later accesses are fast. This could imply that the data is cached.
• Write access is tangibly slower than writing to a local hard disk. Amazon guarantee that when the operation has completed, data has been written to disk in multiple locations.
• Amazon warn that even after data has been written, old values may still be read. This implies that there can be more than one cache of the data, and that every GET request does not trigger a check from the front-end caches to the back-end store.
• Amazon adjusted their initial pricing from a simple flat rate per GB to one that charges small frequently retrieved items per thousand GET or HEAD operations, rather than purely per byte. There may be a flat cost for every entry (indexing, billing entries) and a per-request cost in CPU-time for every access. These fixed overheads are now billed for.

The dynamic per-bucket DNS entries are implemented by having a custom S3 DNS server that returns a hostname for every host under the s3.amazonaws.com subdomain, even if a bucket of that name has never been created: Amazon Web Services logo Amazon Elastic Compute Cloud (EC2) brings scalable computing power to the masses allowing users to launch and terminate server instances on demand, hence the term elastic. EC2 uses Xen Virtualization. ... Amazon Web Services logo Amazon Elastic Compute Cloud (EC2) brings scalable computing power to the masses allowing users to launch and terminate server instances on demand, hence the term elastic. EC2 uses Xen Virtualization. ... A Metro Ethernet is a computer network based on the Ethernet standard and which covers a metropolitan area. ... The Domain Name System (DNS) associates various sorts of information with so-called domain names; most importantly, it serves as the phone book for the Internet by translating human-readable computer hostnames, e. ...

 > nslookup made-up-name-for-wikipedia.s3.amazonaws.com Non-authoritative answer: made-up-name-for-wikipedia.s3.amazonaws.com canonical name = s3-directional-w.amazonaws.com. s3-directional-w.amazonaws.com canonical name = s3-1-w.amazonaws.com. Name: s3-1-w.amazonaws.com Address: 72.21.211.228 

At the identified host, there is a web server that always serves up 404 error pages when a GET request against a nonexistent bucket is issued. A 404 error is presented to the user. ...

 > telnet made-up-name-for-wikipedia.s3.amazonaws.com 80 Trying 72.21.207.212... Connected to s3-1-w.amazonaws.com. Escape character is '^]'. GET / HTTP/1.0 host: made-up-name-for-wikipedia.s3.amazonaws.com HTTP/1.1 404 Not Found x-amz-request-id: F0F7301EF1873635 x-amz-id-2: 9gT/YmUa7EZXIm9FNv7GGThAre8Kn5CEfXpoJpthwuq54Pm+5RRcThAdBa20XsLj Content-Type: application/xml Date: Wed, 27 Feb 2008 18:54:42 GMT Connection: close Server: AmazonS3 <?xml version="1.0" encoding="UTF-8"?> <Error><Code>NoSuchBucket</Code> <Message>The specified bucket does not exist</Message> <RequestId>F0F7301EF1873635</RequestId> <BucketName>made-up-name-for-wikipedia</BucketName> <HostId>9gT/YmUa7EZXIm9FNv7GGThAre8Kn5CEfXpoJpthwuq54Pm+5RRcThAdBa20XsLj</HostId> </Error> Connection closed by foreign host. 

Notice how the XML error text is intended for machine interpretation, rather than end users. The RequestId element can be used in support requests, as Amazon log every request, at least for a few days. The Extensible Markup Language (XML) is a general-purpose markup language. ...

Notable uses

Photo hosting service SmugMug has used S3 since April 2006. In November 2006, they claimed to be saving US$500,000 per year by using S3 instead of their own dedicated storage servers.^[2] SmugMug has noted some problems with S3^[3] but continued to use the service and as of April 30, 2007, had saved almost $1 million in storage costs.^[2] SmugMug is a digital photo sharing website, founded by a father and son team in 2002. ...

Web 2.0 blog GigaOM noted that several other startups have embraced S3 as a means of keeping costs lower. ^{[4] [5]} On September 30, 2005, Tim OReilly wrote a piece summarizing his view of Web 2. ... GigaOM.com is a Web 2. ...

Jeremy Zawodny found that using S3 for home backup would save hundreds of dollars compared with buying and maintaining an array of hard drives^[6] and compiled a list of programs that would automate the process.^[7] Jeremy Zawodny Jeremy Zawodny is currently an employee of Yahoo! in the platform engineering group. ...

There is a Pluggable Storage Engine for MySQL ^[8], making it available as tables and rows in MySQL. There is also a backup utility for MySQL ^[9] that does automatic backup of MySQL databases to S3. MySQL (pronounced (IPA) , my S-Q-L[1]) is a multithreaded, multi-user SQL database management system (DBMS)[2] which has, according to MySQL AB, more than 10 million installations. ...

There is a User Mode File System for Linux that lets EC2-hosted Xen images mount an S3 bucket as a file system. Note that as the semantics of the S3 file system are not that of a Posix file system, the file system may not behave entirely as expected. Xen is a free virtual machine monitor for IA-32, x86-64, IA-64 and PowerPC architectures. ... POSIX or Portable Operating System Interface[1] is the collective name of a family of related standards specified by the IEEE to define the application programming interface (API) for software compatible with variants of the Unix operating system. ...

Apache Hadoop file systems can be hosted on S3, as its requirements of a file system are met by S3. As a result, Hadoop can be used to run MapReduce algorithms on EC2 servers, reading data and writing results back to S3. Hadoop is a collection of Free Java software previously developed by the Nutch project but now maintainted by Lucene[1]. The system includes a distributed filesystem reminiscent of GoogleFS named the Hadoop Distributed File System (or just DFS[1]), a clone of MapReduce called HadoopMapReduce[2] and a few other... MapReduce is a programming tool developed by Google in C++ (Python and Java are supported through interfaces), in which parallel computations over large (> 1 terabyte) data sets are performed. ... Amazon Web Services logo Amazon Elastic Compute Cloud (EC2) brings scalable computing power to the masses allowing users to launch and terminate server instances on demand, hence the term elastic. EC2 uses Xen Virtualization. ...

There is a data deduplication based PC backup ^[10] service built for Amazon S3. Data deduplication is claimed to have a 20-to-1 reduction in storage and bandwidth consumed, which might reduce the cost of using Amazon S3 by that amount. Deduplication refers to the database maintenance task of removing duplicate data from within its databases. ...

References

Amazon. ... Year 2006 (MMVI) was a common year starting on Sunday of the Gregorian calendar. ... is the 73rd day of the year (74th in leap years) in the Gregorian calendar. ... For information on Wikipedia press releases, see Wikipedia:Press releases. ... is the 314th day of the year (315th in leap years) in the Gregorian calendar. ... Year 2006 (MMVI) was a common year starting on Sunday of the Gregorian calendar. ... is the 30th day of the year in the Gregorian calendar. ... Year 2007 (MMVII) is the current year, a common year starting on Monday of the Gregorian calendar and the AD/CE era in the 21st century. ... Year 2006 (MMVI) was a common year starting on Sunday of the Gregorian calendar. ... is the 60th day of the year (61st in leap years) in the Gregorian calendar. ...

External links

• Amazon S3 (official site)
• S3Interface.com (example usable web interface)
• SecoBackup (de-duplicated backup on top of S3)