《谷歌bigtable原文+中文注解》由会员分享，可在线阅读，更多相关《谷歌bigtable原文+中文注解（14页珍藏版）》请在金锄头文库上搜索。
1、Bigtable: A Distributed Storage System for Structured Data Fay Chang, Jeffrey Dean, Sanjay Ghemawat, Wilson C. Hsieh, Deborah A. Wallach Mike Burrows, Tushar Chandra, Andrew Fikes, Robert E. Gruber fay,jeff,sanjay,wilsonh,kerr,m3b,tushar,fi kes,gruber Google, Inc. Abstract Bigtable is a distributed
2、storage system for managing structured data that is designed to scale to a very large size: petabytes of data across thousands of commodity servers. Many projects at Google store data in Bigtable, including web indexing, Google Earth, and Google Fi- nance. These applications place very different dem
3、ands on Bigtable, both in terms of data size (from URLs to web pages to satellite imagery) and latency requirements (frombackendbulkprocessingtoreal-timedataserving). Despite these varied demands, Bigtable has successfully provided a fl exible, high-performance solution for all of theseGoogleproduct
4、s. Inthis paperwedescribethesim- ple data model provided by Bigtable, which gives clients dynamic control over data layout and format, and we de- scribe the design and implementation of Bigtable. 1Introduction Over the last two and a half years we have designed, implemented, and deployed a distribut
5、ed storage system for managing structured data at Google called Bigtable. Bigtable is designed to reliably scale to petabytes of data and thousands of machines. Bigtable has achieved several goals: wide applicability, scalability, high per- formance, and high availability.Bigtable is used by more th
6、an sixty Google products and projects, includ- ing Google Analytics, Google Finance, Orkut, Person- alized Search, Writely, and Google Earth. These prod- ucts use Bigtable for a variety of demanding workloads, which range from throughput-oriented batch-processing jobs to latency-sensitive serving of
7、 data to end users. The Bigtable clusters used by these products span a wide range of confi gurations, from a handful to thousands of servers, andstore upto severalhundredterabytesof data. Inmanyways, Bigtableresemblesadatabase: it shares many implementation strategies with databases. Paral- lel dat
8、abases 14 and main-memorydatabases 13 have achieved scalability and high performance, but Bigtable providesadifferentinterfacethansuchsystems. Bigtable does not support a full relational data model; instead, it provides clients with a simple data model that supports dynamic control over data layout
9、and format, and al- lows clients to reason about the locality properties of the data represented in the underlying storage. Data is in- dexed using row and column names that can be arbitrary strings. Bigtable also treats data as uninterpreted strings, although clients often serialize various forms o
10、f struc- tured and semi-structured data into these strings. Clients can control the locality of their data through careful choices in their schemas. Finally, Bigtable schema pa- rameters let clients dynamically control whether to serve data out of memory or from disk. Section 2 describes the data mo
11、del in more detail, and Section 3 provides an overview of the client API. Sec- tion 4 briefl y describes the underlyingGoogleinfrastruc- ture on which Bigtable depends. Section 5 describes the fundamentals of the Bigtable implementation, and Sec- tion 6 describes some of the refi nements that we mad
12、e to improve Bigtables performance. Section 7 provides measurements of Bigtables performance. We describe several examples of how Bigtable is used at Google in Section 8, and discuss some lessons we learned in designing and supporting Bigtable in Section 9.Fi- nally, Section 10 describes related wor
13、k, and Section 11 presents our conclusions. 2Data Model A Bigtable is a sparse, distributed, persistent multi- dimensional sorted map. The map is indexed by a row key, column key, and a timestamp; each value in the map is an uninterpreted array of bytes. (row:string, column:string, time:int64) strin
14、g To appear in OSDI 20061 “CNN.com“ “CNN“ “.“ “.“ “.“ t9 t6 t3 t5 8 t “anchor: “ “ n.www“ “anchor:my.look.ca“contents:“ Figure 1:A slice of an example table that stores Web pages. The row name is a reversed URL. The contents column family con- tains the page contents, and the anchor column family co
15、ntains the text of any anchors that reference the page. CNNs home page is referenced by both the Sports Illustrated and the MY-look home pages, so the row contains columns named anchor: and anchor:my.look.ca. Each anchor cell has one version; the contents column has three versions, at timestamps t3,
16、 t5, and t6. We settled onthis datamodelafterexamininga variety of potential uses of a Bigtable-like system. As one con- crete example that drove some of our design decisions, suppose we want to keep a copy of a large collection of web pages and related information that could be used by many different projects; let us call this particular table the Webtable. In Webtable, we would use URLs as row keys, variousaspects ofwebpagesas columnnames, and store the contents of the web pages
