青青草原综合久久大伊人导航_色综合久久天天综合_日日噜噜夜夜狠狠久久丁香五月_热久久这里只有精品

隨筆-380  評論-37  文章-0  trackbacks-0

來自startup的垂直搜索引擎http://www.kosmix.com/的開源項目,又一個開源的類似google mapreduce 的分布式文件系統,可以應用在諸如圖片存儲、搜索引擎、網格計算、數據挖掘這樣需要處理大數據量的網絡應用中。與hadoop集成得也比較好,這樣可以充分利用了hadoop一些現成的功能,基于C++。

Introduction

Applications that process large volumes of data (such as, search engines, grid computing applications, data mining applications, etc.) require a backend infrastructure for storing data. Such infrastructure is required to support applications whose workload could be characterized as:

  • Primarily write-once/read-many workloads
  • Few millions of large files, where each file is on the order of a few tens of MB to a few tens of GB in size
  • Mostly sequential access

We have developed the Kosmos Distributed File System (KFS), a high performance distributed file system to meet this infrastructure need.

The system consists of 3 components:

  1. Meta-data server: a single meta-data server that provides a global namespace
  2. Block server: Files are split into blocks orchunksand stored on block servers. Blocks are also known as chunk servers. Chunkserver store the chunks as files in the underlying file system (such as, XFS on Linux)
  3. Client library: that provides the file system API to allow applications to interface with KFS. To integrate applications to use KFS, applications will need to be modified and relinked with the KFS client library.

KFS is implemented in C++. It is built using standard system components such as, TCP sockets, aio (for disk I/O), STL, and boost libraries. It has been tested on 64-bit x86 architectures running Linux FC5.

While KFS can be accessed natively from C++ applications, support is also provided for Java applications. JNI glue code is included in the release to allow Java applications to access the KFS client library APIs.

Features
  • Incremental scalability: New chunkserver nodes can be added as storage needs increase; the system automatically adapts to the new nodes.
  • Availability: Replication is used to provide availability due to chunk server failures. Typically, files are replicated 3-way.
  • Per file degree of replication: The degree of replication is configurable on a per file basis, with a max. limit of 64.
  • Re-replication: Whenever the degree of replication for a file drops below the configured amount (such as, due to an extended chunkserver outage), the metaserver forces the block to be re-replicated on the remaining chunk servers. Re-replication is done in the background without overwhelming the system.
  • Re-balancing: Periodically, the meta-server may rebalance the chunks amongst chunkservers. This is done to help with balancing disk space utilization amongst nodes.
  • Data integrity: To handle disk corruptions to data blocks, data blocks are checksummed. Checksum verification is done on each read; whenever there is a checksum mismatch, re-replication is used to recover the corrupted chunk.
  • File writes: The system follows the standard model. When an application creates a file, the filename becomes part of the filesystem namespace. For performance, writes are cached at the KFS client library. Periodically, the cache is flushed and data is pushed out to the chunkservers. Also, applications can force data to be flushed to the chunkservers. In either case, once data is flushed to the server, it is available for reading.
  • Leases: KFS client library uses caching to improve performance. Leases are used to support cache consistency.
  • Chunk versioning: Versioning is used to detect stale chunks.
  • Client side fail-over: The client library is resilient to chunksever failures. During reads, if the client library determines that the chunkserver it is communicating with is unreachable, the client library will fail-over to another chunkserver and continue the read. This fail-over is transparent to the application.
  • Language support: KFS client library can be accessed from C++, Java, and Python.
  • FUSE support on Linux: By mounting KFS via FUSE, this support allows existing linux utilities (such as, ls) to interface with KFS.
  • Tools: A shell binary is included in the set of tools. This allows users to navigate the filesystem tree using utilities such as, cp, ls, mkdir, rmdir, rm, mv. Tools to also monitor the chunk/meta-servers are provided.
  • Deploy scrīpts: To simplify launching KFS servers, a set of scrīpts to (1) install KFS binaries on a set of nodes, (2) start/stop KFS servers on a set of nodes are also provided.
  • Job placement support: The KFS client library exports an API to determine the location of a byte range of a file. Job placement systems built on top of KFS can leverage this API to schedule jobs appropriately.
  • Local read optimization: When applications are run on the same nodes as chunkservers, the KFS client library contains an optimization for reading data locally. That is, if the chunk is stored on the same node as the one on which the application is executing, data is read from the local node.
KFS with Hadoop

KFS has been integrated with Hadoop using Hadoop’s filesystem interfaces. This allows existing Hadoop applications to use KFS seamlessly. The integration code has been submitted as a patch to Hadoop-JIRA-1963 (this will enable distribution of the integration code with Hadoop). In addition, the code as well as instructions will also be available for download from the KFS project page shortly. As part of the integration, there is job placement support for Hadoop. That is, the Hadoop Map/Reduce job placement system can schedule jobs on the nodes where the chunks are stored.

參考資料:

  • distribute file system

http://lucene.apache.org/hadoop/

http://www.danga.com/mogilefs/

http://www.lustre.org/

http://oss.sgi.com/projects/xfs/

 

http://www.megite.com/discover/filesystem

http://swik.net/distributed+cluster

  • cluster&high availability

http://www.gluster.org/index.php

http://www.linux-ha.org/

http://openssi.org

http://kerrighed.org/

http://openmosix.sourceforge.net/

 

http://www.linux.com/article.pl?sid=06/09/12/1459204

http://labs.google.com/papers/mapreduce.html

posted on 2010-04-01 09:47 小王 閱讀(2122) 評論(2)  編輯 收藏 引用 所屬分類: 分布式系統

評論:
# re: kosmix,又一個開源的類似google mapreduce 的分布式文件系統 2010-04-01 12:55 | 那誰
概念性的錯誤:mapreduce不是分布式文件系統,你說的應該是GFS.
  回復  更多評論
  
# re: kosmix,又一個開源的類似google mapreduce 的分布式文件系統 2010-04-01 21:51 | 小王
感謝那誰的指教,現題目已經改過
  回復  更多評論
  
# re: kosmix,又一個開源的類似GFS的分布式文件系統 2010-05-10 12:51 | CANDYGonzales19
Do not money to buy a house? Worry no more, just because it is real to take the <a href="http://lowest-rate-loans.com/topics/credit-loans">http://www.lowest-rate-loans.com</a> to solve such problems. Hence take a commercial loan to buy all you want.   回復  更多評論
  
青青草原综合久久大伊人导航_色综合久久天天综合_日日噜噜夜夜狠狠久久丁香五月_热久久这里只有精品
  • <ins id="pjuwb"></ins>
    <blockquote id="pjuwb"><pre id="pjuwb"></pre></blockquote>
    <noscript id="pjuwb"></noscript>
          <sup id="pjuwb"><pre id="pjuwb"></pre></sup>
            <dd id="pjuwb"></dd>
            <abbr id="pjuwb"></abbr>
            日韩一级精品| 久久亚洲一区二区| 欧美午夜大胆人体| 亚洲一区久久久| 一区二区三区欧美在线观看| 欧美理论片在线观看| 亚洲午夜国产成人av电影男同| 99精品国产福利在线观看免费 | 久久成人精品无人区| 欧美在线观看www| 亚洲激情在线视频| 一区二区三欧美| 国产农村妇女精品一二区| 狂野欧美一区| 欧美日韩精品免费观看| 欧美专区日韩专区| 免费成人性网站| 午夜精品区一区二区三| 久久视频在线免费观看| 亚洲小视频在线| 久久久久国色av免费看影院| 亚洲精品日韩一| 性欧美xxxx视频在线观看| 亚洲第一色在线| 亚洲一区二区3| 亚洲国产中文字幕在线观看| 这里只有精品电影| 亚洲国产精品一区制服丝袜| 国产精品99久久久久久久久久久久| 国产一区二区电影在线观看| 亚洲国产精品电影| 国产一区二区三区的电影 | 美女精品在线| 午夜天堂精品久久久久| 蜜臀91精品一区二区三区| 亚洲欧美中文在线视频| 免费视频一区二区三区在线观看| 亚洲欧美日韩国产成人| 欧美成人免费在线视频| 久久久国产一区二区三区| 欧美女同视频| 欧美国产亚洲视频| 国产亚洲免费的视频看| 亚洲桃花岛网站| av成人天堂| 美女主播精品视频一二三四| 久久久水蜜桃av免费网站| 国产精品二区三区四区| 亚洲精品视频在线| 亚洲人成网站精品片在线观看| 亚洲欧美日韩人成在线播放| 一区二区三区三区在线| 欧美 日韩 国产在线| 米奇777超碰欧美日韩亚洲| 国产日韩av在线播放| 亚洲香蕉在线观看| 亚洲一区久久久| 欧美日韩国产精品一卡| 最新中文字幕亚洲| 亚洲最新色图| 欧美日韩亚洲一区二区三区四区| 亚洲福利视频一区二区| 亚洲卡通欧美制服中文| 欧美成人午夜激情在线| 亚洲欧洲在线视频| 一本大道av伊人久久综合| 欧美国产在线视频| 亚洲精品美女在线观看| 99在线精品观看| 欧美日韩国产精品| 亚洲视频在线二区| 午夜免费日韩视频| 国产亚洲精品久久久| 久久av在线看| 蜜臀99久久精品久久久久久软件| 亚洲国产91| 欧美日韩视频第一区| 一本色道久久综合亚洲精品小说| 亚洲一区免费| 国产色视频一区| 久久久久久一区| 亚洲级视频在线观看免费1级| 日韩五码在线| 国产美女搞久久| 久久狠狠亚洲综合| 亚洲国产综合视频在线观看| 亚洲视频图片小说| 国模吧视频一区| 男女激情久久| 亚洲一级在线| 欧美成人免费大片| 亚洲视频在线观看网站| 国产视频在线观看一区 | 中国日韩欧美久久久久久久久| 欧美一区二区三区在线观看| 激情欧美一区二区三区| 欧美精品一区二区三| 亚洲专区一区| 亚洲第一免费播放区| 亚洲欧美日韩视频二区| 亚洲第一网站| 国产精品久久网站| 久热这里只精品99re8久| 一区二区三区你懂的| 美日韩精品免费| 亚洲专区国产精品| 亚洲国产视频a| 国产精品资源在线观看| 模特精品裸拍一区| 欧美亚洲综合久久| 亚洲伦伦在线| 免费国产自线拍一欧美视频| 午夜精品成人在线视频| 亚洲精品乱码久久久久久蜜桃麻豆| 国产精品久久久久7777婷婷| 免费观看在线综合| 久久爱另类一区二区小说| 中国成人亚色综合网站| 亚洲黄色毛片| 欧美a级理论片| 久久久久久一区| 亚洲欧美在线网| 一二三四社区欧美黄| 亚洲黄色天堂| 一区精品在线| 国产主播一区二区| 国产欧美在线看| 国产精品家庭影院| 欧美日韩免费看| 欧美经典一区二区| 欧美成人午夜激情视频| 久久中文在线| 久久精品日韩| 欧美在线一二三区| 亚洲欧美日韩第一区| 亚洲午夜日本在线观看| 9色精品在线| 亚洲精品资源| 99re6这里只有精品| 亚洲精品少妇30p| 亚洲精品国精品久久99热| 亚洲国产精品综合| 亚洲激情小视频| 亚洲美女在线一区| 一区二区三区导航| 亚洲婷婷在线| 亚洲一区精彩视频| 性色av一区二区三区在线观看| 亚洲综合日本| 欧美专区在线观看一区| 久久九九免费视频| 另类尿喷潮videofree| 久热精品在线视频| 欧美国产精品va在线观看| 欧美顶级大胆免费视频| 欧美日韩免费高清一区色橹橹| 欧美日韩理论| 国产欧美精品一区二区三区介绍| 国产九九精品视频| 国语对白精品一区二区| 亚洲黄色有码视频| 夜夜夜精品看看| 欧美一区观看| 男女精品视频| 99re6这里只有精品视频在线观看| 亚洲精选在线观看| 午夜精品久久99蜜桃的功能介绍| 午夜精品免费视频| 美国成人直播| 欧美午夜理伦三级在线观看| 国产日韩欧美在线视频观看| 黄色在线一区| 一区二区三区.www| 久久不射网站| 亚洲国产福利在线| 亚洲一区二区精品在线| 久久久久高清| 欧美日韩在线亚洲一区蜜芽| 国产一区二区观看| 一本一本久久a久久精品综合麻豆| 亚洲男女自偷自拍| 欧美福利专区| 午夜久久电影网| 欧美日韩国产片| 韩日精品视频一区| 亚洲图片欧美日产| 欧美国产激情二区三区| 亚洲午夜国产成人av电影男同| 久久美女性网| 国产日本亚洲高清| 中文无字幕一区二区三区| 久久裸体艺术| 亚洲视频电影在线| 欧美激情bt| 尤物精品在线| 欧美专区第一页| 亚洲午夜久久久| 欧美日韩一区综合| 亚洲精品永久免费| 麻豆精品视频在线观看|