青青草原综合久久大伊人导航_色综合久久天天综合_日日噜噜夜夜狠狠久久丁香五月_热久久这里只有精品

隨筆-380  評論-37  文章-0  trackbacks-0

來自startup的垂直搜索引擎http://www.kosmix.com/的開源項目,又一個開源的類似google mapreduce 的分布式文件系統,可以應用在諸如圖片存儲、搜索引擎、網格計算、數據挖掘這樣需要處理大數據量的網絡應用中。與hadoop集成得也比較好,這樣可以充分利用了hadoop一些現成的功能,基于C++。

Introduction

Applications that process large volumes of data (such as, search engines, grid computing applications, data mining applications, etc.) require a backend infrastructure for storing data. Such infrastructure is required to support applications whose workload could be characterized as:

  • Primarily write-once/read-many workloads
  • Few millions of large files, where each file is on the order of a few tens of MB to a few tens of GB in size
  • Mostly sequential access

We have developed the Kosmos Distributed File System (KFS), a high performance distributed file system to meet this infrastructure need.

The system consists of 3 components:

  1. Meta-data server: a single meta-data server that provides a global namespace
  2. Block server: Files are split into blocks orchunksand stored on block servers. Blocks are also known as chunk servers. Chunkserver store the chunks as files in the underlying file system (such as, XFS on Linux)
  3. Client library: that provides the file system API to allow applications to interface with KFS. To integrate applications to use KFS, applications will need to be modified and relinked with the KFS client library.

KFS is implemented in C++. It is built using standard system components such as, TCP sockets, aio (for disk I/O), STL, and boost libraries. It has been tested on 64-bit x86 architectures running Linux FC5.

While KFS can be accessed natively from C++ applications, support is also provided for Java applications. JNI glue code is included in the release to allow Java applications to access the KFS client library APIs.

Features
  • Incremental scalability: New chunkserver nodes can be added as storage needs increase; the system automatically adapts to the new nodes.
  • Availability: Replication is used to provide availability due to chunk server failures. Typically, files are replicated 3-way.
  • Per file degree of replication: The degree of replication is configurable on a per file basis, with a max. limit of 64.
  • Re-replication: Whenever the degree of replication for a file drops below the configured amount (such as, due to an extended chunkserver outage), the metaserver forces the block to be re-replicated on the remaining chunk servers. Re-replication is done in the background without overwhelming the system.
  • Re-balancing: Periodically, the meta-server may rebalance the chunks amongst chunkservers. This is done to help with balancing disk space utilization amongst nodes.
  • Data integrity: To handle disk corruptions to data blocks, data blocks are checksummed. Checksum verification is done on each read; whenever there is a checksum mismatch, re-replication is used to recover the corrupted chunk.
  • File writes: The system follows the standard model. When an application creates a file, the filename becomes part of the filesystem namespace. For performance, writes are cached at the KFS client library. Periodically, the cache is flushed and data is pushed out to the chunkservers. Also, applications can force data to be flushed to the chunkservers. In either case, once data is flushed to the server, it is available for reading.
  • Leases: KFS client library uses caching to improve performance. Leases are used to support cache consistency.
  • Chunk versioning: Versioning is used to detect stale chunks.
  • Client side fail-over: The client library is resilient to chunksever failures. During reads, if the client library determines that the chunkserver it is communicating with is unreachable, the client library will fail-over to another chunkserver and continue the read. This fail-over is transparent to the application.
  • Language support: KFS client library can be accessed from C++, Java, and Python.
  • FUSE support on Linux: By mounting KFS via FUSE, this support allows existing linux utilities (such as, ls) to interface with KFS.
  • Tools: A shell binary is included in the set of tools. This allows users to navigate the filesystem tree using utilities such as, cp, ls, mkdir, rmdir, rm, mv. Tools to also monitor the chunk/meta-servers are provided.
  • Deploy scrīpts: To simplify launching KFS servers, a set of scrīpts to (1) install KFS binaries on a set of nodes, (2) start/stop KFS servers on a set of nodes are also provided.
  • Job placement support: The KFS client library exports an API to determine the location of a byte range of a file. Job placement systems built on top of KFS can leverage this API to schedule jobs appropriately.
  • Local read optimization: When applications are run on the same nodes as chunkservers, the KFS client library contains an optimization for reading data locally. That is, if the chunk is stored on the same node as the one on which the application is executing, data is read from the local node.
KFS with Hadoop

KFS has been integrated with Hadoop using Hadoop’s filesystem interfaces. This allows existing Hadoop applications to use KFS seamlessly. The integration code has been submitted as a patch to Hadoop-JIRA-1963 (this will enable distribution of the integration code with Hadoop). In addition, the code as well as instructions will also be available for download from the KFS project page shortly. As part of the integration, there is job placement support for Hadoop. That is, the Hadoop Map/Reduce job placement system can schedule jobs on the nodes where the chunks are stored.

參考資料:

  • distribute file system

http://lucene.apache.org/hadoop/

http://www.danga.com/mogilefs/

http://www.lustre.org/

http://oss.sgi.com/projects/xfs/

 

http://www.megite.com/discover/filesystem

http://swik.net/distributed+cluster

  • cluster&high availability

http://www.gluster.org/index.php

http://www.linux-ha.org/

http://openssi.org

http://kerrighed.org/

http://openmosix.sourceforge.net/

 

http://www.linux.com/article.pl?sid=06/09/12/1459204

http://labs.google.com/papers/mapreduce.html

posted on 2010-04-01 09:47 小王 閱讀(2122) 評論(2)  編輯 收藏 引用 所屬分類: 分布式系統

評論:
# re: kosmix,又一個開源的類似google mapreduce 的分布式文件系統 2010-04-01 12:55 | 那誰
概念性的錯誤:mapreduce不是分布式文件系統,你說的應該是GFS.
  回復  更多評論
  
# re: kosmix,又一個開源的類似google mapreduce 的分布式文件系統 2010-04-01 21:51 | 小王
感謝那誰的指教,現題目已經改過
  回復  更多評論
  
# re: kosmix,又一個開源的類似GFS的分布式文件系統 2010-05-10 12:51 | CANDYGonzales19
Do not money to buy a house? Worry no more, just because it is real to take the <a href="http://lowest-rate-loans.com/topics/credit-loans">http://www.lowest-rate-loans.com</a> to solve such problems. Hence take a commercial loan to buy all you want.   回復  更多評論
  
青青草原综合久久大伊人导航_色综合久久天天综合_日日噜噜夜夜狠狠久久丁香五月_热久久这里只有精品
  • <ins id="pjuwb"></ins>
    <blockquote id="pjuwb"><pre id="pjuwb"></pre></blockquote>
    <noscript id="pjuwb"></noscript>
          <sup id="pjuwb"><pre id="pjuwb"></pre></sup>
            <dd id="pjuwb"></dd>
            <abbr id="pjuwb"></abbr>
            久久综合一区二区三区| av成人手机在线| 久久久久一区二区三区| 性久久久久久久久| 黑人一区二区| 亚洲电影在线播放| 欧美国产大片| 亚洲女同同性videoxma| 亚洲女人天堂av| 亚洲国产精品999| 亚洲毛片播放| 国产一区清纯| 亚洲国产美国国产综合一区二区| 欧美精品日韩一区| 久久国产精品久久久久久久久久| 久久美女性网| 亚洲一区二区精品在线| 欧美在线观看一区二区| 日韩亚洲一区在线播放| 欧美亚洲三级| 一本色道久久综合| 久久久久高清| 亚洲一区欧美一区| 美女精品视频一区| 欧美一区免费视频| 欧美日韩国产美女| 久久综合给合| 国产精品永久在线| 亚洲激情综合| 在线播放日韩专区| 亚洲影院色无极综合| 亚洲精品一区二区三区婷婷月| 在线中文字幕一区| 亚洲精品视频免费观看| 久久大综合网| 性欧美精品高清| 欧美日韩另类综合| 欧美黄色影院| 影音先锋久久资源网| 亚洲自拍偷拍网址| 亚洲一区二区视频在线观看| 免费成人av资源网| 久久久国产成人精品| 国产精品久久久久久久久久久久久| 免费永久网站黄欧美| 国产午夜精品理论片a级大结局 | 亚洲精品社区| 亚洲福利国产精品| 久久精品青青大伊人av| 午夜亚洲精品| 国产精品久线观看视频| 夜夜爽99久久国产综合精品女不卡| 亚洲国产精品精华液网站| 久久福利影视| 久久久噜噜噜久久中文字幕色伊伊 | 亚洲美女视频| 亚洲区第一页| 美女视频黄a大片欧美| 欧美 日韩 国产在线| 一区二区三区在线观看视频| 欧美在线视频全部完| 久久久久久久久久久一区| 国产亚洲精品综合一区91| 亚洲自拍偷拍网址| 久久精品人人做人人爽| 国产亚洲激情视频在线| 欧美在线free| 免费成人高清视频| 亚洲国产精品成人综合| 美女尤物久久精品| 亚洲国产va精品久久久不卡综合| 亚洲国产精品尤物yw在线观看| 久久久美女艺术照精彩视频福利播放 | 亚洲欧洲日产国产综合网| 女女同性精品视频| 亚洲精品日韩一| 亚洲一区二区三区在线视频| 欧美精品尤物在线| 一区二区三区高清不卡| 欧美一区午夜视频在线观看| 国产色视频一区| 久久久一本精品99久久精品66| 欧美91福利在线观看| 亚洲精品在线视频观看| 欧美视频精品在线| 欧美一区在线看| 欧美国产第二页| 亚洲宅男天堂在线观看无病毒| 国产日韩欧美三级| 欧美阿v一级看视频| 在线亚洲电影| 麻豆精品一区二区av白丝在线| 亚洲人午夜精品免费| 欧美人成网站| 久久岛国电影| 一本大道久久a久久综合婷婷| 欧美伊人久久久久久久久影院| 亚洲高清毛片| 欧美性片在线观看| 久久亚洲不卡| 亚洲午夜一区二区| 欧美高清在线一区| 欧美一区二区大片| 99视频精品全部免费在线| 国产日韩av一区二区| 欧美国产视频在线观看| 在线中文字幕日韩| 亚洲国产精品ⅴa在线观看 | 9久re热视频在线精品| 国产精品一区二区在线观看网站 | 欧美午夜激情在线| 美女精品网站| 久久av红桃一区二区小说| 亚洲毛片在线免费观看| 亚洲一区二区三区乱码aⅴ蜜桃女 亚洲一区二区三区乱码aⅴ | 亚洲国产精品久久91精品| 欧美午夜宅男影院| 欧美成人亚洲成人| 欧美亚洲一区| 亚洲综合社区| 亚洲视频碰碰| 日韩视频第一页| 亚洲国内精品在线| 欧美电影免费| 免费一级欧美在线大片| 久久久久综合一区二区三区| 亚洲免费在线电影| 亚洲视频在线看| 亚洲国产欧美在线人成| 久久久国产91| 欧美一区二区三区视频| 亚洲一二三级电影| 99视频精品免费观看| 亚洲人成在线播放网站岛国| 国产一区二区高清不卡| 国产精品午夜春色av| 欧美日韩一区二区三区免费看| 欧美大片va欧美在线播放| 看欧美日韩国产| 美女日韩在线中文字幕| 老司机成人在线视频| 男人的天堂成人在线| 免费一级欧美片在线观看| 久久一区视频| 欧美激情 亚洲a∨综合| 欧美激情在线狂野欧美精品| 欧美女同视频| 国产精品国色综合久久| 国产精品女主播在线观看| 国产麻豆精品在线观看| 国产欧美一区二区精品忘忧草 | 亚洲欧美日韩国产成人精品影院| 亚洲一区自拍| 欧美在线国产精品| 久久最新视频| 欧美日韩视频在线| 国产精品久久久久久久电影| 国产毛片一区| 亚洲电影下载| 9i看片成人免费高清| 亚洲一区久久| 久热精品在线视频| 亚洲韩国日本中文字幕| 99国产精品99久久久久久粉嫩| 一区二区三区视频在线看| 亚洲欧美一区二区三区极速播放| 久久精品一区二区国产| 欧美寡妇偷汉性猛交| 国产精品国产三级国产普通话三级 | 欧美精品情趣视频| 国产精品福利在线观看网址| 国产农村妇女毛片精品久久麻豆 | 国产精品视频免费观看| 玉米视频成人免费看| 一区二区三区四区五区精品视频| 欧美一区二区三区免费视| 久久综合中文| 在线综合亚洲欧美在线视频| 久久久久国产精品一区三寸| 欧美欧美在线| 一区二区在线观看视频在线观看| 99riav久久精品riav| 久久精品国产久精国产思思| 亚洲高清久久| 欧美自拍偷拍| 国产精品久久久久一区二区三区共| 伊人精品成人久久综合软件| 亚洲永久免费视频| 亚洲国产99精品国自产| 欧美诱惑福利视频| 国产精品日韩高清| 9国产精品视频| 欧美顶级少妇做爰| 亚洲欧美高清| 欧美色视频在线| 亚洲精品中文字幕女同| 久久综合色影院| 性色av一区二区三区红粉影视| 欧美连裤袜在线视频| 亚洲高清毛片|