• <ins id="pjuwb"></ins>
    <blockquote id="pjuwb"><pre id="pjuwb"></pre></blockquote>
    <noscript id="pjuwb"></noscript>
          <sup id="pjuwb"><pre id="pjuwb"></pre></sup>
            <dd id="pjuwb"></dd>
            <abbr id="pjuwb"></abbr>
            隨筆-380  評(píng)論-37  文章-0  trackbacks-0

            來自startup的垂直搜索引擎http://www.kosmix.com/的開源項(xiàng)目,又一個(gè)開源的類似google mapreduce 的分布式文件系統(tǒng),可以應(yīng)用在諸如圖片存儲(chǔ)、搜索引擎、網(wǎng)格計(jì)算、數(shù)據(jù)挖掘這樣需要處理大數(shù)據(jù)量的網(wǎng)絡(luò)應(yīng)用中。與hadoop集成得也比較好,這樣可以充分利用了hadoop一些現(xiàn)成的功能,基于C++。

            Introduction

            Applications that process large volumes of data (such as, search engines, grid computing applications, data mining applications, etc.) require a backend infrastructure for storing data. Such infrastructure is required to support applications whose workload could be characterized as:

            • Primarily write-once/read-many workloads
            • Few millions of large files, where each file is on the order of a few tens of MB to a few tens of GB in size
            • Mostly sequential access

            We have developed the Kosmos Distributed File System (KFS), a high performance distributed file system to meet this infrastructure need.

            The system consists of 3 components:

            1. Meta-data server: a single meta-data server that provides a global namespace
            2. Block server: Files are split into blocks orchunksand stored on block servers. Blocks are also known as chunk servers. Chunkserver store the chunks as files in the underlying file system (such as, XFS on Linux)
            3. Client library: that provides the file system API to allow applications to interface with KFS. To integrate applications to use KFS, applications will need to be modified and relinked with the KFS client library.

            KFS is implemented in C++. It is built using standard system components such as, TCP sockets, aio (for disk I/O), STL, and boost libraries. It has been tested on 64-bit x86 architectures running Linux FC5.

            While KFS can be accessed natively from C++ applications, support is also provided for Java applications. JNI glue code is included in the release to allow Java applications to access the KFS client library APIs.

            Features
            • Incremental scalability: New chunkserver nodes can be added as storage needs increase; the system automatically adapts to the new nodes.
            • Availability: Replication is used to provide availability due to chunk server failures. Typically, files are replicated 3-way.
            • Per file degree of replication: The degree of replication is configurable on a per file basis, with a max. limit of 64.
            • Re-replication: Whenever the degree of replication for a file drops below the configured amount (such as, due to an extended chunkserver outage), the metaserver forces the block to be re-replicated on the remaining chunk servers. Re-replication is done in the background without overwhelming the system.
            • Re-balancing: Periodically, the meta-server may rebalance the chunks amongst chunkservers. This is done to help with balancing disk space utilization amongst nodes.
            • Data integrity: To handle disk corruptions to data blocks, data blocks are checksummed. Checksum verification is done on each read; whenever there is a checksum mismatch, re-replication is used to recover the corrupted chunk.
            • File writes: The system follows the standard model. When an application creates a file, the filename becomes part of the filesystem namespace. For performance, writes are cached at the KFS client library. Periodically, the cache is flushed and data is pushed out to the chunkservers. Also, applications can force data to be flushed to the chunkservers. In either case, once data is flushed to the server, it is available for reading.
            • Leases: KFS client library uses caching to improve performance. Leases are used to support cache consistency.
            • Chunk versioning: Versioning is used to detect stale chunks.
            • Client side fail-over: The client library is resilient to chunksever failures. During reads, if the client library determines that the chunkserver it is communicating with is unreachable, the client library will fail-over to another chunkserver and continue the read. This fail-over is transparent to the application.
            • Language support: KFS client library can be accessed from C++, Java, and Python.
            • FUSE support on Linux: By mounting KFS via FUSE, this support allows existing linux utilities (such as, ls) to interface with KFS.
            • Tools: A shell binary is included in the set of tools. This allows users to navigate the filesystem tree using utilities such as, cp, ls, mkdir, rmdir, rm, mv. Tools to also monitor the chunk/meta-servers are provided.
            • Deploy scrīpts: To simplify launching KFS servers, a set of scrīpts to (1) install KFS binaries on a set of nodes, (2) start/stop KFS servers on a set of nodes are also provided.
            • Job placement support: The KFS client library exports an API to determine the location of a byte range of a file. Job placement systems built on top of KFS can leverage this API to schedule jobs appropriately.
            • Local read optimization: When applications are run on the same nodes as chunkservers, the KFS client library contains an optimization for reading data locally. That is, if the chunk is stored on the same node as the one on which the application is executing, data is read from the local node.
            KFS with Hadoop

            KFS has been integrated with Hadoop using Hadoop’s filesystem interfaces. This allows existing Hadoop applications to use KFS seamlessly. The integration code has been submitted as a patch to Hadoop-JIRA-1963 (this will enable distribution of the integration code with Hadoop). In addition, the code as well as instructions will also be available for download from the KFS project page shortly. As part of the integration, there is job placement support for Hadoop. That is, the Hadoop Map/Reduce job placement system can schedule jobs on the nodes where the chunks are stored.

            參考資料:

            • distribute file system

            http://lucene.apache.org/hadoop/

            http://www.danga.com/mogilefs/

            http://www.lustre.org/

            http://oss.sgi.com/projects/xfs/

             

            http://www.megite.com/discover/filesystem

            http://swik.net/distributed+cluster

            • cluster&high availability

            http://www.gluster.org/index.php

            http://www.linux-ha.org/

            http://openssi.org

            http://kerrighed.org/

            http://openmosix.sourceforge.net/

             

            http://www.linux.com/article.pl?sid=06/09/12/1459204

            http://labs.google.com/papers/mapreduce.html

            posted on 2010-04-01 09:47 小王 閱讀(2042) 評(píng)論(2)  編輯 收藏 引用 所屬分類: 分布式系統(tǒng)

            評(píng)論:
            # re: kosmix,又一個(gè)開源的類似google mapreduce 的分布式文件系統(tǒng) 2010-04-01 12:55 | 那誰
            概念性的錯(cuò)誤:mapreduce不是分布式文件系統(tǒng),你說的應(yīng)該是GFS.
              回復(fù)  更多評(píng)論
              
            # re: kosmix,又一個(gè)開源的類似google mapreduce 的分布式文件系統(tǒng) 2010-04-01 21:51 | 小王
            感謝那誰的指教,現(xiàn)題目已經(jīng)改過
              回復(fù)  更多評(píng)論
              
            # re: kosmix,又一個(gè)開源的類似GFS的分布式文件系統(tǒng) 2010-05-10 12:51 | CANDYGonzales19
            Do not money to buy a house? Worry no more, just because it is real to take the <a href="http://lowest-rate-loans.com/topics/credit-loans">http://www.lowest-rate-loans.com</a> to solve such problems. Hence take a commercial loan to buy all you want.   回復(fù)  更多評(píng)論
              
            无码任你躁久久久久久老妇App| 精品综合久久久久久97超人| 婷婷久久综合九色综合九七| 久久免费视频1| 精品久久久无码人妻中文字幕| 久久精品国产亚洲AV香蕉| 久久香蕉国产线看观看99| 精品视频久久久久| 久久亚洲AV成人出白浆无码国产 | 亚洲精品乱码久久久久久久久久久久 | 久久午夜无码鲁丝片秋霞| 99精品国产在热久久| 亚洲伊人久久综合中文成人网| 人妻无码久久一区二区三区免费| 久久精品成人免费观看97| 国产亚洲婷婷香蕉久久精品| 欧美精品国产综合久久| 久久久WWW免费人成精品| 国内精品久久久久影院优 | 亚洲精品高清一二区久久| 99久久精品毛片免费播放| 亚洲欧美日韩久久精品第一区| 三级韩国一区久久二区综合| 久久久青草青青亚洲国产免观| 色综合久久无码中文字幕| 无码国内精品久久人妻麻豆按摩| 久久精品国产99国产精品澳门 | 久久国产色AV免费看| 久久精品无码一区二区WWW| 色婷婷综合久久久久中文字幕| 国产精品99久久不卡| 国内精品久久久久久久亚洲| 中文字幕一区二区三区久久网站| 91精品国产综合久久久久久| 97精品伊人久久大香线蕉app| 日韩精品久久无码人妻中文字幕| 伊人久久综合无码成人网| 伊人久久精品影院| 久久久国产打桩机| 久久无码人妻一区二区三区午夜| 久久这里只有精品18|