锘??xml version="1.0" encoding="utf-8" standalone="yes"?>
鐢變簬鍏峰浣庢垚鏈拰鍓嶆墍鏈湁鐨勯珮鎵╁睍鎬э紝Hadoop宸茶鍏涓烘槸鏂頒竴浠g殑澶ф暟鎹鐞嗗鉤鍙般傚氨鍍?0騫村墠SQL錛圫tructured Query Language錛夊嚭鐜頒竴鏍鳳紝Hadoop姝e甫鏉ヤ簡鏂頒竴杞殑鏁版嵁闈╁懡銆傚浠奌adoop宸蹭粠鍒濆嚭鑼呭簮鐨勫皬璞″彉鎴愪簡琛屼笟鐨勫法浜猴紝浣咹adoop浠嶉渶緇х畫瀹屽杽銆?/p>
鍩轟簬Java璇█鏋勫緩鐨凥adoop妗嗘灦瀹為檯涓婁竴縐嶅垎甯冨紡澶勭悊澶ф暟鎹鉤鍙幫紝鍏跺寘鎷蔣浠跺拰浼楀瀛愰」鐩傚湪榪戝崄騫翠腑Hadoop宸叉垚涓哄ぇ鏁版嵁闈╁懡鐨勪腑蹇冦侻apReduce浣滀負Hadoop鐨勬牳蹇冩槸涓縐嶅鐞嗗ぇ鍨嬪強瓚呭ぇ鍨嬫暟鎹泦錛圱B綰у埆鐨勬暟鎹傚寘鎷綉緇滅偣鍑諱駭鐢熺殑嫻佹暟鎹佹棩蹇楁枃浠躲佺ぞ浜ょ綉緇滅瓑鎵甯︽潵鐨勬暟鎹級騫剁敓鎴愮浉鍏崇殑鎵ц鐨勭紪紼嬫ā鍨嬨傚叾涓昏鎬濇兂鏄粠鍑芥暟寮忕紪紼嬭璦鍊熼壌鑰屾潵鐨勶紝鍚屾椂涔熷寘鍚簡浠庣煝閲忕紪紼嬭璦鍊熼壌鐨勭壒鎬с?/p>
浜掕仈緗戝法澶碮ahoo錛佷綔涓篐adoop妗嗘灦鐨勫厛椹辯爺絀惰咃紝鍦?騫存椂闂村凡緇忓皢Hadoop濉戦犳垚浜嗘瀬涓烘垚鍔熺殑鎶鏈備絾鐩告瘮浜嶴QL錛孒adoop鍦ㄦ煇浜涙柟闈粛鐒舵樉寰椾笉澶熷畬鍠勩傝繖鐩存帴瀵艱嚧鐜頒粖鎵鏈夌洰鍏夐兘闆嗕腑鍦℉adoop渚涘簲鍟嗙殑韜笂銆傚寘鎷珹mazon銆丆loudera絳夊叕鍙稿甫鏉ヤ紬澶氱殑鍒涙柊騫舵彁渚涘己澶х殑宸ュ叿銆侰loudera鎺ㄥ嚭鐨凜HD3鍖呭惈浼楀鐨勯檮鍔犺蔣浠訛紝鍙互甯姪綆$悊銆佽繍琛孒adoop涓婄殑澶嶆潅浠誨姟錛屼緥濡傦細Apache Mahout銆丗lume銆丼qoop銆丳ig銆丱ozie銆丠ive銆丠Base銆乑ooKeeper銆乄hirr絳夈傚悓鏃禖loudera涔熸槸鐩墠鏈澶х殑鎻愪緵浼佷笟Hadoop鎶鏈敮鎸佸拰鍩硅鐨勫巶鍟嗐傝孉mazon鏄緝鏃╁湪鍏叡浜戜腑榪愯Hadoop鐨勫叕鍙革紝鍏舵彁渚涚殑鍩轟簬MapReduce鐨勫脊鎬ц綆楀彲鎻愪緵嫻烽噺鐨勬暟鎹綆楁湇鍔°?/p>
浣嗘暟鎹鐞嗗彧鏄ぇ鏁版嵁澶勭悊鐨勪竴閮ㄥ垎錛岀粍緇囨渶緇堟兂瑕佸緱鍒扮殑鏄粡榪囧垎鏋愬悗鐨勬湁浠峰肩殑鏁版嵁銆傚晢涓氭櫤鑳藉拰鏁版嵁鍒嗘瀽鍘傚晢濡侱atameer銆丠adapt浠ュ強Karmasphere灝辨樉鐨勪笉鍙垨緙恒?/p>
Hadoop鍦?011騫磋瘉鏄庤嚜韜殑浠峰鹼紝鏈鏄庢樉鐨勮抗璞″氨鏄簲澶ф暟鎹簱綆$悊杞歡渚涘簲鍟咵MC銆両BM銆両nformatica銆丮icrosoft浠ュ強Oracle閮芥姇鍏ヤ簡Hadoop鐨勬鎶便侲MC涓嶮apR灞曞紑鍚堜綔錛岃孧icrosoft鍜孫racle鍒欏垎鍒笌Hortonworks鍜孋loudera灞曞紑浜嗗悎浣溿傝孍MC鍜孫racle宸茬粡鎺ㄥ嚭浜咹adoop涓撴湁璁懼銆備笅闈㈠氨璁╂垜浠潵鐪嬩竴涓婬adoop鍦ㄥぇ鏁版嵁棰嗗煙閮戒繕铏忎簡閭d簺鍏徃鐨勫績銆?/p>
Amazon鍩轟簬MapReduce鐨勬湇鍔?/strong>
Amazon鏃╁湪2009騫村氨鎺ㄥ嚭浜嗗熀浜嶩adoop MapReduce鐨凟C2錛圗lastic Compute Cloud錛夋湇鍔°傚洜姝mazon鍦ㄥ簲瀵圭敤鎴峰簲鐢ㄥ拰闇姹備笂鏄懼緱鑳告湁鎴愮銆傛棤璁烘槸涓皬鍨嬩紒涓氳繕鏄秴澶у瀷鐨勭粍緇囷紝鍩轟簬MapReduce鐨凟C2鏈嶅姟閮界粡鍙椾綇浜嗚冮獙銆傚悓鏃禔WS錛?a title="Powered by Text-Enhance" id="_GPLITA_0" in_rurl="http://www.textsrv.com/click?v=VVM6MTc4MjM6MzM2OmFtYXpvbjozNmU0MGEyNTkzY2Q2YjllMDhkOTg5ZjRhMGEzZWE0MDp6LTEwNDctMTQyMTI6Y2xvdWQuY3Nkbi5uZXQ%3D" style="color: #015fb6; ">Amazon Web Service錛夎繕鍖呮嫭Amazon S3錛圫imple storage Service錛夈侫mazon S3鍙彁渚涢珮浼哥緝鎬с侀潬鍙潬鎬с侀珮鍙敤鎬т互鍙婃瀬浣庣殑瀛樺偍鎴愭湰銆傚埄鐢ˋWS鍙珮鏁堢殑澶勭悊鏁版嵁瀵嗛泦鍨嬬殑浠誨姟錛屽Web绱㈠紩銆佹暟鎹寲鎺樸佹棩蹇楁枃浠跺垎鏋愩佹満鍣ㄥ涔犱互鍙婄鎶鍜岀敓鐗╀俊鎭殑瀛︽湳鐮旂┒銆?/p>
Cloudera鎻愪緵瀹夊叏鐨凥adoop騫沖彴
Cloudera涔熸槸姣旇緝鏃╃殑澶ц妯adoop杞歡鍜屾湇鍔℃彁渚涘晢銆侰loudera涓鐩翠笓娉ㄤ簬灝嗗紑婧愮殑Apache Hadoop瀹屽杽鎴愬彲闈犵殑騫沖彴銆侰loudera鐩墠鎷ユ湁100澶氬瀹㈡埛錛屽茍涓斿湪鏈湀榪樹笌Oracle灞曞紑鍚堜綔錛屽叡鍚岃繘鍐涘ぇ鏁版嵁棰嗗煙銆?/p>
鍦–loudera鎻愪緵浜嗙敤浜庣鐞嗗ぇ鏁版嵁鐨勭鐞嗘帶鍒跺彴鍜岃礋璐g鐞咹adoop閮ㄧ講鐨勫伐鍏蜂互鍙婁紒涓氱駭鐨勬敮鎸併侰loudera鐨勭鐞嗗伐鍏鋒彁渚涘熀浜庡悜瀵煎紡鐨凥adoop瀹夎鍜岄厤緗彍鍗曘傚悓鏃舵彁渚涚浉搴旂殑宸ュ叿錛屼互甯姪緋葷粺綆$悊鍛樼洃鎺у鉤鍙扮殑鍋ュ悍鐘跺喌銆佽瘖鏂棶棰樸佷紭鍖栨ц兘錛屽茍榪涜鎵闇鐨勯厤緗拰瀹夊叏鍙樻洿銆傝孋loudera鐨勪紒涓氱駭鏀寔涓庢湇鍔″寘鎷厤緗鏌ャ佸崌綰у拰涓庣涓夋柟緋葷粺闆嗘垚浠ュ強鍏朵粬鎶鏈祫婧愩傜幇浠奀loudera綆$悊杞歡鐜板湪鐨勪環鏍兼槸姣忚妭鐐規瘡騫?000緹庡厓錛堜笉鍖呮嫭紜歡錛夈?/p>
Datameer灝嗗ぇ鏁版嵁涓庡晢涓氭櫤鑳芥湁鏈虹粨鍚?/strong>
Datameer瀹gО鍏跺叕鍙稿熀浜嶩adoop騫沖彴鐨勪駭鍝佹柟妗圖AS錛圖atameer Analytics Solution錛夐潪甯擱傜敤浜庡晢涓氭櫤鑳斤紙BI錛夈侱atameer鍙氳繃JDBC銆丠ive銆丠ttp榪炴帴浠諱綍鐨勬暟鎹簮銆傚悓鏃跺寘鎷竴涓悜瀵奸┍鍔ㄩ泦鎴愬鉤鍙幫紝鍙畨鎺掕礋杞藉茍浠庝換浣曠粨鏋勫寲銆佸崐緇撴瀯鍖栧拰闈炵粨鏋勫寲鐨勫ぇ鏁版嵁闆嗐侱atameer鐨勫ぇ鏁版嵁鍒嗘瀽瑙e喅鏂規閫氳繃琛ㄦ牸鎺ュ彛鏁村悎Hadoop鐨勬暟鎹寲鎺樿兘鍔涖傚茍閫氳繃REST API鍦ㄧ鏈変簯鍜屽叕鍏變簯涓緭鍏ュ拰杈撳嚭鏁版嵁銆?/p>
EMC鐨勭粺涓鏁版嵁鍒嗘瀽騫沖彴
EMC鎺ㄥ嚭鐢ㄤ簬鏀寔澶ф暟鎹垎鏋愮殑騫沖彴――EMC Greenplum緇熶竴鍒嗘瀽騫沖彴(UAP)銆侴reenplum UAP鏄竴涓敮涓鐨勭粺涓鏁版嵁鍒嗘瀽騫沖彴錛屽彲鎵╁睍鑷沖叾浠栧伐鍏鳳紝鍏剁嫭鐗逛箣澶勫湪浜庯紝瀹冨皢瀵瑰ぇ鏁版嵁鐨勮鐭ュ拰鍒嗕韓璐┛鏁翠釜鍒嗘瀽榪囩▼錛屽疄鐜版瘮浠ュ線鏇撮珮鐨勫晢涓氫環鍊箋俇AP鍖呮嫭EMC Greenplum 鍏崇郴鏁版嵁搴撱丒MC Greenplum HD Hadoop浠ュ強EMC Greenplum Chorus銆俇AP灝卞ソ姣斾竴涓暟鎹垎鏋愬洟闃燂紝鍖呮嫭浜嗕粠鏁版嵁縐戝瀹跺拰BI鍒嗘瀽甯堝埌DBA鍜屽湪綰垮晢涓氱敤鎴峰拰綆$悊鑰呫侲MC閽堝紜歡璁懼DCA錛圖ata Computing Appliance錛夛紝鍏惰凍浠ヨ繍琛孍MC Greenplum 鍏崇郴鏁版嵁搴撳拰EMC Greenplum HD鑺傜偣銆侱CA鎻愪緵鎺у埗綆$悊鐣岄潰錛屾柟渚跨鐞嗕漢鍛樼洃瑙嗐佺鐞咷reenplum鏁版嵁搴撳拰Hadoop緋葷粺鎬ц兘銆?/p>
Hadapt涓嶩adoop鐜鏃犵紳闆嗘垚
Hive浣滀負榪愯鍦℉adoop涓婄殑鏁版嵁浠撳簱緇勪歡騫朵笉鍍廐adoop閭f牱鍙椾漢鍏蟲敞銆傝孒adapt鍒欐彁渚涢泦浼楀鍔熻兘浜庝竴韜殑鏁版嵁鍒嗘瀽鐜錛屾棬鍦ㄥ鐞嗗瓨鍦ㄤ簬Hadoop鍜孲QL鐜涓紶緇熺粨鏋勫寲鐨勬暟鎹侶adapt騫沖彴鍙繍琛屽湪縐佹湁浜戝拰鍏叡浜戜箣涓婏紝騫舵彁渚涗粠涓涓幆澧冭闂暟鎹殑鑳藉姏銆傚寘鎷幇鏈夊熀浜嶴QL鐨勫伐鍏蜂互鍙奙apReduce澶勭悊鍜屽ぇ鏁版嵁鍒嗘瀽銆侶adapt鑷姩鍒嗗紑鎵цHadoop鍜屽叧緋繪暟鎹簱涔嬮棿鐨勬煡璇紝澶勫垎鍒╃敤浜咹adoop鐨勯珮鎵╁睍鎬у拰鍏崇郴鏁版嵁搴撶殑楂橀熸с?/p>
Hortonworks緇ф壙Yahoo錛?Hadoop琛i挼
Yahoo!鍦ㄥ幓騫村墺紱諱簡Hadoop涓氬姟錛屽茍涓庣璋烽鎶曞叕鍙窧enchmark Capital鍚堣祫緇勫緩涓瀹跺悕涓篐ortonworks鐨勫叕鍙搞傛柊鍏徃鍖呭惈鍦╕ahoo錛佽礎鐚渶澶х殑50鍚嶅伐紼嬪笀錛屾棬鍦ㄧ戶緇帹鍔℉adoop鐨勫彂灞曘侶ortonworks楂樼鏂█榪欐敮浠ahoo!寮鍙戝洟闃熶負鐝簳鐨勫叕鍙稿皢浼氳礎鐚洿澶氱殑Hadoop浠g爜錛屽茍鎸囧紩Hadoop騫沖彴鏈潵鐨勫彂灞曘侶ortonworks宸插湪鍘誨勾10鏈堜笌寰蔣鎴愪負鍚堜綔浼欎即鍏崇郴銆侶ortonworks鍙府鍔㎝icrosoft鎺ㄥ嚭Windows騫沖彴涔嬩笂鐨凥adoop銆侶ortonworks鍦ㄥ幓騫?1鏈堜篃鎺ㄥ嚭浜嗚嚜鐢辯殑HDP錛圚ortonworks Data Platform錛塚1錛岃岀粨鍚堜簡鏈鏂?.23鐗圚adoop鐨凥DP V2灝嗗湪2012騫寸涓瀛e害鎺ㄥ嚭銆侶ortonworks榪樻彁渚汬adoop鐨勫煿璁笌鏀寔錛屽姞寮哄湪榪欐柟闈笌Cloudera鍜孧apR鐨勭珵浜夈?/p>
IBM鐨凥adoop涔嬭礬
IBM鍦ㄥ騫翠互鍓嶅氨寮濮嬬爺絀禜adoop銆傜幇浠奍BM鎻愪緵鍩轟簬浜戞湇鍔$殑嫻烽噺鏁版嵁鍒嗘瀽鏂歸潰澶氱鏂規鐨勯夋嫨錛屼絾鐩墠IBM鐨勭瓥鐣ヤ技涔庝富瑕佹槸鍥寸粫Hadoop鍦ㄥ彂灞曘侷BM鍦?鏈堟帹鍑轟簡鍏禨martCloud浜戣綆楀鉤鍙般傚茍鎵胯鏀瑰杽Hadoop宸ヤ綔璐熻澆銆侷BM鎻愪緵浜嗗熀浜嶩adoop鐨処nfoSphere BigInsights錛圛BM InfoSphere BigInsights鏄敤浜庡垎鏋愬拰铏氭嫙鍖栨搗閲忔暟鎹殑杞歡鍜屾湇鍔★紝榪欐鏂頒駭鍝佺敱 ApacheHadoop 鎻愪緵鎶鏈敮鎸併傦級鍩烘湰鐗堝拰浼佷笟鐗堛?InfoSphere BigInsights涔嬪墠浣滀負IBM嫻嬭瘯鍜屽紑鍙戠殑浜戜駭鍝侊紝鐜板湪琚玈martCloud鍙栦唬銆?/p>
Informatica 鍚戜簯鏇磋繘涓姝?/strong>
澶у鏁扮殑鏁版嵁綆$悊杞歡渚涘簲鍟嗭紙濡侷BM銆丱racle銆丼yncsort銆乀alend錛夐兘娑夊強鍒癏adoop銆侷nformatica鍦ㄥ幓騫?0鏈堜篃鎺ㄥ嚭浜咹adoop鐜涓嬬殑鏁版嵁緙栬瘧杞崲瑙e喅鏂規――HParser銆?/p>
璇ユ柟妗堝彲浠ヨ繍琛屽湪鍑犱箮鎵鏈夌殑Apache Hadoop鍒嗗竷寮忕幆澧冧腑錛屼笌MapReduce鏋舵瀯騫寵錛岃兘楂樻晥鐜囧湴鎶婃棤緇撴瀯鐨勫鏉傛暟鎹?#8213;―璇稿緗戠粶璁板綍銆佺ぞ浜ゅ獟浣撴暟鎹侀氳瘽璇︾粏璁板綍浠ュ強鍏朵粬鏁版嵁鏍煎紡――杞崲涓篐adoop涓粨鏋勬垨鍗婄粨鏋勬牸寮忋傚綋鎶婃暟鎹漿鍖栦負鏇村叿緇撴瀯鎬х殑鏍煎紡鍚庯紝渚垮彲浠ュ緱鍒版洿蹇熺殑浣跨敤鍜岀敓鏁堬紝浠庤岄┍鍔ㄤ笟鍔″彂灞曘佹彁楂樿繍钀ユ晥鐜囥?/p>
Karmasphere Hadoop鏁版嵁鍒嗘瀽鍒╁櫒
Karmasphere鎻愪緵浜嗙洿鎺ヨ闂瓾adoop涓粨鏋勫寲鍜岄潪緇撴瀯鍖栨暟鎹互鍙婅繘涓姝ュ垎鏋愭煡璇㈢殑鐗規э紝鍚屾椂Karmasphere榪樻彁渚涚殑鍙鍖栧伐浣滅┖闂淬侹armasphere鎻愪緵鐨勫彲瑙嗗寲宸ュ叿鎻愪緵浜哠QL鎴栧叾浠栫壒瀹氭煡璇㈣璦鍒嗘瀽浣嶄簬Amazon S3銆佸伐浣滄祦浠ュ強鏈湴鏂囦歡緋葷粺涓婄殑緇撴瀯鍖栧拰闈炵粨鏋勬暟鎹殑鐗規с備紒涓氳繕鍙互浣跨敤鏁版嵁搴撴垨鐩稿叧宸ュ叿錛堜緥濡侲xcel錛夋潵鎻愬彇鍒嗘瀽寰楀嚭鐨勬暟鎹?/p>
MapR甯︽潵鏇撮珮鎬ц兘鐨凥adoop
MapR鍦℉adoop鐨勮垶鍙頒笂鏄懼緱鏍煎鑰鐪鹼紝鍏舵彁渚汬adoop闈炲父鐙壒銆侻apR鍩轟簬寮婧怘adoop錛屽湪鍙渶鏈夐檺紜歡鐨勭幆澧冧腑鎻愪緵鏇村揩鐨凥adoop銆傚悓鏃禡apr閰嶅浜嗗揩鐓э紝騫跺彿縐頒笉浼氬嚭鐜癝POF鍗曡妭鐐規晠闅滐紝涓旇璁や負鏄笌鐜版湁HDFS鐨凙PI鍏煎銆傚洜姝ら潪甯稿鏄撴浛鎹㈠師鏈夌殑緋葷粺銆侻apR鏈鏂扮殑0.23鐗堣В鍐寵澶氬紑婧怘adoop鐨勭己闄楓傝孧apR涓嶦MC鐨勫悎浣滀綋鐜板湪浜咵MC Greenplum HD Enterprise Edition涓婏紝鍏跺氨鏄熀浜嶮apR M5鏋勫緩鐨勩?/p>
Microsoft鍏ㄩ潰鎷ユ姳Hadoop
褰揈MC銆両BM銆丱racle閮藉湪2011騫存秹鍙奌adoop鏃訛紝Microsoft鍏ㄩ潰鎷ユ姳Hadoop鐨勪婦鍔ㄥ氨鏄懼緱涓嶈凍涓哄浜嗐傝孒adoop鐨刉indows Server灝嗗湪鍦?012騫存帹鍑猴紝灞婃椂鍏惰繕浼氫笌寰蔣鐜版湁鐨凚I宸ュ叿鑱斿悎澶勭悊浠誨姟銆傚幓騫村井杞〃紺烘帹鍑篧indows Azure涓婄殑Hadoop棰勮鐗堬紝寰蔣榪樹嬌Hadoop鐨勬暟鎹氳繃閮ㄧ講鍦ㄥ熀浜庝簯鐨刉indows Azure鑾峰彇銆傚茍浣垮叾鑳藉涓庝紒涓氱殑鍟嗕笟鏅鴻兘宸ュ叿涓璧峰垎鏋愭暟鎹傚井杞洰鍓嶆涓嶩ortonworks鍚堜綔鏃ㄥ湪鍔姏綆鍖栦笅杞姐佸畨瑁呭拰閰嶇疆絳夊嚑涓狧adoop鐨勭浉鍏蟲妧鏈傚寘鎷琀DFS銆丠ive銆丳ig銆傝繖灝嗘湁鍒╀簬浼佷笟閫氳繃Hadoop鎷撳鑷韓鐨勪笟鍔°傚井杞皢緙栧啓鏂扮殑ODBC椹卞姩紼嬪簭騫舵墿灞曡嚜宸辯幇鏈夌殑鏌ヨ緋葷粺鍒癏ive銆傝繖鏍蜂竴鏉ョ敤鎴峰皢鑳藉鐩存帴浠嶦xcel銆丳owerView鎵цHadoop鏌ヨ銆?/p>
Oracle榪涘啗浜戣綆?/strong>
Oracle鍦?011 Oracle鍏ㄧ悆澶т細涓婂甯冩帹鍑轟簡Oracle Big Data Appliance銆侭ig Data Appliance鏄竴涓泦鎴愪簡Hadoop銆丯oSQL Database銆丱racle鏁版嵁搴揌adoop閫傞厤鍣ㄣ丱racle鏁版嵁搴揌adoop瑁呰澆鍣ㄥ強R璇█鐨勭郴緇熴侽racle榪樺湪浠婂勾1鏈堜笌Cloudera鎴愪負鍚堜綔浼欎即鍏崇郴銆侽racle鐜板凡灝咰loudera Distribution Including Apache Hadoop錛圕DH錛夊拰Cloudera Manager闆嗘垚鍒癘racle澶ф暟鎹満涔嬩腑銆侽racle涔熷皢鍒╃敤Cloudera鍦℉adoop棰嗗煙鐨勪笓涓氱煡璇嗘彁渚涘煿璁強鍜ㄨ涓氬姟銆侽racle澶ф暟鎹満涓繍琛屼簡Oracle Linux鎿嶄綔緋葷粺錛?涓満鏋朵腑鍖呭惈18涓狾racle-Sun鏈嶅姟鍣紝鍏辮216涓牳蹇冿紝鍚屾椂鍏峰864GB鐨勫唴瀛樺拰648TB鐨勫瓨鍌ㄨ兘鍔涳紝鍏跺敭浠蜂負45涓囩編鍏冦傦紙鏉庢櫤/緙栬瘧錛?/p>
濡備粖Apache Hadoop宸叉垚涓哄ぇ鏁版嵁琛屼笟鍙戝睍鑳屽悗鐨勯┍鍔ㄥ姏銆侶ive鍜孭ig絳夋妧鏈篃緇忓父琚彁鍒幫紝浣嗘槸浠栦滑閮芥湁浠涔堝姛鑳斤紝涓轟粈涔堜細闇瑕佸鎬殑鍚嶅瓧錛堝Oozie錛孼ooKeeper銆丗lume錛夈?/p>
Hadoop甯︽潵浜嗗粔浠風殑澶勭悊澶ф暟鎹紙澶ф暟鎹殑鏁版嵁瀹歸噺閫氬父鏄?0-100GB鎴栨洿澶氾紝鍚屾椂鏁版嵁縐嶇被澶氱澶氭牱錛屽寘鎷粨鏋勫寲銆侀潪緇撴瀯鍖栫瓑錛夌殑鑳藉姏銆備絾榪欎笌涔嬪墠鏈変粈涔堜笉鍚岋紵
鐜頒粖浼佷笟鏁版嵁浠撳簱鍜屽叧緋誨瀷鏁版嵁搴撴搮闀垮鐞嗙粨鏋勫寲鏁版嵁錛屽茍涓斿彲浠ュ瓨鍌ㄥぇ閲忕殑鏁版嵁銆備絾鎴愭湰涓婃湁浜涙槀璐點傝繖縐嶅鏁版嵁鐨勮姹傞檺鍒朵簡鍙鐞嗙殑鏁版嵁縐嶇被錛屽悓鏃惰繖縐嶆儻鎬ф墍甯︾殑緙虹偣榪樺獎鍝嶅埌鏁版嵁浠撳簱鍦ㄩ潰瀵規搗閲忓紓鏋勬暟鎹椂瀵逛簬鏁忔嵎鐨勬帰绱€傝繖閫氬父鎰忓懗鐫鏈変環鍊肩殑鏁版嵁婧愬湪緇勭粐鍐呬粠鏈鎸栨帢銆傝繖灝辨槸Hadoop涓庝紶緇熸暟鎹鐞嗘柟寮忔渶澶х殑涓嶅悓銆?/p>
鏈枃灝遍噸鐐規帰璁ㄤ簡Hadoop緋葷粺鐨勭粍鎴愰儴鍒嗭紝騫惰В閲婂悇涓粍鎴愰儴鍒嗙殑鍔熻兘銆?/p>
MapReduce——Hadoop鐨勬牳蹇?/strong>
Google鐨勭綉緇滄悳绱㈠紩鎿庡湪寰楃泭浜庣畻娉曞彂鎸ヤ綔鐢ㄧ殑鍚屾椂錛孧apReduce鍦ㄥ悗鍙板彂鎸ヤ簡鏋佸ぇ鐨勪綔鐢ㄣ侻apReduce妗嗘灦鎴愪負褰撲粖澶ф暟鎹鐞嗚儗鍚庣殑鏈鍏峰獎鍝嶅姏鐨?#8220;鍙戝姩鏈?#8221;銆傞櫎浜咹adoop錛屼綘榪樹細鍦∕apReduce涓婂彂鐜癕PP錛圫ybase IQ鎺ㄥ嚭浜嗗垪紺烘暟鎹簱錛夊拰NoSQL錛堝Vertica鍜孧ongoDB錛夈?/p>
MapReduce鐨勯噸瑕佸垱鏂版槸褰撳鐞嗕竴涓ぇ鏁版嵁闆嗘煡璇㈡椂浼氬皢鍏朵換鍔″垎瑙e茍鍦ㄨ繍琛岀殑澶氫釜鑺傜偣涓鐞嗐傚綋鏁版嵁閲忓緢澶ф椂灝辨棤娉曞湪涓鍙版湇鍔″櫒涓婅В鍐抽棶棰橈紝姝ゆ椂鍒嗗竷寮忚綆椾紭鍔垮氨浣撶幇鍑烘潵銆傚皢榪欑鎶鏈笌Linux鏈嶅姟鍣ㄧ粨鍚堝彲鑾峰緱鎬т環姣旀瀬楂樼殑鏇夸唬澶ц妯¤綆楅樀鍒楃殑鏂規硶銆俌ahoo鍦?006騫寸湅鍒頒簡Hadoop鏈潵鐨勬綔鍔涳紝騫墮個璇稨adoop鍒涘浜篋oug Cutting鐫鎵嬪彂灞旽adoop鎶鏈紝鍦?008騫碒adoop宸茬粡褰㈡垚涓瀹氱殑瑙勬ā銆侶adoop欏圭洰鍐嶄粠鍒濇湡鍙戝睍鐨勬垚鐔熺殑榪囩▼涓悓鏃跺惛綰充簡涓浜涘叾浠栫殑緇勪歡錛屼互渚胯繘涓姝ユ彁楂樿嚜韜殑鏄撶敤鎬у拰鍔熻兘銆?/p>
HDFS鍜孧apReduce
浠ヤ笂鎴戜滑璁ㄨ浜哅apReduce灝嗕換鍔″垎鍙戝埌澶氫釜鏈嶅姟鍣ㄤ笂澶勭悊澶ф暟鎹殑鑳藉姏銆傝屽浜庡垎甯冨紡璁$畻錛屾瘡涓湇鍔″櫒蹇呴』鍏峰瀵規暟鎹殑璁塊棶鑳藉姏錛岃繖灝辨槸HDFS錛圚adoop Distributed File System錛夋墍璧峰埌鐨勪綔鐢ㄣ?/p>
HDFS涓嶮apReduce鐨勭粨鍚堟槸寮哄ぇ鐨勩傚湪澶勭悊澶ф暟鎹殑榪囩▼涓紝褰揌adoop闆嗙兢涓殑鏈嶅姟鍣ㄥ嚭鐜伴敊璇椂錛屾暣涓綆楄繃紼嬪茍涓嶄細緇堟銆傚悓鏃禜FDS鍙繚闅滃湪鏁翠釜闆嗙兢涓彂鐢熸晠闅滈敊璇椂鐨勬暟鎹啑浣欍傚綋璁$畻瀹屾垚鏃跺皢緇撴灉鍐欏叆HFDS鐨勪竴涓妭鐐逛箣涓侶DFS瀵瑰瓨鍌ㄧ殑鏁版嵁鏍煎紡騫舵棤鑻涘埢鐨勮姹傦紝鏁版嵁鍙互鏄潪緇撴瀯鍖栨垨鍏跺畠綾誨埆銆傜浉鍙嶅叧緋繪暟鎹簱鍦ㄥ瓨鍌ㄦ暟鎹箣鍓嶉渶瑕佸皢鏁版嵁緇撴瀯鍖栧茍瀹氫箟鏋舵瀯銆?/p>
寮鍙戜漢鍛樼紪鍐欎唬鐮佽矗浠繪槸浣挎暟鎹湁鎰忎箟銆侶adoop MapReduce綰х殑緙栫▼鍒╃敤Java APIs錛屽茍鍙墜鍔ㄥ姞杞芥暟鎹枃浠跺埌HDFS涔嬩腑銆?/p>
Pig鍜孒ive
瀵逛簬寮鍙戜漢鍛橈紝鐩存帴浣跨敤Java APIs鍙兘鏄箯鍛蟲垨瀹規槗鍑洪敊鐨勶紝鍚屾椂涔熼檺鍒朵簡Java紼嬪簭鍛樺湪Hadoop涓婄紪紼嬬殑榪愮敤鐏墊椿鎬с備簬鏄疕adoop鎻愪緵浜嗕袱涓В鍐蟲柟妗堬紝浣垮緱Hadoop緙栫▼鍙樺緱鏇村姞瀹規槗銆?/p>
•Pig鏄竴縐嶇紪紼嬭璦錛屽畠綆鍖栦簡Hadoop甯歌鐨勫伐浣滀換鍔°侾ig鍙姞杞芥暟鎹佽〃杈捐漿鎹㈡暟鎹互鍙婂瓨鍌ㄦ渶緇堢粨鏋溿侾ig鍐呯疆鐨勬搷浣滀嬌寰楀崐緇撴瀯鍖栨暟鎹彉寰楁湁鎰忎箟錛堝鏃ュ織鏂囦歡錛夈傚悓鏃禤ig鍙墿灞曚嬌鐢↗ava涓坊鍔犵殑鑷畾涔夋暟鎹被鍨嬪茍鏀寔鏁版嵁杞崲銆?/p>
•Hive鍦℉adoop涓壆婕旀暟鎹粨搴撶殑瑙掕壊銆侶ive娣誨姞鏁版嵁鐨勭粨鏋勫湪HDFS錛坔ive superimposes structure on data in HDFS錛夛紝騫跺厑璁鎬嬌鐢ㄧ被浼間簬SQL璇硶榪涜鏁版嵁鏌ヨ銆備笌Pig涓鏍鳳紝Hive鐨勬牳蹇冨姛鑳芥槸鍙墿灞曠殑銆?/p>
Pig鍜孒ive鎬繪槸浠や漢鍥版儜鐨勩侶ive鏇撮傚悎浜庢暟鎹粨搴撶殑浠誨姟錛孒ive涓昏鐢ㄤ簬闈欐佺殑緇撴瀯浠ュ強闇瑕佺粡甯稿垎鏋愮殑宸ヤ綔銆侶ive涓嶴QL鐩鎬技淇冧嬌鍏舵垚涓篐adoop涓庡叾浠朆I宸ュ叿緇撳悎鐨勭悊鎯充氦闆嗐侾ig璧嬩簣寮鍙戜漢鍛樺湪澶ф暟鎹泦棰嗗煙鏇村鐨勭伒媧繪э紝騫跺厑璁稿紑鍙戠畝媧佺殑鑴氭湰鐢ㄤ簬杞崲鏁版嵁嫻佷互渚垮祵鍏ュ埌杈冨ぇ鐨勫簲鐢ㄧ▼搴忋侾ig鐩告瘮Hive鐩稿杞婚噺錛屽畠涓昏鐨勪紭鍔挎槸鐩告瘮浜庣洿鎺ヤ嬌鐢℉adoop Java APIs鍙ぇ騫呭墛鍑忎唬鐮侀噺銆傛鍥犱負濡傛錛孭ig浠嶇劧鏄惛寮曞ぇ閲忕殑杞歡寮鍙戜漢鍛樸?/p>
鏀瑰杽鏁版嵁璁塊棶錛欻Base銆丼qoop浠ュ強Flume
Hadoop鏍稿績榪樻槸涓濂楁壒澶勭悊緋葷粺錛屾暟鎹姞杞借繘HDFS銆佸鐞嗙劧鍚庢绱€傚浜庤綆楄繖鎴栧鎴栧皯鏈変簺鍊掗錛屼絾閫氬父浜掑姩鍜岄殢鏈哄瓨鍙栨暟鎹槸鏈夊繀瑕佺殑銆侶Base浣滀負闈㈠悜鍒楃殑鏁版嵁搴撹繍琛屽湪HDFS涔嬩笂銆侶Base浠oogle BigTable涓鴻摑鏈傞」鐩殑鐩爣灝辨槸蹇熷湪涓繪満鍐呮暟鍗佷嚎琛屾暟鎹腑瀹氫綅鎵闇鐨勬暟鎹茍璁塊棶瀹冦侶Base鍒╃敤MapReduce鏉ュ鐞嗗唴閮ㄧ殑嫻烽噺鏁版嵁銆傚悓鏃禜ive鍜孭ig閮藉彲浠ヤ笌HBase緇勫悎浣跨敤錛孒ive鍜孭ig榪樹負HBase鎻愪緵浜嗛珮灞傝璦鏀寔錛屼嬌寰楀湪HBase涓婅繘琛屾暟鎹粺璁″鐞嗗彉鐨勯潪甯哥畝鍗曘?/p>
浣嗕負浜嗘巿鏉冮殢鏈哄瓨鍌ㄦ暟鎹紝HBase涔熷仛鍑轟簡涓浜涢檺鍒訛細渚嬪Hive涓嶩Base鐨勬ц兘姣斿師鐢熷湪HDFS涔嬩笂鐨凥ive瑕佹參4-5鍊嶃傚悓鏃禜Base澶х害鍙瓨鍌≒B綰х殑鏁版嵁錛屼笌涔嬬浉姣擧DFS鐨勫閲忛檺鍒惰揪鍒?0PB銆侶Base涓嶉傚悎鐢ㄤ簬ad-hoc鍒嗘瀽錛孒Base鏇撮傚悎鏁村悎澶ф暟鎹綔涓哄ぇ鍨嬪簲鐢ㄧ殑涓閮ㄥ垎錛屽寘鎷棩蹇椼佽綆椾互鍙婃椂闂村簭鍒楁暟鎹?/p>
鑾峰彇鏁版嵁涓庤緭鍑烘暟鎹?/strong>
Sqoop鍜孎lume鍙敼榪涙暟鎹殑浜掓搷浣滄у拰鍏朵綑閮ㄥ垎銆係qoop鍔熻兘涓昏鏄粠鍏崇郴鏁版嵁搴撳鍏ユ暟鎹埌Hadoop錛屽茍鍙洿鎺ュ鍏ュ埌HFDS鎴朒ive銆傝孎lume璁捐鏃ㄥ湪鐩存帴灝嗘祦鏁版嵁鎴栨棩蹇楁暟鎹鍏DFS銆?/p>
Hive鍏峰鐨勫弸濂絊QL鏌ヨ鏄笌綣佸鏁版嵁搴撶殑鐞嗘兂緇撳悎鐐癸紝鏁版嵁搴撳伐鍏烽氳繃JDBC鎴朞DBC鏁版嵁搴撻┍鍔ㄧ▼搴忚繛鎺ャ?/p>
璐熻矗鍗忚皟宸ヤ綔嫻佺▼鐨刏ooKeeper鍜孫ozie
闅忕潃瓚婃潵瓚婂鐨勯」鐩姞鍏adoop澶у搴茍鎴愪負闆嗙兢緋葷粺榪愪綔鐨勪竴閮ㄥ垎錛屽ぇ鏁版嵁澶勭悊緋葷粺闇瑕佽礋璐e崗璋冨伐浣滅殑鐨勬垚鍛樸傞殢鐫璁$畻鑺傜偣鐨勫澶氾紝闆嗙兢鎴愬憳闇瑕佸郊姝ゅ悓姝ュ茍浜嗚В鍘誨摢閲岃闂湇鍔″拰濡備綍閰嶇疆錛孼ooKeeper姝f槸涓烘鑰岀敓鐨勩?/p>
鑰屽湪Hadoop鎵ц鐨勪換鍔℃湁鏃跺欓渶瑕佸皢澶氫釜Map/Reduce浣滀笟榪炴帴鍒頒竴璧鳳紝瀹冧滑涔嬮棿鎴栬鎵規渚濊禆銆侽ozie緇勪歡鎻愪緵綆$悊宸ヤ綔嫻佺▼鍜屼緷璧栫殑鍔熻兘錛屽茍鏃犻渶寮鍙戜漢鍛樼紪鍐欏畾鍒剁殑瑙e喅鏂規銆?/p>
Ambari鏄渶鏂板姞鍏adoop鐨勯」鐩紝Ambari欏圭洰鏃ㄥ湪灝嗙洃鎺у拰綆$悊絳夋牳蹇冨姛鑳藉姞鍏adoop欏圭洰銆侫mbari鍙府鍔╃郴緇熺鐞嗗憳閮ㄧ講鍜岄厤緗瓾adoop錛屽崌綰ч泦緹や互鍙婄洃鎺ф湇鍔°傝繕鍙氳繃API闆嗘垚涓庡叾浠栫殑緋葷粺綆$悊宸ュ叿銆?/p>
Apache Whirr鏄竴濂楄繍琛屼簬浜戞湇鍔$殑綾誨簱錛堝寘鎷琀adoop錛夛紝鍙彁渚涢珮搴︾殑浜掕ˉ鎬с俉hirr鐜頒粖鐩稿涓珛錛屽綋鍓嶆敮鎸丄mazon EC2鍜孯ackspace鏈嶅姟銆?/p>
鏈哄櫒瀛︿範錛歁ahout
鍚勭被緇勭粐闇姹傜殑涓嶅悓瀵艱嚧鐩稿叧鐨勬暟鎹艦褰㈣壊鑹詫紝瀵硅繖浜涙暟鎹殑鍒嗘瀽涔熼渶瑕佸鏍峰寲鐨勬柟娉曘侻ahout鎻愪緵涓浜涘彲鎵╁睍鐨勬満鍣ㄥ涔犻鍩熺粡鍏哥畻娉曠殑瀹炵幇錛屾棬鍦ㄥ府鍔╁紑鍙戜漢鍛樻洿鍔犳柟渚垮揩鎹峰湴鍒涘緩鏅鴻兘搴旂敤紼嬪簭銆侻ahout鍖呭惈璁稿瀹炵幇錛屽寘鎷泦緹ゃ佸垎綾匯佹帹鑽愯繃婊ゃ侀綣佸瓙欏規寲鎺樸?/p>
浣跨敤Hadoop
閫氬父鎯呭喌涓嬶紝Hadoop搴旂敤浜庡垎甯冨紡鐜銆傚氨鍍忎箣鍓峀inux鐨勭姸鍐典竴鏍鳳紝鍘傚晢闆嗘垚鍜屾祴璇旳pache Hadoop鐢熸佺郴緇熺殑緇勪歡錛屽茍娣誨姞鑷繁鐨勫伐鍏峰拰綆$悊鍔熻兘銆傦紙鏉庢櫤/緙栬瘧錛?/p>