锘??xml version="1.0" encoding="utf-8" standalone="yes"?>
閫夋嫨棰橈細
1.涓や釜闀垮害鍚勪負N鐨勬湁搴忔暟緇勮繘琛屽悎騫訛紝姹傚彲鑳界殑鏈澶氱殑姣旇緝嬈℃暟錛岋紙2n-1錛?br>2.涓や釜闀垮害涓篘鐨勬湁搴忔暟緇勶紝瑕佹眰鍦ㄨ繖涓や釜鏁扮粍涓帓絎琋鐨勫厓绱狅紝鏈灝忕殑鏃墮棿澶嶆潅搴︼紵 錛?O(logn)錛岀被浼間簩鍒嗘悳绱級
3.閫嗘嘗鍏拌〃杈懼紡姹傚鹼紝錛堢珶鐒剁敾浜嗗緢涔呯殑鍚庣紑琛ㄨ揪寮忔病鐢誨嚭鏉ワ紝鐪熸澂鍏楓傘傜洿鎺ユ眰鍊煎氨琛屼簡錛?br>4.涓涓叧浜庝簩鍙夋爲鐨勯棶棰橈紝澶ф剰鏄鍦ㄤ簩鍙夋爲鏌ユ壘鏌愪釜鍏冪礌錛屾眰閫夐」緇欏嚭鐨勬煡鎵懼簭鍒楀摢涓笉鍙兘鍑虹幇錛燂紙鑰冨療浜屽弶鏍戠殑鎬ц川錛?br>5.excell鐨勫垪琛ㄧず濡侫B...Z, AA AB ....ZZ, AAA AAB .... ZZZ, 姹侱EF鐨勫崄榪涘埗鍊鹼紙姹?6榪涘埗鐨勫鹼紝鐩存帴璁$畻錛?br>6.鍑芥暟鎸囬拡鏁扮粍鐨勫啓娉曢棶棰樸傘?br>7.铏氬嚱鏁伴棶棰橈紝澶ф剰鏄熀綾誨畾涔変簡涓涓繚鎶ゆ垚鍛橈紝鏋勯犲嚱鏁板垵濮嬪寲涓?錛岃繕瀹氫箟浜嗕竴涓櫄鍑芥暟錛屽熀綾繪槸灝嗘垚鍛?-錛岃屽瓙綾誨彧閲嶅畾涔変簡铏氬嚱鏁幫紝灝嗘垚鍛?+錛屼富鍑芥暟閲岋紝new浜嗕竴涓瓙綾誨璞★紝鐒跺悗瀹氫箟涓涓熀綾繪寚閽堟寚鍚戞瀵硅薄錛屽張瀹氫箟浜嗕竴涓熀綾誨紩鐢ㄦ寚鍚戞鍩虹被鎸囬拡鎸囧悜鐨勫璞★紝鐒跺悗鍒嗗埆璋冪敤浜嗚櫄鍑芥暟錛岃姹傚熀綾誨畾涔夌殑鎴愬憳鐨勫箋?br>8.緇欏嚭涓孌電▼搴忥紝瑕佹眰杈撳嚭鍊鹼紝鐩存帴璁$畻銆傜▼搴忛噷璁$畻瀛楃鏁扮粍 char a[]={'a','b','c'}鐨勯暱搴﹂噰鐢╯izeof(a)/sizeof(a[0])鐨勬柟娉曘?br>9.鎸囧嚭緇欏嚭閫夐」涓笉鍙兘瀛樺偍鍦ㄦ爤涓殑鏄傘傘傦紙鍏ㄥ眬闈欐佸彉閲忥紝鏀懼湪闈欐佸尯涓級
10.緇欏嚭char *p="hello world", char a[]="byebye",strncpy(p,a,6),闂繖涓▼搴忚繍琛屽悗p鐨勭粨鏋滄槸浠涔堬紵(榪欓噷*p鏄竴涓瓧絎︿覆甯擱噺錛屼笉鑳藉瀹冪殑鍏冪礌榪涜淇敼錛屾墍浠ョ▼搴忓湪榪愯鏃朵細鍑洪敊)
涓昏棰樼紪紼嬮錛?br>澶ф剰鏄粰鍑轟竴涓暟緇勶紝榪欎釜鏁扮粍姣忎釜鍏冪礌閮戒笉鍚岋紝騫朵笖鍙兘鏄崌搴忕殑錛屾垨鑰呮槸鍗囧簭+鏃嬭漿鍚庣殑緇撴灉錛屼緥濡?,2,3,4,5,鎴栬?4,5,1,2,3 鎴栬?3,4,5,1,2絳夌瓑錛?br>鐒跺悗緇欎竴涓暟錛岃鎵懼嚭榪欎釜鏁板湪鎵緇欐暟緇勪腑鐨勭儲寮曞兼垨鑰呰繑鍥?1錛岃姹傚鏉傚害蹇呴』灝忎簬o(n)銆?br>鐩稿姣旇緝綆鍗曞惂錛岄鍏堟槸鍒ゆ柇鏄惁鏄粠宸﹀埌鍙蟲湁鍗囧簭鐨勶紝鑻ユ槸錛屽垯鐢ㄤ簩鍒嗘煡鎵撅紝澶嶆潅搴︿負o(logn),濡傛灉涓嶆槸錛屽垯鏍規嵁瑕佹壘鐨勫間笌絎竴涓兼瘮杈冪殑緇撴灉錛屽湪宸﹀崐閮ㄥ垎鎴栧彸鍗婇儴鍒嗘煡鎵捐繖涓暟錛屾槗鐭ワ紝鏌ユ壘嬈℃暟鑲畾灝忎簬n錛屽洜鑰屽鏉傚害絎﹀悎瑕佹眰銆?br>絎簩涓皬棰樻槸瑕佺粰鍑轟竴浜涙祴璇曟暟鎹茍鍔犱互璇存槑銆?br>姝e紡鎵懼伐鐨勭涓鍦洪潰璇曪紝涓嶆槸寰堥『鍒╋紝鐗規璁板綍錛屾敀涓婻P, ^.^
鏂規1錛氬彲浠ヤ及璁℃瘡涓枃浠跺畨鐨勫ぇ灝忎負50G×64=320G錛岃繙榪滃ぇ浜庡唴瀛橀檺鍒剁殑4G銆傛墍浠ヤ笉鍙兘灝嗗叾瀹屽叏鍔犺澆鍒板唴瀛樹腑澶勭悊銆傝冭檻閲囧彇鍒嗚屾不涔嬬殑鏂規硶銆?/p>
s 閬嶅巻鏂囦歡a錛屽姣忎釜url姹傚彇錛岀劧鍚庢牴鎹墍鍙栧緱鐨勫煎皢url鍒嗗埆瀛樺偍鍒?a name=baidusnap0>1000涓皬鏂囦歡錛堣涓?a >
錛変腑銆傝繖鏍鋒瘡涓皬鏂囦歡鐨勫ぇ綰︿負300M銆?/p>
s 閬嶅巻鏂囦歡b錛岄噰鍙栧拰a鐩稿悓鐨勬柟寮忓皢url鍒嗗埆瀛樺偍鍒?strong style="BACKGROUND-COLOR: #ffff66; COLOR: black">100
0鍚勫皬鏂囦歡錛堣涓?a >s 姹傛瘡瀵瑰皬鏂囦歡涓浉鍚岀殑url鏃訛紝鍙互鎶婂叾涓竴涓皬鏂囦歡鐨剈rl瀛樺偍鍒癶ash_set涓傜劧鍚庨亶鍘嗗彟涓涓皬鏂囦歡鐨勬瘡涓猽rl錛岀湅鍏舵槸鍚﹀湪鍒氭墠鏋勫緩鐨刪ash_set涓紝濡傛灉鏄紝閭d箞灝辨槸鍏卞悓鐨剈rl錛屽瓨鍒版枃浠墮噷闈㈠氨鍙互浜嗐?/p>
鏂規2錛氬鏋滃厑璁告湁涓瀹氱殑閿欒鐜囷紝鍙互浣跨敤Bloom filter錛?G鍐呭瓨澶ф鍙互琛ㄧず340浜縝it銆傚皢鍏朵腑涓涓枃浠朵腑鐨剈rl浣跨敤Bloom filter鏄犲皠涓鴻繖340浜縝it錛岀劧鍚庢尐涓鍙栧彟澶栦竴涓枃浠剁殑url錛屾鏌ユ槸鍚︿笌Bloom filter錛屽鏋滄槸錛岄偅涔堣url搴旇鏄叡鍚岀殑url錛堟敞鎰忎細鏈変竴瀹氱殑閿欒鐜囷級銆?/p>
2. 鏈?0涓枃浠訛紝姣忎釜鏂囦歡1G錛屾瘡涓枃浠剁殑姣忎竴琛屽瓨鏀劇殑閮芥槸鐢ㄦ埛鐨剄uery錛屾瘡涓枃浠剁殑query閮藉彲鑳介噸澶嶃傝姹備綘鎸夌収query鐨勯搴︽帓搴忋?/strong>
鏂規1錛?/p>
s 欏哄簭璇誨彇10涓枃浠訛紝鎸夌収hash(query)%10鐨勭粨鏋滃皢query鍐欏叆鍒板彟澶?0涓枃浠訛紙璁頒負錛変腑銆傝繖鏍鋒柊鐢熸垚鐨勬枃浠舵瘡涓殑澶у皬澶х害涔?G錛堝亣璁緃ash鍑芥暟鏄殢鏈虹殑錛夈?/p>
s 鎵句竴鍙板唴瀛樺湪2G宸﹀彸鐨勬満鍣紝渚濇瀵?a >鐢╤ash_map(query, query_count)鏉ョ粺璁℃瘡涓猶uery鍑虹幇鐨勬鏁般傚埄鐢ㄥ揩閫?鍫?褰掑茍鎺掑簭鎸夌収鍑虹幇嬈℃暟榪涜鎺掑簭銆傚皢鎺掑簭濂界殑query鍜屽搴旂殑query_cout杈撳嚭鍒版枃浠朵腑銆傝繖鏍峰緱鍒頒簡10涓帓濂藉簭鐨勬枃浠訛紙璁頒負
錛夈?/p>
s 瀵?a >榪?0涓枃浠惰繘琛屽綊騫舵帓搴忥紙鍐呮帓搴忎笌澶栨帓搴忕浉緇撳悎錛夈?/p>
鏂規2錛?/p>
涓鑸琿uery鐨勬婚噺鏄湁闄愮殑錛屽彧鏄噸澶嶇殑嬈℃暟姣旇緝澶氳屽凡錛屽彲鑳藉浜庢墍鏈夌殑query錛屼竴嬈℃у氨鍙互鍔犲叆鍒板唴瀛樹簡銆傝繖鏍鳳紝鎴戜滑灝卞彲浠ラ噰鐢╰rie鏍?hash_map絳夌洿鎺ユ潵緇熻姣忎釜query鍑虹幇鐨勬鏁幫紝鐒跺悗鎸夊嚭鐜版鏁板仛蹇?鍫?褰掑茍鎺掑簭灝卞彲浠ヤ簡銆?/p>
鏂規3錛?/p>
涓庢柟妗?綾諱技錛屼絾鍦ㄥ仛瀹宧ash錛屽垎鎴愬涓枃浠跺悗錛屽彲浠ヤ氦緇欏涓枃浠舵潵澶勭悊錛岄噰鐢ㄥ垎甯冨紡鐨勬灦鏋勬潵澶勭悊錛堟瘮濡侻apReduce錛夛紝鏈鍚庡啀榪涜鍚堝茍銆?/p>
3. 鏈変竴涓?G澶у皬鐨勪竴涓枃浠訛紝閲岄潰姣忎竴琛屾槸涓涓瘝錛岃瘝鐨勫ぇ灝忎笉瓚呰繃16瀛楄妭錛屽唴瀛橀檺鍒跺ぇ灝忔槸1M銆傝繑鍥為鏁版渶楂樼殑100涓瘝銆?/strong>
鏂規1錛氶『搴忚鏂囦歡涓紝瀵逛簬姣忎釜璇峹錛屽彇錛岀劧鍚庢寜鐓ц鍊煎瓨鍒?000涓皬鏂囦歡錛堣涓?a >
錛変腑銆傝繖鏍鋒瘡涓枃浠跺ぇ姒傛槸200k宸﹀彸銆傚鏋滃叾涓殑鏈夌殑鏂囦歡瓚呰繃浜?M澶у皬錛岃繕鍙互鎸夌収綾諱技鐨?strong style="BACKGROUND-COLOR: #880000; COLOR: white">鏂規硶緇х畫寰涓嬪垎錛岀煡閬撳垎瑙e緱鍒扮殑灝忔枃浠剁殑澶у皬閮戒笉瓚呰繃1M銆傚姣忎釜灝忔枃浠訛紝緇熻姣忎釜鏂囦歡涓嚭鐜扮殑璇嶄互鍙婄浉搴旂殑棰戠巼錛堝彲浠ラ噰鐢╰rie鏍?hash_map絳夛級錛屽茍鍙栧嚭鍑虹幇棰戠巼鏈澶х殑100涓瘝錛堝彲浠ョ敤鍚?strong style="BACKGROUND-COLOR: #ffff66; COLOR: black">100涓粨鐐圭殑鏈灝忓爢錛夛紝騫舵妸100璇嶅強鐩稿簲鐨勯鐜囧瓨鍏ユ枃浠訛紝榪欐牱鍙堝緱鍒頒簡5000涓枃浠躲備笅涓姝ュ氨鏄妸榪?000涓枃浠惰繘琛屽綊騫訛紙綾諱技涓庡綊騫舵帓搴忥級鐨勮繃紼嬩簡銆?/p>
4. 嫻烽噺鏃ュ織鏁版嵁錛屾彁鍙栧嚭鏌愭棩璁塊棶鐧懼害嬈℃暟鏈澶氱殑閭d釜IP銆?/strong>
鏂規1錛氶鍏堟槸榪欎竴澶╋紝騫朵笖鏄闂櫨搴︾殑鏃ュ織涓殑IP鍙栧嚭鏉ワ紝閫愪釜鍐欏叆鍒頒竴涓ぇ鏂囦歡涓傛敞鎰忓埌IP鏄?2浣嶇殑錛屾渶澶氭湁涓狪P銆傚悓鏍峰彲浠ラ噰鐢ㄦ槧灝勭殑鏂規硶錛屾瘮濡傛ā1000錛屾妸鏁翠釜澶ф枃浠舵槧灝勪負1000涓皬鏂囦歡錛屽啀鎵懼嚭姣忎釜灝忔枃涓嚭鐜伴鐜囨渶澶х殑IP錛堝彲浠ラ噰鐢╤ash_map榪涜棰戠巼緇熻錛岀劧鍚庡啀鎵懼嚭棰戠巼鏈澶х殑鍑犱釜錛夊強鐩稿簲鐨勯鐜囥傜劧鍚庡啀鍦ㄨ繖1000涓渶澶х殑IP涓紝鎵懼嚭閭d釜棰戠巼鏈澶х殑IP錛屽嵆涓烘墍姹傘?/p>
5. 鍦?.5浜夸釜鏁存暟涓壘鍑轟笉閲嶅鐨勬暣鏁幫紝鍐呭瓨涓嶈凍浠ュ綰寵繖2.5浜夸釜鏁存暟銆?/strong>
鏂規1錛氶噰鐢?-Bitmap錛堟瘡涓暟鍒嗛厤2bit錛?0琛ㄧず涓嶅瓨鍦紝01琛ㄧず鍑虹幇涓嬈★紝10琛ㄧず澶氭錛?1鏃犳剰涔夛級榪涜錛屽叡闇鍐呭瓨鍐呭瓨錛岃繕鍙互鎺ュ彈銆傜劧鍚庢壂鎻忚繖2.5浜夸釜鏁存暟錛屾煡鐪婤itmap涓浉瀵瑰簲浣嶏紝濡傛灉鏄?0鍙?1錛?1鍙?0錛?0淇濇寔涓嶅彉銆傛墍鎻忓畬浜嬪悗錛屾煡鐪媌itmap錛屾妸瀵瑰簲浣嶆槸01鐨勬暣鏁拌緭鍑哄嵆鍙?/p>
鏂規2錛氫篃鍙噰鐢ㄤ笂棰樼被浼肩殑鏂規硶錛岃繘琛屽垝鍒嗗皬鏂囦歡鐨?strong style="BACKGROUND-COLOR: #880000; COLOR: white">鏂規硶銆傜劧鍚庡湪灝忔枃浠朵腑鎵懼嚭涓嶉噸澶嶇殑鏁存暟錛屽茍鎺掑簭銆傜劧鍚庡啀榪涜褰掑茍錛屾敞鎰忓幓闄ら噸澶嶇殑鍏冪礌銆?/p>
6. 嫻烽噺鏁版嵁鍒嗗竷鍦?strong style="BACKGROUND-COLOR: #ffff66; COLOR: black">100鍙扮數鑴戜腑錛屾兂涓姙娉曢珮鏍$粺璁″嚭榪欐壒鏁版嵁鐨凾OP10銆?/strong>
鏂規1錛?/p>
s 鍦ㄦ瘡鍙扮數鑴戜笂姹傚嚭TOP10錛屽彲浠ラ噰鐢ㄥ寘鍚?0涓厓绱犵殑鍫嗗畬鎴愶紙TOP10灝忥紝鐢ㄦ渶澶у爢錛孴OP10澶э紝鐢ㄦ渶灝忓爢錛夈傛瘮濡傛眰TOP10澶э紝鎴戜滑棣栧厛鍙栧墠10涓厓绱犺皟鏁存垚鏈灝忓爢錛屽鏋滃彂鐜幫紝鐒跺悗鎵弿鍚庨潰鐨勬暟鎹紝騫朵笌鍫嗛《鍏冪礌姣旇緝錛屽鏋滄瘮鍫嗛《鍏冪礌澶э紝閭d箞鐢ㄨ鍏冪礌鏇挎崲鍫嗛《錛岀劧鍚庡啀璋冩暣涓烘渶灝忓爢銆傛渶鍚庡爢涓殑鍏冪礌灝辨槸TOP10澶с?/p>
s 姹傚嚭姣忓彴鐢佃剳涓婄殑TOP10鍚庯紝鐒跺悗鎶婅繖100鍙扮數鑴戜笂鐨凾OP10緇勫悎璧鋒潵錛屽叡1000涓暟鎹紝鍐嶅埄鐢ㄤ笂闈㈢被浼肩殑鏂規硶姹傚嚭TOP10灝卞彲浠ヤ簡銆?/p>
7. 鎬庝箞鍦ㄦ搗閲忔暟鎹腑鎵懼嚭閲嶅嬈℃暟鏈澶氱殑涓涓紵
鏂規1錛氬厛鍋歨ash錛岀劧鍚庢眰妯℃槧灝勪負灝忔枃浠訛紝姹傚嚭姣忎釜灝忔枃浠朵腑閲嶅嬈℃暟鏈澶氱殑涓涓紝騫惰褰曢噸澶嶆鏁般傜劧鍚庢壘鍑轟笂涓姝ユ眰鍑虹殑鏁版嵁涓噸澶嶆鏁版渶澶氱殑涓涓氨鏄墍姹傦紙鍏蜂綋鍙傝冨墠闈㈢殑棰橈級銆?/p>
8. 涓婂崈涓囨垨涓婁嚎鏁版嵁錛堟湁閲嶅錛夛紝緇熻鍏朵腑鍑虹幇嬈℃暟鏈澶氱殑閽盢涓暟鎹?/strong>
鏂規1錛氫笂鍗冧竾鎴栦笂浜跨殑鏁版嵁錛岀幇鍦ㄧ殑鏈哄櫒鐨勫唴瀛樺簲璇ヨ兘瀛樹笅銆傛墍浠ヨ冭檻閲囩敤hash_map/鎼滅儲浜屽弶鏍?綰㈤粦鏍戠瓑鏉ヨ繘琛岀粺璁℃鏁般傜劧鍚庡氨鏄彇鍑哄墠N涓嚭鐜版鏁版渶澶氱殑鏁版嵁浜嗭紝鍙互鐢ㄧ6棰樻彁鍒扮殑鍫嗘満鍒跺畬鎴愩?/p>
9. 1000涓囧瓧絎︿覆錛屽叾涓湁浜涙槸閲嶅鐨勶紝闇瑕佹妸閲嶅鐨勫叏閮ㄥ幓鎺夛紝淇濈暀娌℃湁閲嶅鐨勫瓧絎︿覆銆傝鎬庝箞璁捐鍜屽疄鐜幫紵
鏂規1錛氳繖棰樼敤trie鏍戞瘮杈冨悎閫傦紝hash_map涔熷簲璇ヨ兘琛屻?/p>
10. 涓涓枃鏈枃浠訛紝澶х害鏈変竴涓囪錛屾瘡琛屼竴涓瘝錛岃姹傜粺璁″嚭鍏朵腑鏈棰戠箒鍑虹幇鐨勫墠10涓瘝錛岃緇欏嚭鎬濇兂錛岀粰鍑烘椂闂村鏉傚害鍒嗘瀽銆?/strong>
鏂規1錛氳繖棰樻槸鑰冭檻鏃墮棿鏁堢巼銆傜敤trie鏍戠粺璁℃瘡涓瘝鍑虹幇鐨勬鏁幫紝鏃墮棿澶嶆潅搴︽槸O(n*le)錛坙e琛ㄧず鍗曡瘝鐨勫鉤鍑嗛暱搴︼級銆傜劧鍚庢槸鎵懼嚭鍑虹幇鏈棰戠箒鐨勫墠10涓瘝錛屽彲浠ョ敤鍫嗘潵瀹炵幇錛屽墠闈㈢殑棰樹腑宸茬粡璁插埌浜嗭紝鏃墮棿澶嶆潅搴︽槸O(n*lg10)銆傛墍浠ユ葷殑鏃墮棿澶嶆潅搴︼紝鏄疧(n*le)涓嶰(n*lg10)涓緝澶х殑鍝竴涓?/p>
11. 涓涓枃鏈枃浠訛紝鎵懼嚭鍓?0涓粡甯稿嚭鐜扮殑璇嶏紝浣嗚繖嬈℃枃浠舵瘮杈冮暱錛岃鏄笂浜胯鎴栧崄浜胯錛屾諱箣鏃犳硶涓嬈¤鍏ュ唴瀛橈紝闂渶浼樿В銆?/strong>
鏂規1錛氶鍏堟牴鎹敤hash騫舵眰妯★紝灝嗘枃浠跺垎瑙d負澶氫釜灝忔枃浠訛紝瀵逛簬鍗曚釜鏂囦歡鍒╃敤涓婇鐨?strong style="BACKGROUND-COLOR: #880000; COLOR: white">鏂規硶姹傚嚭姣忎釜鏂囦歡浠朵腑10涓渶甯稿嚭鐜扮殑璇嶃傜劧鍚庡啀榪涜褰掑茍澶勭悊錛屾壘鍑烘渶緇堢殑10涓渶甯稿嚭鐜扮殑璇嶃?/p>
12. 100w涓暟涓壘鍑烘渶澶х殑100涓暟銆?/strong>
鏂規1錛氬湪鍓嶉潰鐨勯涓紝鎴戜滑宸茬粡鎻愬埌浜嗭紝鐢ㄤ竴涓惈100涓厓绱犵殑鏈灝忓爢瀹屾垚銆傚鏉傚害涓篛(100w*lg100)銆?/p>
鏂規2錛氶噰鐢ㄥ揩閫熸帓搴忕殑鎬濇兂錛屾瘡嬈″垎鍓蹭箣鍚庡彧鑰冭檻姣旇醬澶х殑涓閮ㄥ垎錛岀煡閬撴瘮杞村ぇ鐨勪竴閮ㄥ垎鍦ㄦ瘮100澶氱殑鏃跺欙紝閲囩敤浼犵粺鎺掑簭綆楁硶鎺掑簭錛屽彇鍓?strong style="BACKGROUND-COLOR: #ffff66; COLOR: black">100涓傚鏉傚害涓篛(100w*100)銆?/p>
鏂規3錛氶噰鐢ㄥ眬閮ㄦ窐姹版硶銆傞夊彇鍓?strong style="BACKGROUND-COLOR: #ffff66; COLOR: black">100涓厓绱狅紝騫舵帓搴忥紝璁頒負搴忓垪L銆傜劧鍚庝竴嬈℃壂鎻忓墿浣欑殑鍏冪礌x錛屼笌鎺掑ソ搴忕殑100涓厓绱犱腑鏈灝忕殑鍏冪礌姣旓紝濡傛灉姣旇繖涓渶灝忕殑瑕佸ぇ錛岄偅涔堟妸榪欎釜鏈灝忕殑鍏冪礌鍒犻櫎錛屽茍鎶妜鍒╃敤鎻掑叆鎺掑簭鐨勬濇兂錛屾彃鍏ュ埌搴忓垪L涓備緷嬈″驚鐜紝鐭ラ亾鎵弿浜嗘墍鏈夌殑鍏冪礌銆傚鏉傚害涓篛(100w*100)銆?/p>
13. 瀵繪壘鐑棬鏌ヨ錛?/strong>
鎼滅儲寮曟搸浼氶氳繃鏃ュ織鏂囦歡鎶婄敤鎴鋒瘡嬈℃绱嬌鐢ㄧ殑鎵鏈夋绱覆閮借褰曚笅鏉ワ紝姣忎釜鏌ヨ涓茬殑闀垮害涓?-255瀛楄妭銆傚亣璁劇洰鍓嶆湁涓鍗冧竾涓褰曪紝榪欎簺鏌ヨ涓茬殑閲嶅璇繪瘮杈冮珮錛岃櫧鐒舵繪暟鏄?鍗冧竾錛屼絾鏄鏋滃幓闄ら噸澶嶅拰錛屼笉瓚呰繃3鐧句竾涓備竴涓煡璇覆鐨勯噸澶嶅害瓚婇珮錛岃鏄庢煡璇㈠畠鐨勭敤鎴瘋秺澶氾紝涔熷氨瓚婄儹闂ㄣ傝浣犵粺璁℃渶鐑棬鐨?0涓煡璇覆錛岃姹備嬌鐢ㄧ殑鍐呭瓨涓嶈兘瓚呰繃1G銆?/p>
(1) 璇鋒弿榪頒綘瑙e喅榪欎釜闂鐨勬濊礬錛?/p>
(2) 璇風粰鍑轟富瑕佺殑澶勭悊嫻佺▼錛岀畻娉曪紝浠ュ強綆楁硶鐨勫鏉傚害銆?/p>
鏂規1錛氶噰鐢╰rie鏍戯紝鍏抽敭瀛楀煙瀛樿鏌ヨ涓插嚭鐜扮殑嬈℃暟錛屾病鏈夊嚭鐜頒負0銆傛渶鍚庣敤10涓厓绱犵殑鏈灝忔帹鏉ュ鍑虹幇棰戠巼榪涜鎺掑簭銆?/p>
14. 涓鍏辨湁N涓満鍣紝姣忎釜鏈哄櫒涓婃湁N涓暟銆傛瘡涓満鍣ㄦ渶澶氬瓨O(N)涓暟騫跺瀹冧滑鎿嶄綔銆傚浣曟壘鍒?a >涓暟涓殑涓暟錛?/strong>
鏂規1錛氬厛澶т綋浼拌涓涓嬭繖浜涙暟鐨勮寖鍥達紝姣斿榪欓噷鍋囪榪欎簺鏁伴兘鏄?2浣嶆棤絎﹀彿鏁存暟錛堝叡鏈?a >涓級銆傛垜浠妸0鍒?a >
鐨勬暣鏁板垝鍒嗕負N涓寖鍥存錛屾瘡涓鍖呭惈
涓暣鏁般傛瘮濡傦紝絎竴涓浣?鍒?a >
錛岀浜屾涓?a >
鍒?a >
錛?#8230;錛岀N涓涓?a >
鍒?a >
銆傜劧鍚庯紝鎵弿姣忎釜鏈哄櫒涓婄殑N涓暟錛屾妸灞炰簬絎竴涓尯孌電殑鏁版斁鍒扮涓涓満鍣ㄤ笂錛屽睘浜庣浜屼釜鍖烘鐨勬暟鏀懼埌絎簩涓満鍣ㄤ笂錛?#8230;錛屽睘浜庣N涓尯孌電殑鏁版斁鍒扮N涓満鍣ㄤ笂銆傛敞鎰忚繖涓繃紼嬫瘡涓満鍣ㄤ笂瀛樺偍鐨勬暟搴旇鏄疧(N)鐨勩備笅闈㈡垜浠緷嬈$粺璁℃瘡涓満鍣ㄤ笂鏁扮殑涓暟錛屼竴嬈$瘡鍔狅紝鐩村埌鎵懼埌絎琸涓満鍣紝鍦ㄨ鏈哄櫒涓婄瘡鍔犵殑鏁板ぇ浜庢垨絳変簬
錛岃屽湪絎琸-1涓満鍣ㄤ笂鐨勭瘡鍔犳暟灝忎簬
錛屽茍鎶婅繖涓暟璁頒負x銆傞偅涔堟垜浠鎵劇殑涓綅鏁板湪絎琸涓満鍣ㄤ腑錛屾帓鍦ㄧ
浣嶃傜劧鍚庢垜浠絎琸涓満鍣ㄧ殑鏁版帓搴忥紝騫舵壘鍑虹
涓暟錛屽嵆涓烘墍姹傜殑涓綅鏁般傚鏉傚害鏄?a >
鐨勩?/p>
鏂規2錛氬厛瀵規瘡鍙版満鍣ㄤ笂鐨勬暟榪涜鎺掑簭銆傛帓濂藉簭鍚庯紝鎴戜滑閲囩敤褰掑茍鎺掑簭鐨勬濇兂錛屽皢榪橬涓満鍣ㄤ笂鐨勬暟褰掑茍璧鋒潵寰楀埌鏈緇堢殑鎺掑簭銆傛壘鍒扮涓究鏄墍姹傘傚鏉傚害鏄?a >
鐨勩?/p>
15. 鏈澶ч棿闅欓棶棰?/strong>
緇欏畾n涓疄鏁?a >錛屾眰鐫n涓疄鏁板湪瀹炶醬涓婂悜閲?涓暟涔嬮棿鐨勬渶澶у樊鍊鹼紝瑕佹眰綰挎х殑鏃墮棿綆楁硶銆?/p>
鏂規1錛氭渶鍏堟兂鍒扮殑鏂規硶灝辨槸鍏堝榪檔涓暟鎹繘琛屾帓搴忥紝鐒跺悗涓閬嶆壂鎻忓嵆鍙‘瀹氱浉閭葷殑鏈澶ч棿闅欍備絾璇?strong style="BACKGROUND-COLOR: #880000; COLOR: white">鏂規硶涓嶈兘婊¤凍綰挎ф椂闂寸殑瑕佹眰銆傛晠閲囧彇濡備笅鏂規硶錛?/p>
s 鎵懼埌n涓暟鎹腑鏈澶у拰鏈灝忔暟鎹甿ax鍜宮in銆?/p>
s 鐢╪-2涓偣絳夊垎鍖洪棿[min, max]錛屽嵆灝哰min, max]絳夊垎涓簄-1涓尯闂達紙鍓嶉棴鍚庡紑鍖洪棿錛夛紝灝嗚繖浜涘尯闂寸湅浣滄《錛岀紪鍙蜂負錛屼笖妗?a >
鐨勪笂鐣屽拰妗秈+1鐨勪笅灞婄浉鍚岋紝鍗蟲瘡涓《鐨勫ぇ灝忕浉鍚屻傛瘡涓《鐨勫ぇ灝忎負錛?a >
銆傚疄闄呬笂錛岃繖浜涙《鐨勮竟鐣屾瀯鎴愪簡涓涓瓑宸暟鍒楋紙棣栭」涓簃in錛屽叕宸負
錛夛紝涓旇涓哄皢min鏀懼叆絎竴涓《錛屽皢max鏀懼叆絎琻-1涓《銆?/p>
s 灝唍涓暟鏀懼叆n-1涓《涓細灝嗘瘡涓厓绱?a >鍒嗛厤鍒版煇涓《錛堢紪鍙蜂負index錛夛紝鍏朵腑
錛屽茍姹傚嚭鍒嗗埌姣忎釜妗剁殑鏈澶ф渶灝忔暟鎹?/p>
s 鏈澶ч棿闅欙細闄ゆ渶澶ф渶灝忔暟鎹甿ax鍜宮in浠ュ鐨刵-2涓暟鎹斁鍏-1涓《涓紝鐢辨娊灞夊師鐞嗗彲鐭ヨ嚦灝戞湁涓涓《鏄┖鐨勶紝鍙堝洜涓烘瘡涓《鐨勫ぇ灝忕浉鍚岋紝鎵浠ユ渶澶ч棿闅欎笉浼氬湪鍚屼竴妗朵腑鍑虹幇錛屼竴瀹氭槸鏌愪釜妗剁殑涓婄晫鍜屾皵鍊欐煇涓《鐨勪笅鐣屼箣闂撮殭錛屼笖璇ラ噺絳掍箣闂寸殑妗訛紙鍗充究濂藉湪璇ヨ繛涓究濂戒箣闂寸殑妗訛級涓瀹氭槸絀烘《銆備篃灝辨槸璇達紝鏈澶ч棿闅欏湪妗秈鐨勪笂鐣屽拰妗秊鐨勪笅鐣屼箣闂翠駭鐢?a >銆備竴閬嶆壂鎻忓嵆鍙畬鎴愩?/p>
16. 灝嗗涓泦鍚堝悎騫舵垚娌℃湁浜ら泦鐨勯泦鍚堬細緇欏畾涓涓瓧絎︿覆鐨勯泦鍚堬紝鏍煎紡濡傦細銆傝姹傚皢鍏朵腑浜ら泦涓嶄負絀虹殑闆嗗悎鍚堝茍錛岃姹傚悎騫跺畬鎴愮殑闆嗗悎涔嬮棿鏃犱氦闆嗭紝渚嬪涓婁緥搴旇緭鍑?a >
銆?/strong>
(1) 璇鋒弿榪頒綘瑙e喅榪欎釜闂鐨勬濊礬錛?/p>
(2) 緇欏嚭涓昏鐨?strong style="BACKGROUND-COLOR: #ff66ff; COLOR: black">澶勭悊嫻佺▼錛岀畻娉曪紝浠ュ強綆楁硶鐨勫鏉傚害錛?/p>
(3) 璇鋒弿榪板彲鑳界殑鏀硅繘銆?/p>
鏂規1錛氶噰鐢ㄥ茍鏌ラ泦銆傞鍏堟墍鏈夌殑瀛楃涓查兘鍦ㄥ崟鐙殑騫舵煡闆嗕腑銆傜劧鍚庝緷鎵弿姣忎釜闆嗗悎錛岄『搴忓悎騫跺皢涓や釜鐩擱偦鍏冪礌鍚堝茍銆備緥濡傦紝瀵逛簬錛岄鍏堟煡鐪媋aa鍜宐bb鏄惁鍦ㄥ悓涓涓茍鏌ラ泦涓紝濡傛灉涓嶅湪錛岄偅涔堟妸瀹冧滑鎵鍦ㄧ殑騫舵煡闆嗗悎騫訛紝鐒跺悗鍐嶇湅bbb鍜宑cc鏄惁鍦ㄥ悓涓涓茍鏌ラ泦涓紝濡傛灉涓嶅湪錛岄偅涔堜篃鎶婂畠浠墍鍦ㄧ殑騫舵煡闆嗗悎騫躲傛帴涓嬫潵鍐嶆壂鎻忓叾浠栫殑闆嗗悎錛屽綋鎵鏈夌殑闆嗗悎閮芥壂鎻忓畬浜嗭紝騫舵煡闆嗕唬琛ㄧ殑闆嗗悎渚挎槸鎵姹傘傚鏉傚害搴旇鏄疧(NlgN)鐨勩傛敼榪涚殑璇濓紝棣栧厛鍙互璁板綍姣忎釜鑺傜偣鐨勬牴緇撶偣錛屾敼榪涙煡璇€傚悎騫剁殑鏃跺欙紝鍙互鎶婂ぇ鐨勫拰灝忕殑榪涜鍚堬紝榪欐牱涔熷噺灝戝鏉傚害銆?/p>
17. 鏈澶у瓙搴忓垪涓庢渶澶у瓙鐭╅樀闂
鏁扮粍鐨勬渶澶у瓙搴忓垪闂錛氱粰瀹氫竴涓暟緇勶紝鍏朵腑鍏冪礌鏈夋錛屼篃鏈夎礋錛屾壘鍑哄叾涓竴涓繛緇瓙搴忓垪錛屼嬌鍜屾渶澶с?/p>
鏂規1錛氳繖涓棶棰樺彲浠ュ姩鎬佽鍒掔殑鎬濇兂瑙e喅銆傝琛ㄧず浠ョi涓厓绱?a >
緇撳熬鐨勬渶澶у瓙搴忓垪錛岄偅涔堟樉鐒?a >
銆傚熀浜庤繖涓鐐瑰彲浠ュ緢蹇敤浠g爜瀹炵幇銆?/p>
鏈澶у瓙鐭╅樀闂錛氱粰瀹氫竴涓煩闃碉紙浜岀淮鏁扮粍錛夛紝鍏朵腑鏁版嵁鏈夊ぇ鏈夊皬錛岃鎵句竴涓瓙鐭╅樀錛屼嬌寰楀瓙鐭╅樀鐨勫拰鏈澶э紝騫惰緭鍑鴻繖涓拰銆?/p>
鏂規1錛氬彲浠ラ噰鐢ㄤ笌鏈澶у瓙搴忓垪綾諱技鐨勬濇兂鏉ヨВ鍐熾傚鏋滄垜浠‘瀹氫簡閫夋嫨絎琲鍒楀拰絎琷鍒椾箣闂寸殑鍏冪礌錛岄偅涔堝湪榪欎釜鑼冨洿鍐咃紝鍏跺疄灝辨槸涓涓渶澶у瓙搴忓垪闂銆傚浣曠‘瀹氱i鍒楀拰絎琷鍒楀彲浠ヨ瘝鐢ㄦ毚鎼滅殑鏂規硶榪涜銆?/p>
1. 瀹夎澧炲己鍔熻兘鍖?Guest Additions)
瀹夎濂?u>ubuntu鍚庯紝榪愯Ubuntu騫剁櫥褰曘傜劧鍚庡湪VirtualBox鐨勮彍鍗曢噷閫夋嫨"璁懼(Devices)" -> "瀹夎澧炲己鍔熻兘鍖?Install Guest Additions)"銆?/p>
浣犱細鍙戠幇鍦║buntu妗岄潰涓婂鍑轟竴涓厜鐩樺浘鏍囷紝榪欏紶鍏夌洏榛樿琚嚜鍔ㄥ姞杞藉埌浜嗘枃浠跺す/media/cdom0銆傝繘鍏ュ懡浠よ緇堢錛岃緭鍏ワ細
cd /media/cdom0
sudo ./VboxLinuxAdditions.run
寮濮嬪畨瑁呭伐鍏峰寘銆傚畨瑁呭畬姣曞悗浼氭彁紺鴻閲嶅惎Ubuntu銆?/p>
2. 璁劇疆鍏變韓鏂囦歡澶?/p>
閲嶅惎瀹屾垚鍚庣偣鍑?璁懼(Devices)" -> 鍏變韓鏂囦歡澶?Shared Folders)鑿滃崟錛屾坊鍔犱竴涓叡浜枃浠跺す錛岄夐」鍥哄畾鍜屼復鏃舵槸鎸囪鏂囦歡澶規槸鍚︽槸鎸佷箙鐨勩傚叡浜悕鍙互浠誨彇涓涓嚜宸卞枩嬈㈢殑錛屾瘮濡?gongxiang"錛屽敖閲忎嬌鐢ㄨ嫳鏂囧悕縐般?/p>
3. 鎸傝澆鍏變韓鏂囦歡澶?/p>
閲嶆柊榪涘叆铏氭嫙Ubuntu錛屽湪鍛戒護琛岀粓绔笅杈撳叆錛?/p>
sudo mkdir /mnt/shared
sudo mount -t vboxsf gongxiang /mnt/shared
鍏朵腑"gongxiang"鏄箣鍓嶅垱寤虹殑鍏變韓鏂囦歡澶圭殑鍚嶅瓧銆侽K錛岀幇鍦║buntu鍜屼富鏈哄彲浠ヤ簰浼犳枃浠朵簡銆?/p>
鍋囧鎮ㄤ笉鎯蟲瘡涓嬈¢兘鎵嬪姩鎸傝澆錛屽彲浠ュ湪/etc/fstab涓坊鍔犱竴欏?/p>
gongxiang /mnt/shared vboxsf rw,gid=100,uid=1000,auto 0 0
榪欐牱灝辮兘澶熻嚜鍔ㄦ寕杞戒簡銆?/p>
4. 鍗歌澆鐨勮瘽浣跨敤涓嬮潰鐨勫懡浠わ細
sudo umount -f /mnt/shared
娉ㄦ剰錛?/p>
鍏變韓鏂囦歡澶圭殑鍚嶇О鍗冧竾涓嶈鍜屾寕杞界偣鐨勫悕縐扮浉鍚屻傛瘮濡傦紝涓婇潰鐨勬寕杞界偣鏄?mnt/shared錛屽鏋滃叡浜枃浠跺す鐨勫悕瀛椾篃鏄痵hared鐨勮瘽錛屽湪鎸傝澆鐨勬椂鍊欏氨浼氬嚭鐜板涓嬬殑閿欒淇℃伅(鐪?a >http://www.virtualbox.org/ticket/2265)錛?/p>
/sbin/mount.vboxsf: mounting failed with the error: Protocol error
鍘熷洜鍒嗘瀽鍙互鐪婽ips on running Sun Virtualbox鐨凷hared Folder on a Linux Guest鑺傘?/p>
1銆佸壇鏈殑瀛樻斁錛屽壇鏈殑瀛樻斁鏄?/span>HDFS鍙潬鎬у拰鎬ц兘鐨勫叧閿?/span>HDFS閲囩敤涓縐嶇О涓?/span>rack-aware鐨勭瓥鐣ユ潵鏀硅繘鏁版嵁鐨勫彲闈犳с佹湁鏁堟у拰緗戠粶甯﹀鐨勫埄鐢ㄣ傝繖涓瓥鐣ュ疄鐜扮殑鐭湡鐩爣鏄獙璇佸湪鐢熶駭鐜涓嬬殑琛ㄧ幇錛岃瀵熷畠鐨勮涓猴紝鏋勫緩嫻嬭瘯鍜岀爺絀剁殑鍩虹錛屼互渚垮疄鐜版洿鍏堣繘鐨勭瓥鐣ャ傚簽澶х殑HDFS瀹炰緥涓鑸繍琛屽湪澶氫釜鏈烘灦鐨勮綆楁満褰㈡垚鐨勯泦緹や笂錛屼笉鍚屾満鏋墮棿鐨勪袱鍙版満鍣ㄧ殑閫氳闇瑕侀氳繃浜ゆ崲鏈猴紝鏄劇劧閫氬父鎯呭喌涓嬶紝鍚屼竴涓満鏋跺唴鐨勪袱涓妭鐐歸棿鐨勫甫瀹戒細姣斾笉鍚屾満鏋墮棿鐨勪袱鍙版満鍣ㄧ殑甯﹀澶с?/span>
閫氳繃涓涓О涓?/span>Rack Awareness鐨勮繃紼嬶紝Namenode鍐沖畾浜嗘瘡涓?/span>Datanode鎵灞炵殑rack id銆備竴涓畝鍗曚絾娌℃湁浼樺寲鐨勭瓥鐣ュ氨鏄皢鍓湰瀛樻斁鍦ㄥ崟鐙殑鏈烘灦涓娿傝繖鏍峰彲浠ラ槻姝㈡暣涓満鏋訛紙闈炲壇鏈瓨鏀撅級澶辨晥鐨勬儏鍐碉紝騫朵笖鍏佽璇繪暟鎹殑鏃跺欏彲浠ヤ粠澶氫釜鏈烘灦璇誨彇銆傝繖涓畝鍗曠瓥鐣ヨ緗彲浠ュ皢鍓湰鍒嗗竷鍦ㄩ泦緹や腑錛屾湁鍒╀簬緇勪歡澶辮觸鎯呭喌涓嬬殑璐熻澆鍧囪 銆備絾鏄紝榪欎釜綆鍗曠瓥鐣ュ姞澶т簡鍐欑殑浠d環錛屽洜涓轟竴涓啓鎿嶄綔闇瑕佷紶杈?/span>block鍒板涓満鏋躲?/span>
鍦ㄥぇ澶氭暟鎯呭喌涓嬶紝replication鍥犲瓙鏄?/span>3錛?/span>HDFS鐨勫瓨鏀劇瓥鐣ユ槸灝嗕竴涓壇鏈瓨鏀懼湪鏈湴鏈烘灦涓婄殑鑺傜偣錛屼竴涓壇鏈斁鍦ㄥ悓涓鏈烘灦涓婄殑鍙︿竴涓妭鐐癸紝鏈鍚庝竴涓壇鏈斁鍦ㄤ笉鍚屾満鏋朵笂鐨勪竴涓妭鐐廣傛満鏋剁殑閿欒榪滆繙姣旇妭鐐圭殑閿欒灝戯紝榪欎釜絳栫暐涓嶄細褰卞搷鍒版暟鎹殑鍙潬鎬у拰鏈夋晥鎬с備笁鍒嗕箣涓鐨勫壇鏈湪涓涓妭鐐逛笂錛屼笁鍒嗕箣浜屽湪涓涓満鏋朵笂錛屽叾浠栦繚瀛樺湪鍓╀笅鐨勬満鏋朵腑錛岃繖涓絳栫暐鏀硅繘浜嗗啓鐨勬ц兘銆?/span>
2銆佸壇鏈殑閫夋嫨錛屼負浜嗛檷浣庢暣浣撶殑甯﹀娑堣楀拰璇誨歡鏃訛紝HDFS浼氬敖閲忚reader璇繪渶榪戠殑鍓湰銆傚鏋滃湪reader鐨勫悓涓涓満鏋朵笂鏈変竴涓壇鏈紝閭d箞灝辮璇ュ壇鏈傚鏋滀竴涓?/span>HDFS闆嗙兢璺ㄨ秺澶氫釜鏁版嵁涓績錛岄偅涔?/span>reader涔熷皢棣栧厛灝濊瘯璇繪湰鍦版暟鎹腑蹇冪殑鍓湰銆?/span>
3銆?/span>SafeMode
Namenode鍚姩鍚庝細榪涘叆涓涓О涓?/span>SafeMode鐨勭壒孌婄姸鎬侊紝澶勫湪榪欎釜鐘舵佺殑Namenode鏄笉浼氳繘琛屾暟鎹潡鐨勫鍒剁殑銆?/span>Namenode浠庢墍鏈夌殑 Datanode鎺ユ敹蹇冭煩鍖呭拰Blockreport銆?/span>Blockreport鍖呮嫭浜嗘煇涓?/span>Datanode鎵鏈夌殑鏁版嵁鍧楀垪琛ㄣ傛瘡涓?/span>block閮芥湁鎸囧畾鐨勬渶灝忔暟鐩殑鍓湰銆傚綋Namenode媯嫻嬬‘璁ゆ煇涓?/span>Datanode鐨勬暟鎹潡鍓湰鐨勬渶灝忔暟鐩紝閭d箞璇?/span>Datanode灝變細琚涓烘槸瀹夊叏鐨勶紱濡傛灉涓瀹氱櫨鍒嗘瘮錛堣繖涓弬鏁板彲閰嶇疆錛夌殑鏁版嵁鍧楁嫻嬬‘璁ゆ槸瀹夊叏鐨勶紝閭d箞Namenode灝嗛鍑?/span>SafeMode鐘舵侊紝鎺ヤ笅鏉ュ畠浼氱‘瀹氳繕鏈夊摢浜涙暟鎹潡鐨勫壇鏈病鏈夎揪鍒版寚瀹氭暟鐩紝騫跺皢榪欎簺block澶嶅埗鍒板叾浠?/span>Datanode銆?/span>
浜斻佹枃浠剁郴緇熷厓鏁版嵁鐨勬寔涔呭寲
Namenode瀛樺偍HDFS鐨勫厓鏁版嵁銆傚浜庝換浣曞鏂囦歡鍏冩暟鎹駭鐢熶慨鏀圭殑鎿嶄綔錛?/span>Namenode閮戒嬌鐢ㄤ竴涓О涓?/span>Editlog鐨勪簨鍔℃棩蹇楄褰曚笅鏉ャ備緥濡傦紝鍦?/span>HDFS涓垱寤轟竴涓枃浠訛紝Namenode灝變細鍦?/span>Editlog涓彃鍏ヤ竴鏉¤褰曟潵琛ㄧず錛涘悓鏍鳳紝淇敼鏂囦歡鐨?/span>replication鍥犲瓙涔熷皢寰 Editlog鎻掑叆涓鏉¤褰曘?/span>Namenode鍦ㄦ湰鍦?/span>OS鐨勬枃浠剁郴緇熶腑瀛樺偍榪欎釜Editlog銆傛暣涓枃浠剁郴緇熺殑namespace錛屽寘鎷?/span>block鍒版枃浠剁殑鏄犲皠銆佹枃浠剁殑灞炴э紝閮藉瓨鍌ㄥ湪縐頒負FsImage鐨勬枃浠朵腑錛岃繖涓枃浠朵篃鏄斁鍦?/span>Namenode鎵鍦ㄧ郴緇熺殑鏂囦歡緋葷粺涓娿?/span>
Namenode鍦ㄥ唴瀛樹腑淇濆瓨鐫鏁翠釜鏂囦歡緋葷粺namespace鍜屾枃浠?/span>Blockmap鐨勬槧鍍忋傝繖涓叧閿殑鍏冩暟鎹璁″緱寰堢揣鍑戯紝鍥犺屼竴涓甫鏈?/span>4G鍐呭瓨鐨?Namenode瓚沖鏀拺嫻烽噺鐨勬枃浠跺拰鐩綍銆傚綋Namenode鍚姩鏃訛紝瀹冧粠紜洏涓鍙?/span>Editlog鍜?/span>FsImage錛屽皢鎵鏈?/span>Editlog涓殑浜嬪姟浣滅敤錛?/span>apply)鍦ㄥ唴瀛樹腑鐨?/span>FsImage 錛屽茍灝嗚繖涓柊鐗堟湰鐨?/span>FsImage浠庡唴瀛樹腑flush鍒扮‖鐩樹笂,鐒跺悗鍐?/span>truncate榪欎釜鏃х殑Editlog錛屽洜涓鴻繖涓棫鐨?/span>Editlog鐨勪簨鍔¢兘宸茬粡浣滅敤鍦?/span>FsImage涓婁簡銆傝繖涓繃紼嬬О涓?/span>checkpoint銆傚湪褰撳墠瀹炵幇涓紝checkpoint鍙彂鐢熷湪Namenode鍚姩鏃訛紝鍦ㄤ笉涔呯殑灝嗘潵鎴戜滑灝嗗疄鐜版敮鎸佸懆鏈熸х殑checkpoint銆?/span>
Datanode騫朵笉鐭ラ亾鍏充簬鏂囦歡鐨勪換浣曚笢瑗匡紝闄や簡灝嗘枃浠朵腑鐨勬暟鎹繚瀛樺湪鏈湴鐨勬枃浠剁郴緇熶笂銆傚畠鎶婃瘡涓?/span>HDFS鏁版嵁鍧楀瓨鍌ㄥ湪鏈湴鏂囦歡緋葷粺涓婇殧紱葷殑鏂囦歡涓?Datanode騫朵笉鍦ㄥ悓涓涓洰褰曞垱寤烘墍鏈夌殑鏂囦歡錛岀浉鍙嶏紝瀹冪敤鍚彂寮忓湴鏂規硶鏉ョ‘瀹氭瘡涓洰褰曠殑鏈浣蟲枃浠舵暟鐩紝騫朵笖鍦ㄩ傚綋鐨勬椂鍊欏垱寤哄瓙鐩綍銆傚湪鍚屼竴涓洰褰曞垱寤烘墍鏈夌殑鏂囦歡涓嶆槸鏈浼樼殑閫夋嫨錛屽洜涓烘湰鍦版枃浠剁郴緇熷彲鑳芥棤娉曢珮鏁堝湴鍦ㄥ崟涓鐩綍涓敮鎸佸ぇ閲忕殑鏂囦歡銆傚綋涓涓?/span>Datanode鍚姩鏃訛紝瀹冩壂鎻忔湰鍦版枃浠剁郴緇燂紝瀵硅繖浜涙湰鍦版枃浠朵駭鐢熺浉搴旂殑涓涓墍鏈?/span>HDFS鏁版嵁鍧楃殑鍒楄〃錛岀劧鍚庡彂閫佹姤鍛婂埌Namenode錛岃繖涓姤鍛婂氨鏄?/span>Blockreport銆?/span>
鍏侀氳鍗忚
鎵鏈夌殑HDFS閫氳鍗忚閮芥槸鏋勫緩鍦?/span>TCP/IP鍗忚涓娿傚鎴風閫氳繃涓涓彲閰嶇疆鐨勭鍙h繛鎺ュ埌Namenode錛岄氳繃ClientProtocol涓?Namenode浜や簰銆傝?/span>Datanode鏄嬌鐢?/span>DatanodeProtocol涓?/span>Namenode浜や簰銆備粠ClientProtocol鍜?Datanodeprotocol鎶借薄鍑轟竴涓繙紼嬭皟鐢?/span>(RPC錛夛紝鍦ㄨ璁′笂錛?/span>Namenode涓嶄細涓誨姩鍙戣搗RPC錛岃屾槸鏄搷搴旀潵鑷鎴風鍜?Datanode 鐨?/span>RPC璇鋒眰銆?/span>
涓冦佸仴澹?/span>
HDFS鐨勪富瑕佺洰鏍囧氨鏄疄鐜板湪澶辮觸鎯呭喌涓嬬殑鏁版嵁瀛樺偍鍙潬鎬с傚父瑙佺殑涓夌澶辮觸錛?/span>Namenode failures, Datanode failures鍜岀綉緇滃垎鍓詫紙network partitions)銆?/span>
1銆佺‖鐩樻暟鎹敊璇佸績璺蟲嫻嬪拰閲嶆柊澶嶅埗
姣忎釜Datanode鑺傜偣閮藉悜Namenode鍛ㄦ湡鎬у湴鍙戦佸績璺沖寘銆傜綉緇滃垏鍓插彲鑳藉鑷翠竴閮ㄥ垎Datanode璺?/span>Namenode澶卞幓鑱旂郴銆?Namenode閫氳繃蹇冭煩鍖呯殑緙哄け媯嫻嬪埌榪欎竴鎯呭喌錛屽茍灝嗚繖浜?/span>Datanode鏍囪涓?/span>dead錛屼笉浼氬皢鏂扮殑IO璇鋒眰鍙戠粰瀹冧滑銆傚瘎瀛樺湪dead Datanode涓婄殑浠諱綍鏁版嵁灝嗕笉鍐嶆湁鏁堛?/span>Datanode鐨勬浜″彲鑳藉紩璧蜂竴浜?/span>block鐨勫壇鏈暟鐩綆浜庢寚瀹氬鹼紝Namenode涓嶆柇鍦拌窡韙渶瑕佸鍒剁殑 block錛屽湪浠諱綍闇瑕佺殑鎯呭喌涓嬪惎鍔ㄥ鍒躲傚湪涓嬪垪鎯呭喌鍙兘闇瑕侀噸鏂板鍒訛細鏌愪釜Datanode鑺傜偣澶辨晥錛屾煇涓壇鏈伃鍒版崯鍧忥紝Datanode涓婄殑紜洏閿欒錛屾垨鑰呮枃浠剁殑replication鍥犲瓙澧炲ぇ銆?/span>
2銆侀泦緹ゅ潎琛?/span>
HDFS鏀寔鏁版嵁鐨勫潎琛¤鍒掞紝濡傛灉鏌愪釜Datanode鑺傜偣涓婄殑絀洪棽絀洪棿浣庝簬鐗瑰畾鐨勪復鐣岀偣錛岄偅涔堝氨浼氬惎鍔ㄤ竴涓鍒掕嚜鍔ㄥ湴灝嗘暟鎹粠涓涓?/span>Datanode鎼Щ鍒扮┖闂茬殑Datanode銆傚綋瀵規煇涓枃浠剁殑璇鋒眰紿佺劧澧炲姞錛岄偅涔堜篃鍙兘鍚姩涓涓鍒掑垱寤鴻鏂囦歡鏂扮殑鍓湰錛屽茍鍒嗗竷鍒伴泦緹や腑浠ユ弧瓚沖簲鐢ㄧ殑瑕佹眰銆傝繖浜涘潎琛¤鍒掔洰鍓嶈繕娌℃湁瀹炵幇銆?/span>
3銆佹暟鎹畬鏁存?/span>
浠庢煇涓?/span>Datanode鑾峰彇鐨勬暟鎹潡鏈夊彲鑳芥槸鎹熷潖鐨勶紝榪欎釜鎹熷潖鍙兘鏄敱浜?/span>Datanode鐨勫瓨鍌ㄨ澶囬敊璇佺綉緇滈敊璇垨鑰呰蔣浠?/span>bug閫犳垚鐨勩?/span>HDFS瀹㈡埛绔蔣浠跺疄鐜頒簡HDFS鏂囦歡鍐呭鐨勬牎楠屽拰銆傚綋鏌愪釜瀹㈡埛绔垱寤轟竴涓柊鐨?/span>HDFS鏂囦歡錛屼細璁$畻榪欎釜鏂囦歡姣忎釜block鐨勬牎楠屽拰錛屽茍浣滀負涓涓崟鐙殑闅愯棌鏂囦歡淇濆瓨榪欎簺鏍¢獙鍜屽湪鍚屼竴涓?/span>HDFS namespace涓嬨傚綋瀹㈡埛绔绱㈡枃浠跺唴瀹癸紝瀹冧細紜浠?/span>Datanode鑾峰彇鐨勬暟鎹窡鐩稿簲鐨勬牎楠屽拰鏂囦歡涓殑鏍¢獙鍜屾槸鍚﹀尮閰嶏紝濡傛灉涓嶅尮閰嶏紝瀹㈡埛绔彲浠ラ夋嫨浠庡叾浠?/span>Datanode鑾峰彇璇?/span>block鐨勫壇鏈?/span>
4銆佸厓鏁版嵁紓佺洏閿欒
FsImage鍜?/span>Editlog鏄?/span>HDFS鐨勬牳蹇冩暟鎹粨鏋勩傝繖浜涙枃浠跺鏋滄崯鍧忎簡錛屾暣涓?/span>HDFS瀹炰緥閮藉皢澶辨晥銆傚洜鑰岋紝Namenode鍙互閰嶇疆鎴愭敮鎸佺淮鎶ゅ涓?/span>FsImage鍜?/span>Editlog鐨勬嫹璐濄備換浣曞FsImage鎴栬?/span>Editlog鐨勪慨鏀癸紝閮藉皢鍚屾鍒板畠浠殑鍓湰涓娿傝繖涓悓姝ユ搷浣滃彲鑳戒細闄嶄綆 Namenode姣忕鑳芥敮鎸佸鐞嗙殑namespace浜嬪姟銆傝繖涓唬浠鋒槸鍙互鎺ュ彈鐨勶紝鍥犱負HDFS鏄暟鎹瘑闆嗙殑錛岃岄潪鍏冩暟鎹瘑闆嗐傚綋Namenode閲嶅惎鐨勬椂鍊欙紝瀹冩繪槸閫夊彇鏈榪戠殑涓鑷寸殑FsImage鍜?/span>Editlog浣跨敤銆?/span>
Namenode鍦?/span>HDFS鏄崟鐐瑰瓨鍦紝濡傛灉Namenode鎵鍦ㄧ殑鏈哄櫒閿欒錛屾墜宸ョ殑騫查鏄繀欏葷殑銆傜洰鍓嶏紝鍦ㄥ彟涓鍙版満鍣ㄤ笂閲嶅惎鍥犳晠闅滆屽仠姝㈡湇鍔$殑Namenode榪欎釜鍔熻兘榪樻病瀹炵幇銆?/span>
5銆佸揩鐓?/span>
蹇収鏀寔鏌愪釜鏃墮棿鐨勬暟鎹嫹璐濓紝褰?/span>HDFS鏁版嵁鎹熷潖鐨勬椂鍊欙紝鍙互鎭㈠鍒拌繃鍘諱竴涓凡鐭ユ紜殑鏃墮棿鐐廣?/span>HDFS鐩墠榪樹笉鏀寔蹇収鍔熻兘銆?/span>
鍏佹暟鎹粍緇?/span>
1銆佹暟鎹潡
鍏煎HDFS鐨勫簲鐢ㄩ兘鏄鐞嗗ぇ鏁版嵁闆嗗悎鐨勩傝繖浜涘簲鐢ㄩ兘鏄啓鏁版嵁涓嬈★紝璇誨嵈鏄竴嬈″埌澶氭錛屽茍涓旇鐨勯熷害瑕佹弧瓚蟲祦寮忚銆?/span>HDFS鏀寔鏂囦歡鐨?/span>write- once-read-many璇箟銆備竴涓吀鍨嬬殑block澶у皬鏄?/span>64MB錛屽洜鑰岋紝鏂囦歡鎬繪槸鎸夌収64M鍒囧垎鎴?/span>chunk錛屾瘡涓?/span>chunk瀛樺偍浜庝笉鍚岀殑 Datanode
2銆佹楠?/span>
鏌愪釜瀹㈡埛绔垱寤烘枃浠剁殑璇鋒眰鍏跺疄騫舵病鏈夌珛鍗沖彂緇?/span>Namenode錛屼簨瀹炰笂錛?/span>HDFS瀹㈡埛绔細灝嗘枃浠舵暟鎹紦瀛樺埌鏈湴鐨勪竴涓復鏃舵枃浠躲傚簲鐢ㄧ殑鍐欒閫忔槑鍦伴噸瀹氬悜鍒拌繖涓復鏃舵枃浠躲傚綋榪欎釜涓存椂鏂囦歡绱Н鐨勬暟鎹秴榪囦竴涓?/span>block鐨勫ぇ灝忥紙榛樿64M)錛屽鎴風鎵嶄細鑱旂郴Namenode銆?/span>Namenode灝嗘枃浠跺悕鎻掑叆鏂囦歡緋葷粺鐨勫眰嬈$粨鏋勪腑錛屽茍涓斿垎閰嶄竴涓暟鎹潡緇欏畠錛岀劧鍚庤繑鍥?/span>Datanode鐨勬爣璇嗙鍜岀洰鏍囨暟鎹潡緇欏鎴風銆傚鎴風灝嗘湰鍦頒復鏃舵枃浠?/span>flush鍒版寚瀹氱殑 Datanode涓娿傚綋鏂囦歡鍏抽棴鏃訛紝鍦ㄤ復鏃舵枃浠朵腑鍓╀綑鐨勬病鏈?/span>flush鐨勬暟鎹篃浼氫紶杈撳埌鎸囧畾鐨?/span>Datanode錛岀劧鍚庡鎴風鍛婅瘔Namenode鏂囦歡宸茬粡鍏抽棴銆傛鏃?/span>Namenode鎵嶅皢鏂囦歡鍒涘緩鎿嶄綔鎻愪氦鍒版寔涔呭瓨鍌ㄣ傚鏋?/span>Namenode鍦ㄦ枃浠跺叧闂墠鎸備簡錛岃鏂囦歡灝嗕涪澶便?/span>
涓婅堪鏂規硶鏄閫氳繃瀵?/span>HDFS涓婅繍琛岀殑鐩爣搴旂敤璁ょ湡鑰冭檻鐨勭粨鏋溿傚鏋滀笉閲囩敤瀹㈡埛绔紦瀛橈紝鐢變簬緗戠粶閫熷害鍜岀綉緇滃牭濉炰細瀵瑰悶浼伴噺閫犳垚姣旇緝澶х殑褰卞搷銆?/span>
3銆佹祦姘寸嚎澶嶅埗
褰撴煇涓鎴風鍚?/span>HDFS鏂囦歡鍐欐暟鎹殑鏃跺欙紝涓寮濮嬫槸鍐欏叆鏈湴涓存椂鏂囦歡錛屽亣璁捐鏂囦歡鐨?/span>replication鍥犲瓙璁劇疆涓?/span>3錛岄偅涔堝鎴風浼氫粠Namenode 鑾峰彇涓寮?/span>Datanode鍒楄〃鏉ュ瓨鏀懼壇鏈傜劧鍚庡鎴風寮濮嬪悜絎竴涓?/span>Datanode浼犺緭鏁版嵁錛岀涓涓?/span>Datanode涓灝忛儴鍒嗕竴灝忛儴鍒嗭紙4kb)鍦版帴鏀舵暟鎹紝灝嗘瘡涓儴鍒嗗啓鍏ユ湰鍦頒粨搴擄紝騫朵笖鍚屾椂浼犺緭璇ラ儴鍒嗗埌絎簩涓?/span>Datanode鑺傜偣銆傜浜屼釜Datanode涔熸槸榪欐牱錛岃竟鏀惰竟浼狅紝涓灝忛儴鍒嗕竴灝忛儴鍒嗗湴鏀訛紝瀛樺偍鍦ㄦ湰鍦頒粨搴擄紝鍚屾椂浼犵粰絎笁涓?/span>Datanode錛岀涓変釜Datanode灝變粎浠呮槸鎺ユ敹騫跺瓨鍌ㄤ簡銆傝繖灝辨槸嫻佹按綰垮紡鐨勫鍒躲?/span>
涔濄佸彲璁塊棶鎬?/span>
HDFS緇欏簲鐢ㄦ彁渚涗簡澶氱璁塊棶鏂瑰紡錛屽彲浠ラ氳繃DFSShell閫氳繃鍛戒護琛屼笌HDFS鏁版嵁榪涜浜や簰錛屽彲浠ラ氳繃java API璋冪敤錛屼篃鍙互閫氳繃C璇█鐨勫皝瑁?/span>API璁塊棶錛屽茍涓旀彁渚涗簡嫻忚鍣ㄨ闂殑鏂瑰紡銆傛鍦ㄥ紑鍙戦氳繃WebDav鍗忚璁塊棶鐨勬柟寮忋傚叿浣撲嬌鐢ㄥ弬鑰冩枃妗c?/span>
鍗併佺┖闂寸殑鍥炴敹
1銆佹枃浠剁殑鍒犻櫎鍜屾仮澶?/span>
鐢ㄦ埛鎴栬呭簲鐢ㄥ垹闄ゆ煇涓枃浠訛紝榪欎釜鏂囦歡騫舵病鏈夌珛鍒諱粠HDFS涓垹闄ゃ傜浉鍙嶏紝HDFS灝嗚繖涓枃浠墮噸鍛藉悕錛屽茍杞Щ鍒?/span>/trash鐩綍銆傚綋鏂囦歡榪樺湪/trash鐩綍鏃訛紝璇ユ枃浠跺彲浠ヨ榪呴熷湴鎭㈠銆傛枃浠跺湪/trash涓繚瀛樼殑鏃墮棿鏄彲閰嶇疆鐨勶紝褰撹秴榪囪繖涓椂闂達紝Namenode灝變細灝嗚鏂囦歡浠?/span>namespace涓垹闄ゃ傛枃浠剁殑鍒犻櫎錛屼篃灝嗛噴鏀懼叧鑱旇鏂囦歡鐨勬暟鎹潡銆傛敞鎰忓埌錛屽湪鏂囦歡琚敤鎴峰垹闄ゅ拰HDFS絀洪棽絀洪棿鐨勫鍔犱箣闂翠細鏈変竴涓瓑寰呮椂闂村歡榪熴?/span>
褰撹鍒犻櫎鐨勬枃浠惰繕淇濈暀鍦?/span>/trash鐩綍涓殑鏃跺欙紝濡傛灉鐢ㄦ埛鎯蟲仮澶嶈繖涓枃浠訛紝鍙互媯绱㈡祻瑙?/span>/trash鐩綍騫舵绱㈣鏂囦歡銆?/span>/trash鐩綍浠呬粎淇濆瓨琚垹闄ゆ枃浠剁殑鏈榪戜竴嬈℃嫹璐濄?/span>/trash鐩綍涓庡叾浠栨枃浠剁洰褰曟病鏈変粈涔堜笉鍚岋紝闄や簡涓鐐癸細HDFS鍦ㄨ鐩綍涓婂簲鐢ㄤ簡涓涓壒孌婄殑絳栫暐鏉ヨ嚜鍔ㄥ垹闄ゆ枃浠訛紝鐩墠鐨勯粯璁ょ瓥鐣ユ槸鍒犻櫎淇濈暀瓚呰繃6灝忔椂鐨勬枃浠訛紝榪欎釜絳栫暐浠ュ悗浼氬畾涔夋垚鍙厤緗殑鎺ュ彛銆?/span>
2銆?/span>Replication鍥犲瓙鐨勫噺灝?/span>
褰撴煇涓枃浠剁殑replication鍥犲瓙鍑忓皬錛?/span>Namenode浼氶夋嫨瑕佸垹闄ょ殑榪囧墿鐨勫壇鏈備笅嬈″績璺蟲嫻嬪氨灝嗚淇℃伅浼犻掔粰Datanode錛?Datanode灝變細縐婚櫎鐩稿簲鐨?/span>block騫墮噴鏀劇┖闂達紝鍚屾牱錛屽湪璋冪敤setReplication鏂規硶鍜岄泦緹や腑鐨勭┖闂茬┖闂村鍔犱箣闂翠細鏈変竴涓椂闂村歡榪熴?/span>
鍙傝冭祫鏂欙細
HDFS Java API: http://hadoop.apache.org/core/docs/current/api/
HDFS source code: http://hadoop.apache.org/core/version_control.html
鍏堝喅鏉′歡
璇峰厛紜Hadoop琚紜畨瑁呫侀厤緗拰姝e父榪愯涓傛洿澶氫俊鎭錛?/p>
Hadoop蹇熷叆闂ㄥ鍒濇浣跨敤鑰呫?
Hadoop闆嗙兢鎼緩瀵瑰ぇ瑙勬ā鍒嗗竷寮忛泦緹ゃ?
姒傝堪
Hadoop Map/Reduce鏄竴涓嬌鐢ㄧ畝鏄撶殑杞歡妗嗘灦錛屽熀浜庡畠鍐欏嚭鏉ョ殑搴旂敤紼嬪簭鑳藉榪愯鍦ㄧ敱涓婂崈涓晢鐢ㄦ満鍣ㄧ粍鎴愮殑澶у瀷闆嗙兢涓婏紝騫朵互涓縐嶅彲闈犲閿欑殑鏂瑰紡騫惰澶勭悊涓奣綰у埆鐨勬暟鎹泦銆?/p>
涓涓狹ap/Reduce 浣滀笟錛坖ob錛?閫氬父浼氭妸杈撳叆鐨勬暟鎹泦鍒囧垎涓鴻嫢騫茬嫭绔嬬殑鏁版嵁鍧楋紝鐢?map浠誨姟錛坱ask錛変互瀹屽叏騫惰鐨勬柟寮忓鐞嗗畠浠傛鏋朵細瀵筸ap鐨勮緭鍑哄厛榪涜鎺掑簭錛?鐒跺悗鎶婄粨鏋滆緭鍏ョ粰reduce浠誨姟銆傞氬父浣滀笟鐨勮緭鍏ュ拰杈撳嚭閮戒細琚瓨鍌ㄥ湪鏂囦歡緋葷粺涓?鏁翠釜妗嗘灦璐熻矗浠誨姟鐨勮皟搴﹀拰鐩戞帶錛屼互鍙婇噸鏂版墽琛屽凡緇忓け璐ョ殑浠誨姟銆?/p>
閫氬父錛孧ap/Reduce妗嗘灦鍜屽垎甯冨紡鏂囦歡緋葷粺鏄繍琛屽湪涓緇勭浉鍚岀殑鑺傜偣涓婄殑錛屼篃灝辨槸璇達紝璁$畻鑺傜偣鍜屽瓨鍌ㄨ妭鐐歸氬父鍦ㄤ竴璧楓傝繖縐嶉厤緗厑璁告鏋跺湪閭d簺宸茬粡瀛樺ソ鏁版嵁鐨勮妭鐐逛笂楂樻晥鍦拌皟搴︿換鍔★紝榪欏彲浠ヤ嬌鏁翠釜闆嗙兢鐨勭綉緇滃甫瀹借闈炲父楂樻晥鍦板埄鐢ㄣ?/p>
Map/Reduce妗嗘灦鐢變竴涓崟鐙殑master JobTracker 鍜屾瘡涓泦緹よ妭鐐逛竴涓猻lave TaskTracker鍏卞悓緇勬垚銆俶aster璐熻矗璋冨害鏋勬垚涓涓綔涓氱殑鎵鏈変換鍔★紝榪欎簺浠誨姟鍒嗗竷鍦ㄤ笉鍚岀殑slave涓婏紝master鐩戞帶瀹冧滑鐨勬墽琛岋紝閲嶆柊鎵ц宸茬粡澶辮觸鐨勪換鍔°傝宻lave浠呰礋璐f墽琛岀敱master鎸囨淳鐨勪換鍔°?/p>
搴旂敤紼嬪簭鑷沖皯搴旇鎸囨槑杈撳叆/杈撳嚭鐨勪綅緗紙璺緞錛夛紝騫墮氳繃瀹炵幇鍚堥傜殑鎺ュ彛鎴栨娊璞$被鎻愪緵map鍜宺educe鍑芥暟銆傚啀鍔犱笂鍏朵粬浣滀笟鐨勫弬鏁幫紝灝辨瀯鎴愪簡浣滀笟閰嶇疆錛坖ob configuration錛夈傜劧鍚庯紝Hadoop鐨?job client鎻愪氦浣滀笟錛坖ar鍖?鍙墽琛岀▼搴忕瓑錛夊拰閰嶇疆淇℃伅緇橨obTracker錛屽悗鑰呰礋璐e垎鍙戣繖浜涜蔣浠跺拰閰嶇疆淇℃伅緇檚lave銆佽皟搴︿換鍔″茍鐩戞帶瀹冧滑鐨勬墽琛岋紝鍚屾椂鎻愪緵鐘舵佸拰璇婃柇淇℃伅緇檍ob-client銆?/p>
铏界劧Hadoop妗嗘灦鏄敤JavaTM瀹炵幇鐨勶紝浣哅ap/Reduce搴旂敤紼嬪簭鍒欎笉涓瀹氳鐢?Java鏉ュ啓 銆?/p>
Hadoop Streaming鏄竴縐嶈繍琛屼綔涓氱殑瀹炵敤宸ュ叿錛屽畠鍏佽鐢ㄦ埛鍒涘緩鍜岃繍琛屼換浣曞彲鎵ц紼嬪簭 錛堜緥濡傦細Shell宸ュ叿錛夋潵鍋氫負mapper鍜宺educer銆?
Hadoop Pipes鏄竴涓笌SWIG鍏煎鐨凜++ API 錛堟病鏈夊熀浜嶫NITM鎶鏈級錛屽畠涔熷彲鐢ㄤ簬瀹炵幇Map/Reduce搴旂敤紼嬪簭銆?
杈撳叆涓庤緭鍑?br>Map/Reduce妗嗘灦榪愯漿鍦?lt;key, value> 閿煎涓婏紝涔熷氨鏄錛?妗嗘灦鎶婁綔涓氱殑杈撳叆鐪嬩負鏄竴緇?lt;key, value> 閿煎錛屽悓鏍蜂篃浜у嚭涓緇?<key, value> 閿煎鍋氫負浣滀笟鐨勮緭鍑猴紝榪欎袱緇勯敭鍊煎鐨勭被鍨嬪彲鑳戒笉鍚屻?/p>
妗嗘灦闇瑕佸key鍜寁alue鐨勭被(classes)榪涜搴忓垪鍖栨搷浣滐紝 鍥犳錛岃繖浜涚被闇瑕佸疄鐜?Writable鎺ュ彛銆?鍙﹀錛屼負浜嗘柟渚挎鏋舵墽琛屾帓搴忔搷浣滐紝key綾誨繀欏誨疄鐜?WritableComparable鎺ュ彛銆?/p>
涓涓狹ap/Reduce 浣滀笟鐨勮緭鍏ュ拰杈撳嚭綾誨瀷濡備笅鎵紺猴細
(input) <k1, v1> -> map -> <k2, v2> -> combine -> <k2, v2> -> reduce -> <k3, v3> (output)
渚嬪瓙錛歐ordCount v1.0
鍦ㄦ繁鍏ョ粏鑺備箣鍓嶏紝璁╂垜浠厛鐪嬩竴涓狹ap/Reduce鐨勫簲鐢ㄧず渚嬶紝浠ヤ究瀵瑰畠浠殑宸ヤ綔鏂瑰紡鏈変竴涓垵姝ョ殑璁よ瘑銆?/p>
WordCount鏄竴涓畝鍗曠殑搴旂敤錛屽畠鍙互璁$畻鍑烘寚瀹氭暟鎹泦涓瘡涓涓崟璇嶅嚭鐜扮殑嬈℃暟銆?/p>
榪欎釜搴旂敤閫傜敤浜?鍗曟満妯″紡錛?浼垎甯冨紡妯″紡 鎴?瀹屽叏鍒嗗竷寮忔ā寮?涓夌Hadoop瀹夎鏂瑰紡銆?/p>
婧愪唬鐮?br> WordCount.java
1. package org.myorg;
2.
3. import java.io.IOException;
4. import java.util.*;
5.
6. import org.apache.hadoop.fs.Path;
7. import org.apache.hadoop.conf.*;
8. import org.apache.hadoop.io.*;
9. import org.apache.hadoop.mapred.*;
10. import org.apache.hadoop.util.*;
11.
12. public class WordCount {
13.
14. public static class Map extends MapReduceBase implements Mapper<LongWritable, Text, Text, IntWritable> {
15. private final static IntWritable one = new IntWritable(1);
16. private Text word = new Text();
17.
18. public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {
19. String line = value.toString();
20. StringTokenizer tokenizer = new StringTokenizer(line);
21. while (tokenizer.hasMoreTokens()) {
22. word.set(tokenizer.nextToken());
23. output.collect(word, one);
24. }
25. }
26. }
27.
28. public static class Reduce extends MapReduceBase implements Reducer<Text, IntWritable, Text, IntWritable> {
29. public void reduce(Text key, Iterator<IntWritable> values, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {
30. int sum = 0;
31. while (values.hasNext()) {
32. sum += values.next().get();
33. }
34. output.collect(key, new IntWritable(sum));
35. }
36. }
37.
38. public static void main(String[] args) throws Exception {
39. JobConf conf = new JobConf(WordCount.class);
40. conf.setJobName("wordcount");
41.
42. conf.setOutputKeyClass(Text.class);
43. conf.setOutputValueClass(IntWritable.class);
44.
45. conf.setMapperClass(Map.class);
46. conf.setCombinerClass(Reduce.class);
47. conf.setReducerClass(Reduce.class);
48.
49. conf.setInputFormat(TextInputFormat.class);
50. conf.setOutputFormat(TextOutputFormat.class);
51.
52. FileInputFormat.setInputPaths(conf, new Path(args[0]));
53. FileOutputFormat.setOutputPath(conf, new Path(args[1]));
54.
55. JobClient.runJob(conf);
57. }
58. }
59.
鐢ㄦ硶
鍋囪鐜鍙橀噺HADOOP_HOME瀵瑰簲瀹夎鏃剁殑鏍圭洰褰曪紝HADOOP_VERSION瀵瑰簲Hadoop鐨勫綋鍓嶅畨瑁呯増鏈紝緙栬瘧WordCount.java鏉ュ垱寤簀ar鍖咃紝鍙涓嬫搷浣滐細
$ mkdir wordcount_classes
$ javac -classpath ${HADOOP_HOME}/hadoop-${HADOOP_VERSION}-core.jar -d wordcount_classes WordCount.java
$ jar -cvf /usr/joe/wordcount.jar -C wordcount_classes/ .
鍋囪錛?/p>
/usr/joe/wordcount/input - 鏄疕DFS涓殑杈撳叆璺緞
/usr/joe/wordcount/output - 鏄疕DFS涓殑杈撳嚭璺緞
鐢ㄧず渚嬫枃鏈枃浠跺仛涓鴻緭鍏ワ細
$ bin/hadoop dfs -ls /usr/joe/wordcount/input/
/usr/joe/wordcount/input/file01
/usr/joe/wordcount/input/file02
$ bin/hadoop dfs -cat /usr/joe/wordcount/input/file01
Hello World Bye World
$ bin/hadoop dfs -cat /usr/joe/wordcount/input/file02
Hello Hadoop Goodbye Hadoop
榪愯搴旂敤紼嬪簭錛?/p>
$ bin/hadoop jar /usr/joe/wordcount.jar org.myorg.WordCount /usr/joe/wordcount/input /usr/joe/wordcount/output
杈撳嚭鏄細
$ bin/hadoop dfs -cat /usr/joe/wordcount/output/part-00000
Bye 1
Goodbye 1
Hadoop 2
Hello 2
World 2
搴旂敤紼嬪簭鑳藉浣跨敤-files閫夐」鏉ユ寚瀹氫竴涓敱閫楀彿鍒嗛殧鐨勮礬寰勫垪琛紝榪欎簺璺緞鏄痶ask鐨勫綋鍓嶅伐浣滅洰褰曘備嬌鐢ㄩ夐」-libjars鍙互鍚憁ap鍜宺educe鐨刢lasspath涓坊鍔爅ar鍖呫備嬌鐢?archives閫夐」紼嬪簭鍙互浼犻掓。妗堟枃浠跺仛涓哄弬鏁幫紝榪欎簺妗f鏂囦歡浼氳瑙e帇騫朵笖鍦╰ask鐨勫綋鍓嶅伐浣滅洰褰曚笅浼氬垱寤轟竴涓寚鍚戣В鍘嬬敓鎴愮殑鐩綍鐨勭鍙烽摼鎺ワ紙浠ュ帇緙╁寘鐨勫悕瀛楀懡鍚嶏級銆?鏈夊叧鍛戒護琛岄夐」鐨勬洿澶氱粏鑺傝鍙傝?Commands manual銆?/p>
浣跨敤-libjars鍜?files榪愯wordcount渚嬪瓙錛?br>hadoop jar hadoop-examples.jar wordcount -files cachefile.txt -libjars mylib.jar input output
瑙i噴
WordCount搴旂敤紼嬪簭闈炲父鐩存埅浜嗗綋銆?/p>
Mapper(14-26琛?涓殑map鏂規硶(18-25琛?閫氳繃鎸囧畾鐨?TextInputFormat(49琛?涓嬈″鐞嗕竴琛屻傜劧鍚庯紝瀹冮氳繃StringTokenizer 浠ョ┖鏍間負鍒嗛殧絎﹀皢涓琛屽垏鍒嗕負鑻ュ共tokens錛屼箣鍚庯紝杈撳嚭< <word>, 1> 褰㈠紡鐨勯敭鍊煎銆?/p>
瀵逛簬紺轟緥涓殑絎竴涓緭鍏ワ紝map杈撳嚭鏄細
< Hello, 1>
< World, 1>
< Bye, 1>
< World, 1>
絎簩涓緭鍏ワ紝map杈撳嚭鏄細
< Hello, 1>
< Hadoop, 1>
< Goodbye, 1>
< Hadoop, 1>
鍏充簬緇勬垚涓涓寚瀹氫綔涓氱殑map鏁扮洰鐨勭‘瀹氾紝浠ュ強濡備綍浠ユ洿綺劇粏鐨勬柟寮忓幓鎺у埗榪欎簺map錛屾垜浠皢鍦ㄦ暀紼嬬殑鍚庣畫閮ㄥ垎瀛︿範鍒版洿澶氱殑鍐呭銆?/p>
WordCount榪樻寚瀹氫簡涓涓猚ombiner (46琛?銆傚洜姝わ紝姣忔map榪愯涔嬪悗錛屼細瀵硅緭鍑烘寜鐓ey榪涜鎺掑簭錛岀劧鍚庢妸杈撳嚭浼犻掔粰鏈湴鐨刢ombiner錛堟寜鐓т綔涓氱殑閰嶇疆涓嶳educer涓鏍鳳級錛岃繘琛屾湰鍦拌仛鍚堛?/p>
絎竴涓猰ap鐨勮緭鍑烘槸錛?br>< Bye, 1>
< Hello, 1>
< World, 2>
絎簩涓猰ap鐨勮緭鍑烘槸錛?br>< Goodbye, 1>
< Hadoop, 2>
< Hello, 1>
Reducer(28-36琛?涓殑reduce鏂規硶(29-35琛? 浠呮槸灝嗘瘡涓猭ey錛堟湰渚嬩腑灝辨槸鍗曡瘝錛夊嚭鐜扮殑嬈℃暟姹傚拰銆?/p>
鍥犳榪欎釜浣滀笟鐨勮緭鍑哄氨鏄細
< Bye, 1>
< Goodbye, 1>
< Hadoop, 2>
< Hello, 2>
< World, 2>
浠g爜涓殑run鏂規硶涓寚瀹氫簡浣滀笟鐨勫嚑涓柟闈紝 渚嬪錛氶氳繃鍛戒護琛屼紶閫掕繃鏉ョ殑杈撳叆/杈撳嚭璺緞銆乲ey/value鐨勭被鍨嬨佽緭鍏?杈撳嚭鐨勬牸寮忕瓑絳塉obConf涓殑閰嶇疆淇℃伅銆傞殢鍚庣▼搴忚皟鐢ㄤ簡JobClient.runJob(55琛?鏉ユ彁浜や綔涓氬茍涓旂洃鎺у畠鐨勬墽琛屻?/p>
鎴戜滑灝嗗湪鏈暀紼嬬殑鍚庣畫閮ㄥ垎瀛︿範鏇村鐨勫叧浜嶫obConf錛?JobClient錛?Tool鍜屽叾浠栨帴鍙e強綾?class)銆?/p>
Map/Reduce - 鐢ㄦ埛鐣岄潰
榪欓儴鍒嗘枃妗d負鐢ㄦ埛灝嗕細闈復鐨凪ap/Reduce妗嗘灦涓殑鍚勪釜鐜妭鎻愪緵浜嗛傚綋鐨勭粏鑺傘傝繖搴旇浼氬府鍔╃敤鎴鋒洿緇嗙矑搴﹀湴鍘誨疄鐜般侀厤緗拰璋冧紭浣滀笟銆傜劧鑰岋紝璇鋒敞鎰忔瘡涓被/鎺ュ彛鐨刯avadoc鏂囨。鎻愪緵鏈鍏ㄩ潰鐨勬枃妗o紱鏈枃鍙槸鎯寵搗鍒版寚鍗楃殑浣滅敤銆?/p>
鎴戜滑浼氬厛鐪嬬湅Mapper鍜孯educer鎺ュ彛銆傚簲鐢ㄧ▼搴忛氬父浼氶氳繃鎻愪緵map鍜宺educe鏂規硶鏉ュ疄鐜板畠浠?/p>
鐒跺悗錛屾垜浠細璁ㄨ鍏朵粬鐨勬牳蹇冩帴鍙o紝鍏朵腑鍖呮嫭錛?JobConf錛孞obClient錛孭artitioner錛?OutputCollector錛孯eporter錛?InputFormat錛孫utputFormat絳夌瓑銆?/p>
鏈鍚庯紝鎴戜滑灝嗛氳繃璁ㄨ妗嗘灦涓竴浜涙湁鐢ㄧ殑鍔熻兘鐐癸紙渚嬪錛欴istributedCache錛?IsolationRunner絳夌瓑錛夋潵鏀跺熬銆?/p>
鏍稿績鍔熻兘鎻忚堪
搴旂敤紼嬪簭閫氬父浼氶氳繃鎻愪緵map鍜宺educe鏉ュ疄鐜?Mapper鍜孯educer鎺ュ彛錛屽畠浠粍鎴愪綔涓氱殑鏍稿績銆?/p>
Mapper
Mapper灝嗚緭鍏ラ敭鍊煎(key/value pair)鏄犲皠鍒頒竴緇勪腑闂存牸寮忕殑閿煎闆嗗悎銆?/p>
Map鏄竴綾誨皢杈撳叆璁板綍闆嗚漿鎹負涓棿鏍煎紡璁板綍闆嗙殑鐙珛浠誨姟銆?榪欑杞崲鐨勪腑闂存牸寮忚褰曢泦涓嶉渶瑕佷笌杈撳叆璁板綍闆嗙殑綾誨瀷涓鑷淬備竴涓粰瀹氱殑杈撳叆閿煎鍙互鏄犲皠鎴?涓垨澶氫釜杈撳嚭閿煎銆?/p>
Hadoop Map/Reduce妗嗘灦涓烘瘡涓涓狪nputSplit浜х敓涓涓猰ap浠誨姟錛岃屾瘡涓狪nputSplit鏄敱璇ヤ綔涓氱殑InputFormat浜х敓鐨勩?/p>
姒傛嫭鍦拌錛屽Mapper鐨勫疄鐜拌呴渶瑕侀噸鍐?JobConfigurable.configure(JobConf)鏂規硶錛岃繖涓柟娉曢渶瑕佷紶閫掍竴涓狫obConf鍙傛暟錛岀洰鐨勬槸瀹屾垚Mapper鐨勫垵濮嬪寲宸ヤ綔銆傜劧鍚庯紝妗嗘灦涓鴻繖涓換鍔$殑InputSplit涓瘡涓敭鍊煎璋冪敤涓嬈?map(WritableComparable, Writable, OutputCollector, Reporter)鎿嶄綔銆傚簲鐢ㄧ▼搴忓彲浠ラ氳繃閲嶅啓Closeable.close()鏂規硶鏉ユ墽琛岀浉搴旂殑娓呯悊宸ヤ綔銆?/p>
杈撳嚭閿煎涓嶉渶瑕佷笌杈撳叆閿煎鐨勭被鍨嬩竴鑷淬備竴涓粰瀹氱殑杈撳叆閿煎鍙互鏄犲皠鎴?涓垨澶氫釜杈撳嚭閿煎銆傞氳繃璋冪敤 OutputCollector.collect(WritableComparable,Writable)鍙互鏀墮泦杈撳嚭鐨勯敭鍊煎銆?/p>
搴旂敤紼嬪簭鍙互浣跨敤Reporter鎶ュ憡榪涘害錛岃瀹氬簲鐢ㄧ駭鍒殑鐘舵佹秷鎭紝鏇存柊Counters錛堣鏁板櫒錛夛紝鎴栬呬粎鏄〃鏄庤嚜宸辮繍琛屾甯搞?/p>
妗嗘灦闅忓悗浼氭妸涓庝竴涓壒瀹歬ey鍏寵仈鐨勬墍鏈変腑闂磋繃紼嬬殑鍊鹼紙value錛夊垎鎴愮粍錛岀劧鍚庢妸瀹冧滑浼犵粰Reducer浠ヤ駭鍑烘渶緇堢殑緇撴灉銆傜敤鎴峰彲浠ラ氳繃 JobConf.setOutputKeyComparatorClass(Class)鏉ユ寚瀹氬叿浣撹礋璐e垎緇勭殑 Comparator銆?/p>
Mapper鐨勮緭鍑鴻鎺掑簭鍚庯紝灝辮鍒掑垎緇欐瘡涓猂educer銆傚垎鍧楃殑鎬繪暟鐩拰涓涓綔涓氱殑reduce浠誨姟鐨勬暟鐩槸涓鏍風殑銆傜敤鎴峰彲浠ラ氳繃瀹炵幇鑷畾涔夌殑 Partitioner鏉ユ帶鍒跺摢涓猭ey琚垎閰嶇粰鍝釜 Reducer銆?/p>
鐢ㄦ埛鍙夋嫨閫氳繃 JobConf.setCombinerClass(Class)鎸囧畾涓涓猚ombiner錛屽畠璐熻矗瀵逛腑闂磋繃紼嬬殑杈撳嚭榪涜鏈湴鐨勮仛闆嗭紝榪欎細鏈夊姪浜庨檷浣庝粠Mapper鍒?Reducer鏁版嵁浼犺緭閲忋?/p>
榪欎簺琚帓濂藉簭鐨勪腑闂磋繃紼嬬殑杈撳嚭緇撴灉淇濆瓨鐨勬牸寮忔槸(key-len, key, value-len, value)錛屽簲鐢ㄧ▼搴忓彲浠ラ氳繃JobConf鎺у埗瀵硅繖浜涗腑闂寸粨鏋滄槸鍚﹁繘琛屽帇緙╀互鍙婃庝箞鍘嬬緝錛屼嬌鐢ㄥ摢縐?CompressionCodec銆?/p>
闇瑕佸灝戜釜Map錛?br>Map鐨勬暟鐩氬父鏄敱杈撳叆鏁版嵁鐨勫ぇ灝忓喅瀹氱殑錛屼竴鑸氨鏄墍鏈夎緭鍏ユ枃浠剁殑鎬誨潡錛坆lock錛夋暟銆?/p>
Map姝e父鐨勫茍琛岃妯″ぇ鑷存槸姣忎釜鑺傜偣錛坣ode錛夊ぇ綰?0鍒?00涓猰ap錛屽浜嶤PU 娑堣楄緝灝忕殑map浠誨姟鍙互璁懼埌300涓乏鍙熾傜敱浜庢瘡涓換鍔″垵濮嬪寲闇瑕佷竴瀹氱殑鏃墮棿錛屽洜姝わ紝姣旇緝鍚堢悊鐨勬儏鍐墊槸map鎵ц鐨勬椂闂磋嚦灝戣秴榪?鍒嗛挓銆?/p>
榪欐牱錛屽鏋滀綘杈撳叆10TB鐨勬暟鎹紝姣忎釜鍧楋紙block錛夌殑澶у皬鏄?28MB錛屼綘灝嗛渶瑕佸ぇ綰?2,000涓猰ap鏉ュ畬鎴愪換鍔★紝闄ら潪浣跨敤 setNumMapTasks(int)錛堟敞鎰忥細榪欓噷浠呬粎鏄妗嗘灦榪涜浜嗕竴涓彁紺?hint)錛屽疄闄呭喅瀹氬洜绱犺榪欓噷錛夊皢榪欎釜鏁板艱緗緱鏇撮珮銆?/p>
Reducer
Reducer灝嗕笌涓涓猭ey鍏寵仈鐨勪竴緇勪腑闂存暟鍊奸泦褰掔害錛坮educe錛変負涓涓洿灝忕殑鏁板奸泦銆?/p>
鐢ㄦ埛鍙互閫氳繃 JobConf.setNumReduceTasks(int)璁懼畾涓涓綔涓氫腑reduce浠誨姟鐨勬暟鐩?/p>
姒傛嫭鍦拌錛屽Reducer鐨勫疄鐜拌呴渶瑕侀噸鍐?JobConfigurable.configure(JobConf)鏂規硶錛岃繖涓柟娉曢渶瑕佷紶閫掍竴涓狫obConf鍙傛暟錛岀洰鐨勬槸瀹屾垚Reducer鐨勫垵濮嬪寲宸ヤ綔銆傜劧鍚庯紝妗嗘灦涓烘垚緇勭殑杈撳叆鏁版嵁涓殑姣忎釜<key, (list of values)>瀵硅皟鐢ㄤ竴嬈?reduce(WritableComparable, Iterator, OutputCollector, Reporter)鏂規硶銆備箣鍚庯紝搴旂敤紼嬪簭鍙互閫氳繃閲嶅啓Closeable.close()鏉ユ墽琛岀浉搴旂殑娓呯悊宸ヤ綔銆?/p>
Reducer鏈?涓富瑕侀樁孌碉細shuffle銆乻ort鍜宺educe銆?/p>
Shuffle
Reducer鐨勮緭鍏ュ氨鏄疢apper宸茬粡鎺掑ソ搴忕殑杈撳嚭銆傚湪榪欎釜闃舵錛屾鏋墮氳繃HTTP涓烘瘡涓猂educer鑾峰緱鎵鏈塎apper杈撳嚭涓笌涔嬬浉鍏崇殑鍒嗗潡銆?/p>
Sort
榪欎釜闃舵錛屾鏋跺皢鎸夌収key鐨勫煎Reducer鐨勮緭鍏ヨ繘琛屽垎緇?錛堝洜涓轟笉鍚宮apper鐨勮緭鍑轟腑鍙兘浼氭湁鐩稿悓鐨刱ey錛夈?/p>
Shuffle鍜孲ort涓や釜闃舵鏄悓鏃惰繘琛岀殑錛沵ap鐨勮緭鍑轟篃鏄竴杈硅鍙栧洖涓杈硅鍚堝茍鐨勩?/p>
Secondary Sort
濡傛灉闇瑕佷腑闂磋繃紼嬪key鐨勫垎緇勮鍒欏拰reduce鍓嶅key鐨勫垎緇勮鍒欎笉鍚岋紝閭d箞鍙互閫氳繃 JobConf.setOutputValueGroupingComparator(Class)鏉ユ寚瀹氫竴涓狢omparator銆傚啀鍔犱笂 JobConf.setOutputKeyComparatorClass(Class)鍙敤浜庢帶鍒朵腑闂磋繃紼嬬殑key濡備綍琚垎緇勶紝鎵浠ョ粨鍚堜袱鑰呭彲浠ュ疄鐜版寜鍊肩殑浜屾鎺掑簭銆?/p>
Reduce
鍦ㄨ繖涓樁孌碉紝妗嗘灦涓哄凡鍒嗙粍鐨勮緭鍏ユ暟鎹腑鐨勬瘡涓?<key, (list of values)>瀵硅皟鐢ㄤ竴嬈?reduce(WritableComparable, Iterator, OutputCollector, Reporter)鏂規硶銆?/p>
Reduce浠誨姟鐨勮緭鍑洪氬父鏄氳繃璋冪敤 OutputCollector.collect(WritableComparable, Writable)鍐欏叆 鏂囦歡緋葷粺鐨勩?/p>
搴旂敤紼嬪簭鍙互浣跨敤Reporter鎶ュ憡榪涘害錛岃瀹氬簲鐢ㄧ▼搴忕駭鍒殑鐘舵佹秷鎭紝鏇存柊Counters錛堣鏁板櫒錛夛紝鎴栬呬粎鏄〃鏄庤嚜宸辮繍琛屾甯搞?/p>
Reducer鐨勮緭鍑烘槸娌℃湁鎺掑簭鐨勩?/p>
闇瑕佸灝戜釜Reduce錛?br>Reduce鐨勬暟鐩緩璁槸0.95鎴?.75涔樹互 (<no. of nodes> * mapred.tasktracker.reduce.tasks.maximum)銆?/p>
鐢?.95錛屾墍鏈塺educe鍙互鍦╩aps涓瀹屾垚鏃跺氨绔嬪埢鍚姩錛屽紑濮嬩紶杈搈ap鐨勮緭鍑虹粨鏋溿傜敤1.75錛岄熷害蹇殑鑺傜偣鍙互鍦ㄥ畬鎴愮涓杞畆educe浠誨姟鍚庯紝鍙互寮濮嬬浜岃疆錛岃繖鏍峰彲浠ュ緱鍒版瘮杈冨ソ鐨勮礋杞藉潎琛$殑鏁堟灉銆?/p>
澧炲姞reduce鐨勬暟鐩細澧炲姞鏁翠釜妗嗘灦鐨勫紑閿錛屼絾鍙互鏀瑰杽璐熻澆鍧囪 錛岄檷浣庣敱浜庢墽琛屽け璐ュ甫鏉ョ殑璐熼潰褰卞搷銆?/p>
涓婅堪姣斾緥鍥犲瓙姣旀暣浣撴暟鐩◢灝忎竴浜涙槸涓轟簡緇欐鏋朵腑鐨勬帹嫻嬫т換鍔★紙speculative-tasks錛?鎴栧け璐ョ殑浠誨姟棰勭暀涓浜況educe鐨勮祫婧愩?/p>
鏃燫educer
濡傛灉娌℃湁褰掔害瑕佽繘琛岋紝閭d箞璁劇疆reduce浠誨姟鐨勬暟鐩負闆舵槸鍚堟硶鐨勩?/p>
榪欑鎯呭喌涓嬶紝map浠誨姟鐨勮緭鍑轟細鐩存帴琚啓鍏ョ敱 setOutputPath(Path)鎸囧畾鐨勮緭鍑鴻礬寰勩傛鏋跺湪鎶婂畠浠啓鍏ileSystem涔嬪墠娌℃湁瀵瑰畠浠繘琛屾帓搴忋?/p>
Partitioner
Partitioner鐢ㄤ簬鍒掑垎閿肩┖闂達紙key space錛夈?/p>
Partitioner璐熻矗鎺у埗map杈撳嚭緇撴灉key鐨勫垎鍓層侹ey錛堟垨鑰呬竴涓猭ey瀛愰泦錛夎鐢ㄤ簬浜х敓鍒嗗尯錛岄氬父浣跨敤鐨勬槸Hash鍑芥暟銆傚垎鍖虹殑鏁扮洰涓庝竴涓綔涓氱殑reduce浠誨姟鐨勬暟鐩槸涓鏍風殑銆傚洜姝わ紝瀹冩帶鍒跺皢涓棿榪囩▼鐨刱ey錛堜篃灝辨槸榪欐潯璁板綍錛夊簲璇ュ彂閫佺粰m涓猺educe浠誨姟涓殑鍝竴涓潵榪涜reduce鎿嶄綔銆?/p>
HashPartitioner鏄粯璁ょ殑 Partitioner銆?/p>
Reporter
Reporter鏄敤浜嶮ap/Reduce搴旂敤紼嬪簭鎶ュ憡榪涘害錛岃瀹氬簲鐢ㄧ駭鍒殑鐘舵佹秷鎭紝 鏇存柊Counters錛堣鏁板櫒錛夌殑鏈哄埗銆?/p>
Mapper鍜孯educer鐨勫疄鐜板彲浠ュ埄鐢≧eporter 鏉ユ姤鍛婅繘搴︼紝鎴栬呬粎鏄〃鏄庤嚜宸辮繍琛屾甯搞傚湪閭g搴旂敤紼嬪簭闇瑕佽姳寰堥暱鏃墮棿澶勭悊涓埆閿煎鐨勫満鏅腑錛岃繖縐嶆満鍒舵槸寰堝叧閿殑錛屽洜涓烘鏋跺彲鑳戒細浠ヤ負榪欎釜浠誨姟瓚呮椂浜嗭紝浠庤屽皢瀹冨己琛屾潃姝匯傚彟涓涓伩鍏嶈繖縐嶆儏鍐靛彂鐢熺殑鏂瑰紡鏄紝灝嗛厤緗弬鏁癿apred.task.timeout璁劇疆涓轟竴涓凍澶熼珮鐨勫鹼紙鎴栬呭共鑴嗚緗負闆訛紝鍒欐病鏈夎秴鏃墮檺鍒朵簡錛夈?/p>
搴旂敤紼嬪簭鍙互鐢≧eporter鏉ユ洿鏂癈ounter錛堣鏁板櫒錛夈?/p>
OutputCollector
OutputCollector鏄竴涓狹ap/Reduce妗嗘灦鎻愪緵鐨勭敤浜庢敹闆?Mapper鎴朢educer杈撳嚭鏁版嵁鐨勯氱敤鏈哄埗 錛堝寘鎷腑闂磋緭鍑虹粨鏋滃拰浣滀笟鐨勮緭鍑虹粨鏋滐級銆?/p>
Hadoop Map/Reduce妗嗘灦闄勫甫浜嗕竴涓寘鍚澶氬疄鐢ㄥ瀷鐨刴apper銆乺educer鍜宲artitioner 鐨勭被搴撱?/p>
浣滀笟閰嶇疆
JobConf浠h〃涓涓狹ap/Reduce浣滀笟鐨勯厤緗?/p>
JobConf鏄敤鎴峰悜Hadoop妗嗘灦鎻忚堪涓涓狹ap/Reduce浣滀笟濡備綍鎵ц鐨勪富瑕佹帴鍙c傛鏋朵細鎸夌収JobConf鎻忚堪鐨勪俊鎭繝瀹炲湴鍘誨皾璇曞畬鎴愯繖涓綔涓氾紝鐒惰岋細
涓浜涘弬鏁板彲鑳戒細琚鐞嗚呮爣璁頒負 final錛岃繖鎰忓懗瀹冧滑涓嶈兘琚洿鏀廣?
涓浜涗綔涓氱殑鍙傛暟鍙互琚洿鎴簡褰撳湴榪涜璁劇疆錛堜緥濡傦細 setNumReduceTasks(int)錛夛紝鑰屽彟涓浜涘弬鏁板垯涓庢鏋舵垨鑰呬綔涓氱殑鍏朵粬鍙傛暟涔嬮棿寰鍦扮浉浜掑獎鍝嶏紝騫朵笖璁劇疆璧鋒潵姣旇緝澶嶆潅錛堜緥濡傦細 setNumMapTasks(int)錛夈?
閫氬父錛孞obConf浼氭寚鏄嶮apper銆丆ombiner(濡傛灉鏈夌殑璇?銆?Partitioner銆丷educer銆両nputFormat鍜?OutputFormat鐨勫叿浣撳疄鐜般侸obConf榪樿兘鎸囧畾涓緇勮緭鍏ユ枃浠?(setInputPaths(JobConf, Path...) /addInputPath(JobConf, Path)) 鍜?setInputPaths(JobConf, String) /addInputPaths(JobConf, String)) 浠ュ強杈撳嚭鏂囦歡搴旇鍐欏湪鍝効 (setOutputPath(Path))銆?/p>
JobConf鍙夋嫨鍦板浣滀笟璁劇疆涓浜涢珮綰ч夐」錛屼緥濡傦細璁劇疆Comparator錛?鏀懼埌DistributedCache涓婄殑鏂囦歡錛涗腑闂寸粨鏋滄垨鑰呬綔涓氳緭鍑虹粨鏋滄槸鍚﹂渶瑕佸帇緙╀互鍙婃庝箞鍘嬬緝錛?鍒╃敤鐢ㄦ埛鎻愪緵鐨勮剼鏈?setMapDebugScript(String)/setReduceDebugScript(String)) 榪涜璋冭瘯錛涗綔涓氭槸鍚﹀厑璁擱闃叉э紙speculative錛変換鍔$殑鎵ц (setMapSpeculativeExecution(boolean))/(setReduceSpeculativeExecution(boolean)) 錛涙瘡涓換鍔℃渶澶х殑灝濊瘯嬈℃暟 (setMaxMapAttempts(int)/setMaxReduceAttempts(int)) 錛涗竴涓綔涓氳兘瀹瑰繊鐨勪換鍔″け璐ョ殑鐧懼垎姣?(setMaxMapTaskFailuresPercent(int)/setMaxReduceTaskFailuresPercent(int)) 錛涚瓑絳夈?/p>
褰撶劧錛岀敤鎴瘋兘浣跨敤 set(String, String)/get(String, String) 鏉ヨ緗垨鑰呭彇寰楀簲鐢ㄧ▼搴忛渶瑕佺殑浠繪剰鍙傛暟銆傜劧鑰岋紝DistributedCache鐨勪嬌鐢ㄦ槸闈㈠悜澶ц妯″彧璇繪暟鎹殑銆?/p>
浠誨姟鐨勬墽琛屽拰鐜
TaskTracker鏄湪涓涓崟鐙殑jvm涓婁互瀛愯繘紼嬬殑褰㈠紡鎵ц Mapper/Reducer浠誨姟錛圱ask錛夌殑銆?/p>
瀛愪換鍔′細緇ф壙鐖禩askTracker鐨勭幆澧冦傜敤鎴峰彲浠ラ氳繃JobConf涓殑 mapred.child.java.opts閰嶇疆鍙傛暟鏉ヨ瀹氬瓙jvm涓婄殑闄勫姞閫夐」錛屼緥濡傦細 閫氳繃-Djava.library.path=<> 灝嗕竴涓潪鏍囧噯璺緞璁句負榪愯鏃剁殑閾炬帴鐢ㄤ互鎼滅儲鍏變韓搴擄紝絳夌瓑銆傚鏋渕apred.child.java.opts鍖呭惈涓涓鍙稝taskid@錛?瀹冧細琚浛鎹㈡垚map/reduce鐨則askid鐨勫箋?/p>
涓嬮潰鏄竴涓寘鍚涓弬鏁板拰鏇挎崲鐨勪緥瀛愶紝鍏朵腑鍖呮嫭錛氳褰昷vm GC鏃ュ織錛?JVM JMX浠g悊紼嬪簭浠ユ棤瀵嗙爜鐨勬柟寮忓惎鍔紝榪欐牱瀹冨氨鑳借繛鎺ュ埌jconsole涓婏紝浠庤屽彲浠ユ煡鐪嬪瓙榪涚▼鐨勫唴瀛樺拰綰跨▼錛屽緱鍒扮嚎紼嬬殑dump錛涜繕鎶婂瓙jvm鐨勬渶澶у爢灝哄璁劇疆涓?12MB錛?騫朵負瀛恓vm鐨刯ava.library.path娣誨姞浜嗕竴涓檮鍔犺礬寰勩?/p>
<property>
<name>mapred.child.java.opts</name>
<value>
-Xmx512M -Djava.library.path=/home/mycompany/lib -verbose:gc -Xloggc:/tmp/@taskid@.gc
-Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false
</value>
</property>
鐢ㄦ埛鎴栫鐞嗗憳涔熷彲浠ヤ嬌鐢╩apred.child.ulimit璁懼畾榪愯鐨勫瓙浠誨姟鐨勬渶澶ц櫄鎷熷唴瀛樸俶apred.child.ulimit鐨勫間互錛圞B)涓哄崟浣嶏紝騫朵笖蹇呴』澶т簬鎴栫瓑浜?Xmx鍙傛暟浼犵粰JavaVM鐨勫鹼紝鍚﹀垯VM浼氭棤娉曞惎鍔ㄣ?/p>
娉ㄦ剰錛歮apred.child.java.opts鍙敤浜庤緗畉ask tracker鍚姩鐨勫瓙浠誨姟銆備負瀹堟姢榪涚▼璁劇疆鍐呭瓨閫夐」璇鋒煡鐪?cluster_setup.html
${mapred.local.dir}/taskTracker/鏄痶ask tracker鐨勬湰鍦扮洰褰曪紝 鐢ㄤ簬鍒涘緩鏈湴緙撳瓨鍜宩ob銆傚畠鍙互鎸囧畾澶氫釜鐩綍錛堣法瓚婂涓鐩橈級錛屾枃浠朵細鍗婇殢鏈虹殑淇濆瓨鍒版湰鍦拌礬寰勪笅鐨勬煇涓洰褰曘傚綋job鍚姩鏃訛紝task tracker鏍規嵁閰嶇疆鏂囨。鍒涘緩鏈湴job鐩綍錛岀洰褰曠粨鏋勫浠ヤ笅鎵紺猴細
${mapred.local.dir}/taskTracker/archive/ :鍒嗗竷寮忕紦瀛樸傝繖涓洰褰曚繚瀛樻湰鍦扮殑鍒嗗竷寮忕紦瀛樸傚洜姝ゆ湰鍦板垎甯冨紡緙撳瓨鏄湪鎵鏈塼ask鍜宩ob闂村叡浜殑銆?
${mapred.local.dir}/taskTracker/jobcache/$jobid/ : 鏈湴job鐩綍銆?
${mapred.local.dir}/taskTracker/jobcache/$jobid/work/: job鎸囧畾鐨勫叡浜洰褰曘傚悇涓換鍔″彲浠ヤ嬌鐢ㄨ繖涓┖闂村仛涓烘殏瀛樼┖闂達紝鐢ㄤ簬瀹冧滑涔嬮棿鍏變韓鏂囦歡銆傝繖涓洰褰曢氳繃job.local.dir 鍙傛暟鏆撮湶緇欑敤鎴楓傝繖涓礬寰勫彲浠ラ氳繃API JobConf.getJobLocalDir()鏉ヨ闂傚畠涔熷彲浠ヨ鍋氫負緋葷粺灞炴ц幏寰椼傚洜姝わ紝鐢ㄦ埛錛堟瘮濡傝繍琛宻treaming錛夊彲浠ヨ皟鐢⊿ystem.getProperty("job.local.dir")鑾峰緱璇ョ洰褰曘?
${mapred.local.dir}/taskTracker/jobcache/$jobid/jars/: 瀛樻斁jar鍖呯殑璺緞錛岀敤浜庡瓨鏀句綔涓氱殑jar鏂囦歡鍜屽睍寮鐨刯ar銆俲ob.jar鏄簲鐢ㄧ▼搴忕殑jar鏂囦歡錛屽畠浼氳鑷姩鍒嗗彂鍒板悇鍙版満鍣紝鍦╰ask鍚姩鍓嶄細琚嚜鍔ㄥ睍寮銆備嬌鐢╝pi JobConf.getJar() 鍑芥暟鍙互寰楀埌job.jar鐨勪綅緗備嬌鐢↗obConf.getJar().getParent()鍙互璁塊棶瀛樻斁灞曞紑鐨刯ar鍖呯殑鐩綍銆?
${mapred.local.dir}/taskTracker/jobcache/$jobid/job.xml錛?涓涓猨ob.xml鏂囦歡錛屾湰鍦扮殑閫氱敤鐨勪綔涓氶厤緗枃浠躲?
${mapred.local.dir}/taskTracker/jobcache/$jobid/$taskid錛?姣忎釜浠誨姟鏈変竴涓洰褰晅ask-id錛屽畠閲岄潰鏈夊涓嬬殑鐩綍緇撴瀯錛?
${mapred.local.dir}/taskTracker/jobcache/$jobid/$taskid/job.xml錛?涓涓猨ob.xml鏂囦歡錛屾湰鍦板寲鐨勪換鍔′綔涓氶厤緗枃浠躲備換鍔℃湰鍦板寲鏄寚涓鴻task璁懼畾鐗瑰畾鐨勫睘鎬у箋傝繖浜涘間細鍦ㄤ笅闈㈠叿浣撹鏄庛?
${mapred.local.dir}/taskTracker/jobcache/$jobid/$taskid/output 涓涓瓨鏀句腑闂磋繃紼嬬殑杈撳嚭鏂囦歡鐨勭洰褰曘傚畠淇濆瓨浜嗙敱framwork浜х敓鐨勪復鏃秏ap reduce鏁版嵁錛屾瘮濡俶ap鐨勮緭鍑烘枃浠剁瓑銆?
${mapred.local.dir}/taskTracker/jobcache/$jobid/$taskid/work錛?task鐨勫綋鍓嶅伐浣滅洰褰曘?
${mapred.local.dir}/taskTracker/jobcache/$jobid/$taskid/work/tmp錛?task鐨勪復鏃剁洰褰曘傦紙鐢ㄦ埛鍙互璁懼畾灞炴apred.child.tmp 鏉ヤ負map鍜宺educe task璁懼畾涓存椂鐩綍銆傜己鐪佸兼槸./tmp銆傚鏋滆繖涓間笉鏄粷瀵硅礬寰勶紝 瀹冧細鎶妕ask鐨勫伐浣滆礬寰勫姞鍒拌璺緞鍓嶉潰浣滀負task鐨勪復鏃舵枃浠惰礬寰勩傚鏋滆繖涓兼槸緇濆璺緞鍒欑洿鎺ヤ嬌鐢ㄨ繖涓箋?濡傛灉鎸囧畾鐨勭洰褰曚笉瀛樺湪錛屼細鑷姩鍒涘緩璇ョ洰褰曘備箣鍚庯紝鎸夌収閫夐」 -Djava.io.tmpdir='涓存椂鏂囦歡鐨勭粷瀵硅礬寰?鎵цjava瀛愪換鍔°?pipes鍜宻treaming鐨勪復鏃舵枃浠惰礬寰勬槸閫氳繃鐜鍙橀噺TMPDIR='the absolute path of the tmp dir'璁懼畾鐨勶級銆?濡傛灉mapred.child.tmp鏈?/tmp鍊鹼紝榪欎釜鐩綍浼氳鍒涘緩銆?
涓嬮潰鐨勫睘鎬ф槸涓烘瘡涓猼ask鎵ц鏃朵嬌鐢ㄧ殑鏈湴鍙傛暟錛屽畠浠繚瀛樺湪鏈湴鍖栫殑浠誨姟浣滀笟閰嶇疆鏂囦歡閲岋細
鍚嶇О 綾誨瀷 鎻忚堪
mapred.job.id String job id
mapred.jar String job鐩綍涓媕ob.jar鐨勪綅緗?
job.local.dir String job鎸囧畾鐨勫叡浜瓨鍌ㄧ┖闂?
mapred.tip.id String task id
mapred.task.id String task灝濊瘯id
mapred.task.is.map boolean 鏄惁鏄痬ap task
mapred.task.partition int task鍦╦ob涓殑id
map.input.file String map璇誨彇鐨勬枃浠跺悕
map.input.start long map杈撳叆鐨勬暟鎹潡鐨勮搗濮嬩綅緗亸縐?
map.input.length long map杈撳叆鐨勬暟鎹潡鐨勫瓧鑺傛暟
mapred.work.output.dir String task涓存椂杈撳嚭鐩綍
task鐨勬爣鍑嗚緭鍑哄拰閿欒杈撳嚭嫻佷細琚鍒癟askTracker涓紝騫朵笖璁板綍鍒?${HADOOP_LOG_DIR}/userlogs
DistributedCache 鍙敤浜巑ap鎴杛educe task涓垎鍙慾ar鍖呭拰鏈湴搴撱傚瓙jvm鎬繪槸鎶?褰撳墠宸ヤ綔鐩綍 鍔犲埌 java.library.path 鍜?LD_LIBRARY_PATH銆?鍥犳錛屽彲浠ラ氳繃 System.loadLibrary鎴?System.load瑁呰澆緙撳瓨鐨勫簱銆傛湁鍏充嬌鐢ㄥ垎甯冨紡緙撳瓨鍔犺澆鍏變韓搴撶殑緇嗚妭璇峰弬鑰?native_libraries.html
浣滀笟鐨勬彁浜や笌鐩戞帶
JobClient鏄敤鎴鋒彁浜ょ殑浣滀笟涓嶫obTracker浜や簰鐨勪富瑕佹帴鍙c?/p>
JobClient 鎻愪緵鎻愪氦浣滀笟錛岃拷韙繘紼嬶紝璁塊棶瀛愪換鍔$殑鏃ュ織璁板綍錛岃幏寰桵ap/Reduce闆嗙兢鐘舵佷俊鎭瓑鍔熻兘銆?/p>
浣滀笟鎻愪氦榪囩▼鍖呮嫭錛?/p>
媯鏌ヤ綔涓氳緭鍏ヨ緭鍑烘牱寮忕粏鑺?
涓轟綔涓氳綆桰nputSplit鍊箋?
濡傛灉闇瑕佺殑璇濓紝涓轟綔涓氱殑DistributedCache寤虹珛蹇呴』鐨勭粺璁′俊鎭?
鎷瘋礉浣滀笟鐨刯ar鍖呭拰閰嶇疆鏂囦歡鍒癋ileSystem涓婄殑Map/Reduce緋葷粺鐩綍涓嬨?
鎻愪氦浣滀笟鍒癑obTracker騫朵笖鐩戞帶瀹冪殑鐘舵併?
浣滀笟鐨勫巻鍙叉枃浠惰褰曞埌鎸囧畾鐩綍鐨?_logs/history/"瀛愮洰褰曚笅銆傝繖涓寚瀹氱洰褰曠敱hadoop.job.history.user.location璁懼畾錛岄粯璁ゆ槸浣滀笟杈撳嚭鐨勭洰褰曘傚洜姝ら粯璁ゆ儏鍐典笅錛屾枃浠朵細瀛樻斁鍦╩apred.output.dir/_logs/history鐩綍涓嬨傜敤鎴峰彲浠ヨ緗甴adoop.job.history.user.location涓簄one鏉ュ仠姝㈡棩蹇楄褰曘?/p>
鐢ㄦ埛浣跨敤涓嬮潰鐨勫懡浠ゅ彲浠ョ湅鍒板湪鎸囧畾鐩綍涓嬬殑鍘嗗彶鏃ュ織璁板綍鐨勬憳瑕併?
$ bin/hadoop job -history output-dir
榪欎釜鍛戒護浼氭墦鍗板嚭浣滀笟鐨勭粏鑺傦紝浠ュ強澶辮觸鐨勫拰琚潃姝葷殑浠誨姟緇嗚妭銆?br>瑕佹煡鐪嬫湁鍏充綔涓氱殑鏇村緇嗚妭渚嬪鎴愬姛鐨勪換鍔°佹瘡涓換鍔″皾璇曠殑嬈℃暟錛坱ask attempt錛夌瓑錛屽彲浠ヤ嬌鐢ㄤ笅闈㈢殑鍛戒護
$ bin/hadoop job -history all output-dir
鐢ㄦ埛鍙互浣跨敤 OutputLogFilter 浠庤緭鍑虹洰褰曞垪琛ㄤ腑絳涢夋棩蹇楁枃浠躲?/p>
涓鑸儏鍐碉紝鐢ㄦ埛鍒╃敤JobConf鍒涘緩搴旂敤紼嬪簭騫墮厤緗綔涓氬睘鎬э紝 鐒跺悗鐢?JobClient 鎻愪氦浣滀笟騫剁洃瑙嗗畠鐨勮繘紼嬨?/p>
浣滀笟鐨勬帶鍒?br>鏈夋椂鍊欙紝鐢ㄤ竴涓崟鐙殑Map/Reduce浣滀笟騫朵笉鑳藉畬鎴愪竴涓鏉傜殑浠誨姟錛岀敤鎴蜂篃璁歌閾炬帴澶氫釜Map/Reduce浣滀笟鎵嶈銆傝繖鏄鏄撳疄鐜扮殑錛屽洜涓轟綔涓氶氬父杈撳嚭鍒板垎甯冨紡鏂囦歡緋葷粺涓婄殑錛屾墍浠ュ彲浠ユ妸榪欎釜浣滀笟鐨勮緭鍑轟綔涓轟笅涓涓綔涓氱殑杈撳叆瀹炵幇涓茶仈銆?/p>
鐒惰岋紝榪欎篃鎰忓懗鐫錛岀‘淇濇瘡涓浣滀笟瀹屾垚(鎴愬姛鎴栧け璐?鐨勮矗浠誨氨鐩存帴钀藉湪浜嗗鎴瘋韓涓娿傚湪榪欑鎯呭喌涓嬶紝鍙互鐢ㄧ殑鎺у埗浣滀笟鐨勯夐」鏈夛細
runJob(JobConf)錛氭彁浜や綔涓氾紝浠呭綋浣滀笟瀹屾垚鏃惰繑鍥炪?
submitJob(JobConf)錛氬彧鎻愪氦浣滀笟錛屼箣鍚庨渶瑕佷綘杞瀹冭繑鍥炵殑 RunningJob鍙ユ焺鐨勭姸鎬侊紝騫舵牴鎹儏鍐佃皟搴︺?
JobConf.setJobEndNotificationURI(String)錛氳緗竴涓綔涓氬畬鎴愰氱煡錛屽彲閬垮厤杞銆?
浣滀笟鐨勮緭鍏?br>InputFormat 涓篗ap/Reduce浣滀笟鎻忚堪杈撳叆鐨勭粏鑺傝鑼冦?/p>
Map/Reduce妗嗘灦鏍規嵁浣滀笟鐨処nputFormat鏉ワ細
媯鏌ヤ綔涓氳緭鍏ョ殑鏈夋晥鎬с?
鎶婅緭鍏ユ枃浠跺垏鍒嗘垚澶氫釜閫昏緫InputSplit瀹炰緥錛?騫舵妸姣忎竴瀹炰緥鍒嗗埆鍒嗗彂緇欎竴涓?Mapper銆?
鎻愪緵RecordReader鐨勫疄鐜幫紝榪欎釜RecordReader浠庨昏緫InputSplit涓幏寰楄緭鍏ヨ褰曪紝 榪欎簺璁板綍灝嗙敱Mapper澶勭悊銆?
鍩轟簬鏂囦歡鐨処nputFormat瀹炵幇錛堥氬父鏄?FileInputFormat鐨勫瓙綾伙級 榛樿琛屼負鏄寜鐓ц緭鍏ユ枃浠剁殑瀛楄妭澶у皬錛屾妸杈撳叆鏁版嵁鍒囧垎鎴愰昏緫鍒嗗潡錛坙ogical InputSplit 錛夈?鍏朵腑杈撳叆鏂囦歡鎵鍦ㄧ殑FileSystem鐨勬暟鎹潡灝哄鏄垎鍧楀ぇ灝忕殑涓婇檺銆備笅闄愬彲浠ヨ緗甿apred.min.split.size 鐨勫箋?/p>
鑰冭檻鍒拌竟鐣屾儏鍐碉紝瀵逛簬寰堝搴旂敤紼嬪簭鏉ヨ錛屽緢鏄庢樉鎸夌収鏂囦歡澶у皬榪涜閫昏緫鍒嗗壊鏄笉鑳芥弧瓚抽渶姹傜殑銆?鍦ㄨ繖縐嶆儏鍐典笅錛屽簲鐢ㄧ▼搴忛渶瑕佸疄鐜頒竴涓猂ecordReader鏉ュ鐞嗚褰曠殑杈圭晫騫朵負姣忎釜浠誨姟鎻愪緵涓涓昏緫鍒嗗潡鐨勯潰鍚戣褰曠殑瑙嗗浘銆?/p>
TextInputFormat 鏄粯璁ょ殑InputFormat銆?/p>
濡傛灉涓涓綔涓氱殑Inputformat鏄疶extInputFormat錛?騫朵笖妗嗘灦媯嫻嬪埌杈撳叆鏂囦歡鐨勫悗緙鏄?gz鍜?lzo錛屽氨浼氫嬌鐢ㄥ搴旂殑CompressionCodec鑷姩瑙e帇緙╄繖浜涙枃浠躲?浣嗘槸闇瑕佹敞鎰忥紝涓婅堪甯﹀悗緙鐨勫帇緙╂枃浠朵笉浼氳鍒囧垎錛屽茍涓旀暣涓帇緙╂枃浠朵細鍒嗙粰涓涓猰apper鏉ュ鐞嗐?/p>
InputSplit
InputSplit 鏄竴涓崟鐙殑Mapper瑕佸鐞嗙殑鏁版嵁鍧椼?/p>
涓鑸殑InputSplit 鏄瓧鑺傛牱寮忚緭鍏ワ紝鐒跺悗鐢盧ecordReader澶勭悊騫惰漿鍖栨垚璁板綍鏍峰紡銆?/p>
FileSplit 鏄粯璁ょ殑InputSplit銆?瀹冩妸 map.input.file 璁懼畾涓鴻緭鍏ユ枃浠剁殑璺緞錛岃緭鍏ユ枃浠舵槸閫昏緫鍒嗗潡鏂囦歡銆?/p>
RecordReader
RecordReader 浠嶪nputSlit璇誨叆<key, value>瀵廣?/p>
涓鑸殑錛孯ecordReader 鎶婄敱InputSplit 鎻愪緵鐨勫瓧鑺傛牱寮忕殑杈撳叆鏂囦歡錛岃漿鍖栨垚鐢盡apper澶勭悊鐨勮褰曟牱寮忕殑鏂囦歡銆?鍥犳RecordReader璐熻矗澶勭悊璁板綍鐨勮竟鐣屾儏鍐靛拰鎶婃暟鎹〃紺烘垚keys/values瀵瑰艦寮忋?/p>
浣滀笟鐨勮緭鍑?br>OutputFormat 鎻忚堪Map/Reduce浣滀笟鐨勮緭鍑烘牱寮忋?/p>
Map/Reduce妗嗘灦鏍規嵁浣滀笟鐨凮utputFormat鏉ワ細
媯楠屼綔涓氱殑杈撳嚭錛屼緥濡傛鏌ヨ緭鍑鴻礬寰勬槸鍚﹀凡緇忓瓨鍦ㄣ?
鎻愪緵涓涓猂ecordWriter鐨勫疄鐜幫紝鐢ㄦ潵杈撳嚭浣滀笟緇撴灉銆?杈撳嚭鏂囦歡淇濆瓨鍦‵ileSystem涓娿?
TextOutputFormat鏄粯璁ょ殑 OutputFormat銆?/p>
浠誨姟鐨凷ide-Effect File
鍦ㄤ竴浜涘簲鐢ㄧ▼搴忎腑錛屽瓙浠誨姟闇瑕佷駭鐢熶竴浜泂ide-file錛岃繖浜涙枃浠朵笌浣滀笟瀹為檯杈撳嚭緇撴灉鐨勬枃浠朵笉鍚屻?/p>
鍦ㄨ繖縐嶆儏鍐典笅錛屽悓涓涓狹apper鎴栬匯educer鐨勪袱涓疄渚嬶紙姣斿棰勯槻鎬т換鍔★級鍚屾椂鎵撳紑鎴栬呭啓 FileSystem涓婄殑鍚屼竴鏂囦歡灝變細浜х敓鍐茬獊銆傚洜姝ゅ簲鐢ㄧ▼搴忓湪鍐欐枃浠剁殑鏃跺欓渶瑕佷負姣忔浠誨姟灝濊瘯錛堜笉浠呬粎鏄瘡嬈′換鍔★紝姣忎釜浠誨姟鍙互灝濊瘯鎵ц寰堝嬈★級閫夊彇涓涓嫭涓鏃犱簩鐨勬枃浠跺悕(浣跨敤attemptid錛屼緥濡倀ask_200709221812_0001_m_000000_0)銆?/p>
涓轟簡閬垮厤鍐茬獊錛孧ap/Reduce妗嗘灦涓烘瘡嬈″皾璇曟墽琛屼換鍔¢兘寤虹珛鍜岀淮鎶や竴涓壒孌婄殑 ${mapred.output.dir}/_temporary/_${taskid}瀛愮洰褰曪紝榪欎釜鐩綍浣嶄簬鏈灝濊瘯鎵ц浠誨姟杈撳嚭緇撴灉鎵鍦ㄧ殑FileSystem涓婏紝鍙互閫氳繃 ${mapred.work.output.dir}鏉ヨ闂繖涓瓙鐩綍銆?瀵逛簬鎴愬姛瀹屾垚鐨勪換鍔″皾璇曪紝鍙湁${mapred.output.dir}/_temporary/_${taskid}涓嬬殑鏂囦歡浼氱Щ鍔ㄥ埌${mapred.output.dir}銆傚綋鐒訛紝妗嗘灦浼氫涪寮冮偅浜涘け璐ョ殑浠誨姟灝濊瘯鐨勫瓙鐩綍銆傝繖縐嶅鐞嗚繃紼嬪浜庡簲鐢ㄧ▼搴忔潵璇存槸瀹屽叏閫忔槑鐨勩?/p>
鍦ㄤ換鍔℃墽琛屾湡闂達紝搴旂敤紼嬪簭鍦ㄥ啓鏂囦歡鏃跺彲浠ュ埄鐢ㄨ繖涓壒鎬э紝姣斿 閫氳繃 FileOutputFormat.getWorkOutputPath()鑾峰緱${mapred.work.output.dir}鐩綍錛?騫跺湪鍏朵笅鍒涘緩浠繪剰浠誨姟鎵ц鏃舵墍闇鐨剆ide-file錛屾鏋跺湪浠誨姟灝濊瘯鎴愬姛鏃朵細椹笂縐誨姩榪欎簺鏂囦歡錛屽洜姝や笉闇瑕佸湪紼嬪簭鍐呬負姣忔浠誨姟灝濊瘯閫夊彇涓涓嫭涓鏃犱簩鐨勫悕瀛椼?/p>
娉ㄦ剰錛氬湪姣忔浠誨姟灝濊瘯鎵ц鏈熼棿錛?{mapred.work.output.dir} 鐨勫煎疄闄呬笂鏄?${mapred.output.dir}/_temporary/_{$taskid}錛岃繖涓兼槸Map/Reduce妗嗘灦鍒涘緩鐨勩?鎵浠ヤ嬌鐢ㄨ繖涓壒鎬х殑鏂規硶鏄紝鍦?FileOutputFormat.getWorkOutputPath() 璺緞涓嬪垱寤簊ide-file鍗沖彲銆?/p>
瀵逛簬鍙嬌鐢╩ap涓嶄嬌鐢╮educe鐨勪綔涓氾紝榪欎釜緇撹涔熸垚绔嬨傝繖縐嶆儏鍐典笅錛宮ap鐨勮緭鍑虹粨鏋滅洿鎺ョ敓鎴愬埌HDFS涓娿?/p>
RecordWriter
RecordWriter 鐢熸垚<key, value> 瀵瑰埌杈撳嚭鏂囦歡銆?/p>
RecordWriter鐨勫疄鐜版妸浣滀笟鐨勮緭鍑虹粨鏋滃啓鍒?FileSystem銆?/p>
鍏朵粬鏈夌敤鐨勭壒鎬?br>Counters
Counters 鏄涓敱Map/Reduce妗嗘灦鎴栬呭簲鐢ㄧ▼搴忓畾涔夌殑鍏ㄥ眬璁℃暟鍣ㄣ?姣忎竴涓狢ounter鍙互鏄換浣曚竴縐?Enum綾誨瀷銆傚悓涓鐗瑰畾Enum綾誨瀷鐨凜ounter鍙互姹囬泦鍒頒竴涓粍錛屽叾綾誨瀷涓篊ounters.Group銆?/p>
搴旂敤紼嬪簭鍙互瀹氫箟浠繪剰(Enum綾誨瀷)鐨凜ounters騫朵笖鍙互閫氳繃 map 鎴栬?reduce鏂規硶涓殑 Reporter.incrCounter(Enum, long)鎴栬?Reporter.incrCounter(String, String, long) 鏇存柊銆備箣鍚庢鏋朵細姹囨昏繖浜涘叏灞counters銆?/p>
DistributedCache
DistributedCache 鍙皢鍏蜂綋搴旂敤鐩稿叧鐨勩佸ぇ灝哄鐨勩佸彧璇葷殑鏂囦歡鏈夋晥鍦板垎甯冩斁緗?/p>
DistributedCache 鏄疢ap/Reduce妗嗘灦鎻愪緵鐨勫姛鑳斤紝鑳藉緙撳瓨搴旂敤紼嬪簭鎵闇鐨勬枃浠?錛堝寘鎷枃鏈紝妗f鏂囦歡錛宩ar鏂囦歡絳夛級銆?/p>
搴旂敤紼嬪簭鍦↗obConf涓氳繃url(hdfs://)鎸囧畾闇瑕佽緙撳瓨鐨勬枃浠躲?DistributedCache鍋囧畾鐢県dfs://鏍煎紡url鎸囧畾鐨勬枃浠跺凡緇忓湪 FileSystem涓婁簡銆?/p>
Map-Redcue妗嗘灦鍦ㄤ綔涓氭墍鏈変換鍔℃墽琛屼箣鍓嶄細鎶婂繀瑕佺殑鏂囦歡鎷瘋礉鍒皊lave鑺傜偣涓娿?瀹冭繍琛岄珮鏁堟槸鍥犱負姣忎釜浣滀笟鐨勬枃浠跺彧鎷瘋礉涓嬈″茍涓斾負閭d簺娌℃湁鏂囨。鐨剆lave鑺傜偣緙撳瓨鏂囨。銆?/p>
DistributedCache 鏍規嵁緙撳瓨鏂囨。淇敼鐨勬椂闂存埑榪涜榪借釜銆?鍦ㄤ綔涓氭墽琛屾湡闂達紝褰撳墠搴旂敤紼嬪簭鎴栬呭閮ㄧ▼搴忎笉鑳戒慨鏀圭紦瀛樻枃浠躲?/p>
distributedCache鍙互鍒嗗彂綆鍗曠殑鍙鏁版嵁鎴栨枃鏈枃浠訛紝涔熷彲浠ュ垎鍙戝鏉傜被鍨嬬殑鏂囦歡渚嬪褰掓。鏂囦歡鍜宩ar鏂囦歡銆傚綊妗f枃浠?zip,tar,tgz鍜宼ar.gz鏂囦歡)鍦╯lave鑺傜偣涓婁細琚В妗o紙un-archived錛夈?榪欎簺鏂囦歡鍙互璁劇疆鎵ц鏉冮檺銆?/p>
鐢ㄦ埛鍙互閫氳繃璁劇疆mapred.cache.{files|archives}鏉ュ垎鍙戞枃浠躲?濡傛灉瑕佸垎鍙戝涓枃浠訛紝鍙互浣跨敤閫楀彿鍒嗛殧鏂囦歡鎵鍦ㄨ礬寰勩備篃鍙互鍒╃敤API鏉ヨ緗灞炴э細 DistributedCache.addCacheFile(URI,conf)/ DistributedCache.addCacheArchive(URI,conf) and DistributedCache.setCacheFiles(URIs,conf)/ DistributedCache.setCacheArchives(URIs,conf) 鍏朵腑URI鐨勫艦寮忔槸 hdfs://host:port/absolute-path#link-name 鍦⊿treaming紼嬪簭涓紝鍙互閫氳繃鍛戒護琛岄夐」 -cacheFile/-cacheArchive 鍒嗗彂鏂囦歡銆?/p>
鐢ㄦ埛鍙互閫氳繃 DistributedCache.createSymlink(Configuration)鏂規硶璁〥istributedCache 鍦ㄥ綋鍓嶅伐浣滅洰褰曚笅鍒涘緩鍒扮紦瀛樻枃浠剁殑絎﹀彿閾炬帴銆?鎴栬呴氳繃璁劇疆閰嶇疆鏂囦歡灞炴apred.create.symlink涓簓es銆?鍒嗗竷寮忕紦瀛樹細鎴彇URI鐨勭墖孌典綔涓洪摼鎺ョ殑鍚嶅瓧銆?渚嬪錛孶RI鏄?hdfs://namenode:port/lib.so.1#lib.so錛?鍒欏湪task褰撳墠宸ヤ綔鐩綍浼氭湁鍚嶄負lib.so鐨勯摼鎺ワ紝 瀹冧細閾炬帴鍒嗗竷寮忕紦瀛樹腑鐨刲ib.so.1銆?/p>
DistributedCache鍙湪map/reduce浠誨姟涓綔涓?涓縐嶅熀紜杞歡鍒嗗彂鏈哄埗浣跨敤銆傚畠鍙互琚敤浜庡垎鍙慾ar鍖呭拰鏈湴搴擄紙native libraries錛夈?DistributedCache.addArchiveToClassPath(Path, Configuration)鍜?DistributedCache.addFileToClassPath(Path, Configuration) API鑳藉琚敤浜?緙撳瓨鏂囦歡鍜宩ar鍖咃紝騫舵妸瀹冧滑鍔犲叆瀛恓vm鐨刢lasspath銆備篃鍙互閫氳繃璁劇疆閰嶇疆鏂囨。閲岀殑灞炴?mapred.job.classpath.{files|archives}杈懼埌鐩稿悓鐨勬晥鏋溿傜紦瀛樻枃浠跺彲鐢ㄤ簬鍒嗗彂鍜岃杞芥湰鍦板簱銆?/p>
Tool
Tool 鎺ュ彛鏀寔澶勭悊甯哥敤鐨凥adoop鍛戒護琛岄夐」銆?/p>
Tool 鏄疢ap/Reduce宸ュ叿鎴栧簲鐢ㄧ殑鏍囧噯銆傚簲鐢ㄧ▼搴忓簲鍙鐞嗗叾瀹氬埗鍙傛暟錛?瑕佹妸鏍囧噯鍛戒護琛岄夐」閫氳繃 ToolRunner.run(Tool, String[]) 濮旀墭緇?GenericOptionsParser澶勭悊銆?/p>
Hadoop鍛戒護琛岀殑甯哥敤閫夐」鏈夛細
-conf <configuration file>
-D <property=value>
-fs <local|namenode:port>
-jt <local|jobtracker:port>
IsolationRunner
IsolationRunner 鏄府鍔╄皟璇昅ap/Reduce紼嬪簭鐨勫伐鍏楓?/p>
浣跨敤IsolationRunner鐨勬柟娉曟槸錛岄鍏堣緗?keep.failed.tasks.files灞炴т負true 錛堝悓鏃跺弬鑰僰eep.tasks.files.pattern錛夈?/p>
鐒跺悗錛岀櫥褰曞埌浠誨姟榪愯澶辮觸鐨勮妭鐐逛笂錛岃繘鍏?TaskTracker鐨勬湰鍦拌礬寰勮繍琛?IsolationRunner錛?br>$ cd <local path>/taskTracker/${taskid}/work
$ bin/hadoop org.apache.hadoop.mapred.IsolationRunner ../job.xml
IsolationRunner浼氭妸澶辮觸鐨勪換鍔℃斁鍦ㄥ崟鐙殑涓涓兘澶熻皟璇曠殑jvm涓婅繍琛岋紝騫朵笖閲囩敤鍜屼箣鍓嶅畬鍏ㄤ竴鏍風殑杈撳叆鏁版嵁銆?/p>
Profiling
Profiling鏄竴涓伐鍏鳳紝瀹冧嬌鐢ㄥ唴緗殑java profiler宸ュ叿榪涜鍒嗘瀽鑾峰緱(2-3涓?map鎴杛educe鏍蜂緥榪愯鍒嗘瀽鎶ュ憡銆?/p>
鐢ㄦ埛鍙互閫氳繃璁劇疆灞炴apred.task.profile鎸囧畾緋葷粺鏄惁閲囬泦profiler淇℃伅銆?鍒╃敤api JobConf.setProfileEnabled(boolean)鍙互淇敼灞炴у箋傚鏋滆涓簍rue錛?鍒欏紑鍚痯rofiling鍔熻兘銆俻rofiler淇℃伅淇濆瓨鍦ㄧ敤鎴鋒棩蹇楃洰褰曚笅銆傜己鐪佹儏鍐碉紝profiling鍔熻兘鏄叧闂殑銆?/p>
濡傛灉鐢ㄦ埛璁懼畾浣跨敤profiling鍔熻兘錛屽彲浠ヤ嬌鐢ㄩ厤緗枃妗i噷鐨勫睘鎬?mapred.task.profile.{maps|reduces} 璁劇疆瑕乸rofile map/reduce task鐨勮寖鍥淬傝緗灞炴у肩殑api鏄?JobConf.setProfileTaskRange(boolean,String)銆?鑼冨洿鐨勭己鐪佸兼槸0-2銆?/p>
鐢ㄦ埛鍙互閫氳繃璁懼畾閰嶇疆鏂囨。閲岀殑灞炴apred.task.profile.params 鏉ユ寚瀹歱rofiler閰嶇疆鍙傛暟銆備慨鏀瑰睘鎬ц浣跨敤api JobConf.setProfileParams(String)銆傚綋榪愯task鏃訛紝濡傛灉瀛楃涓插寘鍚?s銆?瀹冧細琚浛鎹㈡垚profileing鐨勮緭鍑烘枃浠跺悕銆傝繖浜涘弬鏁頒細鍦ㄥ懡浠よ閲屼紶閫掑埌瀛怞VM涓傜己鐪佺殑profiling 鍙傛暟鏄?-agentlib:hprof=cpu=samples,heap=sites,force=n,thread=y,verbose=n,file=%s銆?/p>
璋冭瘯
Map/Reduce妗嗘灦鑳藉榪愯鐢ㄦ埛鎻愪緵鐨勭敤浜庤皟璇曠殑鑴氭湰紼嬪簭銆?褰搈ap/reduce浠誨姟澶辮觸鏃訛紝鐢ㄦ埛鍙互閫氳繃榪愯鑴氭湰鍦ㄤ換鍔℃棩蹇楋紙渚嬪浠誨姟鐨勬爣鍑嗚緭鍑恒佹爣鍑嗛敊璇佺郴緇熸棩蹇椾互鍙婁綔涓氶厤緗枃浠訛級涓婂仛鍚庣畫澶勭悊宸ヤ綔銆傜敤鎴鋒彁渚涚殑璋冭瘯鑴氭湰紼嬪簭鐨勬爣鍑嗚緭鍑哄拰鏍囧噯閿欒浼氳緭鍑轟負璇婃柇鏂囦歡銆傚鏋滈渶瑕佺殑璇濊繖浜涜緭鍑虹粨鏋滀篃鍙互鎵撳嵃鍦ㄧ敤鎴風晫闈笂銆?/p>
鍦ㄦ帴涓嬫潵鐨勭珷鑺傦紝鎴戜滑璁ㄨ濡備綍涓庝綔涓氫竴璧鋒彁浜よ皟璇曡剼鏈備負浜嗘彁浜よ皟璇曡剼鏈紝 棣栧厛瑕佹妸榪欎釜鑴氭湰鍒嗗彂鍑哄幓錛岃屼笖榪樿鍦ㄩ厤緗枃浠墮噷璁劇疆銆?/p>
濡備綍鍒嗗彂鑴氭湰鏂囦歡錛?br>鐢ㄦ埛瑕佺敤 DistributedCache 鏈哄埗鏉ュ垎鍙戝拰閾炬帴鑴氭湰鏂囦歡
濡備綍鎻愪氦鑴氭湰錛?br>涓涓揩閫熸彁浜よ皟璇曡剼鏈殑鏂規硶鏄垎鍒負闇瑕佽皟璇曠殑map浠誨姟鍜宺educe浠誨姟璁劇疆 "mapred.map.task.debug.script" 鍜?"mapred.reduce.task.debug.script" 灞炴х殑鍊箋傝繖浜涘睘鎬т篃鍙互閫氳繃 JobConf.setMapDebugScript(String) 鍜?JobConf.setReduceDebugScript(String) API鏉ヨ緗傚浜巗treaming錛?鍙互鍒嗗埆涓洪渶瑕佽皟璇曠殑map浠誨姟鍜宺educe浠誨姟浣跨敤鍛戒護琛岄夐」-mapdebug 鍜?-reducedegug鏉ユ彁浜よ皟璇曡剼鏈?/p>
鑴氭湰鐨勫弬鏁版槸浠誨姟鐨勬爣鍑嗚緭鍑恒佹爣鍑嗛敊璇佺郴緇熸棩蹇椾互鍙婁綔涓氶厤緗枃浠躲傚湪榪愯map/reduce澶辮觸鐨勮妭鐐逛笂榪愯璋冭瘯鍛戒護鏄細
$script $stdout $stderr $syslog $jobconf
Pipes 紼嬪簭鏍規嵁絎簲涓弬鏁拌幏寰梒++紼嬪簭鍚嶃?鍥犳璋冭瘯pipes紼嬪簭鐨勫懡浠ゆ槸
$script $stdout $stderr $syslog $jobconf $program
榛樿琛屼負
瀵逛簬pipes錛岄粯璁ょ殑鑴氭湰浼氱敤gdb澶勭悊core dump錛?鎵撳嵃 stack trace騫朵笖緇欏嚭姝e湪榪愯綰跨▼鐨勪俊鎭?/p>
JobControl
JobControl鏄竴涓伐鍏鳳紝瀹冨皝瑁呬簡涓緇凪ap/Reduce浣滀笟浠ュ強浠栦滑涔嬮棿鐨勪緷璧栧叧緋匯?/p>
鏁版嵁鍘嬬緝
Hadoop Map/Reduce妗嗘灦涓哄簲鐢ㄧ▼搴忕殑鍐欏叆鏂囦歡鎿嶄綔鎻愪緵鍘嬬緝宸ュ叿錛岃繖浜涘伐鍏峰彲浠ヤ負map杈撳嚭鐨勪腑闂存暟鎹拰浣滀笟鏈緇堣緭鍑烘暟鎹紙渚嬪reduce鐨勮緭鍑猴級鎻愪緵鏀寔銆傚畠榪橀檮甯︿簡涓浜?CompressionCodec鐨勫疄鐜幫紝姣斿瀹炵幇浜?zlib鍜宭zo鍘嬬緝綆楁硶銆?Hadoop鍚屾牱鏀寔gzip鏂囦歡鏍煎紡銆?/p>
鑰冭檻鍒版ц兘闂錛坺lib錛変互鍙奐ava綾誨簱鐨勭己澶憋紙lzo錛夌瓑鍥犵礌錛孒adoop涔熶負涓婅堪鍘嬬緝瑙e帇綆楁硶鎻愪緵鏈湴搴撶殑瀹炵幇銆傛洿澶氱殑緇嗚妭璇峰弬鑰?榪欓噷銆?/p>
涓棿杈撳嚭
搴旂敤紼嬪簭鍙互閫氳繃 JobConf.setCompressMapOutput(boolean)api鎺у埗map杈撳嚭鐨勪腑闂寸粨鏋滐紝騫朵笖鍙互閫氳繃 JobConf.setMapOutputCompressorClass(Class)api鎸囧畾 CompressionCodec銆?/p>
浣滀笟杈撳嚭
搴旂敤紼嬪簭鍙互閫氳繃 FileOutputFormat.setCompressOutput(JobConf, boolean) api鎺у埗杈撳嚭鏄惁闇瑕佸帇緙╁茍涓斿彲浠ヤ嬌鐢?FileOutputFormat.setOutputCompressorClass(JobConf, Class)api鎸囧畾CompressionCodec銆?/p>
濡傛灉浣滀笟杈撳嚭瑕佷繚瀛樻垚 SequenceFileOutputFormat鏍煎紡錛岄渶瑕佷嬌鐢?SequenceFileOutputFormat.setOutputCompressionType(JobConf, SequenceFile.CompressionType)api錛屾潵璁懼畾 SequenceFile.CompressionType (i.e.RECORD / BLOCK - 榛樿鏄疪ECORD)銆?/p>
渚嬪瓙錛歐ordCount v2.0
榪欓噷鏄竴涓洿鍏ㄩ潰鐨刉ordCount渚嬪瓙錛屽畠浣跨敤浜嗘垜浠凡緇忚璁鴻繃鐨勫緢澶歁ap/Reduce妗嗘灦鎻愪緵鐨勫姛鑳姐?/p>
榪愯榪欎釜渚嬪瓙闇瑕丠DFS鐨勬煇浜涘姛鑳斤紝鐗瑰埆鏄?DistributedCache鐩稿叧鍔熻兘銆傚洜姝よ繖涓緥瀛愬彧鑳借繍琛屽湪 浼垎甯冨紡 鎴栬?瀹屽叏鍒嗗竷寮忔ā寮忕殑 Hadoop涓娿?/p>
婧愪唬鐮?br> WordCount.java
1. package org.myorg;
2.
3. import java.io.*;
4. import java.util.*;
5.
6. import org.apache.hadoop.fs.Path;
7. import org.apache.hadoop.filecache.DistributedCache;
8. import org.apache.hadoop.conf.*;
9. import org.apache.hadoop.io.*;
10. import org.apache.hadoop.mapred.*;
11. import org.apache.hadoop.util.*;
12.
13. public class WordCount extends Configured implements Tool {
14.
15. public static class Map extends MapReduceBase implements Mapper<LongWritable, Text, Text, IntWritable> {
16.
17. static enum Counters { INPUT_WORDS }
18.
19. private final static IntWritable one = new IntWritable(1);
20. private Text word = new Text();
21.
22. private boolean caseSensitive = true;
23. private Set<String> patternsToSkip = new HashSet<String>();
24.
25. private long numRecords = 0;
26. private String inputFile;
27.
28. public void configure(JobConf job) {
29. caseSensitive = job.getBoolean("wordcount.case.sensitive", true);
30. inputFile = job.get("map.input.file");
31.
32. if (job.getBoolean("wordcount.skip.patterns", false)) {
33. Path[] patternsFiles = new Path[0];
34. try {
35. patternsFiles = DistributedCache.getLocalCacheFiles(job);
36. } catch (IOException ioe) {
37. System.err.println("Caught exception while getting cached files: " + StringUtils.stringifyException(ioe));
38. }
39. for (Path patternsFile : patternsFiles) {
40. parseSkipFile(patternsFile);
41. }
42. }
43. }
44.
45. private void parseSkipFile(Path patternsFile) {
46. try {
47. BufferedReader fis = new BufferedReader(new FileReader(patternsFile.toString()));
48. String pattern = null;
49. while ((pattern = fis.readLine()) != null) {
50. patternsToSkip.add(pattern);
51. }
52. } catch (IOException ioe) {
53. System.err.println("Caught exception while parsing the cached file '" + patternsFile + "' : " + StringUtils.stringifyException(ioe));
54. }
55. }
56.
57. public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {
58. String line = (caseSensitive) ? value.toString() : value.toString().toLowerCase();
59.
60. for (String pattern : patternsToSkip) {
61. line = line.replaceAll(pattern, "");
62. }
63.
64. StringTokenizer tokenizer = new StringTokenizer(line);
65. while (tokenizer.hasMoreTokens()) {
66. word.set(tokenizer.nextToken());
67. output.collect(word, one);
68. reporter.incrCounter(Counters.INPUT_WORDS, 1);
69. }
70.
71. if ((++numRecords % 100) == 0) {
72. reporter.setStatus("Finished processing " + numRecords + " records " + "from the input file: " + inputFile);
73. }
74. }
75. }
76.
77. public static class Reduce extends MapReduceBase implements Reducer<Text, IntWritable, Text, IntWritable> {
78. public void reduce(Text key, Iterator<IntWritable> values, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {
79. int sum = 0;
80. while (values.hasNext()) {
81. sum += values.next().get();
82. }
83. output.collect(key, new IntWritable(sum));
84. }
85. }
86.
87. public int run(String[] args) throws Exception {
88. JobConf conf = new JobConf(getConf(), WordCount.class);
89. conf.setJobName("wordcount");
90.
91. conf.setOutputKeyClass(Text.class);
92. conf.setOutputValueClass(IntWritable.class);
93.
94. conf.setMapperClass(Map.class);
95. conf.setCombinerClass(Reduce.class);
96. conf.setReducerClass(Reduce.class);
97.
98. conf.setInputFormat(TextInputFormat.class);
99. conf.setOutputFormat(TextOutputFormat.class);
100.
101. List<String> other_args = new ArrayList<String>();
102. for (int i=0; i < args.length; ++i) {
103. if ("-skip".equals(args[i])) {
104. DistributedCache.addCacheFile(new Path(args[++i]).toUri(), conf);
105. conf.setBoolean("wordcount.skip.patterns", true);
106. } else {
107. other_args.add(args[i]);
108. }
109. }
110.
111. FileInputFormat.setInputPaths(conf, new Path(other_args.get(0)));
112. FileOutputFormat.setOutputPath(conf, new Path(other_args.get(1)));
113.
114. JobClient.runJob(conf);
115. return 0;
116. }
117.
118. public static void main(String[] args) throws Exception {
119. int res = ToolRunner.run(new Configuration(), new WordCount(), args);
120. System.exit(res);
121. }
122. }
123.
榪愯鏍蜂緥
杈撳叆鏍蜂緥錛?/p>
$ bin/hadoop dfs -ls /usr/joe/wordcount/input/
/usr/joe/wordcount/input/file01
/usr/joe/wordcount/input/file02
$ bin/hadoop dfs -cat /usr/joe/wordcount/input/file01
Hello World, Bye World!
$ bin/hadoop dfs -cat /usr/joe/wordcount/input/file02
Hello Hadoop, Goodbye to hadoop.
榪愯紼嬪簭錛?/p>
$ bin/hadoop jar /usr/joe/wordcount.jar org.myorg.WordCount /usr/joe/wordcount/input /usr/joe/wordcount/output
杈撳嚭錛?/p>
$ bin/hadoop dfs -cat /usr/joe/wordcount/output/part-00000
Bye 1
Goodbye 1
Hadoop, 1
Hello 2
World! 1
World, 1
hadoop. 1
to 1
娉ㄦ剰姝ゆ椂鐨勮緭鍏ヤ笌絎竴涓増鏈殑涓嶅悓錛岃緭鍑虹殑緇撴灉涔熸湁涓嶅悓銆?/p>
鐜板湪閫氳繃DistributedCache鎻掑叆涓涓ā寮忔枃浠訛紝鏂囦歡涓繚瀛樹簡瑕佽蹇界暐鐨勫崟璇嶆ā寮忋?/p>
$ hadoop dfs -cat /user/joe/wordcount/patterns.txt
\.
\,
\!
to
鍐嶈繍琛屼竴嬈★紝榪欐浣跨敤鏇村鐨勯夐」錛?/p>
$ bin/hadoop jar /usr/joe/wordcount.jar org.myorg.WordCount -Dwordcount.case.sensitive=true /usr/joe/wordcount/input /usr/joe/wordcount/output -skip /user/joe/wordcount/patterns.txt
搴旇寰楀埌榪欐牱鐨勮緭鍑猴細
$ bin/hadoop dfs -cat /usr/joe/wordcount/output/part-00000
Bye 1
Goodbye 1
Hadoop 1
Hello 2
World 2
hadoop 1
鍐嶈繍琛屼竴嬈★紝榪欎竴嬈″叧闂ぇ灝忓啓鏁忔劅鎬э紙case-sensitivity錛夛細
$ bin/hadoop jar /usr/joe/wordcount.jar org.myorg.WordCount -Dwordcount.case.sensitive=false /usr/joe/wordcount/input /usr/joe/wordcount/output -skip /user/joe/wordcount/patterns.txt
杈撳嚭錛?/p>
$ bin/hadoop dfs -cat /usr/joe/wordcount/output/part-00000
bye 1
goodbye 1
hadoop 2
hello 2
world 2
紼嬪簭瑕佺偣
閫氳繃浣跨敤涓浜汳ap/Reduce妗嗘灦鎻愪緵鐨勫姛鑳斤紝WordCount鐨勭浜屼釜鐗堟湰鍦ㄥ師濮嬬増鏈熀紜涓婃湁浜嗗涓嬬殑鏀硅繘錛?/p>
灞曠ず浜嗗簲鐢ㄧ▼搴忓浣曞湪Mapper (鍜孯educer)涓氳繃configure鏂規硶 淇敼閰嶇疆鍙傛暟(28-43琛?銆?
灞曠ず浜嗕綔涓氬浣曚嬌鐢―istributedCache 鏉ュ垎鍙戝彧璇繪暟鎹?榪欓噷鍏佽鐢ㄦ埛鎸囧畾鍗曡瘝鐨勬ā寮忥紝鍦ㄨ鏁版椂蹇界暐閭d簺絎﹀悎妯″紡鐨勫崟璇?104琛?銆?
灞曠ずTool鎺ュ彛鍜孏enericOptionsParser澶勭悊Hadoop鍛戒護琛岄夐」鐨勫姛鑳?(87-116, 119琛?銆?
灞曠ず浜嗗簲鐢ㄧ▼搴忓浣曚嬌鐢–ounters(68琛?錛屽浣曢氳繃浼犻掔粰map錛堝拰reduce錛?鏂規硶鐨凴eporter瀹炰緥鏉ヨ緗簲鐢ㄧ▼搴忕殑鐘舵佷俊鎭?72琛?銆?
Java鍜孞NI鏄疭un Microsystems, Inc.鍦ㄧ編鍥藉拰鍏跺畠鍥藉鐨勬敞鍐屽晢鏍囥?/p>
鏈枃鏉ヨ嚜CSDN鍗氬錛岃漿杞借鏍囨槑鍑哄錛?a >http://blog.csdn.net/superxgl/archive/2010/01/11/5171929.aspx
涓銆?span>
hive 綆浠?/span>hive 鏄竴涓熀浜?/span> hadoop 鐨勫紑婧愭暟鎹粨搴撳伐鍏鳳紝鐢ㄤ簬瀛樺偍鍜屽鐞嗘搗閲忕粨鏋勫寲鏁版嵁銆?/span> 瀹冩妸嫻烽噺鏁版嵁瀛樺偍浜?/span> hadoop 鏂囦歡緋葷粺錛岃屼笉鏄暟鎹簱錛屼絾鎻愪緵浜嗕竴濂楃被鏁版嵁搴撶殑鏁版嵁瀛樺偍鍜屽鐞嗘満鍒訛紝騫墮噰鐢?/span> HQL 錛堢被 SQL 錛夎璦瀵硅繖浜涙暟鎹繘琛岃嚜鍔ㄥ寲綆$悊鍜屽鐞嗐傛垜浠彲浠ユ妸 hive 涓搗閲忕粨鏋勫寲鏁版嵁鐪嬫垚涓涓釜鐨勮〃錛岃屽疄闄呬笂榪欎簺鏁版嵁鏄垎甯冨紡瀛樺偍鍦?/span> HDFS 涓殑銆?/span> Hive 緇忚繃瀵硅鍙ヨ繘琛岃В鏋愬拰杞崲錛屾渶緇堢敓鎴愪竴緋誨垪鍩轟簬 hadoop 鐨?/span> map/reduce 浠誨姟錛岄氳繃鎵ц榪欎簺浠誨姟瀹屾垚鏁版嵁澶勭悊銆?/span>
Hive 璇炵敓浜?/span> facebook 鐨勬棩蹇楀垎鏋愰渶姹傦紝闈㈠嫻烽噺鐨勭粨鏋勫寲鏁版嵁錛?/span> hive 浠ヨ緝浣庣殑鎴愭湰瀹屾垚浜嗕互寰闇瑕佸ぇ瑙勬ā鏁版嵁搴撴墠鑳藉畬鎴愮殑浠誨姟錛屽茍涓斿涔犻棬妲涚浉瀵硅緝浣庯紝搴旂敤寮鍙戠伒媧昏岄珮鏁堛?/span>
Hive 鑷?/span> 2009.4.29 鍙戝竷絎竴涓畼鏂圭ǔ瀹氱増 0.3.0 鑷充粖錛屼笉榪囦竴騫寸殑鏃墮棿錛屾鍦ㄦ參鎱㈠畬鍠勶紝緗戜笂鑳芥壘鍒扮殑鐩稿叧璧勬枡鐩稿綋灝戯紝灝ゅ叾涓枃璧勬枡鏇村皯錛屾湰鏂囩粨鍚堜笟鍔″ hive 鐨勫簲鐢ㄥ仛浜嗕竴浜涙帰绱紝騫舵妸榪欎簺緇忛獙鍋氫竴涓葷粨錛屾墍璋撳墠杞︿箣閴達紝甯屾湜璇昏呰兘灝戣蛋涓浜涘集璺?/span>
Hive 鐨勫畼鏂?/span> wiki 璇峰弬鑰冭繖閲?/span> :
http://wiki.apache.org/hadoop/Hive
瀹樻柟涓婚〉鍦ㄨ繖閲岋細
http://hadoop.apache.org/hive/
hive-0.5.0 婧愮爜鍖呭拰浜岃繘鍒跺彂甯冨寘鐨勪笅杞藉湴鍧
http://labs.renren.com/apache-mirror/hadoop/hive/hive-0.5.0/
浜屻?span>
閮ㄧ講鐢變簬 Hive 鏄熀浜?/span> hadoop 鐨勫伐鍏鳳紝鎵浠?/span> hive 鐨勯儴緗查渶瑕佷竴涓甯歌繍琛岀殑 hadoop 鐜銆備互涓嬩粙緇?/span> hive 鐨勭畝鍗曢儴緗插拰搴旂敤銆?/span>
閮ㄧ講鐜錛?/span>
鎿嶄綔緋葷粺錛?/span> Red Hat Enterprise Linux AS release 4 (Nahant Update 7)
Hadoop 錛?/span> hadoop-0.20.2 錛屾甯歌繍琛?/span>
閮ㄧ講姝ラ濡備笅錛?/span>
1銆?/span> 涓嬭澆鏈鏂扮増鏈彂甯冨寘 hive-0.5.0-dev.tar.gz 錛屼紶鍒?/span> hadoop 鐨?/span> namenode 鑺傜偣涓婏紝瑙e帇寰楀埌 hive 鐩綍銆傚亣璁捐礬寰勪負錛?/span> /opt/hadoop/hive-0.5.0-bin
2銆?/span> 璁劇疆鐜鍙橀噺 HIVE_HOME 錛屾寚鍚?/span> hive 鏍圭洰褰?/span> /opt/hadoop/hive-0.5.0-bin 銆傜敱浜?/span> hadoop 宸茶繍琛岋紝媯鏌ョ幆澧冨彉閲?/span> JAVA_HOME 鍜?/span> HADOOP_HOME 鏄惁姝g‘鏈夋晥銆?/span>
3銆?/span> 鍒囨崲鍒?/span> $HIVE_HOME 鐩綍錛?/span> hive 閰嶇疆榛樿鍗沖彲錛岃繍琛?/span> bin/hive 鍗沖彲鍚姩 hive 錛屽鏋滄甯稿惎鍔紝灝嗕細鍑虹幇“ hive> ”鎻愮ず絎︺?/span>
4銆?/span> 鍦ㄥ懡浠ゆ彁紺虹涓緭鍏?#8220; show tables; ”錛屽鏋滄甯歌繍琛岋紝璇存槑宸查儴緗叉垚鍔燂紝鍙緵浣跨敤銆?/span>
甯歌闂錛?/span>
1銆?/span> 鎵ц“ show tables; ”鍛戒護鎻愮ず“ FAILED: Error in metadata: java.lang.IllegalArgumentException: URI: does not have a scheme ”錛岃繖鏄敱浜?/span> hive 鎵句笉鍒板瓨鏀懼厓鏁版嵁搴撶殑鏁版嵁搴撹屽鑷寸殑錛屼慨鏀?/span> conf/ hive-default.xml 閰嶇疆鏂囦歡涓殑 hive.metastore.local 涓?/span> true 鍗沖彲銆傜敱浜?/span> hive 鎶婄粨鏋勫寲鏁版嵁鐨勫厓鏁版嵁淇℃伅鏀懼湪絎笁鏂規暟鎹簱錛屾澶勮緗負 true 錛?/span> hive 灝嗗湪鏈湴鍒涘緩 derby 鏁版嵁搴撶敤浜庡瓨鏀懼厓鏁版嵁銆傚綋鐒跺鏋滄湁闇瑕佷篃鍙互閲囩敤 mysql 絳夌涓夋柟鏁版嵁搴撳瓨鏀懼厓鏁版嵁錛屼笉榪囪繖鏃?/span> hive.metastore.local 鐨勯厤緗煎簲涓?/span> false 銆?/span>
2銆?/span> 濡傛灉浣犲凡鏈変竴濂?/span> nutch1.0 緋葷粺姝e湪璺戯紝鑰屼綘涓嶆兂鍗曠嫭鍐嶅幓閮ㄧ講涓濂?/span> hadoop 鐜錛屼綘鍙互鐩存帴浣跨敤 nutch1.0 鑷甫鐨?/span> hadoop 鐜錛屼絾榪欐牱鐨勯儴緗蹭細瀵艱嚧 hive 涓嶈兘姝e父榪愯錛屾彁紺烘壘涓嶅埌鏌愪簺鏂規硶銆傝繖鏄敱浜?/span> nutch1.0 浣跨敤浜?/span> commons-lang-2.1.jar 榪欎釜鍖咃紝鑰?/span> hive 闇瑕佺殑鏄?/span> commons-lang-2.4.jar 錛屼笅杞戒竴涓?/span> 2.4 鐗堟湰鐨勫寘鏇挎崲鎺?/span> 2.1 鍗沖彲錛?/span> nutch 鍜?/span> hive 閮借兘姝e父榪愯銆?/span>
涓夈?span> 搴旂敤鍦烘櫙
鏈枃涓昏璁茶堪浣跨敤 hive 鐨勫疄璺碉紝涓氬姟涓嶆槸鍏抽敭錛岀畝瑕佷粙緇嶄笟鍔″満鏅紝鏈鐨勪換鍔℃槸瀵規悳绱㈡棩蹇楁暟鎹繘琛岀粺璁″垎鏋愩?/span>
闆嗗洟鎼滅儲鍒氫笂綰夸笉涔咃紝鏃ュ織閲忓茍涓嶅ぇ 銆傝繖浜涙棩蹇楀垎甯冨湪 5 鍙板墠绔満錛屾寜灝忔椂淇濆瓨錛屽茍浠ュ皬鏃朵負鍛ㄦ湡瀹氭椂灝嗕笂涓灝忔椂浜х敓鐨勬暟鎹悓姝ュ埌鏃ュ織鍒嗘瀽鏈猴紝緇熻鏁版嵁瑕佹眰鎸夊皬鏃舵洿鏂般傝繖浜涚粺璁¢」錛屽寘鎷叧閿瘝鎼滅儲閲?/span> pv 錛岀被鍒闂噺錛屾瘡縐掕闂噺 tps 絳夌瓑銆?/span>
鍩轟簬 hive 錛屾垜浠皢榪欎簺鏁版嵁鎸夊ぉ涓哄崟浣嶅緩琛紝姣忓ぉ涓涓〃錛屽悗鍙拌剼鏈牴鎹椂闂存埑灝嗘瘡灝忔椂鍚屾榪囨潵鐨?/span> 5 鍙板墠绔満鐨勬棩蹇楁暟鎹悎騫舵垚涓涓棩蹇楁枃浠訛紝瀵煎叆 hive 緋葷粺錛屾瘡灝忔椂鍚屾鐨勬棩蹇楁暟鎹榪藉姞鍒板綋澶╂暟鎹〃涓紝瀵煎叆瀹屾垚鍚庯紝褰撳ぉ鍚勯」緇熻欏瑰皢琚噸鏂拌綆楀茍杈撳嚭緇熻緇撴灉銆?/span>
浠ヤ笂闇姹傝嫢鐩存帴鍩轟簬 hadoop 寮鍙戯紝闇瑕佽嚜琛岀鐞嗘暟鎹紝閽堝澶氫釜緇熻闇姹傚紑鍙戜笉鍚岀殑 map/reduce 榪愮畻浠誨姟錛屽鍚堝茍銆佹帓搴忕瓑澶氶」鎿嶄綔榪涜瀹氬埗錛屽茍媯嫻嬩換鍔¤繍琛岀姸鎬侊紝宸ヤ綔閲忓茍涓嶅皬銆備絾浣跨敤 hive 錛屼粠瀵煎叆鍒板垎鏋愩佹帓搴忋佸幓閲嶃佺粨鏋滆緭鍑猴紝榪欎簺鎿嶄綔閮藉彲浠ヨ繍鐢?/span> hql 璇彞鏉ヨВ鍐籌紝涓鏉¤鍙ョ粡榪囧鐞嗚瑙f瀽鎴愬嚑涓換鍔℃潵榪愯錛屽嵆浣挎槸鍏抽敭璇嶈闂噺澧為噺榪欑闇瑕佸悓鏃惰闂澶╂暟鎹殑杈冧負澶嶆潅鐨勯渶姹備篃鑳介氳繃琛ㄥ叧鑱旇繖鏍風殑璇彞鑷姩瀹屾垚錛岃妭鐪佷簡澶ч噺宸ヤ綔閲忋?/span>
鍥涖?span> Hive 瀹炴垬
鍒濇浣跨敤 hive 錛屽簲璇ヨ涓婃墜榪樻槸鎸哄揩鐨勩?/span> Hive 鎻愪緵鐨勭被 SQL 璇彞涓?/span> mysql 璇彞鏋佷負鐩鎬技錛岃娉曚笂鏈夊ぇ閲忕浉鍚岀殑鍦版柟錛岃繖緇欐垜浠笂鎵嬪甫鏉ヤ簡寰堝ぇ鐨勬柟渚匡紝浣嗘槸瑕佸緱蹇冨簲鎵嬪湴鍐欏ソ榪欎簺璇彞錛岃繕闇瑕佸 hive 鏈夎緝濂界殑浜嗚В錛屾墠鑳界粨鍚?/span> hive 鐗硅壊鍐欏嚭綺懼鐨勮鍙ャ?/span>
鍏充簬 hive 璇█鐨勮緇嗚娉曞彲鍙傝冨畼鏂?/span> wiki 鐨勮璦鎵嬪唽 :
http://wiki.apache.org/hadoop/Hive/LanguageManual
铏界劧璇硶椋庢牸涓烘垜浠彁渚涗簡渚垮埄錛屼絾鍒濇浣跨敤閬囧埌鐨勯棶棰樿繕鏄笉灝戠殑錛屼笅闈㈤拡瀵逛笟鍔″満鏅皥璋堟垜浠亣鍒扮殑闂錛屽拰瀵?/span> hive 鍔熻兘鐨勫畾鍒躲?/span>
1銆?/span> 鍒嗛殧絎﹂棶棰?/span>
棣栧厛閬囧埌鐨勬槸鏃ュ織鏁版嵁鐨勫垎闅旂闂錛屾垜浠殑鏃ュ織鏁版嵁鐨勫ぇ鑷存牸寮忓涓嬶細
2010-05-24 00:00:02@$_$@QQ2010@$_$@all@$_$@NOKIA_1681C@$_$@1@$_$@10@$_$@@$_$@-1@$_$@10@$_$@application@$_$@1
浠庢牸寮忓彲瑙佸叾鍒嗛殧絎︽槸“ @$_$@ ”錛岃繖鏄負浜嗗敖鍙兘闃叉鏃ュ織姝f枃鍑虹幇涓庡垎闅旂鐩稿悓鐨勫瓧絎﹁屽鑷存暟鎹販娣嗐傛湰鏉?/span> hive鏀寔鍦ㄥ緩琛ㄧ殑鏃跺欐寚瀹氳嚜瀹氫箟鍒嗛殧絎︾殑錛屼絾緇忚繃澶氭嫻嬭瘯鍙戠幇鍙敮鎸佸崟涓瓧絎︾殑鑷畾涔夊垎闅旂錛屽儚“ @$_$@ ”榪欐牱鐨勫垎闅旂鏄笉鑳借鏀寔鐨勶紝浣嗘槸鎴戜滑鍙互閫氳繃瀵瑰垎闅旂鐨勫畾鍒惰В鍐寵繖涓棶棰橈紝 hive 鐨勫唴閮ㄥ垎闅旂鏄?#8220; \001 ”錛屽彧瑕佹妸鍒嗛殧絎︽浛鎹㈡垚“\001 ”鍗沖彲銆?/span>
緇忚繃鎺㈢儲鎴戜滑鍙戠幇鏈変袱鏉¢斿緞瑙e喅榪欎釜闂銆?/span>
a) 鑷畾涔?/span> outputformat 鍜?/span> inputformat 銆?/span>
Hive 鐨?/span> outputformat/inputformat 涓?/span> hadoop 鐨?/span> outputformat/inputformat 鐩稿綋綾諱技錛?/span> inputformat 璐熻矗鎶婅緭鍏ユ暟鎹繘琛屾牸寮忓寲錛岀劧鍚庢彁渚涚粰 hive 錛?/span> outputformat 璐熻矗鎶?/span> hive 杈撳嚭鐨勬暟鎹噸鏂版牸寮忓寲鎴愮洰鏍囨牸寮忓啀杈撳嚭鍒版枃浠訛紝榪欑瀵規牸寮忚繘琛屽畾鍒剁殑鏂瑰紡杈冧負搴曞眰錛屽鍏惰繘琛屽畾鍒朵篃鐩稿綆鍗曪紝閲嶅啓 InputFormat 涓?/span> RecordReader 綾諱腑鐨?/span> next 鏂規硶鍗沖彲錛岀ず渚嬩唬鐮佸涓嬶細
public boolean next(LongWritable key, BytesWritable value)
throws IOException {
while ( reader .next(key, text ) ) {
String strReplace = text .toString().toLowerCase().replace( "@$_$@" , "\001" );
Text txtReplace = new Text();
txtReplace.set(strReplace );
value.set(txtReplace.getBytes(), 0, txtReplace.getLength());
return true ;
}
return false ;
}
閲嶅啓 HiveIgnoreKeyTextOutputFormat 涓?/span> RecordWriter 涓殑 write 鏂規硶錛岀ず渚嬩唬鐮佸涓嬶細
public void write (Writable w) throws IOException {
String strReplace = ((Text)w).toString().replace( "\001" , "@$_$@" );
Text txtReplace = new Text();
txtReplace.set(strReplace);
byte [] output = txtReplace.getBytes();
bytesWritable .set(output, 0, output. length );
writer .write( bytesWritable );
}
鑷畾涔?/span> outputformat/inputformat 鍚庯紝鍦ㄥ緩琛ㄦ椂闇瑕佹寚瀹?/span> outputformat/inputformat 錛屽涓嬬ず渚嬶細
stored as INPUTFORMAT 'com.aspire.search.loganalysis.hive.SearchLogInputFormat' OUTPUTFORMAT 'com.aspire.search.loganalysis.hive.SearchLogOutputFormat'
b) 閫氳繃 SerDe(serialize/deserialize) 錛屽湪鏁版嵁搴忓垪鍖栧拰鍙嶅簭鍒楀寲鏃舵牸寮忓寲鏁版嵁銆?/span>
榪欑鏂瑰紡紼嶅井澶嶆潅涓鐐癸紝瀵規暟鎹殑鎺у埗鑳藉姏涔熻寮變竴浜涳紝瀹冧嬌鐢ㄦ鍒欒〃杈懼紡鏉ュ尮閰嶅拰澶勭悊鏁版嵁錛屾ц兘涔熶細鏈夋墍褰卞搷銆備絾瀹冪殑浼樼偣鏄彲浠ヨ嚜瀹氫箟琛ㄥ睘鎬т俊鎭?/span> SERDEPROPERTIES 錛屽湪 SerDe 涓氳繃榪欎簺灞炴т俊鎭彲浠ユ湁鏇村鐨勫畾鍒惰涓恒?/span>
2銆?/span> 鏁版嵁瀵煎叆瀵煎嚭
a) 澶氱増鏈棩蹇楁牸寮忕殑鍏煎
鐢變簬 hive 鐨勫簲鐢ㄥ満鏅富瑕佹槸澶勭悊鍐鋒暟鎹紙鍙涓嶅啓錛夛紝鍥犳瀹冨彧鏀寔鎵歸噺瀵煎叆鍜屽鍑烘暟鎹紝騫朵笉鏀寔鍗曟潯鏁版嵁鐨勫啓鍏ユ垨鏇存柊錛屾墍浠ュ鏋滆瀵煎叆鐨勬暟鎹瓨鍦ㄦ煇浜涗笉澶鑼冪殑琛岋紝鍒欓渶瑕佹垜浠畾鍒朵竴浜涙墿灞曞姛鑳藉鍏惰繘琛屽鐞嗐?/span>
鎴戜滑闇瑕佸鐞嗙殑鏃ュ織鏁版嵁瀛樺湪澶氫釜鐗堟湰錛屽悇涓増鏈瘡涓瓧孌電殑鏁版嵁鍐呭瀛樺湪涓浜涘樊寮傦紝鍙兘鐗堟湰 A 鏃ュ織鏁版嵁鐨勭浜屼釜鍒楁槸鎼滅儲鍏抽敭瀛楋紝浣嗙増鏈?/span> B 鐨勭浜屽垪鍗存槸鎼滅儲鐨勭粓绔被鍨嬶紝濡傛灉榪欎袱涓増鏈殑鏃ュ織鐩存帴瀵煎叆 hive 涓紝寰堟槑鏄炬暟鎹皢浼氭販涔憋紝緇熻緇撴灉涔熶笉浼氭紜傛垜浠殑浠誨姟鏄浣垮涓増鏈殑鏃ュ織鏁版嵁鑳藉湪 hive 鏁版嵁浠撳簱涓叡瀛橈紝涓旇〃鐨?/span> input/output 鎿嶄綔鑳藉鏈緇堟槧灝勫埌姝g‘鐨勬棩蹇楃増鏈殑姝g‘瀛楁銆?/span>
榪欓噷鎴戜滑涓嶅叧蹇冭繖閮ㄥ垎綣佺悙鐨勫伐浣滐紝鍙叧蹇冩妧鏈疄鐜扮殑鍏抽敭鐐癸紝榪欎釜鍔熻兘璇ュ湪鍝噷瀹炵幇鎵嶈兘璁?/span> hive 璁ゅ緱榪欎簺涓嶅悓鏍煎紡鐨勬暟鎹憿錛熺粡榪囧鏂瑰皾璇曪紝鍦ㄤ腑闂翠換浣曠幆鑺傚仛榪欎釜鐗堟湰閫傞厤閮藉皢瀵艱嚧澶嶆潅鍖栵紝鏈緇堣繖涓伐浣滆繕鏄湪 inputformat/outputformat 涓畬鎴愭渶涓轟紭闆咃紝姣曠珶 inputformat 鏄簮澶達紝 outputformat 鏄渶緇堝綊瀹褲傚叿浣撴潵璇達紝鏄湪鍓嶉潰鎻愬埌鐨?/span> inputformat 鐨?/span> next 鏂規硶涓拰鍦?/span> outputformat 鐨?/span> write 鏂規硶涓畬鎴愯繖涓傞厤宸ヤ綔銆?/span>
b) Hive 鎿嶄綔鏈湴鏁版嵁
涓寮濮嬶紝鎬繪槸鎶婃湰鍦版暟鎹厛浼犲埌 HDFS 錛屽啀鐢?/span> hive 鎿嶄綔 hdfs 涓婄殑鏁版嵁錛岀劧鍚庡啀鎶婃暟鎹粠 HDFS 涓婁紶鍥炴湰鍦版暟鎹傚悗鏉ュ彂鐜板ぇ鍙笉蹇呭姝わ紝 hive 璇彞閮芥彁渚涗簡“ local ”鍏抽敭瀛楋紝鏀寔鐩存帴浠庢湰鍦板鍏ユ暟鎹埌 hive 錛屼篃鑳戒粠 hive 鐩存帴瀵煎嚭鏁版嵁鍒版湰鍦幫紝涓嶈繃鍏跺唴閮ㄨ綆楁椂褰撶劧鏄敤 HDFS 涓婄殑鏁版嵁錛屽彧鏄嚜鍔ㄤ負鎴戜滑瀹屾垚瀵煎叆瀵煎嚭鑰屽凡銆?/span>
3銆?/span> 鏁版嵁澶勭悊
鏃ュ織鏁版嵁鐨勭粺璁″鐞嗗湪榪欓噷鍙嶅掓病鏈変粈涔堢壒鍒箣澶勶紝灝辨槸涓浜?/span> SQL 璇彞鑰屽凡錛屼篃娌℃湁浠涔堥珮娣辯殑鎶宸э紝涓嶈繃榪樻槸鍒椾婦涓浜涜鍙ョず渚嬶紝浠ョず hive 澶勭悊鏁版嵁鐨勬柟渚夸箣澶勶紝騫跺睍紺?/span> hive 鐨勪竴浜涚敤娉曘?/span>
a) 涓?/span> hive 娣誨姞鐢ㄦ埛瀹氬埗鍔熻兘錛岃嚜瀹氫箟鍔熻兘閮戒綅浜?/span> hive_contrib.jar 鍖呬腑
add jar /opt/hadoop/hive-0.5.0-bin/lib/hive_contrib.jar;
b) 緇熻姣忎釜鍏抽敭璇嶇殑鎼滅儲閲忥紝騫舵寜鎼滅儲閲忛檷搴忔帓鍒楋紝鐒跺悗鎶婄粨鏋滃瓨鍏ヨ〃 keyword_20100603 涓?/span>
create table keyword_20100603 as select keyword,count(keyword) as count from searchlog_20100603 group by keyword order by count desc;
c) 緇熻姣忕被鐢ㄦ埛緇堢鐨勬悳绱㈤噺錛屽茍鎸夋悳绱㈤噺闄嶅簭鎺掑垪錛岀劧鍚庢妸緇撴灉瀛樺叆琛?/span> device_20100603 涓?/span>
create table device_20100603 as select device,count(device) as count from searchlog_20100603 group by device order by count desc;
d) 鍒涘緩琛?/span> time_20100603 錛屼嬌鐢ㄨ嚜瀹氫箟鐨?/span> INPUTFORMAT 鍜?/span> OUTPUTFORMAT 錛屽茍鎸囧畾琛ㄦ暟鎹殑鐪熷疄瀛樻斁浣嶇疆鍦?/span> '/LogAnalysis/results/time_20100603' 錛?/span> HDFS 璺緞錛夛紝鑰屼笉鏄斁鍦?/span> hive 鑷繁鐨勬暟鎹洰褰曚腑
create external table if not exists time_20100603(time string, count int) stored as INPUTFORMAT 'com.aspire.search.loganalysis.hive.XmlResultInputFormat' OUTPUTFORMAT 'com.aspire.search.loganalysis.hive.XmlResultOutputFormat' LOCATION '/LogAnalysis/results/time_20100603';
e) 緇熻姣忕璁塊棶閲?/span> TPS 錛屾寜璁塊棶閲忛檷搴忔帓鍒楋紝騫舵妸緇撴灉杈撳嚭鍒拌〃 time_20100603 涓紝榪欎釜琛ㄦ垜浠湪涓婇潰鍒氬垰瀹氫箟榪囷紝鍏剁湡瀹炰綅緗湪 '/LogAnalysis/results/time_20100603' 錛屽茍涓旂敱浜?/span> XmlResultOutputFormat 鐨勬牸寮忓寲錛屾枃浠跺唴瀹規槸 XML 鏍煎紡銆?/span>
insert overwrite table time_20100603 select time,count(time) as count from searchlog_20100603 group by time order by count desc;
f) 璁$畻姣忎釜鎼滅儲璇鋒眰鍝嶅簲鏃墮棿鐨勬渶澶у鹼紝鏈灝忓煎拰騫沖潎鍊?/span>
insert overwrite table response_20100603 select max(responsetime) as max,min(responsetime) as min,avg(responsetime) as avg from searchlog_20100603;
g) 鍒涘緩涓涓〃鐢ㄤ簬瀛樻斁浠婂ぉ涓庢槰澶╃殑鍏抽敭璇嶆悳绱㈤噺鍜屽閲忓強鍏跺閲忔瘮鐜囷紝琛ㄦ暟鎹綅浜?/span> '/LogAnalysis/results/keyword_20100604_20100603' 錛屽唴瀹瑰皢鏄?/span> XML 鏍煎紡銆?/span>
create external table if not exists keyword_20100604_20100603(keyword string, count int, increment int, incrementrate double) stored as INPUTFORMAT 'com.aspire.search.loganalysis.hive.XmlResultInputFormat' OUTPUTFORMAT 'com.aspire.search.loganalysis.hive.XmlResultOutputFormat' LOCATION '/LogAnalysis/results/keyword_20100604_20100603';
h) 璁劇疆琛ㄧ殑灞炴э紝浠ヤ究 XmlResultInputFormat 鍜?/span> XmlResultOutputFormat 鑳芥牴鎹?/span> output.resulttype 鐨勪笉鍚屽唴瀹硅緭鍑轟笉鍚屾牸寮忕殑 XML 鏂囦歡銆?/span>
alter table keyword_20100604_20100603 set tblproperties ('output.resulttype'='keyword');
i) 鍏寵仈浠婂ぉ鍏抽敭璇嶇粺璁$粨鏋滆〃錛?/span> keyword_20100604 錛変笌鏄ㄥぉ鍏抽敭璇嶇粺璁$粨鏋滆〃錛?/span> keyword_20100603 錛夛紝緇熻浠婂ぉ涓庢槰澶╁悓鏃跺嚭鐜扮殑鍏抽敭璇嶇殑鎼滅儲嬈℃暟錛屼粖澶╃浉瀵規槰澶╃殑澧為噺鍜屽閲忔瘮鐜囷紝騫舵寜澧為噺姣旂巼闄嶅簭鎺掑垪錛岀粨鏋滆緭鍑哄埌鍒氬垰瀹氫箟鐨?/span> keyword_20100604_20100603 琛ㄤ腑錛屽叾鏁版嵁鏂囦歡鍐呭灝嗕負 XML 鏍煎紡銆?/span>
insert overwrite table keyword_20100604_20100603 select cur.keyword, cur.count, cur.count-yes.count as increment, (cur.count-yes.count)/yes.count as incrementrate from keyword_20100604 cur join keyword_20100603 yes on (cur.keyword = yes.keyword) order by incrementrate desc;
j)
4銆?/span> 鐢ㄦ埛鑷畾涔夊嚱鏁?/span> UDF
閮ㄥ垎緇熻緇撴灉闇瑕佷互 CSV 鐨勬牸寮忚緭鍑猴紝瀵逛簬榪欑被鏂囦歡浣撳叏鏄湁鏁堝唴瀹圭殑鏂囦歡錛屼笉闇瑕佸儚 XML 涓鏍峰寘鍚?/span> version 錛?/span> encoding 絳変俊鎭殑鏂囦歡澶達紝鏈閫傚悎鐢?/span> UDF(user define function) 浜嗐?/span>
UDF 鍑芥暟鍙洿鎺ュ簲鐢ㄤ簬 select 璇彞錛屽鏌ヨ緇撴瀯鍋氭牸寮忓寲澶勭悊涔嬪悗錛屽啀杈撳嚭鍐呭銆傝嚜瀹氫箟 UDF 闇瑕佺戶鎵?/span> org.apache.hadoop.hive.ql.exec.UDF 錛屽茍瀹炵幇 evaluate 鍑芥暟錛?/span> Evaluate 鍑芥暟鏀寔閲嶈澆錛岃繕鏀寔鍙彉鍙傛暟銆傛垜浠疄鐜頒簡涓涓敮鎸佸彲鍙樺瓧絎︿覆鍙傛暟鐨?/span> UDF 錛屾敮鎸佹妸 select 寰楀嚭鐨勪換鎰忎釜鏁扮殑涓嶅悓綾誨瀷鏁版嵁杞崲涓哄瓧絎︿覆鍚庯紝鎸?/span> CSV 鏍煎紡杈撳嚭錛岀敱浜庝唬鐮佽緝綆鍗曪紝榪欓噷緇欏嚭婧愮爜紺轟緥錛?/span>
public String evaluate(String... strs) {
StringBuilder sb = new StringBuilder();
for ( int i = 0; i < strs. length ; i++) {
sb.append(ConvertCSVField(strs[i])).append( ',' );
}
sb.deleteCharAt(sb.length()-1);
return sb.toString();
}
闇瑕佹敞鎰忕殑鏄紝瑕佷嬌鐢?/span> UDF 鍔熻兘錛岄櫎浜嗗疄鐜拌嚜瀹氫箟 UDF 澶栵紝榪橀渶瑕佸姞鍏ュ寘鍚?/span> UDF 鐨勫寘錛岀ず渚嬶細
add jar /opt/hadoop/hive-0.5.0-bin/lib/hive_contrib.jar;
鐒跺悗鍒涘緩涓存椂鏂規硶錛岀ず渚嬶細
CREATE TEMPORARY FUNCTION Result2CSv AS ‘com.aspire.search.loganalysis.hive. Result2CSv';
浣跨敤瀹屾瘯榪樿 drop 鏂規硶錛岀ず渚嬶細
DROP TEMPORARY FUNCTION Result2CSv;
5銆?/span> 杈撳嚭 XML 鏍煎紡鐨勭粺璁$粨鏋?/span>
鍓嶉潰鐪嬪埌閮ㄥ垎鏃ュ織緇熻緇撴灉杈撳嚭鍒頒竴涓〃涓紝鍊熷姪 XmlResultInputFormat 鍜?/span> XmlResultOutputFormat 鏍煎紡鍖栨垚 XML 鏂囦歡錛岃冭檻鍒板垱寤鴻繖涓〃鍙槸涓轟簡寰楀埌 XML 鏍煎紡鐨勮緭鍑烘暟鎹紝鎴戜滑鍙渶瀹炵幇 XmlResultOutputFormat 鍗沖彲錛屽鏋滆繕瑕佹敮鎸?/span> select 鏌ヨ錛屽垯鎴戜滑榪橀渶瑕佸疄鐜?/span> XmlResultInputFormat 錛岃繖閲屾垜浠彧浠嬬粛 XmlResultOutputFormat 銆?/span>
鍓嶉潰浠嬬粛榪囷紝瀹氬埗 XmlResultOutputFormat 鎴戜滑鍙渶閲嶅啓 write 鍗沖彲錛岃繖涓柟娉曞皢浼氭妸 hive 鐨勪互 ’\001’ 鍒嗛殧鐨勫瀛楁鏁版嵁鏍煎紡鍖栦負鎴戜滑闇瑕佺殑 XML 鏍煎紡錛岃綆鍖栫殑紺轟緥浠g爜濡備笅錛?/span>
public void write(Writable w) throws IOException {
String[] strFields = ((Text) w).toString().split( "\001" );
StringBuffer sbXml = new StringBuffer();
if ( strResultType .equals( "keyword" )) {
sbXml.append( "<record><keyword>" ).append(strFields[0]).append(
"</keyword><count>" ).append(strFields[1]).append( "</count><increment>" ).append(strFields[2]).append(
"</increment><rate>" ).append(strFields[3]).append(
"</rate></result>" );
}
Text txtXml = new Text();
byte [] strBytes = sbXml.toString().getBytes( "utf-8" );
txtXml.set(strBytes, 0, strBytes. length );
byte [] output = txtXml.getBytes();
bytesWritable .set(output, 0, output. length );
writer .write( bytesWritable );
}
鍏朵腑鐨?/span> strResultType .equals( "keyword" ) 鎸囧畾鍏抽敭璇嶇粺璁$粨鏋滐紝榪欎釜灞炴ф潵鑷互涓嬭鍙ュ緇撴灉綾誨瀷鐨勬寚瀹氾紝閫氳繃榪欎釜灞炴ф垜浠繕鍙互鐢ㄥ悓涓涓?/span> outputformat 杈撳嚭澶氱綾誨瀷鐨勭粨鏋溿?/span>
alter table keyword_20100604_20100603 set tblproperties ('output.resulttype'='keyword');
浠旂粏鐪嬬湅 write 鍑芥暟鐨勫疄鐜頒究鍙彂鐜幫紝鍏跺疄榪欓噷鍙緭鍑轟簡 XML 鏂囦歡鐨勬鏂囷紝鑰?/span> XML 鐨勬枃浠跺ご鍜岀粨鏉熸爣絳懼湪鍝噷杈撳嚭鍛紵鎵騫告垜浠噰鐢ㄧ殑鏄熀浜?/span> outputformat 鐨勫疄鐜幫紝鎴戜滑鍙互鍦ㄦ瀯閫犲嚱鏁拌緭鍑?/span> version 錛?/span> encoding 絳夋枃浠跺ご淇℃伅錛屽湪 close() 鏂規硶涓緭鍑虹粨鏉熸爣絳俱?/span>
榪欎篃鏄垜浠負浠涔堜笉浣跨敤 UDF 鏉ヨ緭鍑虹粨鏋滅殑鍘熷洜錛岃嚜瀹氫箟 UDF 鍑芥暟涓嶈兘杈撳嚭鏂囦歡澶村拰鏂囦歡灝撅紝瀵逛簬 XML 鏍煎紡鐨勬暟鎹棤娉曡緭鍑哄畬鏁存牸寮忥紝鍙兘杈撳嚭 CSV 榪欑被鎵鏈夎閮芥槸鏈夋晥鏁版嵁鐨勬枃浠躲?/span>
浜斻?span> 鎬葷粨
Hive 鏄竴涓彲鎵╁睍鎬ф瀬寮虹殑鏁版嵁浠撳簱宸ュ叿錛屽熷姪浜?/span> hadoop 鍒嗗竷寮忓瓨鍌ㄨ綆楀鉤鍙板拰 hive 瀵?/span> SQL 璇彞鐨勭悊瑙h兘鍔涳紝鎴戜滑鎵瑕佸仛鐨勫ぇ閮ㄥ垎宸ヤ綔灝辨槸杈撳叆鍜岃緭鍑烘暟鎹殑閫傞厤錛屾伆鎭拌繖涓ら儴鍒?/span> IO 鏍煎紡鏄崈鍙樹竾鍖栫殑錛屾垜浠彧闇瑕佸畾鍒舵垜浠嚜宸辯殑杈撳叆杈撳嚭閫傞厤鍣紝 hive灝嗕負鎴戜滑閫忔槑鍖栧瓨鍌ㄥ拰澶勭悊榪欎簺鏁版嵁錛屽ぇ澶х畝鍖栨垜浠殑宸ヤ綔銆傛湰鏂囩殑閲嶅績涔熸鍦ㄤ簬姝わ紝榪欓儴鍒嗗伐浣滅浉淇℃瘡涓涓仛鏁版嵁鍒嗘瀽鐨勬湅鍙嬮兘浼氶潰瀵圭殑錛屽笇鏈涘鎮ㄦ湁鐩娿?/span>
鏈枃浠嬬粛浜嗕竴嬈$浉褰撶畝鍗曠殑鍩轟簬 hive 鐨勬棩蹇楃粺璁″疄鎴橈紝瀵?/span> hive 鐨勮繍鐢ㄨ繕澶勪簬涓涓浉瀵硅緝嫻呯殑灞傞潰錛岀洰鍓嶅皻鑳芥弧瓚抽渶姹傘傚浜庝竴浜涜緝澶嶆潅鐨勬暟鎹垎鏋愪換鍔★紝浠ヤ笂鎵浠嬬粛鐨勭粡楠屽緢鍙兘鏄笉澶熺敤鐨勶紝鐢氳嚦鏄?/span> hive 鍋氫笉鍒扮殑錛?/span> hive 榪樻湁寰堝榪涢樁鍔熻兘錛岄檺浜庣瘒騫呮湰鏂囨湭鑳芥秹鍙婏紝寰呮棩鍚庣粨鍚堝叿浣撲換鍔″啀璇︾粏闃愯堪銆?/span>
濡傛偍瀵規湰鏂囨湁浠諱綍寤鴻鎴栨寚鏁欙紝璇瘋瘎璁猴紝璋㈣阿銆?/span>