锘??xml version="1.0" encoding="utf-8" standalone="yes"?>99久久免费国产精品,日韩久久无码免费毛片软件,久久精品国产亚洲精品2020 http://www.shnenglu.com/whspecial/category/19559.htmlzh-cnSun, 27 Oct 2013 15:52:56 GMTSun, 27 Oct 2013 15:52:56 GMT60璺ㄦ満鎴跨殑hadoop闆嗙兢http://www.shnenglu.com/whspecial/archive/2013/10/27/203940.htmlwhspecialwhspecialSun, 27 Oct 2013 15:28:00 GMThttp://www.shnenglu.com/whspecial/archive/2013/10/27/203940.htmlhttp://www.shnenglu.com/whspecial/comments/203940.htmlhttp://www.shnenglu.com/whspecial/archive/2013/10/27/203940.html#Feedback0http://www.shnenglu.com/whspecial/comments/commentRss/203940.htmlhttp://www.shnenglu.com/whspecial/services/trackbacks/203940.html

榪欐槸鏉ヨ嚜浜庨樋閲屾妧鏈槈騫村崕鐨勪竴涓垎浜紝鍥犱負鍦ㄧ櫨搴︿篃鑰冭檻榪囩被浼肩殑浜嬫儏錛屾墍浠ュ惉寰楁瘮杈冩湁鎰熸偀錛岃繖閲屾妸鐩稿叧鍐呭鏁寸悊涓涓嬨?/span>

棣栧厛灝婇噸鐗堟潈錛岃繕鏄妸鍘熼摼鎺ュ拰浣滆呰創涓婏細

http://adc.alibabatech.org/carnival/history/schedule/2013/detail/main/286?video=0

鏉ヨ嚜浜庨樋閲屽惔濞佸伐紼嬪笀鐨勫垎浜?/span>

 

棣栧厛闇瑕佽鏄庝竴鐐癸紝璺ㄦ満鎴?/span>hadoop鍙兘搴旂敤鍦烘櫙騫朵笉鏄緢澶氾紝鍥藉唴鍍?/span>BAT榪欑宸ㄥご涔熻闇瑕侊紝浣嗘槸澶ч儴鍒嗙殑涓皬鍏徃涔熻騫朵笉闇瑕佽繖涓紝涔熻榪欐槸涓睜榫欎箣鎶錛屽懙鍛點?/span>

鎶婅繖涓棶棰樺垎涓夋鏉ヨ錛岀涓孌墊槸闂鍑虹幇鐨勮儗鏅紝絎簩孌墊槸瑙e喅璇ラ棶棰樼殑闅劇偣錛岀涓夋鏄渶緇堢殑瑙e喅鏂規銆?/span>

錛堜竴錛?nbsp;鑳屾櫙錛?/span>

鍏堣鐪嬩笅涓轟粈涔堥渶瑕佸仛涓涓法鏈烘埧鐨勫ぇ闆嗙兢錛?/span>

澶ч泦緹ょ殑浼樼偣鍦ㄤ簬鏁版嵁綆$悊鍜屾巿鏉冨鏄擄紙榪欎釜闂鍦ㄤ竴涓閮ㄩ棬鐨勫ぇ鍏徃榪樻槸寰堥噸瑕佺殑錛夛紱璺ㄩ儴闂ㄧ殑浣跨敤鏁版嵁瀹規槗錛屾棤闇閲嶅鎷夊彇鏁版嵁銆?/span>

鍦ㄩ泦緹よ揪鍒頒竴瀹氳妯℃椂錛屽崟鏈烘埧錛堟満鎴垮唴鐨勫閲忔槸鏈夐檺鐨勶級宸茬粡鏃犳硶婊¤凍闆嗙兢鐨勯渶姹備簡錛岃鎯充竴鍔蟲案閫哥殑瑙e喅闂錛岄渶瑕佸緩璁句竴涓法鏈烘埧鐨?/span>hadoop闆嗙兢銆?/span>

錛堜簩錛夋妧鏈寫鎴橈細

2.1 NameNode鐨勬ц兘闂錛?/span>

         鍦ㄧ鐞嗕竴涓法澶х殑hadoop闆嗙兢鏃訛紝鐢變簬鍘熷鐨?/span>Namenode鏄崟鑺傜偣錛屽洜姝や細鎴愪負涓涓ц兘鐡墮錛岄亣鍒扮殑鎬ц兘闂涓昏鍖呮嫭涓ゆ柟闈細瀛樺偍瀹歸噺闂錛堝瓨鍌ㄥ厓鏁版嵁錛夊拰璁$畻鍘嬪姏錛堝鐞?/span>rpc璇鋒眰錛屼慨鏀瑰唴瀛樻爲鏃跺欓渶瑕佸叏灞閿侊級闂銆?/span>

         鍏朵腑瀛樺偍瀹歸噺闂鍙互渚濊禆鍐呭瓨鐨勫瀭鐩存墿灞曟潵瑙e喅錛屼絾鏄綆楀帇鍔涘嵈寰堥毦閫氳繃鎻愬崌紜歡鏉ヨВ鍐籌紙鍥犱負鐩墠鍘傚晢鐨勪富瑕佸彂灞曟柟鍚戞槸澶氭牳錛岃岄潪鎻愰珮涓婚錛?/span>

2.2鏈烘埧涔嬮棿鐨勭綉緇滈檺鍒訛細

         鏈烘埧涔嬮棿鐨勭綉緇滄案榪滄槸涓‖浠舵潯浠剁殑闄愬埗錛岃法鏈烘埧鐨勭綉緇滀紶杈撳甫鏉ヤ簡鏁版嵁寤舵椂鍜屽甫瀹介檺鍒訛細

1錛?nbsp;寤舵椂涓鑸槸鍦?/span>10ms涔嬪唴錛岃?/span>hadoop涓婂ぇ閮ㄥ垎榪愯鐨勬槸紱葷嚎浣滀笟錛屽熀鏈彲鎺ュ彈

2錛?nbsp;甯﹀闄愬埗鐨勯棶棰樻瘮杈冨ぇ錛屽洜涓哄崟鏈烘埧鍐呯殑鐐瑰鐐瑰甫瀹戒竴鑸槸鍦?/span>1Gbps錛岃屾満鎴夸箣闂寸殑甯﹀紜湪20Mbps宸﹀彸錛岄潪甯告湁闄愩?/span>

2.3璧勬簮緇勪箣闂寸殑綆$悊

         姣忎釜閮ㄩ棬鍙互鐪嬪仛涓涓祫婧愮粍錛屽畠浠彲鑳戒細浜掔浉浣跨敤瀵規柟鐨勬暟鎹紝鍥犳濡備綍瑙勫垝璁$畻鍜屽瓨鍌ㄧ殑浣嶇疆灝卞緢閲嶈錛屽惁鍒欎細鍦ㄥ涓満鎴夸箣闂村嚭鐜板ぇ閲忕殑鏁版嵁鎷瘋礉銆?/span>

錛堜笁錛夎В鍐蟲柟妗堬細

鍏堢湅涓嬫暣涓法闆嗙兢hadoop鐨勬灦鏋勫浘錛?/span>


 

閲嶇偣浠嬬粛閲岄潰涓夌偣錛屼篃灝辨槸鍜屼笂闈笁涓棶棰樼浉瀵瑰簲鐨勶細

1錛?nbsp;鍙互鐪嬪埌榪欓噷鐢誨嚭浜嗕袱涓?/span>NN錛?/span>namenode錛夛紝瀹冧滑瀹為檯涓婅繕鏄睘浜庝竴涓?/span>hadoop闆嗙兢錛岃繖鏄笟鐣岄噷鐨勪竴涓В鍐蟲柟妗堬細HDFS Fedaration錛屽畠涓轟簡瑙e喅鍏冩暟鎹妭鐐規ц兘闂錛?/span>

2錛?nbsp;鍙互鐪嬪埌榪欓噷鏈変竴涓?/span>cross node鑺傜偣錛屽畠鏄敤鏉ュ湪涓や釜鏈烘埧涔嬮棿鍚屾鏁版嵁鐨勶紝瀹冪殑璁捐鑰冭檻鍒頒簡鏈烘埧闂寸殑緗戠粶闄愬埗錛?/span>

3錛?nbsp;鏈鍚庢槸groupA銆?/span>groupB錛岃繖鏄負浜嗚В鍐蟲暟鎹駭鍑烘柟鍜屼嬌鐢ㄦ柟鍏崇郴鏉ョ敤鐨勩?/span>

3.1 Federation

Federation鐩稿叧璧勬枡瑙侊細

http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/Federation.html#HDFS_Federation


涓轟簡姘村鉤鎵╁睍Namenode錛?/span>federation浣跨敤浜嗗涓簰鐩哥嫭绔嬬殑namenode銆傚畠浠箣闂翠簰鐩鎬笉闇瑕侀氫俊錛屾瘡涓?/span>datenode闇瑕佸悜鍏ㄩ儴namenode娉ㄥ唽騫跺彂閫佷俊鎭?/span>

BlockPool鏄睘浜庝竴涓?/span>namenode鐨?/span>block闆嗗悎錛屾瘡涓?/span>blockpool涔嬮棿涔熸槸浜掔浉鐙珛鐨勩?/span>

         鍦?/span>federation閲岋紝鏈変竴涓渶瑕佸叧娉ㄧ殑闂錛屽氨鏄涓?/span>namenode鐨勫湴鍧濡備綍瀵圭敤鎴瘋繘琛岄忔槑錛熷畠閲囩敤鐨勮В鍐蟲柟妗堟槸鐩綍鏍戞寕杞界殑鏂規錛堢ぞ鍖烘湁涓?/span>viewFS錛屽簲璇ュ氨鏄負浜嗚В鍐寵繖涓棶棰橈級錛氱啛鎮?/span>linux鎴栬?/span>nfs鐨勬湅鍙嬪簲璇ラ兘鐭ラ亾mount榪欎釜姒傚康錛岀洰褰曟爲鎸傝澆灝辨槸榪欎釜鎰忔濄?/span>

涓嶈繃浣跨敤鐩綍鏍戞寕杞戒篃瀛樺湪鐫涓涓棶棰橈紝灝辨槸鍚勪釜瀛愮洰褰曚笅鐨勫瓨鍌ㄨ祫婧愰渶瑕佷漢涓虹殑浠嬪叆綆$悊錛屼笉鑳藉嚭鐜頒弗閲嶇殑涓嶅潎銆?/span>

3.2 crossNode

         鏈烘埧闂寸殑緗戠粶闄愬埗瑕佹眰涓嶈兘鍑虹幇澶ц妯°侀暱鏃墮棿鐨勬暟鎹嫹璐濓紝闇瑕佷竴涓笓闂ㄧ鐞嗘満鎴塊棿鏁版嵁鎷瘋礉鐨勮繘紼嬶紝鍙仛crossNode銆傚畠鏄嫭绔嬮儴緗茬殑涓涓妭鐐癸紝鍜屽厓鏁版嵁鑺傜偣鏄垎紱葷殑銆?/span>

         瀹冭兘鎻愪緵鐨勫姛鑳芥鎷潵璇翠富瑕佸寘鎷互涓嬩笁鐐癸細

a錛?nbsp;鏍規嵁棰勭疆鐨勮法鏈烘埧鏂囦歡錛岃繘琛屾暟鎹嫹璐?/span>

b錛?nbsp;澶勭悊瀹炴椂鐨勬暟鎹嫹璐濊姹?/span>

c錛?nbsp;榪涜璺ㄦ満鎴跨殑鏁版嵁嫻侀噺鎺у埗

濡備綍寰楃煡璺ㄦ満鎴挎枃浠跺垪琛紵

         鐢變簬紱葷嚎浠誨姟鍩烘湰閮芥槸瀹氭椂瑙﹀彂鐨勶紝鍙互鏍規嵁瀵瑰巻鍙蹭綔涓氱殑鍒嗘瀽鏉ュ艦鎴愪竴涓法鏈烘埧鏂囦歡鍒楄〃

3.3   璧勬簮緇勪箣闂寸殑綆$悊

鍚勪釜璧勬簮緇勪箣闂村瓨鍦ㄦ暟鎹殑渚濊禆錛屾垜浠笇鏈涢氳繃璧勬簮緇勭鐞嗭紝鑳藉疄鐜板ぇ閮ㄥ垎浠誨姟鍦ㄦ湰鏈烘埧鍐呬駭鍑烘暟鎹紝鍙湁灝戦噺璺ㄦ満鎴夸駭鍑烘暟鎹紱澶ч儴鍒嗕換鍔¤鍙栨湰鏈烘埧鐨勬暟鎹壇鏈紝鍙湁灝戦噺璺ㄦ満鎴胯鍙栨暟鎹?/span>

涓轟簡鏍囪瘑璧勬簮緇勪箣闂寸殑鏁版嵁渚濊禆鎬э紝瀹氫箟涓涓祫婧愮粍涔嬮棿鐨勮窛紱繪蹇碉細涓涓祫婧愮粍璁塊棶鍙︿竴涓祫婧愮粍鐨勬暟鎹噺瓚婂錛屽垯涓よ呯殑璺濈瓚婅繎錛屽簲璇ュ皢璺濈鎺ヨ繎鐨勮祫婧愮粍鏀懼湪鍚屼竴涓満鎴垮唴銆?/span>

涓轟簡璁╄綆楀拰浜у嚭灝藉彲鑳藉湴闈犺繎錛屼嬌鐢ㄤ竴涓?/span>MRProxy錛屽浜庝笉鍚岀被鍨嬬殑浠誨姟鍋氫笉鍚屽鐞嗭細

a錛?span style="font-size: 7pt; line-height: normal; font-family: 'Times New Roman';">            紱葷嚎璁$畻錛氳法鏈烘埧鍒楄〃涓殑鏁版嵁姝e湪浼犺緭涓紙DC1->DC2錛夛紝DC2涓婄殑 Job 琚殏鍋滆皟搴︼紝絳夊緟浼犺緭瀹屾瘯

b錛?span style="font-size: 7pt; line-height: normal; font-family: 'Times New Roman';">            Ad-hoc鏌ヨ錛?/span>DC2涓婄殑 Job 闇瑕佽DC1涓婄殑鏁版嵁錛?/span>Job鏆傚仠璋冨害錛岄氱煡 CrossNode錛屾暟鎹紶杈撳畬姣曞悗緇х畫璋冨害

c錛?span style="font-size: 7pt; line-height: normal; font-family: 'Times New Roman';">             鐗規畩鎯呭喌錛氳法鏈烘埧鏁版嵁 Join錛?/span>DC1澶ц〃錛?/span>DC2灝忚〃錛?/span>Job 璋冨害鍒?/span>DC1涓婏紝璺ㄦ満鎴跨洿鎺ヨ鍙?/span>DC2鏁版嵁錛屾棤闇絳夊緟

 

鐢變簬鏄牴鎹棰戝拰ppt鏁寸悊錛屽茍娌℃湁浠g爜鎴栬呮枃妗o紝鎵浠ュ彲鑳芥湁浜涘湴鏂圭殑鐞嗚В鏈夊亸宸紝嬈㈣繋鏉ユ彁鎰忚~



whspecial 2013-10-27 23:28 鍙戣〃璇勮
]]>
Dremel瀛樺偍鏍煎紡瑙f瀽http://www.shnenglu.com/whspecial/archive/2013/08/14/202546.htmlwhspecialwhspecialWed, 14 Aug 2013 15:17:00 GMThttp://www.shnenglu.com/whspecial/archive/2013/08/14/202546.htmlhttp://www.shnenglu.com/whspecial/comments/202546.htmlhttp://www.shnenglu.com/whspecial/archive/2013/08/14/202546.html#Feedback1http://www.shnenglu.com/whspecial/comments/commentRss/202546.htmlhttp://www.shnenglu.com/whspecial/services/trackbacks/202546.html    Dremel鏄痝oogle鎺ㄥ嚭鐨勫張涓紲炲櫒錛宲aper涓縐拌兘澶熷湪3s鍐呭垎鏋?PB鐨勬暟鎹紝涓昏鏄潰鍚戜氦浜掑紡鏌ヨ銆傝繖綃噋aper瀵瑰祵濂楃被鍨嬬殑瀛樺偍鏂瑰紡鏂歸潰錛屾濈淮紜疄鏈変簺璺寵穬錛岃繖綃囨枃绔犱富瑕佽璁茶繖涓紝涓鏂歸潰鏄柟渚垮悗鏉ヨ呯悊瑙o紝鍙︿竴鏂歸潰鏄鑷繁涔熸暣鐞嗕笅鎬濊礬銆?br />

    棣栧厛Dremel浣跨敤鐨勬槸鍒楀瓨妯″瀷錛屽浜庡熀鏈被鍨嬪垪瀛樿緝瀹規槗鍋氬埌錛涗絾鏄浜庡祵濂楃被鍨嬶紝Dremel涔熻兘鍋氬埌灝嗗叾鎷嗚В鎴愬熀鏈被鍨嬪茍榪涜鍒楀瓨錛岃繖鏄煎緱鎴戜滑鐮旂┒鐨勩?br />

    鐩磋鐪嬩笅宓屽綾誨瀷鎸夎瀛樺偍鍜屾媶瑙e悗鎸夊垪瀛樺偍鐨勫姣旀晥鏋滐細

    鐒跺悗瀵逛簬宓屽鏁版嵁綾誨瀷錛孌remel閲岄潰瀹氫箟浜嗛噷闈笁縐嶇被鍨嬬殑瀛楁

    1錛屽繀欏誨嚭鐜?嬈¤屼笖浠呭嚭鐜?嬈$殑瀛楁錛歳equired

    2錛屽彲鑳藉嚭鐜?嬈℃垨鑰?嬈$殑瀛楁錛歰ptional

    3錛屽彲鑳藉嚭鐜?嬈℃垨鑰匩嬈″瓧孌碉細repeated

    涓嬮潰浠aper鐨勪緥瀛愭潵璁茶堪鍚э細

    鍏朵腑DocId鏄痳equired瀛楁錛屽洜姝ゅ湪r1,r2涓繀欏誨嚭鐜?嬈★紱url瀛楁鏄痮ptional瀛楁錛屽洜姝ゅ湪r1鐨勭涓変釜Name閲屾湭鍑虹幇錛屽湪r1鐨勫墠涓や釜Name閲屽嚭鐜頒簡1嬈★紱Backward瀛楁鏄痳epeated瀛楁錛屽洜姝ゅ湪r1鐨凩inks閲屾湭鍑虹幇錛屽湪r2鐨凩inks閲屽嚭鐜頒簡2嬈°?br />

    鐞嗚В浜嗕笂闈㈣繖浜涳紝鐩存帴鏉ョ湅涓婦remel鏄庝箞鏉ュ瓨瀹冪殑鍚э細

    涓婅〃涓殑姣忔潯璁板綍閮芥湁涓や釜灞炴э紝"r"浠h〃repetition level錛?d"浠h〃definition level錛屽畾涔夊涓嬶細

    repetition level:what repeated field in the field’s path the value has repeated錛岃褰曡瀛楁鏄湪鍝釜repeated綰у埆涓婇噸澶嶇殑

    definition level:how many fields inpthat could be undefined (because they are optional or repeated) are actually present錛岃褰曡瀛楁涔嬩笂鏈夊灝戜釜optional鎴栬卹epeated瀛楁瀹為檯鏄湁鍊肩殑錛堟湰鏉ュ彲浠ヤ負null鐨勶級

    鐪嬪埌榪欓噷錛屽悇浣嶅彲鑳藉凡緇忓湪蹇冮噷榛樺康浜嗭細WTF錛佸埆鎬ワ紝鍙互緇撳悎涓涓緥瀛愭潵鐪嬶細

鍏堢湅repetition level錛堜笅闈互r鏇夸唬錛夛紝浠ame.Language.Code涓轟緥錛?/p>

    1)瀵圭1涓嚭鐜扮殑鍊鹼紝鍏秗濮嬬粓涓?錛屽洜姝?en-us'鐨剅涓?

    2)瀵逛簬絎?涓?en'錛屽叾涓婁竴涓兼槸'en-us'錛屽畠浠槸鍦↙anguage綰у埆鍙戠敓鐨勯噸澶嶏紝Name.Language鏄袱綰х殑repeated瀛楁錛屽洜姝涓?

    3)瀵逛簬絎?涓糿ull錛屾槸涓轟簡璁板綍'en-gb'鏄嚭鐜板湪絎笁涓狽ame鑰岄潪絎簩涓狽ame閲岋紝鐗規剰鍗犱綅鐢ㄧ殑銆俷ull鐨勪笂涓涓兼槸'en'錛屽畠浠槸鍦∟ame綰у埆鍙戠敓鐨勯噸澶嶏紝鍥犳r鏄?

    4)瀵逛簬絎?涓?en-gb'錛屽叾涓婁竴涓兼槸null錛屽畠浠篃鏄湪Name綰у埆鍙戠敓鐨勯噸澶嶏紝鍥犳r鏄?

    5)瀵逛簬絎?涓糿ull錛屽叾涓婁竴涓兼槸'en-gb'錛屽畠浠嚭鐜板湪涓や釜涓嶅悓Document閲岋紝鍥犳r鏄?

    鎬葷粨涓嬶紝鐪媟epetition level娉ㄦ剰涓ょ偣錛?,鍙瘮杈冭鍊煎拰涓婁竴涓鹼紱2,鍙渶瑕佺湅榪欎袱涓肩殑閲嶅浣嶇疆涓婃湁鍑犱釜repeated瀛楁

鍐嶇湅definition level錛堜笅闈互d鏇夸唬錛夛紝涔熶互Name.Language.Code涓轟緥錛?/p>

    1)瀵逛簬'en-us'錛屽叾涓婄殑Name錛孡anguage閮藉嚭鐜頒簡錛屽洜姝涓?錛堝叾瀹炲浜庨潪null鍊肩殑瀛楁錛屽叾涓婄殑optional鎴栬卹epeated瀛楁鑲畾鏄嚭鐜頒簡錛屾墍浠ラ兘鏄浉鍚岀殑錛屽彧鏄痭ull瀛楁鐨刣鍊兼湁宸埆錛?br />

    2)瀵逛簬'en'錛屽悓鐞哾涔熶負2

    3)瀵逛簬null錛屽叾涓婂彧鍑虹幇浜哊ame錛屾病鏈夊嚭鐜癓anguage錛屽洜姝涓?

    4)瀵逛簬'en-gb',d涔熶負2

    5)瀵逛簬鏈鍚庝竴涓猲ull錛屽叾涓婁篃鍙嚭鐜頒簡Name錛屾病鏈夊嚭鐜癓anguage錛屽洜姝涓?


    浠ヤ笂鍙槸璁蹭簡dremel鎬庝箞鍘誨瓨宓屽綾誨瀷錛岃嚦浜庤繖縐嶅瓨娉曟槸鎬庝箞鎯沖嚭鏉ョ殑錛岀湡闈炴垜杈堣兘鐞嗚В鐨勪簡銆傘傘傛洿澶氬唴瀹癸紝璇峰弬鑰冨師钁梡aper鍙婄綉涓婅В鏋愩?br />



whspecial 2013-08-14 23:17 鍙戣〃璇勮
]]>
Orcfile鏂囦歡鏍煎紡瑙f瀽錛?錛?/title><link>http://www.shnenglu.com/whspecial/archive/2013/08/14/202545.html</link><dc:creator>whspecial</dc:creator><author>whspecial</author><pubDate>Wed, 14 Aug 2013 15:13:00 GMT</pubDate><guid>http://www.shnenglu.com/whspecial/archive/2013/08/14/202545.html</guid><wfw:comment>http://www.shnenglu.com/whspecial/comments/202545.html</wfw:comment><comments>http://www.shnenglu.com/whspecial/archive/2013/08/14/202545.html#Feedback</comments><slash:comments>0</slash:comments><wfw:commentRss>http://www.shnenglu.com/whspecial/comments/commentRss/202545.html</wfw:commentRss><trackback:ping>http://www.shnenglu.com/whspecial/services/trackbacks/202545.html</trackback:ping><description><![CDATA[<p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    涓婄瘒鏂囩珷浠庢暣浣撲粙緇嶄簡Orcfile鐨勫瓨鍌ㄦ牸寮忥紝鎺ヤ笅鏉ラ噸鐐逛粙緇嶄笅Orc閲岀敤鍒扮殑鍑犵緙栫爜鏍煎紡錛?/p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    瀛楀吀緙栫爜錛氱敤浜嶴tring綾誨瀷鐨勫瓧孌?br /></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    Run-Length緙栫爜錛氱敤浜巌nt錛宭ong錛宻hort絳夌被鍨嬬殑緙栫爜</p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    Bit緙栫爜錛氬彲浠ョ敤浜庡悇縐嶆暟鎹被鍨?br /></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;"><strong>1錛屽瓧鍏哥紪鐮侊細</strong></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    瀵逛簬String綾誨瀷鐨勬瘡涓瓧孌靛垎鍒繚瀛樹竴涓瓧鍏革紝璁板綍姣忎釜鍊煎湪瀛楀吀涓殑浣嶇疆錛屼繚瀛樺瓧鍏哥殑鏁版嵁緇撴瀯閲囩敤涓媯電孩榛戞爲銆傚浜庢瘡涓猄tring瀛楁錛屾渶緇堜細鏈変笁涓緭鍑篠tream錛屽垎鍒槸StringOuptut(璁板綍瀛楀吀涓殑鍊?錛孡engthOutput(璁板綍姣忎釜瀛楀吀鍊肩殑闀垮害)錛孯owOutput(璁板綍瀛楁鍦ㄥ瓧鍏鎬腑鐨勪綅緗?銆?/p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    鎬濊?錛氫負浠涔堣鐢ㄧ孩榛戞爲錛?br /></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    鍥犱負綰㈤粦鏍戞棤璁烘槸鎻掑叆錛屽垹闄わ紝鏌ユ壘鐨勬ц兘閮芥瘮杈冨鉤鍧囷紝閮芥槸O(logN)錛岃屼笖鏄鉤琛℃煡鎵炬爲錛屾渶鍧忔儏鍐典篃涓嶄細閫鍖栨垚O(N)<br /></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    鎬濊?錛氬叾瀹炰竴鑸瓨鍌ㄦ椂榪樹細浣跨敤LZO涔嬬被鐨勫帇緙╋紝瀹冧滑鏈韓灝辨槸涓縐嶅瓧鍏稿帇緙╋紝涓轟粈涔圤rc閲岄潰瑕佽嚜宸卞仛瀛楀吀鍘嬬緝錛?br /></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    鍥犱負LZO涔嬬被鐨勫帇緙╃獥鍙d竴鑸瘮杈冨皬錛圠ZO榛樿鏄?4KB錛夛紝鑰孫rc鐨勫瓧鍏稿帇緙╂槸浠ユ暣涓瓧孌典負鑼冨洿鏉ュ帇緙╃殑錛屽帇緙╃巼浼氭洿濂姐?br /></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;"><strong>2錛孯un-Length緙栫爜錛?/strong><br /></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    瀵逛簬int,long,short綾誨瀷鐨勫瓧孌碉紝浣跨敤Run-Length緙栫爜銆傝Run-Length鑳藉瀵圭瓑宸暟鍒楋紙瀹屽叏鐩哥瓑涔熷睘浜庣瓑宸暟鍒楋級榪涜鍘嬬緝錛岃絳夊樊鏁板垪闇瑕佹弧瓚充互涓嬩袱涓潯浠訛細</p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    1錛岃嚦灝戝寘鍚?涓厓绱?/p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    2錛屽樊鍊煎湪-128~127涔嬮棿錛堝洜涓哄樊鍊肩敤1Byte鏉ヨ〃紺猴級</p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    瀵逛簬涓嶆弧瓚崇瓑宸暟鍒楃殑鏁板瓧錛孯un-Length緙栫爜涔熻兘瀛樺偍錛屼絾鏄病鏈夊帇緙╂晥鏋滐紝Run-Length鐨勫叿浣撳瓨鍌ㄥ涓嬶細</p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    絎竴涓狟yte鏄疌ontrol Byte錛屽彇鍊煎湪-128~127涔嬮棿錛屽叾涓?1~-128浠h〃鍚庨潰瀛樺偍鐫1~128涓笉婊¤凍絳夊樊鏁板垪鐨勬暟瀛楋紝0~127浠h〃鍚庨潰瀛樺偍鐫3~130涓瓑宸暟鍒楃殑鏁板瓧錛?/p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    濡傛灉Control Byte>=0錛屽垯鍚庨潰璺熺潃涓涓狟yte瀛樺偍宸鹼紝鍚﹀垯涓嶅瓨鍌ㄨByte錛?/p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    濡傛灉Control Byte>=0錛屽垯鍚庨潰璺熺潃絳夊樊鏁板垪鐨勭涓涓暟錛屽惁鍒欒窡鐫-Control Byte涓暟瀛椼?/p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    渚嬪瓙錛?/p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    鍘熷鏁板瓧錛?2,12,12,12,12,10,7,13</p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    緇忚繃Run-Length鐨勬暟瀛楋細<span style="color: #ff0000;">2</span>,<span style="color: #ffc000;">0</span>,12,<span style="color: #ff0000;">-3</span>,10,7,13</p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    綰㈣壊浠h〃Control Byte錛岄粍鑹蹭唬琛ㄥ樊鍊鹼紝榛戣壊浠h〃鍏蜂綋鐨勬暟瀛椼?br /></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;"><strong>3錛孊it緙栫爜錛?/strong></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">瀵規墍鏈夌被鍨嬬殑瀛楁閮藉彲浠ラ噰鐢˙it緙栫爜鏉ヨ〃紺鴻鍊兼槸鍚︿負null銆傚湪鍐欎換浣曠被鍨嬪瓧孌典箣鍓嶏紝鍏堝垽鏂瀛楁鍊兼槸澶熶負null錛屽鏋滀負null鍒檅it鍊煎瓨涓?錛屽惁鍒欏瓨涓?錛屽浜庝負null鐨勫瓧孌靛湪瀹為檯緙栫爜鏃朵笉闇瑕佸瓨鍌ㄤ簡銆傜粡榪嘊it緙栫爜涔嬪悗錛屽彲浠ュ浜?涓猙it緇勬垚涓涓狟yte錛屽啀瀵瑰叾榪涜Run-Length緙栫爜銆?/p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    鍏跺疄闄や簡榪欎笁縐嶇紪鐮佹牸寮忎箣澶栵紝Orc瀵逛簬hive鐨勫鏉傜被鍨媋rray,map,list絳夛紝灝嗗叾闄嶇淮鎴愬熀鏈被鍨嬫潵瀛樺偍錛岃繖涓篃鏄煎緱鍊熼壌鐨勶紝濡傛灉鏈夌┖涔嬪悗浼氳繘琛屽垎鏋愩?/p><img src ="http://www.shnenglu.com/whspecial/aggbug/202545.html" width = "1" height = "1" /><br><br><div align=right><a style="text-decoration:none;" href="http://www.shnenglu.com/whspecial/" target="_blank">whspecial</a> 2013-08-14 23:13 <a href="http://www.shnenglu.com/whspecial/archive/2013/08/14/202545.html#Feedback" target="_blank" style="text-decoration:none;">鍙戣〃璇勮</a></div>]]></description></item><item><title>Orcfile鏂囦歡鏍煎紡瑙f瀽錛?錛?/title><link>http://www.shnenglu.com/whspecial/archive/2013/08/14/202544.html</link><dc:creator>whspecial</dc:creator><author>whspecial</author><pubDate>Wed, 14 Aug 2013 15:12:00 GMT</pubDate><guid>http://www.shnenglu.com/whspecial/archive/2013/08/14/202544.html</guid><wfw:comment>http://www.shnenglu.com/whspecial/comments/202544.html</wfw:comment><comments>http://www.shnenglu.com/whspecial/archive/2013/08/14/202544.html#Feedback</comments><slash:comments>0</slash:comments><wfw:commentRss>http://www.shnenglu.com/whspecial/comments/commentRss/202544.html</wfw:commentRss><trackback:ping>http://www.shnenglu.com/whspecial/services/trackbacks/202544.html</trackback:ping><description><![CDATA[<p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    Orcfile(<em>Optimized Row Columnar</em>)鏄痟ive 0.11鐗堥噷寮曞叆鐨勬柊鐨勫瓨鍌ㄦ牸寮忥紝鏄涔嬪墠鐨凴CFile瀛樺偍鏍煎紡鐨勪紭鍖栥傚啓榪欎釜鐨勫摜浠潵鑷簬HortonWorks錛屼唬鐮佸啓鐨勫緢涓嶉敊錛屾瘮涔嬪墠鐨剅cfile寮哄浜嗭紙鎹rcfile鏄釜涓闄㈢殑绔ラ瀷璺戝幓facebook鍐欑殑錛岀湅鏉ヤ腑鍥界殑璁$畻鏈烘暀鑲叉按騫寵繕鏄湁闄愬晩銆傘傘傚洤錛岃窇棰樹簡錛?/p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    鍏堜粙緇嶄笅Orc鐨勬枃浠舵牸寮忥紝鎴竴寮犲畼鏂圭殑鍥撅細<br /></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;"><a target="_blank" data_ue_src="http://blog.51cto.com/attachment/201306/234709762.png"><img src="http://blog.51cto.com/attachment/201306/234709762.png" title="OrcFileLayout.png" data_ue_src="http://blog.51cto.com/attachment/201306/234709762.png" alt="" /></a></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    鍙互鐪嬪埌姣忎釜Orc鏂囦歡鐢?涓垨澶氫釜stripe緇勬垚錛屾瘡涓猻tripe250MB澶у皬錛岃繖涓猄tripe瀹為檯鐩稿綋浜庝箣鍓嶇殑rcfile閲岀殑RowGroup姒傚康錛屼笉榪囧ぇ灝忕敱4MB->250MB錛岃繖鏍峰簲璇ヨ兘鎻愬崌欏哄簭璇葷殑鍚炲悙鐜囥傛瘡涓猄tripe閲屾湁涓夐儴鍒嗙粍鎴愶紝鍒嗗埆鏄疘ndex Data,Row Data,Stripe Footer錛?/p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    1錛孖ndex Data錛氫竴涓交閲忕駭鐨刬ndex錛岄粯璁ゆ槸姣忛殧1W琛屽仛涓涓儲寮曘傝繖閲屽仛鐨勭儲寮曞簲璇ュ彧鏄褰曟煇琛岀殑鍚勫瓧孌靛湪Row Data涓殑offset錛屾嵁璇磋繕鍖呮嫭姣忎釜Column鐨刴ax鍜宮in鍊鹼紝鍏蜂綋娌$粏鐪嬩唬鐮併?/p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    2錛孯ow Data錛氬瓨鐨勬槸鍏蜂綋鐨勬暟鎹紝鍜孯Cfile涓鏍鳳紝鍏堝彇閮ㄥ垎琛岋紝鐒跺悗瀵硅繖浜涜鎸夊垪榪涜瀛樺偍銆備笌RCfile涓嶅悓鐨勫湴鏂瑰湪浜庢瘡涓垪榪涜浜嗙紪鐮侊紝鍒嗘垚澶氫釜Stream鏉ュ瓨鍌紝鍏蜂綋濡備綍緙栫爜鍦ㄤ笅涓綃囪В鏋愰噷浼氳銆?/p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    3錛孲tripe Footer錛氬瓨鐨勬槸鍚勪釜Stream鐨勭被鍨嬶紝闀垮害絳変俊鎭?/p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    姣忎釜鏂囦歡鏈変竴涓狥ile Footer錛岃繖閲岄潰瀛樼殑鏄瘡涓猄tripe鐨勮鏁幫紝姣忎釜Column鐨勬暟鎹被鍨嬩俊鎭瓑錛涙瘡涓枃浠剁殑灝鵑儴鏄竴涓狿ostScript錛岃繖閲岄潰璁板綍浜嗘暣涓枃浠剁殑鍘嬬緝綾誨瀷浠ュ強FileFooter鐨勯暱搴︿俊鎭瓑銆傚湪璇誨彇鏂囦歡鏃訛紝浼歴eek鍒版枃浠跺熬閮ㄨPostScript錛屼粠閲岄潰瑙f瀽鍒癋ile Footer闀垮害錛屽啀璇籉ileFooter錛屼粠閲岄潰瑙f瀽鍒板悇涓猄tripe淇℃伅錛屽啀璇誨悇涓猄tripe錛屽嵆浠庡悗寰鍓嶈銆?/p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    鎺ヤ笅鏉ョ湅涓婳Rcfile鐩稿浜嶳Cfile鍋氫簡鍝簺鏀硅繘錛屼粠Orc浣滆呯殑ppt閲屾埅浜嗗紶鍥撅紝鍒嗗埆瑙i噴涓嬪悇琛岋細<br /></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;"><a target="_blank" data_ue_src="http://blog.51cto.com/attachment/201306/003840639.png"><img src="http://blog.51cto.com/attachment/201306/003840639.png" title="鍥劇墖1.png" width="500" height="345" border="0" hspace="0" vspace="0" data_ue_src="http://blog.51cto.com/attachment/201306/003840639.png" style="width: 500px; height: 345px;" alt="" /></a></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    Hive type model:RCfile鍦ㄥ簳灞傚瓨鍌ㄦ椂涓嶄繚瀛樼被鍨嬶紝閮藉綋鍋欱yte嫻佹潵瀛樺偍</p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    Separtor complex columns:Orc灝嗗鏉傜被鍨嬫媶寮瀛樺偍</p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    Splits Found Quickly錛氫笉寰堢悊瑙?br /></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    Default Column group size錛氫笉鐢ㄨВ閲婁簡</p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    Files per a bucket錛氫笉寰堢悊瑙?/p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    Store min錛宮ax錛宑ount錛宻um錛氬瓨浜嗚繖浜涗究浜庡揩閫熷湴skip鎺変竴涓猻tripe</p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    Versioned metadata:涓嶅緢鐞嗚В<br /></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    Run-Length Data-coding錛氭暣鏁扮被鍨嬪仛Run-Length鍙橀暱緙栫爜</p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    Store Strings in dictionary錛歋tring綾誨瀷鍋氬瓧鍏哥紪鐮?/p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    Store Row Count錛氭瘡涓猄tripe浼氬瓨鍌ㄨ鏁?/p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    Skip Compressed blocks:鍙互鐩存帴skip鎺夊帇緙╄繃鐨刡lock</p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    Store internal indexes:瀛樺偍浜嗕竴涓交閲忕駭鐨刬ndex</p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;"><br /></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    鏁翠釜Orc鐪嬩笅鏉ワ紝浠g爜鍐欑殑榪樻槸姣旇緝娓呮櫚鏄庝簡鐨勶紝鑰屼笖鎴戜滑涔熻繘琛屼簡嫻嬭瘯錛屽帇緙╂晥鏋滄瘮RCfile鎻愬崌浜嗕笉灝戯紝鏈夊叴瓚g殑鏈嬪弸鍙互鏉ョ湅涓嬶紝涔嬪悗浼氬啓絎簩綃囪В鏋愶紝涓昏鏄Orc鐢ㄥ埌鐨勫嚑縐嶇紪鐮佹牸寮忋?br /></p><div></div><img src ="http://www.shnenglu.com/whspecial/aggbug/202544.html" width = "1" height = "1" /><br><br><div align=right><a style="text-decoration:none;" href="http://www.shnenglu.com/whspecial/" target="_blank">whspecial</a> 2013-08-14 23:12 <a href="http://www.shnenglu.com/whspecial/archive/2013/08/14/202544.html#Feedback" target="_blank" style="text-decoration:none;">鍙戣〃璇勮</a></div>]]></description></item><item><title>鍏充簬map/reduce鐨刢ombiner榪愯鏃舵満鐨勯棶棰?/title><link>http://www.shnenglu.com/whspecial/archive/2012/11/06/194819.html</link><dc:creator>whspecial</dc:creator><author>whspecial</author><pubDate>Tue, 06 Nov 2012 15:52:00 GMT</pubDate><guid>http://www.shnenglu.com/whspecial/archive/2012/11/06/194819.html</guid><wfw:comment>http://www.shnenglu.com/whspecial/comments/194819.html</wfw:comment><comments>http://www.shnenglu.com/whspecial/archive/2012/11/06/194819.html#Feedback</comments><slash:comments>0</slash:comments><wfw:commentRss>http://www.shnenglu.com/whspecial/comments/commentRss/194819.html</wfw:commentRss><trackback:ping>http://www.shnenglu.com/whspecial/services/trackbacks/194819.html</trackback:ping><description><![CDATA[<blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;"></blockquote><span style="font-size: 12pt;">   </span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">map/reduce</span><span style="font-size: 12pt; line-height: 150%; font-family: 瀹嬩綋;">鐨?/span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">combiner</span><span style="font-size: 12pt; line-height: 150%; font-family: 瀹嬩綋;">鍒板簳鍦ㄤ粈涔堟椂鍊欒繍琛岋紵</span> <p align="left" style="text-indent: 21pt; line-height: 150%;"><span style="font-size: 12pt; line-height: 150%; font-family: 瀹嬩綋;">鍦ㄧ綉涓婂ぇ澶氭暟璧勬枡涓紝閮芥槸璇?/span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">combiner</span><span style="font-size: 12pt; line-height: 150%; font-family: 瀹嬩綋;">鍦?/span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">map</span><span style="font-size: 12pt; line-height: 150%; font-family: 瀹嬩綋;">绔繍琛岋紝鍙戠敓鍦?/span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">map</span><span style="font-size: 12pt; line-height: 150%; font-family: 瀹嬩綋;">杈撳嚭鏁版嵁涔嬪悗錛岀粡榪?/span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">combiner</span><span style="font-size: 12pt; line-height: 150%; font-family: 瀹嬩綋;">鍐嶄紶閫掔粰</span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">reducer</span><span style="font-size: 12pt; line-height: 150%; font-family: 瀹嬩綋;">銆備絾鏄箣鍓嶅湪宸ヤ綔涓嚭鐜扮殑涓涓棶棰樺鑷存垜鍙戠幇鍘熸潵</span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">combiner</span><span style="font-size: 12pt; line-height: 150%; font-family: 瀹嬩綋;">灞呯劧涔熶細鍦?/span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">reducer</span><span style="font-size: 12pt; line-height: 150%; font-family: 瀹嬩綋;">绔繍琛岋紝騫朵笖浼氬嬈¤繍琛屻?/span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;"><br /> </span><span style="font-size: 8pt; line-height: 150%; font-family: Verdana, sans-serif;">   </span><span style="font-size: 12pt; line-height: 150%; font-family: 瀹嬩綋;">鍦ㄧ綉涓婃煡浜嗕箣鍚庡彂鐜幫紝榪欐槸</span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">hadoop-0.18</span><span style="font-size: 12pt; line-height: 150%; font-family: 瀹嬩綋;">鐗堟湰寮曞叆鐨勬柊</span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">feature</span><span style="font-size: 12pt; line-height: 150%; font-family: 瀹嬩綋;">錛?/span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;"><br /> </span><span style="font-size: 8pt; line-height: 150%; font-family: Verdana, sans-serif;">   </span><span style="font-size:12.0pt;line-height:150%;font-family:"Arial","sans-serif";color:#333333;background:white;">Changed policy for running combiner. The combiner may be run multiple times as the map's output is sorted and merged. Additionally, it may be run on the reduce side as data is merged. The old semantics are available in Hadoop 0.18 if the user calls: job.setCombineOnlyOnce(true)</span><span style="font-size:12.0pt;line-height:150%;font-family:瀹嬩綋;color:#333333; background:white;">銆?/span><span style="font-size: 12pt; line-height: 150%; font-family: Verdana, sans-serif;"><br /> </span><span style="font-size: 8pt; line-height: 150%; font-family: Verdana, sans-serif;">   </span><span style="font-size: 12pt; line-height: 150%; font-family: 瀹嬩綋;">瀹為檯涓?/span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">combiner</span><span style="font-size: 12pt; line-height: 150%; font-family: 瀹嬩綋;">浼氬湪</span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">mapper</span><span style="font-size: 12pt; line-height: 150%; font-family: 瀹嬩綋;">绔拰</span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">reducer</span><span style="font-size: 12pt; line-height: 150%; font-family: 瀹嬩綋;">绔垎鍒繍榪愯錛岀湅浜嗕笅浠g爜錛屽彂鐢?/span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">combine</span><span style="font-size: 12pt; line-height: 150%; font-family: 瀹嬩綋;">鐨勬椂鏈哄湪浠ヤ笅錛?/span><span style="font-size: 8pt; line-height: 150%; font-family: Verdana, sans-serif;"><br /> </span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">   1</span><span style="font-size: 12pt; line-height: 150%; font-family: 瀹嬩綋;">錛?/span> <span style="font-size: 12pt; line-height: 150%; font-family: 瀹嬩綋;">鍦?/span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">mapper</span><span style="font-size: 12pt; line-height: 150%; font-family: 瀹嬩綋;">绔殑</span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">spill</span><span style="font-size: 12pt; line-height: 150%; font-family: 瀹嬩綋;">闃舵錛屽湪緙撳瓨涓殑璁板綍瓚呰繃闃堝兼椂浼氳繘琛?/span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">combine</span></p> <p align="left" style="margin: 2.9pt 0cm; line-height: 150%;"><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">if (spstart != spindex) {</span></p> <p align="left" style="margin: 2.9pt 0cm; line-height: 150%;"><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">         …</span></p> <p align="left" style="margin: 2.9pt 0cm; text-indent: 18pt; line-height: 150%;"><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">combineAndSpill(kvIter, combineInputCounter);</span></p> <p align="left" style="margin: 2.9pt 0cm; line-height: 150%;"><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">}</span></p> <p align="left" style="margin: 2.9pt 0cm 2.9pt 18pt; text-indent: -18pt; line-height: 150%;"><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">2</span><span style="font-size: 12pt; line-height: 150%; font-family: 瀹嬩綋;">錛?/span> <span style="font-size: 12pt; line-height: 150%; font-family: 瀹嬩綋;">鍦?/span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">mapper</span><span style="font-size: 12pt; line-height: 150%; font-family: 瀹嬩綋;">绔殑</span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">merge</span><span style="font-size: 12pt; line-height: 150%; font-family: 瀹嬩綋;">闃舵錛岃繘琛?/span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">merge</span><span style="font-size: 12pt; line-height: 150%; font-family: 瀹嬩綋;">鐨?/span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">spill</span><span style="font-size: 12pt; line-height: 150%; font-family: 瀹嬩綋;">鏂囦歡鏁扮洰</span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">>=3</span><span style="font-size: 12pt; line-height: 150%; font-family: 瀹嬩綋;">鏃朵細榪涜</span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">combine</span></p> <p align="left" style="margin: 2.9pt 0cm; line-height: 150%;"><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">if (null == combinerClass || numSpills < minSpillsForCombine) {</span></p> <p align="left" style="margin: 2.9pt 0cm; text-indent: 21pt; line-height: 150%;"><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">Merger.writeFile(kvIter, writer, reporter);</span></p> <p align="left" style="margin: 2.9pt 0cm; line-height: 150%;"><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">} else {</span></p> <p align="left" style="margin: 2.9pt 0cm; text-indent: 21pt; line-height: 150%;"><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">combineCollector.setWriter(writer);</span></p> <p align="left" style="margin: 2.9pt 0cm; text-indent: 21pt; line-height: 150%;"><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">combineAndSpill(kvIter, combineInputCounter);</span></p> <p align="left" style="margin: 2.9pt 0cm; line-height: 150%;"><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">}</span></p> <p style="line-height:150%"><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">3</span><span style="font-size: 12pt; line-height: 150%; font-family: 瀹嬩綋;">錛?/span> <span style="font-size: 12pt; line-height: 150%; font-family: 瀹嬩綋;">鍦?/span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">reducer</span><span style="font-size: 12pt; line-height: 150%; font-family: 瀹嬩綋;">绔紝涓瀹氫細榪涜</span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">combine</span></p><span style="font-size: 12pt; font-family: Arial; line-height: 17px; background-color: #ffffff; "><br /></span><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;"></blockquote><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;"></blockquote><div class="nj97xbt" id="isChromeWebToolbarDiv" style="display:none"></div> <div class="nfht7xt" id="isChromeWebToolbarDiv" style="display:none"></div> <div class="7rlz9z9" id="isChromeWebToolbarDiv" style="display:none"></div><img src ="http://www.shnenglu.com/whspecial/aggbug/194819.html" width = "1" height = "1" /><br><br><div align=right><a style="text-decoration:none;" href="http://www.shnenglu.com/whspecial/" target="_blank">whspecial</a> 2012-11-06 23:52 <a href="http://www.shnenglu.com/whspecial/archive/2012/11/06/194819.html#Feedback" target="_blank" style="text-decoration:none;">鍙戣〃璇勮</a></div>]]></description></item></channel></rss> <footer> <div class="friendship-link"> <p>感谢您访问我们的网站,您可能还对以下资源感兴趣:</p> <a href="http://www.shnenglu.com/" title="精品视频久久久久">精品视频久久久久</a> <div class="friend-links"> </div> </div> </footer> <a href="http://www.alilinfen.cn" target="_blank">大香伊人久久精品一区二区</a>| <a href="http://www.zgktjj.cn" target="_blank">久久不见久久见免费影院www日本</a>| <a href="http://www.s1307.cn" target="_blank">日韩中文久久</a>| <a href="http://www.oftr.cn" target="_blank">久久久久亚洲av无码专区</a>| <a href="http://www.wanrengou.cn" target="_blank">久久超乳爆乳中文字幕</a>| <a href="http://www.linglonggegame.cn" target="_blank">高清免费久久午夜精品</a>| <a href="http://www.ikxc.cn" target="_blank">欧美久久一级内射wwwwww.</a>| <a href="http://www.oahk.cn" target="_blank">久久久久久久精品妇女99</a>| <a href="http://www.jswlgs.cn" target="_blank">久久香蕉一级毛片</a>| <a href="http://www.191c.cn" target="_blank">久久人人爽人人爽人人爽</a>| <a href="http://www.bayercas.cn" target="_blank">人人狠狠综合久久亚洲婷婷</a>| <a href="http://www.niutuan.com.cn" target="_blank">午夜肉伦伦影院久久精品免费看国产一区二区三区 </a>| <a href="http://www.feschain.cn" target="_blank">久久精品无码一区二区三区日韩 </a>| <a href="http://www.ywxcj.cn" target="_blank">久久se精品一区精品二区</a>| <a href="http://www.cpfang.cn" target="_blank">色偷偷88欧美精品久久久</a>| <a href="http://www.crolab.cn" target="_blank">国产亚洲婷婷香蕉久久精品</a>| <a href="http://www.gb9948-2013.cn" target="_blank">欧美精品一区二区久久 </a>| <a href="http://www.39xw.cn" target="_blank">性高湖久久久久久久久AAAAA</a>| <a href="http://www.cq-fa.org.cn" target="_blank">久久久久人妻一区二区三区vr </a>| <a href="http://www.uhqw.cn" target="_blank">四虎国产精品免费久久</a>| <a href="http://www.deiden.cn" target="_blank">久久99精品国产麻豆蜜芽</a>| <a href="http://www.yonganwl.cn" target="_blank">国产精品99精品久久免费</a>| <a href="http://www.25552l.cn" target="_blank">欧美日韩久久中文字幕</a>| <a href="http://www.cooyu.cn" target="_blank">久久免费99精品国产自在现线</a>| <a href="http://www.todouba.cn" target="_blank">精品无码久久久久久尤物</a>| <a href="http://www.cahe8.cn" target="_blank">久久中文字幕人妻熟av女</a>| <a href="http://www.05958.cn" target="_blank">色播久久人人爽人人爽人人片aV</a>| <a href="http://www.guangzb.cn" target="_blank">精品国产福利久久久</a>| <a href="http://www.yushiji505.cn" target="_blank">久久国产精品99精品国产</a>| <a href="http://www.taobaoke.net.cn" target="_blank">久久久亚洲裙底偷窥综合</a>| <a href="http://www.qeckf.cn" target="_blank">伊人久久大香线蕉AV一区二区</a>| <a href="http://www.xeqw.cn" target="_blank">99久久精品国产一区二区</a>| <a href="http://www.moisky.com.cn" target="_blank">国产精品久久毛片完整版</a>| <a href="http://www.ccgangjiegou.cn" target="_blank">亚洲国产精品成人久久</a>| <a href="http://www.e8ux.cn" target="_blank">中文字幕无码免费久久</a>| <a href="http://www.lzjsyg.cn" target="_blank">99久久99久久精品国产片果冻</a>| <a href="http://www.reboc.cn" target="_blank">色狠狠久久综合网</a>| <a href="http://www.love5000.cn" target="_blank">97精品伊人久久久大香线蕉</a>| <a href="http://www.zhe38.cn" target="_blank">亚洲中文久久精品无码</a>| <a href="http://www.dgchengxin.cn" target="_blank">久久香蕉国产线看观看精品yw</a>| <a href="http://www.g753.cn" target="_blank">精产国品久久一二三产区区别</a>| <script> (function(){ var bp = document.createElement('script'); var curProtocol = window.location.protocol.split(':')[0]; if (curProtocol === 'https') { bp.src = 'https://zz.bdstatic.com/linksubmit/push.js'; } else { bp.src = 'http://push.zhanzhang.baidu.com/push.js'; } var s = document.getElementsByTagName("script")[0]; s.parentNode.insertBefore(bp, s); })(); </script> </body>