�??xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:trackback="http://madskills.com/public/xml/rss/module/trackback/" xmlns:wfw="http://wellformedweb.org/CommentAPI/" xmlns:slash="http://purl.org/rss/1.0/modules/slash/"><channel><title>&#22269;&#20135;&#31934;&#21697;&#20037;&#20037;&#33258;&#22312;&#33258;&#32447;&#35266;&#30475;&#44;&#27431;&#32654;&#31934;&#21697;&#19968;&#21306;&#20108;&#21306;&#20037;&#20037;&#44;&#21320;&#22812;&#31934;&#21697;&#20037;&#20037;&#20037;&#20037;&#57;&#57;&#57;&#57;&#39640;&#28165;</title><link>http://www.shnenglu.com/whspecial/category/19559.html</link><description /><language>zh-cn</language><lastBuildDate>Sun, 27 Oct 2013 15:52:56 GMT</lastBuildDate><pubDate>Sun, 27 Oct 2013 15:52:56 GMT</pubDate><ttl>60</ttl><item><title>跨机房的hadoop集群</title><link>http://www.shnenglu.com/whspecial/archive/2013/10/27/203940.html</link><dc:creator>whspecial</dc:creator><author>whspecial</author><pubDate>Sun, 27 Oct 2013 15:28:00 GMT</pubDate><guid>http://www.shnenglu.com/whspecial/archive/2013/10/27/203940.html</guid><wfw:comment>http://www.shnenglu.com/whspecial/comments/203940.html</wfw:comment><comments>http://www.shnenglu.com/whspecial/archive/2013/10/27/203940.html#Feedback</comments><slash:comments>0</slash:comments><wfw:commentRss>http://www.shnenglu.com/whspecial/comments/commentRss/203940.html</wfw:commentRss><trackback:ping>http://www.shnenglu.com/whspecial/services/trackbacks/203940.html</trackback:ping><description><![CDATA[<div><p style="text-indent:21.0pt;line-height:150%"><span style="font-family:宋体;">�q�是来自于阿里技术嘉�q�华的一个分享，因�ؓ(f��)在百度也考虑�q�类似的事情�Q�所以听得比较有感�?zh��n)��Q�这里把相关内容整理一下�?/span></p>  <p style="text-indent:21.0pt;line-height:150%"><span style="font-family:宋体;">首先���重版权�Q�还是把原链接和作者脓(chu��ng)上：(x��)</span></p>  <p style="line-height:150%"><a >http://adc.alibabatech.org/carnival/history/schedule/2013/detail/main/286?video=0</a></p>  <p style="text-indent:21.0pt;line-height:150%"><span style="font-family:宋体;">来自于阿里吴威工�E�师的分�?/span></p>  <p style="line-height:150%">&nbsp;</p>  <p style="text-indent:21.0pt;line-height:150%"><span style="font-family:宋体;">首先需要说明一点，跨机�?/span>hadoop<span style="font-family:宋体;">可能应用场景�q�不是很多，国内�?/span>BAT<span style="font-family:宋体;">�q�种巨头也许需要，但是大部分的中小公司也许�q�不需要这个，也许�q�是个屠龙之技�Q�呵��c(di��n)�?/span></p>  <p style="text-indent:21.0pt;line-height:150%"><span style="font-family:宋体;">把这个问题分三段来讲�Q�第一�D�|��问题出现的背景，�W�二�D�|��解决该问题的隄����Q�第三段是最�l�的解决�Ҏ(gu��)���?/span></p>  <p style="margin-left:36.0pt;text-indent:-36.0pt;line-height:150%;"><strong>�Q�一�Q?nbsp;</strong><strong><span style="font-family:宋体;">背景�Q?/span></strong></p>  <p style="line-height:150%"><span style="font-family:宋体;">先要看下��Z��么需要做一个跨机房的大集群�Q?/span></p>  <p style="text-indent:21.0pt;line-height:150%"><span style="font-family:宋体;">大集���的优点在于数据���理和授权容易（�q�个问题在一个多部门的大公司�q�是很重要的�Q�；跨部门的使用数据�Ҏ(gu��)���Q�无需重复拉取数据�?/span></p>  <p style="text-indent:21.0pt;line-height:150%"><span style="font-family:宋体;">在集���达��C��定规模时�Q�单机房�Q�机房内的容量是有限的）(j��)已经无法满��集群的需求了(ji��n)�Q�要想一��x��逸的解决问题�Q�需要徏设一个跨机房�?/span>hadoop<span style="font-family:宋体;">集群�?/span></p>  <p style="line-height:150%"><strong><span style="font-family:宋体;">�Q�二�Q�技术挑战：(x��)</span></strong></p>  <p style="line-height:150%"><strong>2.1 NameNode</strong><strong><span style="font-family:宋体;">的性能问题�Q?/span></strong></p>  <p style="line-height:150%">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <span style="font-family:宋体;">在管理一个巨大的</span>hadoop<span style="font-family:宋体;">集群�Ӟ���׃��原始�?/span>Namenode<span style="font-family:宋体;">是单节点�Q�因此会(x��)成�ؓ(f��)一个性能瓉����Q�遇到的性能问题主要包括两方面：(x��)存储定w��问题�Q�存储元数据�Q�和计算压力�Q�处�?/span>rpc<span style="font-family:宋体;">��h���Q�修改内存树(w��i)时候需要全局锁）(j��)问题�?/span></p>  <p style="line-height:150%">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <span style="font-family:宋体;">其中存储定w��问题可以依赖内存的垂直扩展来解决�Q�但是计���压力却很难通过提升����g来解冻I��因�ؓ(f��)目前厂商的主要发展方向是多核�Q�而非提高主频�Q?/span></p>  <p style="line-height:150%"><strong>2.2</strong><strong><span style="font-family:宋体;">机房之间的网�l�限�Ӟ��(x��)</span></strong></p>  <p style="line-height:150%">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <span style="font-family:宋体;">机房之间的网�l�永�q�是个硬件条件的限制�Q�跨机房的网�l�传输带来了(ji��n)数据延时和带宽限�Ӟ��(x��)</span></p>  <p style="margin-left:39.0pt;text-indent:-18.0pt;line-height:150%;">1�Q?nbsp;<span style="font-family:宋体;">延时一般是�?/span>10ms<span style="font-family:宋体;">之内�Q��?/span>hadoop<span style="font-family:宋体;">上大部分�q�行的是��ȝ��作业�Q�基本可接受</span></p>  <p style="margin-left:39.0pt;text-indent:-18.0pt;line-height:150%;">2�Q?nbsp;<span style="font-family:宋体;">带宽限制的问题比较大�Q�因为单机房内的点对点带宽一般是�?/span>1Gbps<span style="font-family:宋体;">�Q�而机房之间的带宽���在</span>20Mbps<span style="font-family:宋体;">左右�Q�非常有限�?/span></p>  <p style="line-height:150%"><strong>2.3</strong><strong><span style="font-family:宋体;">资源�l�之间的���理</span></strong></p>  <p style="line-height:150%">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <span style="font-family:宋体;">每个部门可以看做一个资源组�Q�它们可能会(x��)互相使用�Ҏ(gu��)��的数据，因此如何规划计算和存储的位置���很重要�Q�否则会(x��)在多个机房之间出现大量的数据拯����?/span></p>  <p style="line-height:150%"><strong><span style="font-family:宋体;">�Q�三�Q�解��x��案：(x��)</span></strong></p>  <p style="text-indent:21.0pt;line-height:150%"><span style="font-family:宋体;">先看下整个跨集群</span>hadoop<span style="font-family:宋体;">的架构图�Q?/span></p>  <p style="line-height:150%"><img src="http://www.shnenglu.com/images/cppblog_com/whspecial/QQ囄���20131027233042.jpg" width="558" height="315" alt="" /><br /></p>  <p style="line-height:150%">&nbsp;</p>  <p style="text-indent:18.0pt;line-height:150%"><span style="font-family:宋体;">重点介绍里面三点�Q�也���是和上面三个问题相对应的：(x��)</span></p>  <p style="margin-left:18.0pt;text-indent:-18.0pt;line-height:150%;">1�Q?nbsp;<span style="font-family:宋体;">可以看到�q�里��d���?ji��n)两�?/span>NN<span style="font-family:宋体;">�Q?/span>namenode<span style="font-family:宋体;">�Q�，它们实际上还是属于一�?/span>hadoop<span style="font-family:宋体;">集群�Q�这是业界里的一个解��x��案：(x��)</span>HDFS Fedaration<span style="font-family:宋体;">�Q�它��Z��(ji��n)解决元数据节�Ҏ(gu��)��能问题�Q?/span></p>  <p style="margin-left:18.0pt;text-indent:-18.0pt;line-height:150%;">2�Q?nbsp;<span style="font-family:宋体;">可以看到�q�里有一�?/span>cross node<span style="font-family:宋体;">节点�Q�它是用来在两个机房之间同步数据的，它的设计考虑��C��(ji��n)机房间的�|�络限制�Q?/span></p>  <p style="margin-left:18.0pt;text-indent:-18.0pt;line-height:150%;">3�Q?nbsp;<span style="font-family:宋体;">最后是</span>groupA<span style="font-family:宋体;">�?/span>groupB<span style="font-family:宋体;">�Q�这是�ؓ(f��)�?ji��n)解��x��据��出方和��用方关系来用的�?/span></p>  <p style="line-height:150%"><strong>3.1 Federation</strong></p>  <p style="text-indent:18.0pt;line-height:150%">Federation<span style="font-family:宋体;">相关资料见：(x��)</span></p>  <p style="text-indent:18.0pt;line-height:150%"><a >http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/Federation.html#HDFS_Federation</a></p>  <p style="text-indent:18.0pt;line-height:150%"><img src="http://www.shnenglu.com/images/cppblog_com/whspecial/QQ囄���20131027233052.jpg" width="547" height="381" alt="" /><br /></p>  <p style="text-indent:21.0pt;line-height:150%"><span style="font-family:宋体;">��Z��(ji��n)水��^扩展</span>Namenode<span style="font-family:宋体;">�Q?/span>federation<span style="font-family:宋体;">使用�?ji��n)多个互相独立�?/span>namenode<span style="font-family:宋体;">。它们之间互�怸�需要通信�Q�每�?/span>datenode<span style="font-family:宋体;">需要向全部</span>namenode<span style="font-family:宋体;">注册�q�发送信息�?/span></p>  <p style="text-indent:21.0pt;line-height:150%">BlockPool<span style="font-family:宋体;">是属于一�?/span>namenode<span style="font-family:宋体;">�?/span>block<span style="font-family:宋体;">集合�Q�每�?/span>blockpool<span style="font-family:宋体;">之间也是互相独立的�?/span></p>  <p style="line-height:150%">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <span style="font-family:宋体;">�?/span>federation<span style="font-family:宋体;">里，有一个需要关注的问题�Q�就是多�?/span>namenode<span style="font-family:宋体;">的地址如何对用戯���行透明�Q�它采用的解��x��案是目录�?w��i)挂载的��?gu��)���Q�社区有�?/span>viewFS<span style="font-family:宋体;">�Q�应该就是�ؓ(f��)�?ji��n)解册���个问题�?j��)�Q�熟�(zh��n)?/span>linux<span style="font-family:宋体;">或�?/span>nfs<span style="font-family:宋体;">的朋友应该都知道</span>mount<span style="font-family:宋体;">�q�个概念�Q�目录树(w��i)挂蝲���是�q�个意思�?/span></p>  <p style="text-indent:21.0pt;line-height:150%"><span style="font-family:宋体;">不过使用目录�?w��i)挂载也存在着一个问题，���是各个子目录下的存储资源需要�h为的介入���理�Q�不能出��C��重的不均�?/span></p>  <p style="line-height:150%"><strong>3.2 crossNode</strong></p>  <p style="line-height:150%">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <span style="font-family:宋体;">机房间的�|�络限制要求不能出现大规模、长旉���的数据拷贝，需要一个专门管理机戉K��数据拯���的进�E�，叫做</span>crossNode<span style="font-family:宋体;">。它是独立部�|�的一个节点，和元数据节点是分��ȝ���?/span></p>  <p style="line-height:150%">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <span style="font-family:宋体;">它能提供的功能概括来说主要包括以下三点：(x��)</span></p>  <p style="margin-left:39.0pt;text-indent:-18.0pt;line-height:150%;">a�Q?nbsp;<span style="font-family:宋体;">�Ҏ(gu��)��预置的跨机房文�g�Q�进行数据拷�?/span></p>  <p style="margin-left:39.0pt;text-indent:-18.0pt;line-height:150%;">b�Q?nbsp;<span style="font-family:宋体;">处理实时的数据拷贝请�?/span></p>  <p style="margin-left:39.0pt;text-indent:-18.0pt;line-height:150%;">c�Q?nbsp;<span style="font-family:宋体;">�q�行跨机房的数据���量控制</span></p>  <p style="margin-left:21.0pt;line-height:150%"><span style="font-family:宋体;">如何得知跨机房文件列表？</span></p>  <p style="line-height:150%">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <span style="font-family:宋体;">�׃����ȝ����d��基本都是定时触发的，可以�Ҏ(gu��)��对历史作业的分析来�Ş成一个跨机房文�g列表</span></p>  <p style="margin-left:18.0pt;text-indent:-18.0pt;line-height:150%;"><strong>3.3<span style="font-weight: normal; font-size: 7pt; line-height: normal; font-family: 'Times New Roman';">&nbsp;&nbsp; </span></strong><strong><span style="font-family:宋体;">资源�l�之间的���理</span></strong></p>  <p style="text-indent:18.0pt;line-height:150%"><span style="font-family:宋体;">各个资源�l�之间存在数据的依赖�Q�我们希望通过资源�l�管理，能实现大部分��d��在本机房内��出数据，只有���量跨机房��出数据；大部分�Q务读取本机房的数据副本，只有���量跨机房读取数据�?/span></p>  <p style="text-indent:18.0pt;line-height:150%"><span style="font-family:宋体;">��Z��(ji��n)标识资源�l�之间的数据依赖性，定义一个资源组之间的距���L��念：(x��)一个资源组讉K��另一个资源组的数据量���多�Q�则两者的距离���近�Q�应该将距离接近的资源组攑֜�同一个机房内�?/span></p>  <p style="text-indent:18.0pt;line-height:150%"><span style="font-family:宋体;">��Z��(ji��n)让计���和产出���可能地靠近�Q���用一�?/span>MRProxy<span style="font-family:宋体;">�Q�对于不同类型的��d��做不同处理：(x��)</span></p>  <p style="margin-left:51.75pt;text-indent:-33.75pt;line-height:150%;">a�Q?span style="font-size: 7pt; line-height: normal; font-family: 'Times New Roman';">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span><span style="font-family:宋体;">��ȝ��计算�Q�跨机房列表中的数据正在传输中（</span>DC1-&gt;DC2<span style="font-family:宋体;">�Q�，</span>DC2<span style="font-family:宋体;">上的</span> Job <span style="font-family:宋体;">被暂停调度，�{�待传输完毕</span> </p>  <p style="margin-left:51.75pt;text-indent:-33.75pt;line-height:150%;">b�Q?span style="font-size: 7pt; line-height: normal; font-family: 'Times New Roman';">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span>Ad-hoc<span style="font-family:宋体;">查询�Q?/span>DC2<span style="font-family:宋体;">上的</span> Job <span style="font-family:宋体;">需要读</span>DC1<span style="font-family:宋体;">上的数据�Q?/span>Job<span style="font-family:宋体;">暂停调度�Q�通知</span> CrossNode<span style="font-family:宋体;">�Q�数据传输完毕后�l�箋调度</span> </p>  <p style="margin-left:51.75pt;text-indent:-33.75pt;line-height:150%;">c�Q?span style="font-size: 7pt; line-height: normal; font-family: 'Times New Roman';">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span><span style="font-family:宋体;">�Ҏ(gu��)��情况�Q�跨机房数据</span> Join<span style="font-family:宋体;">�Q?/span>DC1<span style="font-family:宋体;">大表�Q?/span>DC2<span style="font-family:宋体;">���表�Q?/span>Job <span style="font-family:宋体;">调度�?/span>DC1<span style="font-family:宋体;">上，跨机房直接读�?/span>DC2<span style="font-family:宋体;">数据�Q�无需�{�待</span> </p>  <p style="margin-left:18.0pt;line-height:150%">&nbsp;</p>  <p style="text-indent:18.0pt;line-height:150%"><span style="font-family:宋体;">�׃��是根据视频和</span>ppt<span style="font-family:宋体;">整理�Q��ƈ没有代码或者文档，所以可能有些地方的理解有偏差，�Ƣ迎来提意见</span>~</p></div><img src ="http://www.shnenglu.com/whspecial/aggbug/203940.html" width = "1" height = "1" /><br><br><div align=right><a style="text-decoration:none;" href="http://www.shnenglu.com/whspecial/" target="_blank">whspecial</a> 2013-10-27 23:28 <a href="http://www.shnenglu.com/whspecial/archive/2013/10/27/203940.html#Feedback" target="_blank" style="text-decoration:none;">发表评论</a></div>]]></description></item><item><title>Dremel存储格式解析</title><link>http://www.shnenglu.com/whspecial/archive/2013/08/14/202546.html</link><dc:creator>whspecial</dc:creator><author>whspecial</author><pubDate>Wed, 14 Aug 2013 15:17:00 GMT</pubDate><guid>http://www.shnenglu.com/whspecial/archive/2013/08/14/202546.html</guid><wfw:comment>http://www.shnenglu.com/whspecial/comments/202546.html</wfw:comment><comments>http://www.shnenglu.com/whspecial/archive/2013/08/14/202546.html#Feedback</comments><slash:comments>1</slash:comments><wfw:commentRss>http://www.shnenglu.com/whspecial/comments/commentRss/202546.html</wfw:commentRss><trackback:ping>http://www.shnenglu.com/whspecial/services/trackbacks/202546.html</trackback:ping><description><![CDATA[<p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp; &nbsp; Dremel是google推出的又一���器�Q�paper中宣�U�能够在3s内分�?PB的数据，主要是面向交互式查询。这���paper对嵌套类型的存储方式斚w���Q�思维���实有些跌����Q�这���文章主要讲讲这个，一斚w��是方便后来者理解，另一斚w��是让自己也整理下思�\�?br /></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp;&nbsp;&nbsp;&nbsp;首先Dremel使用的是列存模型�Q�对于基本类型列存较�Ҏ(gu��)��做到�Q�但是对于嵌套类型，Dremel也能做到���其拆解成基本类型�ƈ�q�行列存�Q�这是值得我们研究的�?br /></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp;&nbsp;&nbsp;&nbsp;直观看下嵌套�c�d��按行存储和拆解后按列存储的对比效果：(x��)<br /></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;"><a target="_blank"  data_ue_src="http://blog.51cto.com/attachment/201306/230027199.jpg"><img src="http://blog.51cto.com/attachment/201306/230027199.jpg" title="11.jpg" width="450" height="205" border="0" hspace="0" vspace="0" data_ue_src="http://blog.51cto.com/attachment/201306/230027199.jpg" style="width: 450px; height: 205px;"  alt="" /></a></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp;&nbsp;&nbsp;&nbsp;然后对于嵌套数据�c�d���Q�Dremel里面定义�?ji��n)里面三�U�类型的字段<br /></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp;&nbsp;&nbsp;&nbsp;1�Q�必���d���?�ơ而且仅出�?�ơ的字段�Q�required<br /></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp;&nbsp;&nbsp;&nbsp;2�Q�可能出�?�ơ或�?�ơ的字段�Q�optional<br /></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp;&nbsp;&nbsp;&nbsp;3�Q�可能出�?�ơ或者N�ơ字�D�：(x��)repeated<br /></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp;&nbsp;&nbsp;&nbsp;下面以paper的例子来讲述吧：(x��)</p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;"><a target="_blank"  data_ue_src="http://blog.51cto.com/attachment/201306/225240357.jpg"><img src="http://blog.51cto.com/attachment/201306/225240357.jpg" title="2.jpg" width="450" height="360" border="0" hspace="0" vspace="0" data_ue_src="http://blog.51cto.com/attachment/201306/225240357.jpg" style="width: 450px; height: 360px;"  alt="" /></a></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp;&nbsp;&nbsp;&nbsp;其中DocId是required字段�Q�因此在r1,r2中必���d���?�ơ；url字段是optional字段�Q�因此在r1的第三个Name里未出现�Q�在r1的前两个Name里出��C��(ji��n)1�ơ；Backward字段是repeated字段�Q�因此在r1的Links里未出现�Q�在r2的Links里出��C��(ji��n)2�ơ�?br /></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp;&nbsp;&nbsp;&nbsp;理解�?ji��n)上面这些，直接来看下Dremel是怎么来存它的吧：(x��)<br /></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;"><a target="_blank"  data_ue_src="http://blog.51cto.com/attachment/201306/231032285.jpg"><img src="http://blog.51cto.com/attachment/201306/231032285.jpg" title="3.jpg" width="450" height="363" border="0" hspace="0" vspace="0" data_ue_src="http://blog.51cto.com/attachment/201306/231032285.jpg" style="width: 450px; height: 363px;"  alt="" /></a></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp;&nbsp;&nbsp;&nbsp;上表中的每条记录都有两个属性，"r"代表repetition&nbsp;level�Q?d"代表definition&nbsp;level�Q�定义如下：(x��)<br /></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp;&nbsp;&nbsp;&nbsp;repetition&nbsp;level:what&nbsp;repeated&nbsp;field&nbsp;in&nbsp;the&nbsp;field&#8217;s&nbsp;path&nbsp;the&nbsp;value&nbsp;has&nbsp;repeated�Q�记录该字段是在哪个repeated�U�别上重复的</p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp;&nbsp;&nbsp;&nbsp;definition&nbsp;level:how&nbsp;many&nbsp;fields&nbsp;inpthat&nbsp;could&nbsp;be&nbsp;undefined&nbsp;(because&nbsp;they&nbsp;are&nbsp;optional&nbsp;or&nbsp;repeated)&nbsp;are&nbsp;actually&nbsp;present�Q�记录该字段之上有多���个optional或者repeated字段实际是有值的�Q�本来可以�ؓ(f��)null的）(j��)</p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp;&nbsp;&nbsp;&nbsp;看到�q�里�Q�各位可能已�l�在�?j��)里默念了(ji��n)�?x��)WTF�Q�别急，可以�l�合一个例子来看：(x��)<br /></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;"><strong>先看repetition&nbsp;level</strong>�Q�下面以r替代�Q�，以Name.Language.Code��Z���Q?/p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp;&nbsp;&nbsp;&nbsp;1)对第1个出现的��|��其r始终�?�Q�因�?en-us'的r�?<br /></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp;&nbsp;&nbsp;&nbsp;2)对于�W?个�?en'�Q�其上一个值是'en-us'�Q�它们是在Language�U�别发生的重复，Name.Language是两�U�的repeated字段�Q�因此r�?<br /></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp;&nbsp;&nbsp;&nbsp;3)对于�W?个值null�Q�是��Z��(ji��n)记录'en-gb'是出现在�W�三个Name而非�W�二个Name里，�Ҏ(gu��)��占位用的。null的上一个值是'en'�Q�它们是在Name�U�别发生的重复，因此r�?<br /></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp;&nbsp;&nbsp;&nbsp;4)对于�W?个�?en-gb'�Q�其上一个值是null�Q�它们也是在Name�U�别发生的重复，因此r�?<br /></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp;&nbsp;&nbsp;&nbsp;5)对于�W?个值null�Q�其上一个值是'en-gb'�Q�它们出现在两个不同Document里，因此r�?<br /></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp;&nbsp;&nbsp;&nbsp;�ȝ��下，看repetition&nbsp;level注意两点�Q?,只比较该值和上一个��|��2,只需要看�q�两个值的重复位置上有几个repeated字段<br /></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;"><strong>再看definition&nbsp;level</strong>�Q�下面以d替代�Q�，也以Name.Language.Code��Z���Q?/p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp;&nbsp;&nbsp;&nbsp;1)对于'en-us'�Q�其上的Name�Q�Language都出��C��(ji��n)�Q�因此d�?�Q�其实对于非null值的字段�Q�其上的optional或者repeated字段肯定是出��C��(ji��n)�Q�所以都是相同的�Q�只是null字段的d值有差别�Q?br /></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp;&nbsp;&nbsp;&nbsp;2)对于'en'�Q�同理d也�ؓ(f��)2<br /></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp;&nbsp;&nbsp;&nbsp;3)对于null�Q�其上只出现�?ji��n)Name�Q�没有出现Language�Q�因此d�?<br /></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp;&nbsp;&nbsp;&nbsp;4)对于'en-gb',d也�ؓ(f��)2<br /></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp;&nbsp;&nbsp;&nbsp;5)对于最后一个null�Q�其上也只出��C��(ji��n)Name�Q�没有出现Language�Q�因此d�?<br /></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;"><br /></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp;&nbsp;&nbsp;&nbsp;以上只是讲了(ji��n)dremel怎么��d��嵌套�c�d���Q�至于这�U�存法是怎么惛_��来的�Q�真非我辈能理解的了(ji��n)。。。更多内容，请参考原著paper�?qi��ng)网上解析�?br /></p><div></div><img src ="http://www.shnenglu.com/whspecial/aggbug/202546.html" width = "1" height = "1" /><br><br><div align=right><a style="text-decoration:none;" href="http://www.shnenglu.com/whspecial/" target="_blank">whspecial</a> 2013-08-14 23:17 <a href="http://www.shnenglu.com/whspecial/archive/2013/08/14/202546.html#Feedback" target="_blank" style="text-decoration:none;">发表评论</a></div>]]></description></item><item><title>Orcfile文�g格式解析�Q?�Q?/title><link>http://www.shnenglu.com/whspecial/archive/2013/08/14/202545.html</link><dc:creator>whspecial</dc:creator><author>whspecial</author><pubDate>Wed, 14 Aug 2013 15:13:00 GMT</pubDate><guid>http://www.shnenglu.com/whspecial/archive/2013/08/14/202545.html</guid><wfw:comment>http://www.shnenglu.com/whspecial/comments/202545.html</wfw:comment><comments>http://www.shnenglu.com/whspecial/archive/2013/08/14/202545.html#Feedback</comments><slash:comments>0</slash:comments><wfw:commentRss>http://www.shnenglu.com/whspecial/comments/commentRss/202545.html</wfw:commentRss><trackback:ping>http://www.shnenglu.com/whspecial/services/trackbacks/202545.html</trackback:ping><description><![CDATA[<p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp; &nbsp; 上篇文章从整体介�l�了(ji��n)Orcfile的存储格式，接下来重点介�l�下Orc里用到的几种�~�码格式�Q?/p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp;&nbsp;&nbsp;&nbsp;字典�~�码�Q�用于String�c�d��的字�D?br /></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp;&nbsp;&nbsp;&nbsp;Run-Length�~�码�Q�用于int�Q�long�Q�short�{�类型的�~�码</p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp;&nbsp;&nbsp;&nbsp;Bit�~�码�Q�可以用于各�U�数据类�?br /></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;"><strong>1�Q�字典编码：(x��)</strong></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp;&nbsp;&nbsp;&nbsp;对于String�c�d��的每个字�D�分别保存一个字典，记录每个值在字典中的位置�Q�保存字典的数据�l�构采用一��늺�黑树(w��i)。对于每个String字段�Q�最�l�会(x��)有三个输出Stream�Q�分别是StringOuptut(记录字典中的�?�Q�LengthOutput(记录每个字典值的长度)�Q�RowOutput(记录字段在字�怸�的位�|?�?/p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp;&nbsp;&nbsp;&nbsp;思�?�Q��ؓ(f��)什么要用红黑树(w��i)�Q?br /></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp;&nbsp;&nbsp;&nbsp;因�ؓ(f��)�U�黑�?w��i)无论是插入�Q�删除，查找的性能都比较��^均，都是O(logN)�Q�而且是��^衡查找树(w��i)�Q�最坏情况也不会(x��)退化成O(N)<br /></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp;&nbsp;&nbsp;&nbsp;思�?�Q�其实一般存储时�q�会(x��)使用LZO之类的压�~�，它们本��n���是一�U�字典压�~�，��Z��么Orc里面要自己做字典压羃�Q?br /></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp;&nbsp;&nbsp;&nbsp;因�ؓ(f��)LZO之类的压�~�窗口一般比较小�Q�LZO默认�?4KB�Q�，而Orc的字典压�~�是以整个字�D��ؓ(f��)范围来压�~�的�Q�压�~�率�?x��)更好�?br /></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;"><strong>2�Q�Run-Length�~�码�Q?/strong><br /></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp;&nbsp;&nbsp;&nbsp;对于int,long,short�c�d��的字�D�，使用Run-Length�~�码。该Run-Length能够对等差数列（完全相等也属于等差数列）(j��)�q�行压羃�Q�该�{�差数列需要满���以下两个条�Ӟ��(x��)</p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp;&nbsp;&nbsp;&nbsp;1�Q�至���包�?个元�?/p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp;&nbsp;&nbsp;&nbsp;2�Q�差值在-128~127之间�Q�因为差值用1Byte来表�C�）(j��)</p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp;&nbsp;&nbsp;&nbsp;对于不满���等差数列的数字�Q�Run-Length�~�码也能存储�Q�但是没有压�~�效果，Run-Length的具体存储如下：(x��)</p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp;&nbsp;&nbsp;&nbsp;�W�一个Byte是Control&nbsp;Byte�Q�取值在-128~127之间�Q�其�?1~-128代表后面存储着1~128个不满���{�差数列的数字，0~127代表后面存储着3~130个等差数列的数字�Q?/p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp;&nbsp;&nbsp;&nbsp;如果Control&nbsp;Byte&gt;=0�Q�则后面跟着一个Byte存储差��|��否则不存储该Byte�Q?/p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp;&nbsp;&nbsp;&nbsp;如果Control&nbsp;Byte&gt;=0�Q�则后面跟着�{�差数列的第一个数�Q�否则跟着-Control&nbsp;Byte个数字�?/p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp;&nbsp;&nbsp;&nbsp;例子�Q?/p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp;&nbsp;&nbsp;&nbsp;原始数字�Q?2,12,12,12,12,10,7,13</p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp;&nbsp;&nbsp;&nbsp;�l�过Run-Length的数字：(x��)<span style="color: #ff0000;">2</span>,<span style="color: #ffc000;">0</span>,12,<span style="color: #ff0000;">-3</span>,10,7,13</p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp;&nbsp;&nbsp;&nbsp;�U�色代表Control&nbsp;Byte�Q�黄色代表差��|��黑色代表具体的数字�?br /></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;"><strong>3�Q�Bit�~�码�Q?/strong></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">�Ҏ(gu��)��有类型的字段都可以采用Bit�~�码来表�C����值是否�ؓ(f��)null。在写�Q何类型字�D�之前，先判断该字段值是够�ؓ(f��)null�Q�如果�ؓ(f��)null则bit值存�?�Q�否则存�?�Q�对于�ؓ(f��)null的字�D�在实际�~�码时不需要存储了(ji��n)。经�q�Bit�~�码之后�Q�可以对�?个bit�l�成一个Byte�Q�再对其�q�行Run-Length�~�码�?/p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp;&nbsp;&nbsp;&nbsp;其实除了(ji��n)�q�三�U�编码格式之外，Orc对于hive的复杂类型array,map,list�{�，���其降维成基本类型来存储�Q�这个也是值得借鉴的，如果有空之后�?x��)进行分析�?/p><img src ="http://www.shnenglu.com/whspecial/aggbug/202545.html" width = "1" height = "1" /><br><br><div align=right><a style="text-decoration:none;" href="http://www.shnenglu.com/whspecial/" target="_blank">whspecial</a> 2013-08-14 23:13 <a href="http://www.shnenglu.com/whspecial/archive/2013/08/14/202545.html#Feedback" target="_blank" style="text-decoration:none;">发表评论</a></div>]]></description></item><item><title>Orcfile文�g格式解析�Q?�Q?/title><link>http://www.shnenglu.com/whspecial/archive/2013/08/14/202544.html</link><dc:creator>whspecial</dc:creator><author>whspecial</author><pubDate>Wed, 14 Aug 2013 15:12:00 GMT</pubDate><guid>http://www.shnenglu.com/whspecial/archive/2013/08/14/202544.html</guid><wfw:comment>http://www.shnenglu.com/whspecial/comments/202544.html</wfw:comment><comments>http://www.shnenglu.com/whspecial/archive/2013/08/14/202544.html#Feedback</comments><slash:comments>0</slash:comments><wfw:commentRss>http://www.shnenglu.com/whspecial/comments/commentRss/202544.html</wfw:commentRss><trackback:ping>http://www.shnenglu.com/whspecial/services/trackbacks/202544.html</trackback:ping><description><![CDATA[<p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp; &nbsp; Orcfile(<em>Optimized&nbsp;Row&nbsp;Columnar</em>)是hive&nbsp;0.11版里引入的新的存储格式，是对之前的RCFile存储格式的优化。写�q�个的哥们来自于HortonWorks�Q�代码写的很不错�Q�比之前的rcfile强多�?ji��n)（据说rcfile是个中科院的童鞋跑去facebook写的�Q�看来中国的计算机教育水�q����是有限啊。。。囧�Q�跑题了(ji��n)�Q?/p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp;&nbsp;&nbsp;&nbsp;先介�l�下Orc的文件格式，截一张官方的图：(x��)<br /></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;"><a target="_blank"  data_ue_src="http://blog.51cto.com/attachment/201306/234709762.png"><img src="http://blog.51cto.com/attachment/201306/234709762.png" title="OrcFileLayout.png" data_ue_src="http://blog.51cto.com/attachment/201306/234709762.png"  alt="" /></a></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp;&nbsp;&nbsp;&nbsp;可以看到每个Orc文�g�?个或多个stripe�l�成�Q�每个stripe250MB大小�Q�这个Stripe实际相当于之前的rcfile里的RowGroup概念�Q�不�q�大���由4MB-&gt;250MB�Q�这样应该能提升��序�ȝ��吞吐率。每个Stripe里有三部分组成，分别是Index&nbsp;Data,Row&nbsp;Data,Stripe&nbsp;Footer�Q?/p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp;&nbsp;&nbsp;&nbsp;1�Q�Index&nbsp;Data�Q�一个轻量��的index�Q�默认是每隔1W行做一个烦(ch��)引。这里做的烦(ch��)引应该只是记录某行的各字�D�在Row&nbsp;Data中的offset�Q�据说还包括每个Column的max和min��|��具体没细看代码�?/p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp;&nbsp;&nbsp;&nbsp;2�Q�Row&nbsp;Data�Q�存的是具体的数据，和RCfile一��P��先取部分行，然后对这些行按列�q�行存储。与RCfile不同的地方在于每个列�q�行�?ji��n)编码，分成多个Stream来存储，具体如何�~�码在下一���解析里�?x��)讲�?/p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp;&nbsp;&nbsp;&nbsp;3�Q�Stripe&nbsp;Footer�Q�存的是各个Stream的类型，长度�{�信息�?/p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp;&nbsp;&nbsp;&nbsp;每个文�g有一个File&nbsp;Footer�Q�这里面存的是每个Stripe的行敎ͼ�每个Column的数据类型信息等�Q�每个文件的���N��是一个PostScript�Q�这里面记录�?ji��n)整个文件的压羃�c�d��以及(qi��ng)FileFooter的长度信息等。在��d��文�g�Ӟ���?x��)seek到文件尾部读PostScript�Q�从里面解析到File&nbsp;Footer长度�Q�再读FileFooter�Q�从里面解析到各个Stripe信息�Q�再��d��个Stripe�Q�即从后往前读�?/p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp;&nbsp;&nbsp;&nbsp;接下来看下ORcfile相对于RCfile做了(ji��n)哪些改进�Q�从Orc作者的ppt里截�?ji��n)张图，分别解释下各行�?x��)<br /></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;"><a target="_blank"  data_ue_src="http://blog.51cto.com/attachment/201306/003840639.png"><img src="http://blog.51cto.com/attachment/201306/003840639.png" title="囄���1.png" width="500" height="345" border="0" hspace="0" vspace="0" data_ue_src="http://blog.51cto.com/attachment/201306/003840639.png" style="width: 500px; height: 345px;"  alt="" /></a></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp;&nbsp;&nbsp;&nbsp;Hive&nbsp;type&nbsp;model:RCfile在底层存储时不保存类型，都当做Byte���来存储</p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp;&nbsp;&nbsp;&nbsp;Separtor&nbsp;complex&nbsp;columns:Orc���复杂类型拆开存储</p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp;&nbsp;&nbsp;&nbsp;Splits&nbsp;Found&nbsp;Quickly�Q�不很理�?br /></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp;&nbsp;&nbsp;&nbsp;Default&nbsp;Column&nbsp;group&nbsp;size�Q�不用解释了(ji��n)</p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp;&nbsp;&nbsp;&nbsp;Files&nbsp;per&nbsp;a&nbsp;bucket�Q�不很理�?/p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp;&nbsp;&nbsp;&nbsp;Store&nbsp;min�Q�max�Q�count�Q�sum�Q�存�?ji��n)这些便于快速地skip掉一个stripe</p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp;&nbsp;&nbsp;&nbsp;Versioned&nbsp;metadata:不很理解<br /></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp;&nbsp;&nbsp;&nbsp;Run-Length&nbsp;Data-coding�Q�整数类型做Run-Length变长�~�码</p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp;&nbsp;&nbsp;&nbsp;Store&nbsp;Strings&nbsp;in&nbsp;dictionary�Q�String�c�d��做字典编�?/p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp;&nbsp;&nbsp;&nbsp;Store&nbsp;Row&nbsp;Count�Q�每个Stripe�?x��)存储行�?/p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp;&nbsp;&nbsp;&nbsp;Skip&nbsp;Compressed&nbsp;blocks:可以直接skip掉压�~�过的block</p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp;&nbsp;&nbsp;&nbsp;Store&nbsp;internal&nbsp;indexes:存储�?ji��n)一个轻量��的index</p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;"><br /></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">&nbsp;&nbsp;&nbsp;&nbsp;整个Orc看下来，代码写的�q�是比较清晰明了(ji��n)的，而且我们也进行了(ji��n)���试�Q�压�~�效果比RCfile提升�?ji��n)不���，有兴���的朋友可以来看下，之后会(x��)写�W�二���解析，主要是讲Orc用到的几�U�编码格式�?br /></p><div></div><img src ="http://www.shnenglu.com/whspecial/aggbug/202544.html" width = "1" height = "1" /><br><br><div align=right><a style="text-decoration:none;" href="http://www.shnenglu.com/whspecial/" target="_blank">whspecial</a> 2013-08-14 23:12 <a href="http://www.shnenglu.com/whspecial/archive/2013/08/14/202544.html#Feedback" target="_blank" style="text-decoration:none;">发表评论</a></div>]]></description></item><item><title>关于map/reduce的combiner�q�行时机的问�?/title><link>http://www.shnenglu.com/whspecial/archive/2012/11/06/194819.html</link><dc:creator>whspecial</dc:creator><author>whspecial</author><pubDate>Tue, 06 Nov 2012 15:52:00 GMT</pubDate><guid>http://www.shnenglu.com/whspecial/archive/2012/11/06/194819.html</guid><wfw:comment>http://www.shnenglu.com/whspecial/comments/194819.html</wfw:comment><comments>http://www.shnenglu.com/whspecial/archive/2012/11/06/194819.html#Feedback</comments><slash:comments>0</slash:comments><wfw:commentRss>http://www.shnenglu.com/whspecial/comments/commentRss/194819.html</wfw:commentRss><trackback:ping>http://www.shnenglu.com/whspecial/services/trackbacks/194819.html</trackback:ping><description><![CDATA[<blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;"></blockquote><span style="font-size: 12pt;">&nbsp; &nbsp;</span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">map/reduce</span><span style="font-size: 12pt; line-height: 150%; font-family: 宋体;">�?/span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">combiner</span><span style="font-size: 12pt; line-height: 150%; font-family: 宋体;">到底在什么时候运行？</span>  <p align="left" style="text-indent: 21pt; line-height: 150%;"><span style="font-size: 12pt; line-height: 150%; font-family: 宋体;">在网上大多数资料中，都是�?/span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">combiner</span><span style="font-size: 12pt; line-height: 150%; font-family: 宋体;">�?/span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">map</span><span style="font-size: 12pt; line-height: 150%; font-family: 宋体;">端运行，发生�?/span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">map</span><span style="font-size: 12pt; line-height: 150%; font-family: 宋体;">输出数据之后�Q�经�q?/span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">combiner</span><span style="font-size: 12pt; line-height: 150%; font-family: 宋体;">再传递给</span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">reducer</span><span style="font-size: 12pt; line-height: 150%; font-family: 宋体;">。但是之前在工作中出现的一个问题导致我发现原来</span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">combiner</span><span style="font-size: 12pt; line-height: 150%; font-family: 宋体;">居然也会(x��)�?/span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">reducer</span><span style="font-size: 12pt; line-height: 150%; font-family: 宋体;">端运行，�q�且�?x��)多�ơ运行�?/span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;"><br /> </span><span style="font-size: 8pt; line-height: 150%; font-family: Verdana, sans-serif;">&nbsp; &nbsp;</span><span style="font-size: 12pt; line-height: 150%; font-family: 宋体;">在网上查�?ji��n)之后发玎ͼ��q�是</span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">hadoop-0.18</span><span style="font-size: 12pt; line-height: 150%; font-family: 宋体;">版本引入的新</span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">feature</span><span style="font-size: 12pt; line-height: 150%; font-family: 宋体;">�Q?/span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;"><br /> </span><span style="font-size: 8pt; line-height: 150%; font-family: Verdana, sans-serif;">&nbsp; &nbsp;</span><span style="font-size:12.0pt;line-height:150%;font-family:&quot;Arial&quot;,&quot;sans-serif&quot;;color:#333333;background:white;">Changed policy for running combiner. The combiner may be run multiple times as the map's output is sorted and merged. Additionally, it may be run on the reduce side as data is merged. The old semantics are available in Hadoop 0.18 if the user calls:&nbsp;job.setCombineOnlyOnce(true)</span><span style="font-size:12.0pt;line-height:150%;font-family:宋体;color:#333333; background:white;">�?/span><span style="font-size: 12pt; line-height: 150%; font-family: Verdana, sans-serif;"><br /> </span><span style="font-size: 8pt; line-height: 150%; font-family: Verdana, sans-serif;">&nbsp; &nbsp;</span><span style="font-size: 12pt; line-height: 150%; font-family: 宋体;">实际�?/span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">combiner</span><span style="font-size: 12pt; line-height: 150%; font-family: 宋体;">�?x��)�?/span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">mapper</span><span style="font-size: 12pt; line-height: 150%; font-family: 宋体;">端和</span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">reducer</span><span style="font-size: 12pt; line-height: 150%; font-family: 宋体;">端分别运�q�行�Q�看�?ji��n)下代码�Q�发�?/span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">combine</span><span style="font-size: 12pt; line-height: 150%; font-family: 宋体;">的时机在以下�Q?/span><span style="font-size: 8pt; line-height: 150%; font-family: Verdana, sans-serif;"><br /> </span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">&nbsp; &nbsp;1</span><span style="font-size: 12pt; line-height: 150%; font-family: 宋体;">�Q?/span>&nbsp;<span style="font-size: 12pt; line-height: 150%; font-family: 宋体;">�?/span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">mapper</span><span style="font-size: 12pt; line-height: 150%; font-family: 宋体;">端的</span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">spill</span><span style="font-size: 12pt; line-height: 150%; font-family: 宋体;">阶段�Q�在�~�存中的记录���过阈值时�?x��)进�?/span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">combine</span></p>  <p align="left" style="margin: 2.9pt 0cm; line-height: 150%;"><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">if (spstart != spindex) {</span></p>  <p align="left" style="margin: 2.9pt 0cm; line-height: 150%;"><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &#8230;</span></p>  <p align="left" style="margin: 2.9pt 0cm; text-indent: 18pt; line-height: 150%;"><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">combineAndSpill(kvIter, combineInputCounter);</span></p>  <p align="left" style="margin: 2.9pt 0cm; line-height: 150%;"><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">}</span></p>  <p align="left" style="margin: 2.9pt 0cm 2.9pt 18pt; text-indent: -18pt; line-height: 150%;"><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">2</span><span style="font-size: 12pt; line-height: 150%; font-family: 宋体;">�Q?/span>&nbsp;<span style="font-size: 12pt; line-height: 150%; font-family: 宋体;">�?/span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">mapper</span><span style="font-size: 12pt; line-height: 150%; font-family: 宋体;">端的</span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">merge</span><span style="font-size: 12pt; line-height: 150%; font-family: 宋体;">阶段�Q�进�?/span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">merge</span><span style="font-size: 12pt; line-height: 150%; font-family: 宋体;">�?/span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">spill</span><span style="font-size: 12pt; line-height: 150%; font-family: 宋体;">文�g数目</span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">&gt;=3</span><span style="font-size: 12pt; line-height: 150%; font-family: 宋体;">时会(x��)�q�行</span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">combine</span></p>  <p align="left" style="margin: 2.9pt 0cm; line-height: 150%;"><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">if (null == combinerClass || numSpills &lt; minSpillsForCombine) {</span></p>  <p align="left" style="margin: 2.9pt 0cm; text-indent: 21pt; line-height: 150%;"><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">Merger.writeFile(kvIter, writer, reporter);</span></p>  <p align="left" style="margin: 2.9pt 0cm; line-height: 150%;"><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">} else {</span></p>  <p align="left" style="margin: 2.9pt 0cm; text-indent: 21pt; line-height: 150%;"><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">combineCollector.setWriter(writer);</span></p>  <p align="left" style="margin: 2.9pt 0cm; text-indent: 21pt; line-height: 150%;"><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">combineAndSpill(kvIter, combineInputCounter);</span></p>  <p align="left" style="margin: 2.9pt 0cm; line-height: 150%;"><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">}</span></p>  <p style="line-height:150%"><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">3</span><span style="font-size: 12pt; line-height: 150%; font-family: 宋体;">�Q?/span>&nbsp;<span style="font-size: 12pt; line-height: 150%; font-family: 宋体;">�?/span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">reducer</span><span style="font-size: 12pt; line-height: 150%; font-family: 宋体;">端，一定会(x��)�q�行</span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">combine</span></p><span style="font-size: 12pt; font-family: Arial; line-height: 17px; background-color: #ffffff; "><br /></span><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;"></blockquote><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;"></blockquote><div   class="uea8a6c"   id="isChromeWebToolbarDiv" style="display:none"></div> <div   class="w0mw8ya"   id="isChromeWebToolbarDiv" style="display:none"></div> <div   class="0aikg0o"   id="isChromeWebToolbarDiv" style="display:none"></div><img src ="http://www.shnenglu.com/whspecial/aggbug/194819.html" width = "1" height = "1" /><br><br><div align=right><a style="text-decoration:none;" href="http://www.shnenglu.com/whspecial/" target="_blank">whspecial</a> 2012-11-06 23:52 <a href="http://www.shnenglu.com/whspecial/archive/2012/11/06/194819.html#Feedback" target="_blank" style="text-decoration:none;">发表评论</a></div>]]></description></item></channel></rss>                                                                                                                                                                        <footer>
<div class="friendship-link">
<p>��л���������ǵ���վ�������ܻ���������Դ����Ȥ��</p>
<a href="http://www.shnenglu.com/" title="&#31934;&#21697;&#35270;&#39057;&#20037;&#20037;&#20037;&#20037;">&#31934;&#21697;&#35270;&#39057;&#20037;&#20037;&#20037;&#20037;</a>

<div class="friend-links">


</div>
</div>

</footer>

<a href="http://www.abcdds.cn" target="_blank">�þ����ü����ۺ�ɫ�ۺϰ�Ҳȥ</a>|
<a href="http://www.gg4493.cn" target="_blank">��Ʒ�þþþþ�99����</a>|
<a href="http://www.mmmj.net.cn" target="_blank">���þþ��ۺ�</a>|
<a href="http://www.r3665.cn" target="_blank">����AV���������վ�þþ�Ʒ��</a>|
<a href="http://www.zhidaow.com.cn" target="_blank">�þþþ�����AV������վ</a>|
<a href="http://www.jobkey.com.cn" target="_blank">�þþ�91��Ʒ����һ����������</a>|
<a href="http://www.adddi.cn" target="_blank">�պ����޹����ۺϾþþ�</a>|
<a href="http://www.shamba.com.cn" target="_blank">����va�þþþ�</a>|
<a href="http://www.ozxt.cn" target="_blank">ŷ�����㽶�þ��ۺ�����վ</a>|
<a href="http://www.thegraces.com.cn" target="_blank">�þ��㽶һ��ëƬ</a>|
<a href="http://www.zjhgkjjt.com.cn" target="_blank">����Ӱ��ɫ�����ۺϾþ�</a>|
<a href="http://www.yczu.cn" target="_blank">97�þþ�Ʒ������Ʒ���</a>|
<a href="http://www.cnyaxin.cn" target="_blank">���޹���һ���˾þþ�Ʒ</a>|
<a href="http://www.yes365cc.cn" target="_blank">��Ʒ�ۺϾþþþþ�97����</a>|
<a href="http://www.b24193.cn" target="_blank">����AV�պ���Ʒ�þþþþþ�</a>|
<a href="http://www.andz77.cn" target="_blank">�þèD�ձ���ɫ�ۺϾþ�</a>|
<a href="http://www.52shadu.cn" target="_blank">�þþþþþþþþ�</a>|
<a href="http://www.ksjhyt.cn" target="_blank">�þ�Ӱ�ӹ�������</a>|
<a href="http://www.dyfls.cn" target="_blank">�þ����޹�����Ʒ123��</a>|
<a href="http://www.kaczw3.cn" target="_blank">�þþ�Ʒ����һ����������ԡ��</a>|
<a href="http://www.pynov.cn" target="_blank">������Ʒ�þþ�������Ļһ��</a>|
<a href="http://www.cnmango.cn" target="_blank">������Ʒ�þþþþþþ�</a>|
<a href="http://www.ptrjmfv.cn" target="_blank">����a����������þ�</a>|
<a href="http://www.ogpx95.cn" target="_blank">2020�þþ�Ʒ�������ۺ�һ��</a>|
<a href="http://www.s9459.cn" target="_blank">���޾�ƷNV�þþþþþþþ�</a>|
<a href="http://www.xyjsj88.com.cn" target="_blank">�������ݾ�ƷŮ�˾þþþþ�</a>|
<a href="http://www.84448.com.cn" target="_blank">�þ����ù����ۺϾ�Ʒ</a>|
<a href="http://www.okfupo.cn" target="_blank">�����츾��������þþ�</a>|
<a href="http://www.nancanxie.cn" target="_blank">���޹����պ�ŷ���ۺϾþ�</a>|
<a href="http://www.csffh.cn" target="_blank">���ھ�Ʒ�þþþþ�������Ļ</a>|
<a href="http://www.xzxms.cn" target="_blank">��ѹۿ��þþ�����Ƶ</a>|
<a href="http://www.qzlhscqt.cn" target="_blank">������TOKYO�ۺϾþþ�Ʒ</a>|
<a href="http://www.aygdxx.cn" target="_blank">ɫ�ۺϾþ�����ɫ����</a>|
<a href="http://www.yangrendong.com.cn" target="_blank">�þþ�Ʒ����������Ļ����
</a>|
<a href="http://www.mobilepop.cn" target="_blank">777����þ����µ�ַ</a>|
<a href="http://www.unicity642.com.cn" target="_blank">�þ�����������ˬ������Ƭţţ</a>|
<a href="http://www.ppynw.cn" target="_blank">�þ��㽶�����߿��ۿ�è��?v</a>|
<a href="http://www.lrv9.cn" target="_blank">�þ�www����˳ɿ�����Ƭ</a>|
<a href="http://www.qunfazongzhan.cn" target="_blank">����þþþþþþþ�</a>|
<a href="http://www.pygsbdg.cn" target="_blank">�þù�������</a>|
<a href="http://www.deshizhai.cn" target="_blank">���ij������뾫Ʒ�þþò���
</a>|
<script>
(function(){
    var bp = document.createElement('script');
    var curProtocol = window.location.protocol.split(':')[0];
    if (curProtocol === 'https') {
        bp.src = 'https://zz.bdstatic.com/linksubmit/push.js';
    }
    else {
        bp.src = 'http://push.zhanzhang.baidu.com/push.js';
    }
    var s = document.getElementsByTagName("script")[0];
    s.parentNode.insertBefore(bp, s);
})();
</script>
</body>