数据压羃可分成两U类型，一U叫做无损压~，另一U叫做有损压~?br />? 损压~是指用压~后的数据进行重?或者叫做还原，解压~?Q重构后的数据与原来的数据完全相同；无损压羃用于要求重构的信号与原始信号完全一致的? 合。磁盘文件的压羃是一个很常见的例子。根据目前的技术水qI无损压羃法一般可以把普通文件的数据压羃到原来的1/2?/4?br />有损压羃是指使用压羃后的数据q行重构Q重构后的数据与原来的数据有所不同Q但不会让h对原始资?br />? 辄信息造成误解。有损压~适用于重构信号不一定非要和原始信号完全相同的场合。例如，囑փ和声音的压羃可以采用有损压~，因ؓ其中包含的数据往往多于 我们的视觉系l和听觉pȝ所能接收的信息Q丢掉一些数据而不至于对声x者图像所表达的意思生误解，但可大大提高压羃比?br />
压羃技术大致可以按照以下的Ҏ分类Q?br /> 压羃技?br /> |
/------------------------------\
通用无损数据压羃 多媒体数据压~?大多为有损压~?
| |
/----------------\ /------------------------------------\
Zl计 Z字典 音频压羃 囑փ压羃 视频压羃
模型的压 模型的压 | | |
~技? ~技? MP3{? /-------------------\ AVI
| | 二?灰度 彩色 矢量 MPEG2{?br /> /------\ /-------------\ 囑փ 囑փ 囑փ 囑փ
Huffman 术 LZ77 LZ78 LZW | | | \
~码 ~码 \-------------/ 传真?FELICS GIF PostScript
| | | 标准 JPEG{?JPEG{?Windows WMF{?br /> UNIX? 接近无损 PKZIP、LHarc、ARJ?br /> 的COMPACT 压羃极限 UNIX下的COMPRESS
E序{ ?的高U应用 程序等
通用无损数据压羃的历?br />U学家在研究中发玎ͼ大多C息的表达都存在着一定的冗余度，通过采用一定的模型和编码方法，可以
降低q种冗余度。贝实验室?Claude Shannon ?MIT ?R.M.Fano 几乎同时提出了最早的对符可行有
效编码从而实现数据压~的 Shannon-Fano ~码Ҏ?br />
D.A.Huffman ?952q第一ơ发表了他的论文“最冗余代码的构造方法?A Method for the Construction of Minimum Redundancy Codes)。从此，数据压羃开始在商业E序中实现ƈ被应用在许多技术领域。UNIX pȝ上一个压~程?COMPACT 是 Huffman 0 阶自适应~码的具体实现?0 q代初，Huffman~码又在CP/M 和DOS pȝ中实玎ͼ其代表程序叫 SQ。在数据压羃领域QHuffman 的这一论文事实上开创了数据压羃技术一个值得回忆的时代，60 q代?0 q代乃至 80 q代的早期，数据压羃领域几乎一直被 Huffman ~码及其分支所垄断。如果不是后面将要提到的那两个以色列人，也许我们今天q要? Huffman~码?0 ?1 的组合中连忘返?br />
80q代Q数学家们不满?Huffman ~码中的某些致命qQ他们从新的角度入手Q遵?Huffman ~码的主导思想Q设计出另一U更为精，更能接近信息Z“熵”极限的~码Ҏ——算术编码。凭借算术编码的_֦设计和卓表玎ͼZl于可以向着数据? ~的极限前进了。可以证明，术~码得到的压~效果可以最大地减小信息的冗余度Q用最量的符L表辑֎始信息内宏V当Ӟ术~码同时也给E序员和? 机带来了新的挑战：要实现和q行术~码Q需要更苦的~程力_和更加快速的计算机系l。也是Q在同样的计机pȝ上，术~码虽然可以得到最好的 压羃效果Q但却要消耗也许几十倍的计算时。这是Z么算术编码不能在我们日常使用的压~工具中实现的主要原因?br />
那么Q能不能既在压羃? 果上越 HuffmanQ又不增加程序对pȝ资源和时间的需求呢Q我们必L谢下面将要介l的两个以色列h。直?1977 q_数据压羃的研I工作主要集中于c字W和单词频率以及l计模型{方面，研究者们一直在l尽脑汁Z用Huffman~码的程序找出更快、更好的改进? 法?977 q以后，一切都改变了?br />
1977 q_以色列h Jacob Ziv ?Abraham Lempel 发表了论文“顺序数据压~的一个通用法?A Universal Alogrithem for Sequential Data Compression)?978 q_他们发表了该论文的箋“通过可变比率~码的独立序列的压羃?Compression of Individual Sequences via Variable-Rate Coding)。在q两论文中提出的压~技术分别被UCؓ LZ77 ?LZ78 (不知Z么，作者名字的首字母被倒置?。简单地_q两U压~方法的思\完全不同于从 Shannon ?Huffman 到算术压~的传统思\Qh们将Zq一思\的编码方法称作“字典”式~码。字典式~码不但在压~效果上大大过? HuffmanQ而且Q对于好的实玎ͼ其压~和解压~的速度也异常惊人?br />
1984 q_Terry Welch 发表了名为“高性能数据压羃技术?A Technique for High-Performance Data Compression)的论文，描述了他?Sperry Research Center(现在是Unisys的一部分)的研I成果。他实现了LZ78 法的一个变U?—?LZW。LZW l承?LZ77 ?LZ78 压羃效果好、速度快的优点Q而且在算法描qC更容易被Z接受Q有的研I者认为是׃ Welch 的论文比 Ziv ?Lempel 的更Ҏ理解Q，实现也比较简单。不久，UNIX 上出C使用 LZW 法?Compress E序Q该E序性能优良Qƈ有高水^的文档，很快成ؓ?UNIX 世界的压~程序标准。紧随其后的?MS-DOS 环境下的ARCE序( System Enhancement Associates, 1985 )Q还有象 PKWare、PKARC {仿制品。LZ78和LZW一旉l治了UNIX和DOS两大q_?br />
80 q代中期以后Qh们对 LZ77 q行了改q，随之诞生了一Ҏ们今天还在大量用的压羃E序。Haruyasu Yoshizaki(Yoshi)的LHarc和Robert Jung的ARJ是其中两个著名的例子。LZ77得以和LZ78、LZW一起垄断当今的通用数据压羃领域?br />
目前Q基于字典方式的压羃已经有了一个被q泛认可的标准，从古老的PKZip到现在的WinZipQ特别是?br />着Internet上文件传输的行QZIP 格式成ؓ了事实上的标准，没有哪一U通用的文件压~、归档系l不?br />?ZIP 格式。本章主要介l目前用得最多和技术最成熟的无损压~编码技术，包括包含霍夫?Huffman)~码、算术编码、RLE~码和词典编?。注意有一部分压羃法受到国专利法的保护Q例?LZW 法的某些部分和高阶术压羃法的某些细节等Q?br />
4.1 仙农-范诺与霍夫曼~码
4.1.1 仙农-范诺QShannon-FanoQ编?br />仙农-范诺~码法需要用C面两个基本概念：
1. Entropy(?
(1) 熉|信息量的度量ҎQ表CZ条信息中真正需要编码的信息量。事件发生的可能性越（数学上就
是概率越）Q表C某一事g出现的消息越多?br />(2) 某个事g的信息量用Ii=-log2 pi表示Q有时称为surpriseQ， 其中pi为第i个事件的概率Q? pi 1Ҏ?为底Ӟ늚单位?bits"?br />2. 信源S的熵
按照仙农(Shannon)的理论，信源S的熵定义?br />其中pi是符号si在S中出现的概率Qlog2(1/ pi)表示包含在si中的信息量，也就是编码si所需要的位数?br />例如Q一q用256U灰度表C的囑փQ如果每一个象素点灰度的概率均为pi=1/256Q编码每一个象素点需
?位。（最大熵分布Q?br />最熵分布Q?除了一个符号外其余W号的概率全?QHQ?bits.Q定?log20Q?Q?br />
例如Q对下面q条只出C a b c 三个字符的字W串QaabbaccbaaQ字W串长度?10Q字W?a b c ?br />别出C 5 3 2 ơ，?a b c 在信息中出现的概率分别ؓ 0.5 0.3 0.2Q他们的熵分别ؓQ?br />Ea = -log2(0.5) = 1
Eb = -log2(0.3) = 1.737
Ec = -log2(0.2) = 2.322
整条信息的熵也即表达整个字符串需要的位数为：
Ea * 5 + Eb * 3 + Ec * 2 = 14.855 ?br />如果用计机中常用的 ASCII ~码Q表CZ面的字符串需要整?0位！信息Z么能被压~而不丢失?br />有的信息内容呢？单地Ԍ用较的位数表示较频J出现的W号Q这是数据压羃的基本准则。（怎样?0
1 q样的二q制数码表示零点几个二进制位呢？实很困难，但不是没有办法。一旦找C准确表示零点几个
二进制位的方法，接q无损压~的极限了。）
[?.1] 有一q?0个象素组成的灰度囑փQ灰度共?U，分别用符号A、B、C、D和E表示Q?0个象素中?br />现灰度A的象素数?5个，出现灰度B的象素数?个，出现灰度C的象素数?个等{，如表4-01所C?br />如果?个位表示q?个等U的灰度|也就是每个象素用3位表C?{长~码)Q编码这q图像d需?20
位?br />?-01 W号在图像中出现的数?br />按照仙农理论Q这q图像的熵ؓ H(S)=(15/40)×log2(40/15) + (7/40)×log2(40/7) +?+ (5/40)
×log2(40/5) = 2.196
q就是说每个W号?.196位表C，40个象素需?7.84位?br />最早阐q和实现q种~码的是Shannon(1948q?和Fano(1949q?Q因此被UCؓ仙农-范诺(Shannon-Fano)?br />法。这U方法采用从上到下的Ҏq行~码?br />首先按照W号出现的频度或概率排序Q例如，AQBQCQD和EQ如?-02所C?br />然后使用递归Ҏ分成两个部分Q每一部分hq似相同的次敎ͼ如图4-01所C?br />按照q种Ҏq行~码得到的MCؓ91Q实际的压羃比约?.3 : 1?br />?-02 Shannon-Fano法举例?br />W??A B C D E
出现的次?15 7 7 6 5
W号 出现的次?pi) log2(1/pi) 分配的代?需要的位数
A 15 (0.375) 1.4150 00 30
B 7 (0.175) 2.5145 01 14
C 7 (0.175) 2.5145 10 14
D 6 (0.150) 2.7369 110 18
E 5 (0.125) 3.0000 111 15
W?4 ?br />4
?-01 仙农-范诺法~码举例
4.1.2 霍夫?Huffman)~码
霍夫曼在1952q提Z另一U编码方法，即从下到上的~码Ҏ。现以一个具体的例子说明它的~码?br />骤：
(1) 初始化，ҎW号概率的大按由大到小序对符可行排序，如表4-03和图4-02所C?br />(2) 把概率最的两个W号l成一个节点，如图4-02中的D和El成节点P1?br />(3) 重复步骤2Q得到节点P2、P3和P4QŞ成一“树”，其中的P4UCؓ根节炏V?br />(4) 从根节点P4开始到相应于每个符L“树叶”，从上C标上??上枝)或者??下枝)Q至于哪
个ؓ?”哪个ؓ?”则无关紧要Q最后的l果仅仅是分配的代码不同Q而代码的q_长度是相同的?br />(5) 从根节点P4开始顺着树枝到每个叶子分别写出每个符L代码Q如?-03所C?br />(6) 按照仙农理论Q这q图像的熵ؓ
H(S)=(15/39)×log2(39/15) + (7/39)×log2(39/7) + ?+ (5/39)×log2(39/5) = 2.1859
压羃?.37:1?br />?-03 霍夫曼编码D?br />?-02 霍夫曼编码方?br />霍夫曼码的码长虽然是可变的，但却不需要另外附加同步代码（前缀代码Q。例如，码串中的W?位ؓ0Q?br />那末肯定是符号AQ因C其他符L代码没有一个是?开始的Q因此下一位就表示下一个符号代码的W?
W号 出现的次?pi) log2(1/pi) 分配的代?需要的位数
A 15(0.3846) 1.38 0 15
B 7(0.1795) 2.48 100 21
C 6(0.1538) 2.70 101 18
D 6(0.1538) 2.70 110 18
E 5(0.1282) 2.96 111 15
W?5 ?br />4
位。同P如果出现?10”，那么它就代表W号D。如果事先编写出一本解释各U代码意义的“词典”，即码
，那么可以根据码一个码一个码Cơ进行译码?br />与仙?范诺~码相同Q这两种Ҏ都自含同步码Q在~码之后的码串中不需要另外添加标记符P卛_
译码时分割符LҎ代码Q?br />采用霍夫曼编码时有两个问题值得注意Q?br />①霍夫曼码没有错误保护功能，在译码时Q如果码串中没有错误Q那么就能一个接一个地正确译出代码?br />但如果码串中有错误，哪怕仅仅是1位出现错误，不但q个码本w译错，更糟p的是一错一大串Q全׃套，
q种现象UCؓ错误传播(error propagation)。计机对这U错误也无能为力Q说不出错在哪里Q更谈不上去
U正它?br />②霍夫曼码是可变长度码，因此很难随意查找或调用压~文件中间的内容Q然后再译码Q这需要在存储
代码之前加以考虑?br />管如此Q霍夫曼码还是得到广泛应用?霍夫曼编码方法的~码效率比仙?范诺~码效率高一些?br />4.2 术~码
术~码在图像数据压~标?如JPEGQJBIG)中扮演了重要的角艌Ӏ在术~码中，消息??之间的实
数进行编码，术~码用到两个基本的参敎ͼW号的概率和~码间隔。信源符L概率军_压羃~码的效率，
也决定编码过E中信源W号的间隔，而这些间隔包含在0?之间。编码过E中的间隔决定了W号压羃后的?br />出。算术编码器的编码过E可用下面的例子加以解释?br />[?.2] 假设信源W号为{00, 01, 10, 11}Q这些符L概率分别为{ 0.1, 0.4, 0.2, 0.3 }Q根据这?br />概率可把间隔[0, 1)分成4个子间隔Q[0, 0.1), [0.1, 0.5), [0.5, 0.7), [0.7, 1)Q其中[x,y)表示半开?br />间隔Q即包含x不包含y。上面的信息可综合在?-04中?br />?-04 信源W号Q概率和初始~码间隔
如果二进制消息序列的输入为：10 00 11 00 10 11 01。编码时首先输入的符h10Q找到它的编码范?br />是[0.5, 0.7)。消息中W二个符?0的编码范围是[0, 0.1)Q因此就取[0.5, 0.7)的第一个十分之一作ؓ新间
隔[0.5, 0.52)。依此类推，~码W?个符?1时取新间隔ؓ[0.514, 0.52)Q编码第4个符?0Ӟ取新间隔?br />[0.514, 0.5146)Q?。消息的~码输出可以是最后一个间隔中的Q意数。整个编码过E如?-03所C?br />W号 00 01 10 11
概率 0.1 0.4 0.2 0.3
初始~码间隔 [0, 0.1) [0.1, 0.5) [0.5, 0.7) [0.7, 1)
W?6 ?br />4
?-03 术~码q程举例
q个例子的编码和译码的全q程分别表示在表4-05和表4-06中?br />Ҏ上面所丄例子Q可把计过Eȝ如下?br />考虑一个有M个符号i=(1,2,?M)的字W表集，假设概率p( i)=piQ?br />。输入符Lxn表示Q第n个子间隔的范围用
表示。其中l0=0Qd0=1和p0=0Qln表示间隔左边界的?rn 表示?br />隔右边界的|dn=rn-ln表示间隔长度。编码步骤如下：
步骤1Q首先在1?之间l每个符号分配一个初始子间隔Q子间隔的长度等于它的概率，初始子间隔的?br />围用I1=[l1Qr1)=[ Q?)表示。od1=r1-l1QL=l1和R=r1?br />步骤2QL和R的二q制表达式分别表CZؓQ?br />?br />其中ui 和vi {于?”或者?”?br />①如果u1
≠v1 Q不发送Q何数据，转到步骤3Q?br />②如果u1=v1Q就发送二q制W号u1?br />比较u2
和v2Q如果u2≠v2 Q不发送Q何数据，转到步骤3Q?br />如果u2=v2Q就发送二q制W号u2?br />?br />q种比较一直进行到两个W号不相同ؓ止，然后q入步骤3?br />步骤3Qn?Q读下一个符受假讄n个输入符号ؓxn= iQ按照以前的步骤把这个间隔分成如下所C的
子间隔：
W?7 ?br />4
令L=lnQR=rn ?dn=rn-lnQ然后{到步??br />?-05 ~码q程
?-06 译码q程
[?.3] 假设?个符L信源Q它们的概率如表4-07所C：
?-07 W号概率
输入序列为xnQ?2Q?1Q?3Q…。它的编码过E如?-04所C，现说明如下?br />输入W?个符hx1= 2Q可知i=2Q定义初始间隔＝[0.5, 0.75)Q由此可?br />d1=0.25Q左双界的二进制数分别表示为：LQ?.5=0.1(B)QRQ?.7Q?.11?(B) 。按照步?Qu1=v1Q发
?。因u2≠v2Q因此{到步??br />输入W?个字Wx2= 1Qi=1Q它的子间隔Q[0.5, 0.625)Q由此可
得d2=0.125。左双界的二进制数分别表示为：LQ?.5=0.100 ?(B)QRQ?.101?(B)。按照步?Q?br />u2=v2=0Q发?Q而u3和v3不相同，因此在发?之后p{到步??br />输入W?个字W，x3= 3Qi=3Q它的子间隔Q[0.59375, 0.609375)
Q由此可得d3=0.015625。左双界的二进制数分别表示为：LQ?.59375=0.10011 (B)QRQ?br />步骤 输入W号 ~码间隔 ~码判决
1 10 [0.5, 0.7) W号的间隔范围[0.5, 0.7)
2 00 [0.5, 0.52) [0.5, 0.7)间隔的第一?/10
3 11 [0.514, 0.52) [0.5, 0.52)间隔的最后三?/10
4 00 [0.514, 0.5146) [0.514, 0.52)间隔的第一?/10
5 10 [0.5143, 0.51442) [0.514, 0.5146)间隔的第五个1/10开始，二个1/10
6 11 [0.514384, 0.51442 [0.5143, 0.51442)间隔的最??/10
7 01 [0.5143836, 0.514402) [0.514384, 0.51442)间隔??/10Q从W??/10开?br />8 从[0.5143876, 0.514402)中选择一个数作ؓ输出Q?.5143876
步骤 间隔 译码W号译码判决
1 [0.5, 0.7) 10 0.51439在间?[0.5, 0.7)
2 [0.5, 0.52) 00 0.51439在间?[0.5, 0.7)的第1?/10
3 [0.514, 0.52) 11 0.51439在间隔[0.5, 0.52)的第7?/10
4 [0.514, 0.5146) 00 0.51439在间隔[0.514, 0.52)的第1?/10
5 [0.5143, 0.51442) 10 0.51439在间隔[0.514, 0.5146)的第5?/10
6 [0.514384, 0.51442) 11 0.51439在间隔[0.5143, 0.51442)的第7?/10
7 [0.51439, 0.5143948) 01 0.51439在间隔[0.51439, 0.5143948)的第1?/10
8 译码的消息：10 00 11 00 10 11 01
信源W号ai 1 2 3 4
概率pi p1=0.5 p2=0.25 p3=0.125 p4=0.125
初始~码间隔 [0, 0.5) [0.5, 0.75) [0.75, 0.875) [0.875, 1)
W?8 ?br />4
0.609375=0.100111 (B)。按照步?Qu3=v3=0Qu4=v4=1Qu5=v5=1Q但u6和v6不相同，因此在发?11之后转到
步骤3?br />?br />发送的W号是：10011…。被~码的最后的W号是结束符受?br />?-04 术~码概念
p个例子而言Q算术编码器接受的第1位是?”，它的间隔范围限制在[0.5, 1)Q但在这个范围里?br />3U可能的码符2Q?3?Q因此第1位没有包含够的译码信息。在接受W?位之后就变成?0”，它落?br />[0.5, 0.75)的间隔里Q由于这两位表示的符号都指向2开始的间隔Q因此就可断定第一个符h2。在接受
每位信息之后的译码情况如下表4-08所C?br />?-08 译码q程?br />在上面的例子中，我们假定~码器和译码器都知道消息的长度，因此译码器的译码q程不会无限制地q行
下去。实际上在译码器中需要添加一个专门的l止W，当译码器看到l止W时停止译码?br />在算术编码中需要注意的几个问题Q?br />(1) ׃实际的计机的精度不可能无限长，q算中出现溢出是一个明昄问题Q但多数机器都有16位?br />32位或?4位的_ֺQ因此这个问题可使用比例~放Ҏ解决?br />(2) 术~码器对整个消息只生一个码字，q个码字是在间隔[0, 1)中的一个实敎ͼ因此译码器在接受
到表C个实数的所有位之前不能q行译码?br />(3) 术~码也是一U对错误很敏感的~码ҎQ如果有一位发生错误就会导致整个消息译错?br />术~码可以是静态的或者自适应的。在静态算术编码中Q信源符L概率是固定的。在自适应术~码
中，信源W号的概率根据编码时W号出现的频J程度动态地q行修改Q在~码期间估算信源W号概率的过E叫
做徏模。需要开发动态算术编码的原因是因Z先知道精的信源概率是很隄Q而且是不切实际的。当压羃
消息Ӟ我们不能期待一个算术编码器获得最大的效率Q所能做的最有效的方法是在编码过E中估算概率。因
此动态徏模就成ؓ定~码器压~效率的关键?br />接受的数?间隔 译码输出
1 [0.5, 1) -
0 [0.5, 0.75) 2
0 [0.5, 0.609375) 1
1 [0.5625, 0.609375) -
1 [0.59375, 0.609375) 3
???br />W?9 ?br />4
4.3 RLE~码
在一q图像中l常包含有许多颜色相同的囑֝。在q些囑֝中，许多行上都具有相同的颜色Q或者在一?br />上有许多q箋的像素都h相同的颜色倹{在q种情况下就不需要存储每一个像素的颜色|而仅仅存储一?br />像素的颜色|以及h相同颜色的像素数目就可以Q或者存储一个像素的颜色|以及h相同颜色值的?br />数。这U压~编码称E编?run length encodingQRLE)Q具有相同颜色ƈ且是q箋的像素数目称E?br />长度?br />假定有一q灰度图像，Wn行的像素值如?-05所C：
?-05 RLE~码的概?br />用RLE~码Ҏ得到的代码ؓQ?0315084180。代码中用黑体表C的数字是行E长度，黑体字后面的数字?br />表像素的颜色倹{例如黑体字50代表有连l?0个像素具有相同的颜色|它的颜色值是8?br />ҎRLE~码前后的代码数可以发现Q在~码前要?3个代码表C一行的数据Q而编码后只要?1个代
码表CZ表原来的73个代码，压羃前后的数据量之比Uؓ7:1Q即压羃比ؓ7:1。这说明RLE实是一U压~技
术，而且q种~码技术相当直观，也非常经?br />译码时按照与~码旉用的相同规则q行Q还原后得到的数据与压羃前的数据完全相同?br />RLE所能获得的压羃比有多大Q这主要是取决于囑փ本n的特炏V如果图像中h相同颜色的图像块?br />大，囑փ块数目越，获得的压~比p高。反之，压羃比就小?br />RLE压羃~码其适用于计机生成的图像，对减图像文件的存储I间非常有效。然而，RLE寚w色丰?br />的自然图像就昑־力不从心Q在同一行上h相同颜色的连l像素往往很少Q而连l几行都h相同颜色值的
q箋行数更。如果仍然用RLE~码ҎQ不仅不能压~图像数据，反而可能原来的图像数据变得更
大。请注意Q这q不是说RLE~码Ҏ不适用于自然图像的压羃Q相反，在自然图像的压羃中还真少不了RLEQ?br />只不q是不能单纯使用RLE一U编码方法，需要和其他的压~编码技术联合应用?br />4.4 词典~码
有许多场合，开始时不知道要~码数据的统计特性，也不一定允怽事先知道它们的统计特性。因此，?br />们提Z许许多多的数据压~方法，可能获得最大的压羃比。这些技术统UCؓ通用~码技术。词典编?br />(Dictionary Encoding)技术就属于q一cR?br />4.4.1 词典~码的思想
词典~码(dictionary encoding)的根据是数据本n包含有重复代码这个特性。例如文本文件和光栅囑փ
具有这U特性。词典编码法的种cd多，归纳h大致有两cR?br />W一c词典算法是企图查找正在压羃的字W序列是否在以前输入的数据中出现q，然后输出仅仅是指向早
期出现过的字W串的“指针”。这U编码概念如?-06所C?br />W?10 ?br />4
?-06 W一c词典法~码概念
q里所指的“词典”是指用以前处理q的数据来表C编码过E中遇到的重复部分。这cȝ码算法都是以
Abraham Lempel和Jakob Ziv?977q开发和发表的称为LZ77法为基的，例如1982q由Storer和Szymanski
改进的称为LZSS法 ?br />W二c词典算法是企图从输入的数据中创Z个“短语词?dictionary of the phrases)”，q种短语
不一定是h具体含义的短语，可以是Q意字W的l合。编码过E中遇到已经在词怸出现的“短语”时Q编
码器p个词怸的短语的“烦引号”，而不是短语本w。这个概念如?-07所C?br />?-07 W二c词典法~码概念
J.Ziv和A.Lempel?978q首ơ发表了介绍q种~码Ҏ的文章。在他们的研I基上，Terry A.Weltch
?984q发表了改进q种~码法的文章，因此把这U编码方法称为LZW(Lempel-Ziv Walch)压羃~码Q在?br />速硬盘控制器?首先应用了这U算法?br />4.4.2 LZ77法
Z更好地说明LZ77法的原理，首先介绍法中用到的几个术语Q?br />(1) 输入数据?input stream)Q要被压~的字符序列?br />(2) 字符(character)Q输入数据流中的基本单元?br />(3) ~码位置(coding position)Q输入数据流中当前要~码的字W位|，指前向缓冲存储器中的开始字
W?br />(4) 前向~冲存储?Lookahead buffer)Q存放从~码位置到输入数据流l束的字W序列的存储器?br />(5) H口(window)Q指包含W个字W的H口Q字W是从编码位|开始向后数Q也是最后处理的W个字
W??滑动H口)
(6) 指针(pointer)Q指向窗口中的匹配串的开始位|且含长度的指针?br />LZ77~码法的核心是查找从前向缓冲存储器开始的与窗口中最长的匚w丌Ӏ编码算法的具体执行步骤?br />下：
W?11 ?br />4
(1) 把编码位|设|到输入数据的开始位|?br />(2) 查找H口中最长的匚w丌Ӏ?br />(3) 以?Pointer, Length) Character”三元组的格式输出，其中Pointer是指向窗口中匚w串的指针Q?br />Length表示匚w字符的长度，Characters是前向缓冲存储器中的不匹配的W?个字W。没有匹配的字符串时Q?br />输出?0, 0) Character?br />(4) 如果前向~冲存储器不是空的，则把~码位置和窗口向前移(Length+1)个字W，然后q回到步??br />[?.4] 待编码的数据如?-09所C，~码q程如表4-10所C。现作如下说明：
(1) “步骤”栏表示~码步骤?br />(2) “位|”栏表示~码位置Q输入数据流中的W?个字Wؓ~码位置1?br />(3) “匹配串”栏表示H口中找到的最长的匚w丌Ӏ?br />(4) “字W”栏表示匚w之后在前向缓冲存储器中的W?个字W?br />(5) “输出”栏以?Back_chars, Chars_length) Explicit_character”格式输出。其中，
(Back_chars, Chars_length)是指向匹配串的指针，告诉译码器“在q个H口中向后退Back_chars个字W然?br />拯Chars_length个字W到输出”，Explicit_character是真实字W。例如，?-10中的输出?5,2) C”告
诉译码器回退5个字W，然后拯2个字W“AB?br />?-09待编码的数据?br />?-10 ~码q程
4.4.3 LZSS法
LZ77通过输出真实字符解决了在H口中出现没有匹配串的问题，但这个解x案包含有冗余信息。冗余信
息表现在两个斚wQ一是空指针Q二是编码器输出的字W可能包含在下一个匹配串中的字符?br />LZSS法以比较有效的Ҏ解决q个问题Q思想是如果匹配串的长度比指针本n的长?Q最匹配串?br />度）长就输出指针Q否则就输出真实字符。由于输出的压羃数据中包含有指针和字符本nQؓ了区分它们就
需要有额外的标志位Q即ID位?br />LZSS~码法的具体执行步骤如下：
(1) 把编码位|置于输入数据流的开始位|?br />(2) 在前向缓冲存储器中查找与H口中最长的匚w?br />?Pointer Q?匚w串指针?br />?Length Q?匚w串长度?br />(3) 判断匚w串长度Length是否大于{于最匹配串长度(Length≥MIN_LENGTH)Q?br />如果“是”：输出指针Q然后把~码位置向前UdLength个字W?br />位置 1 2 3 4 5 6 7 8 9
字符 A A B C B B A B C
步骤 位置 匚w?字符 输出
1 1 -- A (0,0) A
2 2 A B (1,1) B
3 4 -- C (0,0) C
4 5 B B (2,1) B
5 7 A B C (5,2) C
W?12 ?br />4
如果“否”：输出前向~冲存储器中的第1个字W，然后把编码位|向前移动一个字W?br />(4) 如果前向~冲存储器不是空的，p回到步骤2?br />[?.5] ~码字符串如?-11所C，~码q程如表4-12所C。现说明如下Q?br />(1) “步骤”栏表示~码步骤?br />(2) “位|”栏表示~码位置Q输入数据流中的W?个字Wؓ~码位置1?br />(3) “匹配”栏表示H口中找到的最长的匚w丌Ӏ?br />(4) “字W”栏表示匚w之后在前向缓冲存储器中的W?个字W?br />(5) “输出”栏的输ZؓQ?br />?如果匚w串本w的长度Length≥MIN_LENGTHQ输出指向匹配串的指针，格式?Back_chars,
Chars_length)。该指针告诉译码器“在q个H口中向后退Back_chars个字W然后拷贝Chars_length个字W到
输出”?br />?如果匚w串本w的长度Length≤MIN_LENGTHQ则输出真实的匹配串?br />?-11 输入数据?br />?-12 ~码q程(MIN_LENGTH = 2)
在相同的计算环境下，LZSS法比LZ77可获得比较高的压~比Q而译码同L单。这也就是ؓ什么这U算
法成为开发新法的基Q许多后来开发的文档压羃E序都用了LZSS的思想。例如，PKZip, ARJ, LHArc?br />ZOO{等Q其差别仅仅是指针的长短和窗口的大小{有所不同?br />LZSS同样可以和熵~码联合使用Q例如ARJ׃霍夫曼编码联用，而PKZip则与Shannon-Fano联用Q它的后
l版本也采用霍夫曼编码?br />4.4.4 LZ78法
在介lLZ78法之前Q首先说明在法中用到的几个术语和符P
(1) 字符?Charstream)Q要被编码的数据序列?br />(2) 字符(Character)Q字W流中的基本数据单元?br />(3) 前缀(Prefix)Q?在一个字W之前的字符序列?br />(4) ~-W串(String)Q前~Q字W?br />(5) 码字(Code word)Q码字流中的基本数据单元Q代表词怸的一串字W?br />(6) 码字?Codestream)Q?码字和字W组成的序列Q是~码器的输出?br />(7) 词典(Dictionary)Q?~-W串表。按照词怸的烦引号Ҏ条缀-W串(String)指定一个码?Code
位置 1 2 3 4 5 6 7 8 9 10 11
字符 A A B B C B B A A B C
步骤 位置 匚w?输出
1 1 -- A
2 2 A A
3 3 -- B
4 4 B B
5 5 -- C
6 6 B B (3,2)
7 8 A A B (7,3)
8 11 C C
W?13 ?br />4
word)?br />(8) 当前前缀(Current prefix)Q在~码法中用，指当前正在处理的前缀Q用W号P表示?br />(9) 当前字符(Current character)Q在~码法中用，指当前前~之后的字W，用符号C表示?br />(10) 当前码字(Current code word)Q?在译码算法中使用Q指当前处理的码字，用W表示当前码字Q?br />String.W表示当前码字的缀-W串?br />1. ~码法
LZ78的编码思想是不断地从字W流中提取新的缀-W串(String)Q通俗地理解ؓ新“词条”，然后用“代
号”也是码字(Code word)表示q个“词条”。这样一来，对字W流的编码就变成了用码字(Code word)L
换字W流(Charstream)Q生成码字流(Codestream)Q从而达到压~数据的目的?br />在编码开始时词典是空的，不包含Q何缀-W串(string)。在q种情况下编码器pZ个表C空字符?br />的特D码?例如??和字W流?Charstream)的第一个字WCQƈ把这个字WCd到词怸作ؓ一个由一
个字W组成的~-W串(string)。在~码q程中，如果出现cM的情况，也照此办理?br />在词怸已经包含某些~-W串(String)之后Q如果“当前前~P +当前字符C”已l在词典中，q字符C
来扩展这个前~Q这L扩展操作一直重复到获得一个在词典中没有的~-W串(String)为止。此时就输出?br />C当前前~P的码?Code word)和字WCQƈ把P+Cd到词怸Q然后开始处理字W流(Charstream)中的下一
个前~?br />LZ78~码器的输出是码?字符(W,C)对，每次输出一对到码字中Qƈ用字WC扩展与码字W相对应的~-
W串(String)Q生成新的缀-W串(String)Q然后添加到词典中?br />LZ78~码的具体算法如下：
步骤1Q?在开始时Q词典和当前前缀P都是I的?br />步骤2Q?当前字符C Q? 字符中的下一个字W?br />步骤3Q?判断P+C是否在词怸Q?br />(1) 如果“是”：用C扩展PQ让P Q? P+C Q?br />(2) 如果“否”：
?输出与当前前~P相对应的码字和当前字WCQ?br />?把字W串P+C d到词怸?br />?令P Q? I倹{?br />(3) 判断字符中是否q有字符需要编?br />?如果“是”：q回到步??br />?如果“否”：若当前前~P不是I的Q输出相应于当前前缀P的码字，然后l束~码?br />2. 译码法
在译码开始时译码词典是空的，它将在译码过E中从码字流中重构。每当从码字中d一对码?字符
(W,C)ҎQ码字就参考已l在词典中的~-W串Q然后把当前码字的缀-W串string.W 和字WC输出到字W流
(Charstream)Q而把当前~-W串(string.W+C)d到词怸。在译码l束之后Q重构的词典与编码时生成?br />词典完全相同?br />LZ78译码的具体算法如下：
步骤1Q?在开始时词典是空的?br />步骤2Q?当前码字W Q? 码字中的下一个码字?br />步骤3Q?当前字符C Q? 紧随码字之后的字W?br />步骤4Q?把当前码字的~-W串(string.W)输出到字W流(Charstream)Q然后输出字WC?br />步骤5Q?把string.W+Cd到词怸?br />步骤6Q?判断码字中是否q有码字要译
W?14 ?br />4
(1) 如果“是”，p回到步骤2?br />(2) 如果“否”，则结束?br />[?.6] ~码字符串如?-13所C，~码q程如表4-14所C。现说明如下Q?br />(1) “步骤”栏表示~码步骤?br />(2) “位|”栏表示在输入数据中的当前位|?br />(3) “词典”栏表示d到词怸的缀-W串Q缀-W串的烦引等于“步骤”序受?br />(4) “输出”栏?当前码字W, 当前字符C)化ؓ(W, C)的Ş式输出?br />?-13 ~码字符?br />?-14 ~码q程
与LZ77相比QLZ78的最大优Ҏ在每个编码步骤中减少了缀-W串(String)比较的数目，而压~率与LZ77
cM?br />4.4.5 LZW法
在LZW法中用的术语与LZ78使用的相同，仅增加了一个术语—前~?Root)Q它是由单个字符l成?br />~-W串(String)。在~码原理上，LZW与LZ78相比有如下差别：
?LZW只输Z表词怸的缀-W串(String)的码?code word)。这意呛_开始时词典不能是空的，?br />必须包含可能在字W流出现中的所有单个字W，卛_~?Root)?br />?׃所有可能出现的单个字符都事先包含在词典中，每次~码开始时都用一个字W前~(onecharacter
prefix)Q因此在词典中增加的W?个缀-W串有两个字W?br />现将LZW~码法和译码算法介l如下?br />1. ~码法
LZW~码是围l称典的转换表来完成的。这张{换表存放UCؓ前缀(Prefix)的字W序列，qؓ每个?br />分配一个码?Code word)Q或者叫做序P如表4-15所C。这张{换表实际上是?位ASCII字符集进行扩
充，增加的符L来表C在文本或图像中出现的可变长度ASCII字符丌Ӏ扩充后的代码可?位?0位?1位?br />12位甚x多的位来表示。Welch的论文中用了12位，12位可以有4096个不同的12位代码，q就是说Q{换表
?096个表，其中256个表用来存攑ַ定义的字W，剩下3840个表用来存攑։~(Prefix)?br />?-15 词典
位置 1 2 3 4 5 6 7 8 9
字符 A B B C B C A B A
步骤 位置 词典 输出
1 1 A (0,A)
2 2 B (0,B)
3 3 B C (2,C)
4 5 B C A (3,A)
5 8 B A (2,A)
码字(Code word) 前缀(Prefix)
1
W?15 ?br />4
LZW~码?软g~码器或g~码?是通过理q个词典完成输入与输Z间的转换。LZW~码器的?br />入是字符?Charstream)Q字W流可以是用8位ASCII字符l成的字W串Q而输出是用n?例如12?表示的码
字流(Codestream)Q码字代表单个字W或多个字符l成的字W串?br />LZW~码器用了一U很实用的分?parsing)法Q称婪分析算?greedy parsing algorithm)。在
贪婪分析法中，每一ơ分析都要串行地查来自字W流(Charstream)的字W串Q从中分解出已经识别的最?br />的字W串Q也是已经在词怸出现的最长的前缀(Prefix)。用已知的前~(Prefix)加上下一个输入字WC?br />是当前字符(Current character)作ؓ该前~的扩展字W，形成新的扩展字符东y—缀-W串(String)Q?br />Prefix+C。这个新的缀-W串(String)是否要加到词怸Q还要看词典中是否存有和它相同的~-W串String?br />如果有，那么q个~-W串(String)变成前~(Prefix)Ql输入新的字W，否则把q个~-W串(String)
写到词典中生成一个新的前~(Prefix)Qƈ分配l一个代码?br />LZW~码法的具体执行步骤如下：
步骤1Q?开始时的词典包含所有可能的?Root)Q而当前前~P是空的；
步骤2Q?当前字符(C) Q?字符中的下一个字W；
步骤3Q?判断~-W串P+C是否在词怸
(1) 如果“是”：P Q? P+C // (用C扩展P) Q?br />(2) 如果“否?br />?把代表当前前~P的码字输出到码字?
?把缀-W串P+Cd到词?
?令P Q? C //(现在的P仅包含一个字WC);
步骤4Q?判断码字中是否q有码字要译
(1) 如果“是”，p回到步骤2Q?br />(2) 如果“否?br />?把代表当前前~P的码字输出到码字?
?l束?br />LZW~码法可用伪码表示。开始时假设~码词典包含若干个已l定义的单个码字。例如，256个字W的?br />字，用伪码可以表C成Q?br />??br />193 A
194 B
??br />255
??br />1305 abcdefxyF01234
??br />Dictionary[j] ?all n single-characterQ?jQ?, 2Q?…，n
j ?n+1
Prefix ?read first Character in Charstream
while((C ?next Character)!=NULL)
W?16 ?br />4
2. 译码法
LZW译码法中还用到另外两个术语Q?br />?当前码字(Current code word)Q指当前正在处理的码字，用cW表示Q用string.cW表示当前~-W串Q?br />?先前码字(Previous code word)Q指先于当前码字的码字，用pW表示Q用string.pW表示先前~-W?br />丌Ӏ?br />LZW译码法开始时Q译码词怸~码词典相同Q它包含所有可能的前缀?roots)。LZW法在译码过E?br />中会C先前码字(pW)Q从码字中d前码?cW)之后输出当前~-W串string.cWQ然后把用string.cW?br />W一个字W扩展的先前~-W串string.pWd到词怸?br />LZW译码法的具体执行步骤如下：
步骤1Q?在开始译码时词典包含所有可能的前缀?Root)?br />步骤2Q?cW Q? 码字中的第一个码字?br />步骤3Q?输出当前~-W串string.cW到码字流?br />步骤4Q?先前码字pW Q? 当前码字cW?br />步骤5Q?当前码字cW Q? 码字中的下一个码字?br />步骤6Q?判断当前~-W串string.cW是否在词怸
(1) 如果“是”，则：
?把当前缀-W串string.cW输出到字W流?br />?把先前缀-W串string.pW + 当前前缀-W串string.cW的第一个字WCd到词典?br />(2) 如果“否”，则：
?输出先前~-W串string.pW + 先前~-W串string.pW的第一个字W到字符，
?把它d到词怸?br />步骤7Q?判断码字中是否q有码字要译
(1) 如果“是”，p回到步骤4?br />(2) 如果“否? l束?br />LZW译码法可用伪码表示如下Q?br />Codestream ?cW for Prefix
Dictionary[j] ?all n single-characterQ?jQ?, 2Q?…，n
j ?n+1
cW ?first code from Codestream
Charstream ?Dictionary[cW]
pW ?cW
While((cW ?next Code word)!=NULL)
W?17 ?br />4
[?.7] ~码字符串如?-16所C，~码q程如表4-17所C。现说明如下Q?br />(1) “步骤”栏表示~码步骤Q?br />(2) “位|”栏表示在输入数据中的当前位|；
(3) “词典”栏表示d到词怸的缀-W串Q它的烦引在括号中；
(4) “输出”栏表示码字输出?br />?-16 被编码的字符?br />?-17 LZW的编码过E?br />?-18解释了译码过E。每个译码步骤译码器M个码字，输出相应的缀-W串Qƈ把它d到词怸?br />例如Q在步骤4中，先前码字(2)存储在先前码?pW)中，当前码字(cW)?4)Q当前缀-W串string.cW是输?br />(“A B?Q先前缀-W串string.pW ("B")是用当前~-W串string.cW ("A")的第一个字W，其结?"B A")
d到词怸Q它的烦引号?6)
?-18 LZW的译码过E?br />LZW法得到普遍采用Q它的速度比用LZ77法的速度快，因ؓ它不需要执行那么多的缀-W串比较?br />位置 1 2 3 4 5 6 7 8 9
字符 A B B A B A B A C
步骤 位置 词典 输出
(1) A
(2) B
(3) C
1 1 (4) A B (1)
2 2 (5) B B (2)
3 3 (6) B A (2)
4 4 (7) A B A (4)
5 6 (8) A B A C (7)
6 -- -- -- (3)
步骤 代码 词典 输出
(1) A
(2) B
(3) C
1 (1) -- -- A
2 (2) (4) A B B
3 (2) (5) B B B
4 (4) (6) B A A B
5 (7) (7) A B A A B A
6 (3) (8) A B A C C
W?18 ?br />4
作。对LZW法q一步的改进是增加可变的码字长度Q以及在词典中删除老的~-W串。在GIF囑փ格式和UNIX
的压~程序中已经采用了这些改q措施之后的LZW法?br />LZW法取得了专利，专利权的所有者是国的一个大型计机公司—Unisys(优利pȝ公司)Q除了商?br />软g生公司之外Q可以免费用LZW法?/font>
]]>
最 直观的搜索方式是序搜烦Q以待压~部分的W一个字节与H口中的每一个字节依ơ比较，当找C个相{的字节Ӟ再比较后l的字节…?遍历了窗口后得出最长匹配?span lang="EN-US">gzip 用的是被UC?/span>哈希?span lang="EN-US">?/span>的方法来实现较高效的搜烦?span lang="EN-US">?/span>哈希Q?span lang="EN-US">hashQ?span lang="EN-US">?/span>是分散的意思，把待搜烦的数据按照字节值分散到一个个?/span>?span lang="EN-US">?/span>中，搜烦时再Ҏ字节 值到相应?span lang="EN-US">?/span>?span lang="EN-US">?/span>中去L。短语式压羃的最短匹配ؓ 3 个字节，gzip ?span lang="EN-US"> 3 个字节的g为哈希表的烦引，?span lang="EN-US"> 3 个字节共?span lang="EN-US"> 2 ?span lang="EN-US"> 24 ơ方U取|需?span lang="EN-US"> 16M 个桶Q桶里存攄是窗口中的位||H口的大ؓ 32KQ所以每个桶臛_要有大于两个字节的空_哈希表将大于 32MQ作?span lang="EN-US"> 90 q代开发的E序Q这个要求是太大了，而且随着H口的移动，哈希表里的数据会不断q时Q维护这么大的表Q会降低E序的效率，gzip 定义哈希表ؓ 2 ?span lang="EN-US"> 15 ơ方Q?span lang="EN-US">32KQ个Ӟq设计了一个哈希函数把 16M U取值对应到 32K 个桶中，不同的D对应到相同的桶中是不可避免的Q哈希函数的d?span lang="EN-US">
1.使各U取值尽可能均匀地分布到各个桶中Q避免许多不同的值集中到某些桶中Q而另一些是I桶Q搜烦的效率降低?/span>
2.函数的计尽可能地简单，因ؓ每次 ?/span>插入?/span>?span lang="EN-US">?/span>搜寻?/span>哈希表都要执行哈希函敎ͼ哈希函数的复杂度直接影响E序的执行效率，Ҏ惛_的哈希函数是?span lang="EN-US"> 3 个字节的左边Q或双Q?span lang="EN-US">15 位二q制|但这样只要左边（或右边）2 个字节相同，׃被放到同一个桶中，?span lang="EN-US"> 2 个字节相同的概率是比较高的，不符?span lang="EN-US">?/span>q_分布?/span>的要求?span lang="EN-US">
gzip 采用的算法是Q?span lang="EN-US">A(4,5) + A(6,7,8) ^ B(1,2,3) + B(4,5) + B(6,7,8) ^ C(1,2,3) +
C(4,5,6,7,8) Q说明：A ?span lang="EN-US"> 3 个字节中的第 1 个字节，B 指第 2 个字节，C 指第 3 个字节，A(4,5) 指第一个字节的W?span lang="EN-US"> 4,5 位二q制码，“^?/span>是二q制位的异或操作Q?span lang="EN-US">??/span>?span lang="EN-US">?/span>q接?/span>而不?span lang="EN-US">?/span>?span lang="EN-US">?/span>Q?span lang="EN-US">“^?/span>优先?span lang="EN-US">??/span>Q这样 3 个字节都量?/span>参与?/span>到最后的l果中来Q而且每个l果?span lang="EN-US"> h 都等?span lang="EN-US"> ((?span lang="EN-US">1?span lang="EN-US">h << 5)
^ c)取右 15 位，计算也还单?span lang="EN-US">
哈希表的具体实现也值得探讨,因ؓ无法预先知道每一?span lang="EN-US">?/span>?span lang="EN-US">?/span>会存攑֤个元素Q所以最单的Q会惛_用链表来实现Q哈希表里存攄每个桶的W一?元素Q每个元素除了存攄自n的|q存攄一个指针，指向同一个桶中的下一个元素，可以着指针链来遍历该桶中的每一个元素，插入元素Ӟ先用哈希函数
出该放到第几个桶中Q再把它挂到相应链表的最后?/span>
q个Ҏ的缺Ҏ频繁地申请和释放内存会降低运行速度Q内存指针的存放占据了额外的内存开销?/span>
有更内 存开销和更快速的Ҏ来实现哈希表Qƈ且不需要频J的内存甌和释放：gzip 在内存中甌了两个数l，一个叫 head[]Q一个叫 pre[]Q大都?span lang="EN-US"> 32KQ根据当前位|?span lang="EN-US"> strstart 开始的 3 个字节，用哈希函数计出?span lang="EN-US"> head[] 中的位置 ins_hQ然后把 head[ins_h] 中的D?span lang="EN-US"> pre[strstart]Q再把当前位|?span lang="EN-US"> strstart 记入 head[ins_h]?/span>
随着压羃的进行，head[]里记载着最q的可能的匹配的位置Q如果有匚w的话Q?span lang="EN-US">head[ins_h]不ؓ 0Q，pre[]中的所有位|与原始数据的位|相对应Q但每一个位|保存的值是前一个最q的可能的匹配的位置?/span>
Q?span lang="EN-US">?/span>可能的匹?span lang="EN-US">?/span>是指哈希函数计算出的
ins_h 相同。）着 pre[] 中的指示找下去，直到遇到
0Q可以得到所有匹配在原始数据中的位置Q?span lang="EN-US">0 表示不再有更q的匚w?span lang="EN-US">
接下来很自然地要观察 gzip 具体是如何判断哈希表中数据的q时Q如何清理哈希表的，因ؓ pre[] 里只能存?span lang="EN-US"> 32K 个元素，所以这工作是必须要做的?span lang="EN-US">
gzip 从原始文件中d两个H口大小的内容（?span lang="EN-US"> 64K
字节Q到一块内存中Q这块内存也是一个数l，UC Window[]Q申?span lang="EN-US"> head[]?span lang="EN-US">pre[] q清Ӟstrstart
|ؓ 0?/span>
然后 gzip Ҏ索边插入Q搜索时通过计算 ins_hQ检?span lang="EN-US"> head[] 中是否有匚wQ如果有匚wQ判?span lang="EN-US"> strstart ?span lang="EN-US"> head[] 中的位置是否大于 1 个窗口的大小Q如果大?span lang="EN-US"> 1 个窗口的大小Q就不到 pre[] 中去搜烦了，因ؓ pre[] 中保存的位置更远了，如果不大于，顺着 pre[] 的指C到 Window[] 中逐个匚w位置开始，逐个字节与当前位|的数据比较Q以扑և最长匹配，pre[] 中的位置也要判断是否出一个窗口，如遇到超Z个窗口的位置或?span lang="EN-US"> 0 ׃再找下去Q找不到匚wp出当前位|的单个字节到另外的内存Q输出方法在后文中会介绍Q，q把 strstart 插入哈希表，strstart 递增Q如果找C匚wQ就输出匚w位置和匹配长度这两个数字到另外的内存中，q把 strstart 开始的Q直?span lang="EN-US"> strstart + 匚w长度 为止的所有位|都插入哈希表，strstart += 匚w长度。插入哈希表的方法ؓQ?span lang="EN-US">
pre[strstart % 32K] = head[ins_h];
head[ins_h] = strstart;
?以看出，pre[] 是@环利用的Q所有的位置都在一个窗口以内，但每一个位|保存的g一定是一个窗口以内的?/span>
在搜索时Q?span lang="EN-US">head[] ?span lang="EN-US"> pre[] 中的位置值对应到 pre[] 时也?span lang="EN-US"> % 32K。当
Window[] 中的原始数据要处理完毕Ӟ要把 Window[] 中后一H的数据复制到前一H，再读?span lang="EN-US"> 32K 字节的数据到后一H，strstart -= 32KQ遍?span lang="EN-US"> head[]Q值小于等?span lang="EN-US"> 32K 的，|ؓ 0Q大?span lang="EN-US"> 32K 的，-= 32KQ?span lang="EN-US">pre[] ?span lang="EN-US"> head[] 一样处理。然后同前面一样处理新一H的数据?span lang="EN-US">
分析Q现在可?看到Q虽?span lang="EN-US"> 3 个字节有 16M U取|但实际上一个窗口只?span lang="EN-US"> 32K 个取值需要插入哈希表Q由于短语式重复的存在，实际只有 < 32K U取值插入哈希表?span lang="EN-US"> 32K ?span lang="EN-US">?/span>?span lang="EN-US">?/span>中，而且哈希函数又符?span lang="EN-US">?/span>q_分布?/span>的要求，所以哈希表中实际存在的?/span>冲突?/span>一般不会多Q对搜烦效率的媄响不大。可以预计，?span lang="EN-US">?/span>一般情?span lang="EN-US">?/span>下，??span lang="EN-US">?/span>?span lang="EN-US">?/span>中存攄数据Q正是我们要扄?/span>
哈希表在各种搜烦法中，实现相对的比较简单，Ҏ理解Q?span lang="EN-US">?/span>q_搜烦速度?/span>最快，哈希函数的设计是搜烦速度的关 键，只要W合?/span>q_分布?/span>?span lang="EN-US">?/span>计算?span lang="EN-US">?/span>Q就常常能成U搜索算法中的首选，所以哈希表是最行的一U搜索算法?/span>
但在某些Ҏ情况下，它也有缺点，? 如：1.当键?span lang="EN-US"> k 不存在时Q要求找出小?span lang="EN-US"> k 的最大键码或大于 k 的最键码，哈希表无法有效率地满U要求?span lang="EN-US">2.哈希表的?/span>q_搜烦速度?/span>是徏立在概率论的基础上的Q因Z先不能预知待搜烦的数据集合，我们只能?/span>??span lang="EN-US">?/span>搜烦速度?span lang="EN-US">?/span>q_?span lang="EN-US">?/span>Q而不?span lang="EN-US">?/span>保证?/span>搜烦速度?span lang="EN-US">?/span>上限?/span>。在同hcL命攸关的应用中Q如ȝ或宇航领域）Q将是不合适的?/span>
q些情况及其他一些特D情
况下Q我们必L助其?span lang="EN-US">?/span>q_速度?/span>较低Q但能满相应的Ҏ要求的算法。（见《计机E序设计艺术》第3?排序与查找）。幸?span lang="EN-US">?/span>在窗口中搜烦匚w字节?span lang="EN-US">?/span>不属于特D情c?span lang="EN-US">
旉与压~率的^衡：
gzip 定义了几U可供选择?span lang="EN-US"> levelQ越低的 level
压羃旉快但压~率低Q越高的 level 压羃旉慢但压~率高?span lang="EN-US">
不同?span lang="EN-US"> level 对下面四个变量有不同的取|
nice_length
max_chain
max_lazy
good_length
nice_lengthQ?前面说过Q搜索匹配时Q顺着 pre[] 的指C到 Window[] 中逐个匚w位置开始，扑և最长匹配，但在q过E中Q如果遇C个匹配的长度辑ֈ或超q?span lang="EN-US"> nice_lengthQ就不再试图L更长的匹配。最低的 level 定义 nice_length ?span lang="EN-US"> 8Q最高的
level 定义 nice_length ?span lang="EN-US"> 258Q即一个字节能表示的最大短语匹配长?span lang="EN-US"> 3 + 255Q?span lang="EN-US">
max_chainQ这个D定了着 pre[] 的指C往前回溯的最大次数。最低的 level 定义 max_chain ?span lang="EN-US"> 4Q最高的 level 定义
max_chain ?span lang="EN-US"> 4096。当 max_chain ?span lang="EN-US"> nice_length 有冲H时Q以先达到的为准?/span>
]]>
(1) lzw.h 定义了一些基本的数据l构Q常量，q有变量的初始化{?br />
#ifndef __LZW_H__
#define __LZW_H__
//------------------------------------------------------------------------------
#include <stdio.h>
#include <stdlib.h>
#include <windows.h>
#include <memory.h>
//------------------------------------------------------------------------------
#define LZW_BASE 0x102// The code base
#define CODE_LEN 12 // Max code length
#define TABLE_LEN 4099 // It must be prime number and bigger than 2^CODE_LEN=4096.
// Such as 5051 is also ok.
#define BUFFERSIZE 1024
//------------------------------------------------------------------------------
typedef struct
{
HANDLE h_sour; // Source file handle.
HANDLE h_dest; // Destination file handle.
HANDLE h_suffix; // Suffix table handle.
HANDLE h_prefix; // Prefix table handle.
HANDLE h_code; // Code table handle.
LPWORD lp_prefix; // Prefix table head pointer.
LPBYTE lp_suffix; // Suffix table head pointer.
LPWORD lp_code; // Code table head pointer.
WORD code;
WORD prefix;
BYTE suffix;
BYTE cur_code_len; // Current code length.[ used in Dynamic-Code-Length mode ]
}LZW_DATA,*PLZW_DATA;
typedef struct
{
WORD top;
WORD index;
LPBYTE lp_buffer;
HANDLE h_buffer;
BYTE by_left;
DWORD dw_buffer;
BOOL end_flag;
}BUFFER_DATA,*PBUFFER_DATA;
typedef struct //Stack used in decode
{
WORD index;
HANDLE h_stack;
LPBYTE lp_stack;
}STACK_DATA,*PSTACK_DATA;
//------------------------------------------------------------------------------
VOID stack_create( PSTACK_DATA stack )
{
stack->h_stack = GlobalAlloc( GHND , TABLE_LEN*sizeof(BYTE) );
stack->lp_stack = GlobalLock( stack->h_stack );
stack->index = 0;
}
//------------------------------------------------------------------------------
VOID stack_destory( PSTACK_DATA stack )
{
GlobalUnlock( stack->h_stack );
GlobalFree ( stack->h_stack );
}
//------------------------------------------------------------------------------
VOID buffer_create( PBUFFER_DATA buffer )
{
buffer->h_buffer = GlobalAlloc( GHND, BUFFERSIZE*sizeof(BYTE) );
buffer->lp_buffer = GlobalLock( buffer->h_buffer );
buffer->top = 0;
buffer->index = 0;
buffer->by_left = 0;
buffer->dw_buffer = 0;
buffer->end_flag = FALSE;
}
//------------------------------------------------------------------------------
VOID buffer_destory( PBUFFER_DATA buffer )
{
GlobalUnlock( buffer->h_buffer );
GlobalFree ( buffer->h_buffer );
}
//------------------------------------------------------------------------------
VOID re_init_lzw( PLZW_DATA lzw ) //When code table reached its top it should
{ //be reinitialized.
memset( lzw->lp_code, 0xFFFF, TABLE_LEN*sizeof(WORD) );
lzw->code = LZW_BASE;
lzw->cur_code_len = 9;
}
//------------------------------------------------------------------------------
VOID lzw_create(PLZW_DATA lzw, HANDLE h_sour, HANDLE h_dest)
{
WORD i;
lzw->h_code = GlobalAlloc( GHND, TABLE_LEN*sizeof(WORD) );
lzw->h_prefix = GlobalAlloc( GHND, TABLE_LEN*sizeof(WORD) );
lzw->h_suffix = GlobalAlloc( GHND, TABLE_LEN*sizeof(BYTE) );
lzw->lp_code = GlobalLock( lzw->h_code );
lzw->lp_prefix = GlobalLock( lzw->h_prefix );
lzw->lp_suffix = GlobalLock( lzw->h_suffix );
lzw->code = LZW_BASE;
lzw->cur_code_len = 9;
lzw->h_sour = h_sour;
lzw->h_dest = h_dest;
memset( lzw->lp_code, 0xFFFF, TABLE_LEN*sizeof(WORD) );
}
//------------------------------------------------------------------------------
VOID lzw_destory(PLZW_DATA lzw)
{
GlobalUnlock( lzw->h_code );
GlobalUnlock( lzw->h_prefix );
GlobalUnlock( lzw->h_suffix );
GlobalFree( lzw->h_code );
GlobalFree( lzw->h_prefix );
GlobalFree( lzw->h_suffix );
}
//------------------------------------------------------------------------------
#endif
(2) fileio.h 定义了一些文件操?br />
#ifndef __FILEIO_H__
#define __FILEIO_H__
//------------------------------------------------------------------------------
#include <stdio.h>
#include <stdlib.h>
#include <windows.h>
//------------------------------------------------------------------------------
HANDLE file_handle(CHAR* file_name)
{
HANDLE h_file;
h_file = CreateFile(file_name,
GENERIC_READ|GENERIC_WRITE,
FILE_SHARE_READ|FILE_SHARE_WRITE,
NULL,
OPEN_ALWAYS,
0,
NULL
);
return h_file;
}
//------------------------------------------------------------------------------
WORD load_buffer(HANDLE h_sour, PBUFFER_DATA buffer) // Load file to buffer
{
DWORD ret;
ReadFile(h_sour,buffer->lp_buffer,BUFFERSIZE,&ret,NULL);
buffer->index = 0;
buffer->top = (WORD)ret;
return (WORD)ret;
}
//------------------------------------------------------------------------------
WORD empty_buffer( PLZW_DATA lzw, PBUFFER_DATA buffer)// Output buffer to file
{
DWORD ret;
if(buffer->end_flag) // The flag mark the end of decode
{
if( buffer->by_left )
{
buffer->lp_buffer[ buffer->index++ ] = (BYTE)( buffer->dw_buffer >> 32-buffer->by_left )<<(8-buffer->by_left);
}
}
WriteFile(lzw->h_dest, buffer->lp_buffer,buffer->index,&ret,NULL);
buffer->index = 0;
buffer->top = ret;
return (WORD)ret;
}
//------------------------------------------------------------------------------
#endif
(3) hash.h 定义了压~时所用的码表操作函数Qؓ了快速查找用了hash法Q还有处理hash冲突的函?br />
#ifndef __HASH_H__
#define __HASH_H__
//------------------------------------------------------------------------------
#include <stdio.h>
#include <stdlib.h>
#include <windows.h>
//------------------------------------------------------------------------------
#define DIV TABLE_LEN
#define HASHSTEP 13 // It should bigger than 0.
//------------------------------------------------------------------------------
WORD get_hash_index( PLZW_DATA lzw )
{
DWORD tmp;
WORD result;
DWORD prefix;
DWORD suffix;
prefix = lzw->prefix;
suffix = lzw->suffix;
tmp = prefix<<8 | suffix;
result = tmp % DIV;
return result;
}
//------------------------------------------------------------------------------
WORD re_hash_index( WORD hash ) // If hash conflict occured we must recalculate
{ // hash index .
WORD result;
result = hash + HASHSTEP;
result = result % DIV;
return result;
}
//------------------------------------------------------------------------------
BOOL in_table( PLZW_DATA lzw ) // To find whether current code is already in table.
{
BOOL result;
WORD hash;
hash = get_hash_index( lzw );
if( lzw->lp_code[ hash ] == 0xFFFF )
{
result = FALSE;
}
else
{
if( lzw->lp_prefix[ hash ] == lzw->prefix &&
lzw->lp_suffix[ hash ] == lzw->suffix )
{
result = TRUE;
}
else
{
result = FALSE;
while( lzw->lp_code[ hash ] != 0xFFFF )
{
if( lzw->lp_prefix[ hash ] == lzw->prefix &&
lzw->lp_suffix[ hash ] == lzw->suffix )
{
result = TRUE;
break;
}
hash = re_hash_index( hash );
}
}
}
return result;
}
//------------------------------------------------------------------------------
WORD get_code( PLZW_DATA lzw )
{
WORD hash;
WORD code;
hash = get_hash_index( lzw );
if( lzw->lp_prefix[ hash ] == lzw->prefix &&
lzw->lp_suffix[ hash ] == lzw->suffix )
{
code = lzw->lp_code[ hash ];
}
else
{
while( lzw->lp_prefix[ hash ] != lzw->prefix ||
lzw->lp_suffix[ hash ] != lzw->suffix )
{
hash = re_hash_index( hash );
}
code = lzw->lp_code[ hash ];
}
return code;
}
//------------------------------------------------------------------------------
VOID insert_table( PLZW_DATA lzw )
{
WORD hash;
hash = get_hash_index( lzw );
if( lzw->lp_code[ hash ] == 0xFFFF )
{
lzw->lp_prefix[ hash ] = lzw->prefix;
lzw->lp_suffix[ hash ] = lzw->suffix;
lzw->lp_code[ hash ] = lzw->code;
}
else
{
while( lzw->lp_code[ hash ] != 0xFFFF )
{
hash = re_hash_index( hash );
}
lzw->lp_prefix[ hash ] = lzw->prefix;
lzw->lp_suffix[ hash ] = lzw->suffix;
lzw->lp_code[ hash ] = lzw->code;
}
}
//------------------------------------------------------------------------------
#endif
(4) encode.h 压羃E序d?br />
#ifndef __ENCODE_H__
#define __ENCODE_H__
//------------------------------------------------------------------------------
#include <stdio.h>
#include <stdlib.h>
#include <windows.h>
//------------------------------------------------------------------------------
VOID output_code( DWORD code ,PBUFFER_DATA out, PLZW_DATA lzw)
{
out->dw_buffer |= code << ( 32 - out->by_left - lzw->cur_code_len );
out->by_left += lzw->cur_code_len;
while( out->by_left >= 8 )
{
if( out->index == BUFFERSIZE )
{
empty_buffer( lzw,out);
}
out->lp_buffer[ out->index++ ] = (BYTE)( out->dw_buffer >> 24 );
out->dw_buffer <<= 8;
out->by_left -= 8;
}
}
//------------------------------------------------------------------------------
VOID do_encode( PBUFFER_DATA in, PBUFFER_DATA out, PLZW_DATA lzw)
{
WORD prefix;
while( in->index != in->top )
{
if( !in_table(lzw) )
{
// current code not in code table
// then add it to table and output prefix
insert_table(lzw);
prefix = lzw->suffix;
output_code( lzw->prefix ,out ,lzw );
lzw->code++;
if( lzw->code == (WORD)1<< lzw->cur_code_len )
{
// code reached current code top(1<<cur_code_len)
// then current code length add one
lzw->cur_code_len++;
if( lzw->cur_code_len == CODE_LEN + 1 )
{
re_init_lzw( lzw );
}
}
}
else
{
// current code already in code table
// then output nothing
prefix = get_code(lzw);
}
lzw->prefix = prefix;
lzw->suffix = in->lp_buffer[ in->index++ ];
}
}
//------------------------------------------------------------------------------
VOID encode(HANDLE h_sour,HANDLE h_dest)
{
LZW_DATA lzw;
BUFFER_DATA in ;
BUFFER_DATA out;
BOOL first_run = TRUE;
lzw_create( &lzw ,h_sour,h_dest );
buffer_create( &in );
buffer_create( &out );
while( load_buffer( h_sour, &in ) )
{
if( first_run )
{// File length should be considered but here we simply
// believe file length bigger than 2 bytes.
lzw.prefix = in.lp_buffer[ in.index++ ];
lzw.suffix = in.lp_buffer[ in.index++ ];
first_run = FALSE;
}
do_encode(&in , &out, &lzw);
}
output_code(lzw.prefix, &out , &lzw);
output_code(lzw.suffix, &out , &lzw);
out.end_flag = TRUE;
empty_buffer( &lzw,&out);
lzw_destory( &lzw );
buffer_destory( &in );
buffer_destory( &out );
}
//------------------------------------------------------------------------------
#endif
(5) decode.h 解压函数d?br />
#ifndef __DECODE_H__
#define __DECODE_H__
//------------------------------------------------------------------------------
#include <stdio.h>
#include <stdlib.h>
#include <windows.h>
//------------------------------------------------------------------------------
VOID out_code( WORD code ,PBUFFER_DATA buffer,PLZW_DATA lzw,PSTACK_DATA stack)
{
WORD tmp;
if( code < 0x100 )
{
stack->lp_stack[ stack->index++ ] = code;
}
else
{
stack->lp_stack[ stack->index++ ] = lzw->lp_suffix[ code ];
tmp = lzw->lp_prefix[ code ];
while( tmp > 0x100 )
{
stack->lp_stack[ stack->index++ ] = lzw->lp_suffix[ tmp ];
tmp = lzw->lp_prefix[ tmp ];
}
stack->lp_stack[ stack->index++ ] = (BYTE)tmp;
}
while( stack->index )
{
if( buffer->index == BUFFERSIZE )
{
empty_buffer(lzw,buffer);
}
buffer->lp_buffer[ buffer->index++ ] = stack->lp_stack[ --stack->index ] ;
}
}
//------------------------------------------------------------------------------
VOID insert_2_table(PLZW_DATA lzw )
{
lzw->lp_code[ lzw->code ] = lzw->code;
lzw->lp_prefix[ lzw->code ] = lzw->prefix;
lzw->lp_suffix[ lzw->code ] = lzw->suffix;
lzw->code++;
if( lzw->code == ((WORD)1<<lzw->cur_code_len)-1 )
{
lzw->cur_code_len++;
if( lzw->cur_code_len == CODE_LEN+1 )
lzw->cur_code_len = 9;
}
if(lzw->code >= 1<<CODE_LEN )
{
re_init_lzw(lzw);
}
}
//------------------------------------------------------------------------------
WORD get_next_code( PBUFFER_DATA buffer , PLZW_DATA lzw )
{
BYTE next;
WORD code;
while( buffer->by_left < lzw->cur_code_len )
{
if( buffer->index == BUFFERSIZE )
{
load_buffer( lzw->h_sour, buffer );
}
next = buffer->lp_buffer[ buffer->index++ ];
buffer->dw_buffer |= (DWORD)next << (24-buffer->by_left);
buffer->by_left += 8;
}
code = buffer->dw_buffer >> ( 32 - lzw->cur_code_len );
buffer->dw_buffer <<= lzw->cur_code_len;
buffer->by_left -= lzw->cur_code_len;
return code;
}
//------------------------------------------------------------------------------
VOID do_decode( PBUFFER_DATA in, PBUFFER_DATA out, PLZW_DATA lzw, PSTACK_DATA stack)
{
WORD code;
WORD tmp;
while( in->index != in->top )
{
code = get_next_code( in ,lzw );
if( code < 0x100 )
{
// code already in table
// then simply output the code
lzw->suffix = (BYTE)code;
}
else
{
if( code < lzw->code )
{
// code also in table
// then output code chain
tmp = lzw->lp_prefix[ code ];
while( tmp > 0x100 )
{
tmp = lzw->lp_prefix[ tmp ];
}
lzw->suffix = (BYTE)tmp;
}
else
{
// code == lzw->code
// code not in table
// add code into table
// and out put code
tmp = lzw->prefix;
while( tmp > 0x100 )
{
tmp = lzw->lp_prefix[ tmp ];
}
lzw->suffix = (BYTE)tmp;
}
}
insert_2_table( lzw );
out_code(code,out,lzw,stack);
lzw->prefix = code;
}
}
//------------------------------------------------------------------------------
VOID decode( HANDLE h_sour, HANDLE h_dest )
{
LZW_DATA lzw;
BUFFER_DATA in ;
BUFFER_DATA out;
STACK_DATA stack;
BOOL first_run;
first_run = TRUE;
lzw_create( &lzw ,h_sour,h_dest );
buffer_create( &in );
buffer_create( &out );
stack_create(&stack );
while( load_buffer( h_sour, &in ) )
{
if( first_run )
{
lzw.prefix = get_next_code( &in, &lzw );
lzw.suffix = lzw.prefix;
out_code(lzw.prefix, &out, &lzw , &stack);
first_run = FALSE;
}
do_decode(&in , &out, &lzw, &stack);
}
empty_buffer( &lzw,&out);
lzw_destory( &lzw );
buffer_destory( &in );
buffer_destory( &out );
stack_destory( &stack);
}
#endif
2 下面l出一个应用上面模块的单例?br />
#include <stdio.h>
#include <stdlib.h>
//------------------------------------------------------------------------------
#include "lzw.h"
#include "hash.h"
#include "fileio.h"
#include "encode.h"
#include "decode.h"
//------------------------------------------------------------------------------
HANDLE h_file_sour;
HANDLE h_file_dest;
HANDLE h_file;
CHAR* file_name_in = "d:\\code.c";
CHAR* file_name_out= "d:\\encode.e";
CHAR* file_name = "d:\\decode.d";
//------------------------------------------------------------------------------
int main(int argc, char *argv[])
{
h_file_sour = file_handle(file_name_in);
h_file_dest = file_handle(file_name_out);
h_file = file_handle(file_name);
encode(h_file_sour, h_file_dest);
// decode(h_file_dest,h_file);
CloseHandle(h_file_sour);
CloseHandle(h_file_dest);
CloseHandle(h_file);
return 0;
}
3 后语
之前研究gif文g格式时偶然接触了lzw压羃法Q于是就惌己动手实现。从一开始看人家的原码，然后跟着模仿Q到现在用自q语言表达出来Q从理解原理C码的实现p了不时间与_֊Q但是真正的快乐也就在这里，现在把她拿出来跟大家分n也就是分享快乐?img src ="http://www.shnenglu.com/swo2006/aggbug/11431.html" width = "1" height = "1" />
]]>
1 gzip所使用压羃法的基本原?br />
gzip 对于要压~的文gQ首先用lz77法q行压羃Q对得到的结果再使用huffman~码的方法进行压~。所以我们分别对lz77和huffman~码的原理进行说明?br />
2 gzip压羃法实现Ҏ
2.1 LZ77法的gzip实现
首先Qgzip 从要压羃的文件中d64KB的内容到一个叫window的缓冲区中。ؓ了简单v见，我们?2KB以下文g的压~ؓ例做说明。对于我们这里?2KB 以下文gQgzip整个文件读入到window~冲Z。然后用一个叫strstart的变量在window数组中，?开始一直向后移动? strstart在每一个位|上Q都在它之前的区域中Q寻扑֒当前strstart开始的串的?个字节匹配的Ԍq试图从q些匚w串中扑ֈ最长的匚w 丌Ӏ?br />
如果当前的strstart开始的Ԍ可以扑ֈ最ؓ3个字节的匚w串的话，当前的strstart开始的匚w长度那么长的Ԍ会被一?lt;匚w长度,到匹配串开头的距离>Ҏ换?br />
如果当前的strstart开始的Ԍ找不CQ何的最ؓ3个字节的匚w串的话，那么当前strstart的所在字节将不作改动?br />
Z区分是一?lt;匚w长度,到匹配串开头的距离>对，q是一个没有被改动的字节，q需要ؓ每一个没有被改动的字节或?lt;匚w长度,到匹配串开头的距离>对，另外再占用一
位，来进行区分。这位如果ؓ1Q表C是一?lt;匚w长度,到匹配串开头的距离>对，q位如果?Q表C是一个没有被改动的字节?br />
现在来说明一下，Z么最匹配ؓ3个字节。这是由于，gzip 中，<匚w长度,到匹配串开头的距离>对中Q?匚w长度"的范围ؓ3-258Q也是256U可能|需?bit来保存?到匹配串开头的 距离"的范围ؓ0-32KQ需?5bit来保存。所以一?lt;匚w长度,到匹配串开头的距离>寚w?3位，差一?个字节。如果匹配串于 3个字节的话，使用<匚w长度,到匹配串开头的距离>对进行替换，不但没有压羃Q反而还会增大。所以保?lt;匚w长度,到匹配串开头的? ?gt;Ҏ需要的位数Q决定了最匹配长度至要?个字节?br />
下面我们来介绍gzip如何实现L当前strstart开始的串的最长匹配串?br />
如果每次为当前串L匚w串时Q都要和之前的每个串的至?个字节进行比较的话，那么比较量将是非帔R常大的。ؓ了提高比较速度Qgzip使用了哈? 表。这是gzip实现LZ77的关键。这个哈希表是一个叫head的数l（后面我们看Cؓ什么这个缓冲区叫headQ。gzip对windows中的? 个串Q用串的头三个字节Q也是strstart,strstart+1,strstart+2Q用一个设计好的哈希函数来q行计算Q得C个插入位|? ins_h。也是用串的头三个字节来确定一个插入位|。然后把串的位置Q也是 strstart的|保存在head数组的第ins_h中。我们马上就可以看到Z么要q样做。head数组在没有插入Q何值时Q全部ؓ0?br />
当某处的当前串的三个字节定了一个ins_hQƈ把当时当前串的位|也是当时的strstart保存在了head[ins_h]中。之后另一处，当另 一处的当前串的头三个字节，再ؓ那三个字节时Q再使用那个哈希函数来计，׃是同L三个字节Q同L哈希函数Q得到的ins_h必然和前面得到的 ins_h是相同的。于是就会发现head[ins_h]不ؓ0。这p明了Q有一个头三个字节和自q同的串把自己的位|保存在了这里，现在head [ins_h]中保存的|也就是那个串的开始位|，我们可以找到那个串Q那个串臛_?个字节和当前串的?个字节相同（E后我们可以看到这U说? 不准，q里是ؓ了说明方便）Q我们可以找到那个串Q做q一步比较，看到底能有多长的匚w?br />
我们现在来说明一下，相同的三个字节，通过哈希函数得到的ins_h必然是相同的。而不同的三个字节Q通过哈希函数有没有可能得到同一个ins_hQ? 我没有对q个哈希函数做研IӞq不清楚Q不q一般的哈希函数都是q样的，所以极大可能这里的也会是这U情况，即不同的三个字节Q通过哈希函数有可能得到同 一个ins_hQ不q这q不要紧Q我们发现有可能是匹配串之后Q还会进行串的比较?br />
一个文件中Q可能有很多个串的头三个字节都是相同的，也就是说他们计算得到的ins_h都是相同的，如何能保证找C们中的每一个串呢？gzip使用 一个链把他们链在一赗gzip每次把当前串的位|插入head的当前串头三个字节算出的ins_h处时Q都会首先把原来的head[ins_h]的| 保存C个叫prev的数l中Q保存的位置在现在的strstart处。这样当以后某处的当前串计算出ins_hQ发现head[ins_h]不空Ӟ 可以到prev[ head[ins_h] ]中找到更前一个的头三个字节相同的串的位置。对此我们D例说明?br />
例，?br /> 0abcdabceabcfabcg
^^^^^^^^^^^^^^^^^
01234567890123456
整个串被压羃E序处理之后?br />
由abc出ins_h?br /> q时的head[ins_h]中ؓ 13,?abcg"的开始位|?br /> q时prev[13]中ؓ 9Q即"abcfabcg"的开始位|?br /> q时prev[9]中ؓ 5Q即"abceabcfabcg"的开始位|?br /> q时prev[5]中ؓ 1Q即"abcdabceabcfabcg"的开始位|?br /> q时prev[1]中ؓ 0?br />
我们看到所有头三个字母为abc的串Q被铑֜了一P从head可以一直找下去Q直到找??br />
现在我们也就知道了，三个字节通过哈希函数计算得到同一ins_h的所有的串被铑֜了一Phead[ins_h]为链_prev数组中放着的更早的丌Ӏ这也就是head和prev名称的由
来?br />
gzipL匚w串的另外一个值得注意的实现是Qgq匹配。会q行两次试。比如当前串为str,那么str发生匚w以后Qƈ不发生压~，q会对str+1串进行匹配，然后看哪U?
匚w效果好?br />
例子 ...
从这个例子中我们qC做另外一ơ尝试的原因。如果碰到的一个匹配就使用了的话，可能错过更长匚w的机会。现在做两次会有所改善?br />
...
2.2 问题讨论
我在q里对gzip压羃法做出了一些说明，是希望可以和对gzip或者压~解压羃感兴的朋友q行交流?br /> 我对gzip的了解要比这里说的更多一些，也有更多的例子。如果哪位朋友愿意对下面的问题进行研IӞ以及其他压羃解压~的问题q行研究Q来q里 http://jiurl.cosoft.org.cn/forum/ 和我交流的话Q我也愿意就我知道的内容q行更多的说明?br />
下面是几个问?br />
q种匚w法Q即?个字?最匹?来计一个整敎ͼ是否比用串比较来得高效，高效C么程度?br />
哈希函数的讨论。不同的三个字节Q是否可能得到同一个ins_h。ins_h和计它的三个字节的关系?br />
几次延迟试比较好？
用gq，两次试是否对压~率的改善是非常有限的？
影响lz77压羃率的因素?br />
压羃的极限?br />
2.3 ...
3 gzip源码分析
main() 中调用函?treat_file() ?br /> treat_file() 中打开文gQ调用函?zip()。注意这里的 work 的用法，q是一个函数指针?br /> zip() 中输出gzip文g格式的头Q调?bi_initQct_initQlm_initQ?br /> 其中在lm_init中将 head 初始化清0。初始化strstart?。从文g中读?4KB的内容到window~冲Z?br /> ׃计算strstart=0时的ins_hQ需?,1,2q三个字节和哈希函数发生关系Q所以在lm_init中，预读0,1两个字节Qƈ和哈希函数发生关pR?br />
然后lm_init调用 deflate()?br /> deflate() gzip的LZ77的实C要deflate()中?br /> ...
]]>
The origins of zlib can be found in the history of Info-ZIP. Info-ZIP is loosely organized group of programmers who give the following reason for their existence:
Info-ZIP's purpose is to provide free, portable, high-quality versions of the Zip and UnZip compressor-archiver utilities that are compatible with the DOS-based PKZIP by PKWARE, Inc.
These free versions of Zip and UnZip are world class programs, and are in wide use on platforms ranging from the orphaned Amiga through MS-DOS PCs up to high powered RISC workstations. But these programs are designed to be used as command line utilities, not as library routines. People have found that porting the Info-ZIP source into an application could be a grueling exercise.
Fortunately for all of us, two of the Info-ZIP gurus took it upon themselves to solve this problem. Mark Adler and Jean-loup Gailly single-handedly created zlib, a set of library routines that provide a safe, free, and unpatented implementation of the deflate compression algorithm.
One of the driving reasons behind zlib's creation was for use as the compressor for PNG format graphics. After Unisys belatedly began asserting their patent rights to LZW compression, programmers all over the world were thrown into a panic over the prospect of paying royalties on their GIF decoding programs. The PNG standard was created to provide an unencumbered format for graphics interchange. The zlib version of the deflate algorithm was embraced by PNG developers, not only because it was free, but it also compressed better than the original LZW compressor used in GIF files.
zlib turns out to be good for more than graphics developers, however. The deflate algorithm makes an excellent general purpose compressor, and as such can be incorporated into all sorts of different software. For example, I use zlib as the compression engine in Greenleaf's ArchiveLib, a data compression library that work with ZIP archives. It's performance and compatibility mean I didn't have to reinvent the wheel, saving precious months of development time.
zlib's interface
As a library developer, I know that interfaces make or break a library. Performance issues are important, but if an awkward API makes it impossible to integrate a library into your program, you've got a problem.
zlib's interface is confined to just a few simple function calls. The entire state of a given compression or decompression session is encapsulated in a C structure of type z_stream, whose definition is shown in Figure 1.
typedef struct z_stream_s {
Bytef *next_in; /* next input byte */
uInt avail_in; /* number of bytes available at next_in */
uLong total_in; /* count of input bytes read so far */
Bytef *next_out; /* next output byte should be put there */
uInt avail_out; /* remaining free space at next_out */
uLong total_out; /* count of bytes output so far */
char *msg; /* last error message, NULL if no error */
struct internal_state *state; /* not visible by applications*/
alloc_func zalloc; /* used to allocate the internal state*/
free_func zfree; /* used to free the internal state */
voidpf opaque; /* private data passed to zalloc and zfree*/
int data_type; /* best guess about the data: ascii or binary*/
uLong adler; /* adler32 value of the uncompressed data */
uLong reserved; /* reserved for future use */
} z_stream;
Figure 1
The z_stream object definition
Using the library to compress or decompress a file or other data object consists of three main steps:
- Creating a z_stream object.
- Processing input and output, using the z_stream object to communicate with zlib.
- Destroying the z_stream object.
An overview of the process is shown in Figure 2.
Figure 2
The compression or decompression process
Steps 1 and 3 of the compression process are done using conventional
function calls. The zlib API, documented in header file zlib.h,
prototypes the following functions for initialization and termination of
the compression or decompression process:
- deflateInit()
- inflateInit()
- deflateEnd()
- inflateEnd()
Step 2 is done via repeated calls to either inflate() or deflate(), passing the z_stream object as a parameter. The entire state of the process is contained in that object, so there are no global flags or variables, which allows the library to be completely reentrant. Storing the state of the process in a single object also cuts down on the number of parameters that must be passed to the API functions.
When performing compression or decompression, zlib doesn't perform any I/O on its own. Instead, it reads data from an input buffer pointer that you supply in the z_stream object. You simply set up a pointer to the next block of input data in member next_in, and place the number of available bytes in the avail_in member. Likewise, zlib writes its output data to a memory buffer you set up in the next_out member. As it writes output bytes, zlib decrements the avail_out member until it drops to 0.
Given this interface, Step 2 of the compression process for an input file and an output file might look something like this:
z_stream z;
char input_buffer[ 1024 ];
char output_buffer[ 1024 ];
FILE *fin;
FILE *fout;
int status;
...
z.avail_in = 0;
z.next_out = output_buffer;
z.avail_out = 1024;
for ( ; ; ) {
if ( z.avail_in == 0 ) {
z.next_in = input_buffer;
z.avail_in = fread( input_buffer, 1, 1024, fin );
}
if ( z.avail_in == 0 )
break;
status = deflate( &z, Z_NO_FLUSH );
int count = 1024 - z.avail_out;
if ( count )
fwrite( output_buffer, 1, count, fout );
z.next_out = output_buffer;
z.avail_out = 1024;
}
Figure 3
The code to implement file compression
This method of handling I/O frees zlib from having to implement system dependent read and write code, and it insures that you can use the library to compress any sort of input stream, not just files. It's simply a matter of replacing the wrapper code shown above with a version customized for your data stream.
Wrapping it up
zlib's versatility is one of its strengths, but I don't always need all that flexibility. For example, to perform the simple file compression task Scott asked about at the start of this article, it would be nice to just be able to call a single function to compress a file, and another function to decompress. To make this possible, I created a wrapper class called zlibEngine.
zlibEngine provides a simple API that automates the compression and decompression of files and uses virtual functions to let you customize your user interface to zlib. The class definition is shown in its entirety in Figure 4. There are two different groups of members that are important to you in ZlibEngine. The first is the set of functions providing the calling interface to the engine. The second is the set of functions and data members used to create a user interface that is active during the compression process.
class ZlibEngine : public z_stream {
public :
ZlibEngine();
int compress( const char *input,
const char *output,
int level = 6 );
int decompress( const char *input,
const char *output );
void set_abort_flag( int i ){ m_AbortFlag = i; }
protected :
int percent();
int load_input();
int flush_output();
protected :
virtual void progress( int percent ){};
virtual void status( char *message ){};
protected :
int m_AbortFlag;
FILE *fin;
FILE *fout;
long length;
int err;
enum { input_length = 4096 };
unsigned char input_buffer[ input_length ];
enum { output_length = 4096 };
unsigned char output_buffer[ output_length ];
};
Figure 4
The ZlibEngine wrapper class
The Calling API
There are three C++ functions that implement the API needed to perform simple compression and decompression. Before using the engine, you must call the constructor, the first function. Since ZlibEngine is derived from the z_stream object used as the interface to zlib, the constructor is in effect also creating a z_stream object that will be used to communicate with zlib. In addition, the constructor initializes some of the z_stream member variables that will be used in either compression or decompression.
The two remaining functions are nice and simple: compress() compresses a file using the deflate algorithm. An optional level parameter sets a compression factor between 9 (maximum compression) and 0 (no compression.) decompress() decompresses a file, as you would expect. The compression level parameter isn't necessary when decompressing, due to the nature of the deflate algorithm. Both of these functions return an integer status code, defined in the zlib header file zlib.h. Z_OK is returned when everything works as expected. Note that I added an additional code, Z_USER_ABORT, used for an end user abort of the compression or decompression process.
The wrapper class makes it much easier to compress or decompress files using zlib. You only need to remember three things:
- Include the header file for the wrapper class, zlibengn.h.
- Construct a ZlibEngine object.
- Call the member functions compress() or decompress() to do the actual work.
This means you can now perform compression with code this simple:
#include <zlibengn.h>
int foo()
{
ZlibEngine engine;
return engine.compress( "INPUT.DAT", "INPUT.DA_");
}
That's about as simple as you could ask for, isn't it?
The User Interface API
The calling API doesn't really make much of a case for creating the ZlibEngine class. Based on what you've seen so far, the compress() and decompress() functions don't really need to be members of a class. In theory, a global compress() function could just instantiate a z_stream object when called, without the caller even being aware of it.
The reason for creating this engine class is found in a completely different area: the user interface. It's really nice to be able to track the progress of your compression job while it's running. Conventional C libraries have to make do with callback functions or inflexible standardized routines in order to provide feedback, but C++ offers a better alternative through the use of virtual functions.
The ZlibEngine class has two virtual functions that are used to create a useful user interface: progress() is called periodically during the compression or decompression process, with a single integer argument that tells what percentage of the input file has been processed. status() is called with status messages during processing.
Both of these virtual functions have access to the ZlibEngine protected data element, m_AbortFlag. Setting this flag to a non-zero value will cause the compression or decompression routine to abort immediately. This easily takes care of another sticky user interface problem found when using library code.
Writing your own user interface then becomes a simple exercise. You simply derive a new class from ZlibEngine, and define your own versions of one or both of these virtual functions. Instantiate an object of your class instead of ZlibEngine, and your user interface can be as spiffy and responsive as you like!
Command line compression
I wrote a simple command line test program to demonstrate the use of class ZlibEngine. zlibtest.cpp does a simple compress/decompress cycle of the input file specified on the command line. I implement a progress function that simply prints out the current percent towards completion as the file is processed:
class MyZlibEngine : public ZlibEngine {
public :
void progress( int percent )
{
printf( "%3d%%\b\b\b\b", percent );
if ( kbhit() ) {
getch();
m_AbortFlag = 1;
}
}
};
Since class ZlibEngine is so simple, the derived class doesn't even have to implement a constructor or destructor. The derived version of progress() is able to provide user feedback as well as an abort function with just a few lines of code. zlibtest.cpp is shown in its entirety in Listing 1.
The OCX
To provide a slightly more complicated test of class ZlibEngine, I created a 32 bit OCX using Visual C++ 4.1. The interface to an OCX is defined in terms of methods, events, and properties. ZlibTool.ocx has the following interface:
|
Properties: |
InputFile |
|
|
OutputFile |
|
|
Level |
|
|
Status |
|
|
|
|
Methods: |
Compress() |
|
|
Decompress() |
|
|
Abort() |
|
|
|
|
Events: |
Progress() |
(Note that I chose to pass status information from the OCX using a property, not an event.)
ZlibTool.ocx is a control derived from a standard Win32 progress bar. The progress care gets updated automatically while compressing or decompressing, so you get some user interface functionality for free. Using it with Visual Basic 4.0 or Delphi 2.0 becomes a real breeze. After registering the OCX, you can drop a copy of it onto your form and use it with a minimal amount of coding.
Both the source code for the OCX and a sample Delphi 2.0 program are available on the DDJ listing service. A screen shot of the Delphi program in action is shown in Figure 5.
Figure 5
The Delphi 2.0 OCX test program
Reference material
The source code that accompanies this article can be downloaded from this Web page. It contains the following source code collections:
- The complete source for zlib
- The Visual C++ 4.1 project for the ZlibTool OCX
- The Delphi 2.0 project that exercises the OCX
- The Console test program that exercises the ZlibEngine class
Each of the subdirectories contains a README.TXT file with documentation describing how to build and use the programs.
The source is split into two archives:
|
|
All source code and the OCX file. |
|
|
The supporting MFC and VC++ DLLs. Many people will already have these files on their systems: MFC40.DLL, MSVCRT40.DLL, and OLEPRO32.DLL. |
I haven't discussed the zlib code itself in this article. The best place to start gathering information about how to use zlib and the Info-ZIP products can be found on their home pages. Both pages have links to the most current versions of their source code as well:
|
Info-ZIP |
|
|
zlib |
|
Once you download the Info-ZIP code, the quick start documentation is found in source file zlib.h. If you cook up any useful code that uses zlib, you might want to forward copies to Greg Roelofs for inclusion on the zlib home page. Greg maintains the zlib pages, and you can reach him via links found there.
Feel-good plug
zlib can do a lot more than just compress files. Its versatile interface can be used for streaming I/O, in-memory compression, and more. Since Jean-loup Gailly and Mark Adler were good enough to make this capable tool available to the public, it only makes sense that we take advantage of it. I know I have, and I encourage you to do the same.
]]>
Status of This Memo
This memo provides information for the Internet community. This memo
does not specify an Internet standard of any kind. Distribution of
this memo is unlimited.
IESG Note:
The IESG takes no position on the validity of any Intellectual
Property Rights statements contained in this document.
Notices
Copyright (c) 1996 L. Peter Deutsch and Jean-Loup Gailly
Permission is granted to copy and distribute this document for any
purpose and without charge, including translations into other
languages and incorporation into compilations, provided that the
copyright notice and this notice are preserved, and that any
substantive changes or deletions from the original are clearly
marked.
A pointer to the latest version of this and related documentation in
HTML format can be found at the URL
<ftp://ftp.uu.net/graphics/png/documents/zlib/zdoc-index.html>.
Abstract
This specification defines a lossless compressed data format. The
data can be produced or consumed, even for an arbitrarily long
sequentially presented input data stream, using only an a priori
bounded amount of intermediate storage. The format presently uses
the DEFLATE compression method but can be easily extended to use
other compression methods. It can be implemented readily in a manner
not covered by patents. This specification also defines the ADLER-32
checksum (an extension and improvement of the Fletcher checksum),
used for detection of data corruption, and provides an algorithm for
computing it.
Table of Contents
1. Introduction ................................................... 2
1.1. Purpose ................................................... 2
1.2. Intended audience ......................................... 3
1.3. Scope ..................................................... 3
1.4. Compliance ................................................ 3
1.5. Definitions of terms and conventions used ................ 3
1.6. Changes from previous versions ............................ 3
2. Detailed specification ......................................... 3
2.1. Overall conventions ....................................... 3
2.2. Data format ............................................... 4
2.3. Compliance ................................................ 7
3. References ..................................................... 7
4. Source code .................................................... 8
5. Security Considerations ........................................ 8
6. Acknowledgements ............................................... 8
7. Authors' Addresses ............................................. 8
8. Appendix: Rationale ............................................ 9
9. Appendix: Sample code ..........................................10
1. Introduction
1.1. Purpose
The purpose of this specification is to define a lossless
compressed data format that:
* Is independent of CPU type, operating system, file system,
and character set, and hence can be used for interchange;
* Can be produced or consumed, even for an arbitrarily long
sequentially presented input data stream, using only an a
priori bounded amount of intermediate storage, and hence can
be used in data communications or similar structures such as
Unix filters;
* Can use a number of different compression methods;
* Can be implemented readily in a manner not covered by
patents, and hence can be practiced freely.
The data format defined by this specification does not attempt to
allow random access to compressed data.
1.2. Intended audience
This specification is intended for use by implementors of software
to compress data into zlib format and/or decompress data from zlib
format.
The text of the specification assumes a basic background in
programming at the level of bits and other primitive data
representations.
1.3. Scope
The specification specifies a compressed data format that can be
used for in-memory compression of a sequence of arbitrary bytes.
1.4. Compliance
Unless otherwise indicated below, a compliant decompressor must be
able to accept and decompress any data set that conforms to all
the specifications presented here; a compliant compressor must
produce data sets that conform to all the specifications presented
here.
1.5. Definitions of terms and conventions used
byte: 8 bits stored or transmitted as a unit (same as an octet).
(For this specification, a byte is exactly 8 bits, even on
machines which store a character on a number of bits different
from 8.) See below, for the numbering of bits within a byte.
1.6. Changes from previous versions
Version 3.1 was the first public release of this specification.
In version 3.2, some terminology was changed and the Adler-32
sample code was rewritten for clarity. In version 3.3, the
support for a preset dictionary was introduced, and the
specification was converted to RFC style.
2. Detailed specification
2.1. Overall conventions
In the diagrams below, a box like this:
+---+
| | <-- the vertical bars might be missing
+---+
represents one byte; a box like this:
+==============+
| |
+==============+
represents a variable number of bytes.
Bytes stored within a computer do not have a "bit order", since
they are always treated as a unit. However, a byte considered as
an integer between 0 and 255 does have a most- and least-
significant bit, and since we write numbers with the most-
significant digit on the left, we also write bytes with the most-
significant bit on the left. In the diagrams below, we number the
bits of a byte so that bit 0 is the least-significant bit, i.e.,
the bits are numbered:
+--------+
|76543210|
+--------+
Within a computer, a number may occupy multiple bytes. All
multi-byte numbers in the format described here are stored with
the MOST-significant byte first (at the lower memory address).
For example, the decimal number 520 is stored as:
0 1
+--------+--------+
|00000010|00001000|
+--------+--------+
^ ^
| |
| + less significant byte = 8
+ more significant byte = 2 x 256
2.2. Data format
A zlib stream has the following structure:
0 1
+---+---+
|CMF|FLG| (more-->)
+---+---+
(if FLG.FDICT set)
0 1 2 3
+---+---+---+---+
| DICTID | (more-->)
+---+---+---+---+
+=====================+---+---+---+---+
|...compressed data...| ADLER32 |
+=====================+---+---+---+---+
Any data which may appear after ADLER32 are not part of the zlib
stream.
CMF (Compression Method and flags)
This byte is divided into a 4-bit compression method and a 4-
bit information field depending on the compression method.
bits 0 to 3 CM Compression method
bits 4 to 7 CINFO Compression info
CM (Compression method)
This identifies the compression method used in the file. CM = 8
denotes the "deflate" compression method with a window size up
to 32K. This is the method used by gzip and PNG (see
references [1] and [2] in Chapter 3, below, for the reference
documents). CM = 15 is reserved. It might be used in a future
version of this specification to indicate the presence of an
extra field before the compressed data.
CINFO (Compression info)
For CM = 8, CINFO is the base-2 logarithm of the LZ77 window
size, minus eight (CINFO=7 indicates a 32K window size). Values
of CINFO above 7 are not allowed in this version of the
specification. CINFO is not defined in this specification for
CM not equal to 8.
FLG (FLaGs)
This flag byte is divided as follows:
bits 0 to 4 FCHECK (check bits for CMF and FLG)
bit 5 FDICT (preset dictionary)
bits 6 to 7 FLEVEL (compression level)
The FCHECK value must be such that CMF and FLG, when viewed as
a 16-bit unsigned integer stored in MSB order (CMF*256 + FLG),
is a multiple of 31.
FDICT (Preset dictionary)
If FDICT is set, a DICT dictionary identifier is present
immediately after the FLG byte. The dictionary is a sequence of
bytes which are initially fed to the compressor without
producing any compressed output. DICT is the Adler-32 checksum
of this sequence of bytes (see the definition of ADLER32
below). The decompressor can use this identifier to determine
which dictionary has been used by the compressor.
FLEVEL (Compression level)
These flags are available for use by specific compression
methods. The "deflate" method (CM = 8) sets these flags as
follows:
0 - compressor used fastest algorithm
1 - compressor used fast algorithm
2 - compressor used default algorithm
3 - compressor used maximum compression, slowest algorithm
The information in FLEVEL is not needed for decompression; it
is there to indicate if recompression might be worthwhile.
compressed data
For compression method 8, the compressed data is stored in the
deflate compressed data format as described in the document
"DEFLATE Compressed Data Format Specification" by L. Peter
Deutsch. (See reference [3] in Chapter 3, below)
Other compressed data formats are not specified in this version
of the zlib specification.
ADLER32 (Adler-32 checksum)
This contains a checksum value of the uncompressed data
(excluding any dictionary data) computed according to Adler-32
algorithm. This algorithm is a 32-bit extension and improvement
of the Fletcher algorithm, used in the ITU-T X.224 / ISO 8073
standard. See references [4] and [5] in Chapter 3, below)
Adler-32 is composed of two sums accumulated per byte: s1 is
the sum of all bytes, s2 is the sum of all s1 values. Both sums
are done modulo 65521. s1 is initialized to 1, s2 to zero. The
Adler-32 checksum is stored as s2*65536 + s1 in most-
significant-byte first (network) order.
2.3. Compliance
A compliant compressor must produce streams with correct CMF, FLG
and ADLER32, but need not support preset dictionaries. When the
zlib data format is used as part of another standard data format,
the compressor may use only preset dictionaries that are specified
by this other data format. If this other format does not use the
preset dictionary feature, the compressor must not set the FDICT
flag.
A compliant decompressor must check CMF, FLG, and ADLER32, and
provide an error indication if any of these have incorrect values.
A compliant decompressor must give an error indication if CM is
not one of the values defined in this specification (only the
value 8 is permitted in this version), since another value could
indicate the presence of new features that would cause subsequent
data to be interpreted incorrectly. A compliant decompressor must
give an error indication if FDICT is set and DICTID is not the
identifier of a known preset dictionary. A decompressor may
ignore FLEVEL and still be compliant. When the zlib data format
is being used as a part of another standard format, a compliant
decompressor must support all the preset dictionaries specified by
the other format. When the other format does not use the preset
dictionary feature, a compliant decompressor must reject any
stream in which the FDICT flag is set.
3. References
[1] Deutsch, L.P.,"GZIP Compressed Data Format Specification",
available in ftp://ftp.uu.net/pub/archiving/zip/doc/
[2] Thomas Boutell, "PNG (Portable Network Graphics) specification",
available in ftp://ftp.uu.net/graphics/png/documents/
[3] Deutsch, L.P.,"DEFLATE Compressed Data Format Specification",
available in ftp://ftp.uu.net/pub/archiving/zip/doc/
[4] Fletcher, J. G., "An Arithmetic Checksum for Serial
Transmissions," IEEE Transactions on Communications, Vol. COM-30,
No. 1, January 1982, pp. 247-252.
[5] ITU-T Recommendation X.224, Annex D, "Checksum Algorithms,"
November, 1993, pp. 144, 145. (Available from
gopher://info.itu.ch). ITU-T X.244 is also the same as ISO 8073.
4. Source code
Source code for a C language implementation of a "zlib" compliant
library is available at ftp://ftp.uu.net/pub/archiving/zip/zlib/.
5. Security Considerations
A decoder that fails to check the ADLER32 checksum value may be
subject to undetected data corruption.
6. Acknowledgements
Trademarks cited in this document are the property of their
respective owners.
Jean-Loup Gailly and Mark Adler designed the zlib format and wrote
the related software described in this specification. Glenn
Randers-Pehrson converted this document to RFC and HTML format.
7. Authors' Addresses
L. Peter Deutsch
Aladdin Enterprises
203 Santa Margarita Ave.
Menlo Park, CA 94025
Phone: (415) 322-0103 (AM only)
FAX: (415) 322-1734
EMail: <ghost@aladdin.com>
Jean-Loup Gailly
EMail: <gzip@prep.ai.mit.edu>
Questions about the technical content of this specification can be
sent by email to
Jean-Loup Gailly <gzip@prep.ai.mit.edu> and
Mark Adler <madler@alumni.caltech.edu>
Editorial comments on this specification can be sent by email to
L. Peter Deutsch <ghost@aladdin.com> and
Glenn Randers-Pehrson <randeg@alumni.rpi.edu>
8. Appendix: Rationale
8.1. Preset dictionaries
A preset dictionary is specially useful to compress short input
sequences. The compressor can take advantage of the dictionary
context to encode the input in a more compact manner. The
decompressor can be initialized with the appropriate context by
virtually decompressing a compressed version of the dictionary
without producing any output. However for certain compression
algorithms such as the deflate algorithm this operation can be
achieved without actually performing any decompression.
The compressor and the decompressor must use exactly the same
dictionary. The dictionary may be fixed or may be chosen among a
certain number of predefined dictionaries, according to the kind
of input data. The decompressor can determine which dictionary has
been chosen by the compressor by checking the dictionary
identifier. This document does not specify the contents of
predefined dictionaries, since the optimal dictionaries are
application specific. Standard data formats using this feature of
the zlib specification must precisely define the allowed
dictionaries.
8.2. The Adler-32 algorithm
The Adler-32 algorithm is much faster than the CRC32 algorithm yet
still provides an extremely low probability of undetected errors.
The modulo on unsigned long accumulators can be delayed for 5552
bytes, so the modulo operation time is negligible. If the bytes
are a, b, c, the second sum is 3a + 2b + c + 3, and so is position
and order sensitive, unlike the first sum, which is just a
checksum. That 65521 is prime is important to avoid a possible
large class of two-byte errors that leave the check unchanged.
(The Fletcher checksum uses 255, which is not prime and which also
makes the Fletcher check insensitive to single byte changes 0 <->
255.)
The sum s1 is initialized to 1 instead of zero to make the length
of the sequence part of s2, so that the length does not have to be
checked separately. (Any sequence of zeroes has a Fletcher
checksum of zero.)
9. Appendix: Sample code
The following C code computes the Adler-32 checksum of a data buffer.
It is written for clarity, not for speed. The sample code is in the
ANSI C programming language. Non C users may find it easier to read
with these hints:
& Bitwise AND operator.
>> Bitwise right shift operator. When applied to an
unsigned quantity, as here, right shift inserts zero bit(s)
at the left.
<< Bitwise left shift operator. Left shift inserts zero
bit(s) at the right.
++ "n++" increments the variable n.
% modulo operator: a % b is the remainder of a divided by b.
#define BASE 65521 /* largest prime smaller than 65536 */
/*
Update a running Adler-32 checksum with the bytes buf[0..len-1]
and return the updated checksum. The Adler-32 checksum should be
initialized to 1.
Usage example:
unsigned long adler = 1L;
while (read_buffer(buffer, length) != EOF) {
adler = update_adler32(adler, buffer, length);
}
if (adler != original_adler) error();
*/
unsigned long update_adler32(unsigned long adler,
unsigned char *buf, int len)
{
unsigned long s1 = adler & 0xffff;
unsigned long s2 = (adler >> 16) & 0xffff;
int n;
for (n = 0; n < len; n++) {
s1 = (s1 + buf[n]) % BASE;
s2 = (s2 + s1) % BASE;
}
return (s2 << 16) + s1;
}
/* Return the adler32 of the bytes buf[0..len-1] */
unsigned long adler32(unsigned char *buf, int len)
{
return update_adler32(1L, buf, len);
}
]]>