• <ins id="pjuwb"></ins>
    <blockquote id="pjuwb"><pre id="pjuwb"></pre></blockquote>
    <noscript id="pjuwb"></noscript>
          <sup id="pjuwb"><pre id="pjuwb"></pre></sup>
            <dd id="pjuwb"></dd>
            <abbr id="pjuwb"></abbr>

            O(1) 的小樂

            Job Hunting

            公告

            記錄我的生活和工作。。。
            <2011年1月>
            2627282930311
            2345678
            9101112131415
            16171819202122
            23242526272829
            303112345

            統(tǒng)計(jì)

            • 隨筆 - 182
            • 文章 - 1
            • 評論 - 41
            • 引用 - 0

            留言簿(10)

            隨筆分類(70)

            隨筆檔案(182)

            文章檔案(1)

            如影隨形

            搜索

            •  

            最新隨筆

            最新評論

            閱讀排行榜

            評論排行榜

            Kullback–Leibler divergence KL散度

            In probability theory and information theory, the Kullback–Leibler divergence[1][2][3] (also information divergence,information gain, relative entropy, or KLIC) is a non-symmetric measure of the difference between two probability distributions P and Q. KL measures the expected number of extra bits required to code samples from P when using a code based on Q, rather than using a code based on P. Typically P represents the "true" distribution of data, observations, or a precise calculated theoretical distribution. The measure Q typically represents a theory, model, description, or approximation of P.

            Although it is often intuited as a distance metric, the KL divergence is not a true metric – for example, the KL from P to Q is not necessarily the same as the KL from Q to P.

            KL divergence is a special case of a broader class of divergences called f-divergences. Originally introduced by Solomon Kullbackand Richard Leibler in 1951 as the directed divergence between two distributions, it is not the same as a divergence incalculus. However, the KL divergence can be derived from the Bregman divergence.

             

             

            注意P通常指數(shù)據(jù)集,我們已有的數(shù)據(jù)集,Q表示理論結(jié)果,所以KL divergence 的物理含義就是當(dāng)用Q來編碼P中的采樣時(shí),比用P來編碼P中的采用需要多用的位數(shù)!

             

            KL散度,也有人稱為KL距離,但是它并不是嚴(yán)格的距離概念,其不滿足三角不等式

             

            KL散度是不對稱的,當(dāng)然,如果希望把它變對稱,

            Ds(p1, p2) = [D(p1, p2) + D(p2, p1)] / 2

             

            下面是KL散度的離散和連續(xù)定義!

            D_{\mathrm{KL}}(P\|Q) = \sum_i P(i) \log \frac{P(i)}{Q(i)}. \!

            D_{\mathrm{KL}}(P\|Q) = \int_{-\infty}^\infty p(x) \log \frac{p(x)}{q(x)} \; dx, \!

            注意的一點(diǎn)是p(x) 和q(x)分別是pq兩個(gè)隨機(jī)變量的PDF,D(P||Q)是一個(gè)數(shù)值,而不是一個(gè)函數(shù),看下圖!

             

            注意:KL Area to be Integrated!

             

            File:KL-Gauss-Example.png

             

            KL 散度一個(gè)很強(qiáng)大的性質(zhì):

            The Kullback–Leibler divergence is always non-negative,

            D_{\mathrm{KL}}(P\|Q) \geq 0, \,

            a result known as , with DKL(P||Q) zero if and only if P = Q.

             

            計(jì)算KL散度的時(shí)候,注意問題是在稀疏數(shù)據(jù)集上KL散度計(jì)算通常會出現(xiàn)分母為零的情況!

             

             

            Matlab中的函數(shù):KLDIV給出了兩個(gè)分布的KL散度

            Description

            KLDIV Kullback-Leibler or Jensen-Shannon divergence between two distributions.

            KLDIV(X,P1,P2) returns the Kullback-Leibler divergence between two distributions specified over the M variable values in vector X. P1 is a length-M vector of probabilities representing distribution 1, and P2 is a length-M vector of probabilities representing distribution 2. Thus, the probability of value X(i) is P1(i) for distribution 1 and P2(i) for distribution 2. The Kullback-Leibler divergence is given by:

               KL(P1(x),P2(x)) = sum[P1(x).log(P1(x)/P2(x))]

            If X contains duplicate values, there will be an warning message, and these values will be treated as distinct values. (I.e., the actual values do not enter into the computation, but the probabilities for the two duplicate values will be considered as probabilities corresponding to two unique values.) The elements of probability vectors P1 and P2 must each sum to 1 +/- .00001.

            A "log of zero" warning will be thrown for zero-valued probabilities. Handle this however you wish. Adding 'eps' or some other small value to all probabilities seems reasonable. (Renormalize if necessary.)

            KLDIV(X,P1,P2,'sym') returns a symmetric variant of the Kullback-Leibler divergence, given by [KL(P1,P2)+KL(P2,P1)]/2. See Johnson and Sinanovic (2001).

            KLDIV(X,P1,P2,'js') returns the Jensen-Shannon divergence, given by [KL(P1,Q)+KL(P2,Q)]/2, where Q = (P1+P2)/2. See the Wikipedia article for "Kullback–Leibler divergence". This is equal to 1/2 the so-called "Jeffrey divergence." See Rubner et al. (2000).

            EXAMPLE: Let the event set and probability sets be as follow:
               X = [1 2 3 3 4]';
               P1 = ones(5,1)/5;
               P2 = [0 0 .5 .2 .3]' + eps;
            Note that the event set here has duplicate values (two 3's). These will be treated as DISTINCT events by KLDIV. If you want these to be treated as the SAME event, you will need to collapse their probabilities together before running KLDIV. One way to do this is to use UNIQUE to find the set of unique events, and then iterate over that set, summing probabilities for each instance of each unique event. Here, we just leave the duplicate values to be treated independently (the default):
               KL = kldiv(X,P1,P2);
               KL =
                    19.4899

            Note also that we avoided the log-of-zero warning by adding 'eps' to all probability values in P2. We didn't need to renormalize because we're still within the sum-to-one tolerance.

            REFERENCES:
            1) Cover, T.M. and J.A. Thomas. "Elements of Information Theory," Wiley, 1991.
            2) Johnson, D.H. and S. Sinanovic. "Symmetrizing the Kullback-Leibler distance." IEEE Transactions on Information Theory (Submitted).
            3) Rubner, Y., Tomasi, C., and Guibas, L. J., 2000. "The Earth Mover's distance as a metric for image retrieval." International Journal of Computer Vision, 40(2): 99-121.
            4) <a href="
            http://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence"&gt;Kullback–Leibler divergence</a>. Wikipedia, The Free Encyclopedia.

            posted on 2010-10-16 15:04 Sosi 閱讀(10016) 評論(2)  編輯 收藏 引用 所屬分類: Taps in Research

            評論

            # re: Kullback&ndash;Leibler divergence KL散度 2010-11-30 16:17 tintin0324

            博主,本人的研究方向需要了解kl距離,有些問題想請教下,怎么聯(lián)系呢?
              回復(fù)  更多評論    

            # re: Kullback&ndash;Leibler divergence KL散度 2010-12-05 22:37 Sosi

            @tintin0324
            KL 距離本身很簡單,如果就是那樣子定義的,意義也如上面所說。。如果你想深入了解的話,可以讀以下相關(guān)文獻(xiàn)
              回復(fù)  更多評論    
            統(tǒng)計(jì)系統(tǒng)
            亚洲美日韩Av中文字幕无码久久久妻妇 | 亚洲国产成人精品91久久久 | 国产福利电影一区二区三区久久久久成人精品综合 | 久久99久久99精品免视看动漫| 久久精品中文騷妇女内射| 色成年激情久久综合| 亚洲国产精品嫩草影院久久 | 久久久久精品国产亚洲AV无码| 国内精品九九久久精品| 久久99中文字幕久久| 国产成人精品综合久久久久 | 国产高潮国产高潮久久久| 99久久国产综合精品成人影院| 久久天天婷婷五月俺也去| 亚洲国产成人久久综合碰碰动漫3d| 无码乱码观看精品久久| 蜜桃麻豆www久久| 久久精品麻豆日日躁夜夜躁| 无码任你躁久久久久久| 色综合合久久天天综合绕视看| 色综合久久久久无码专区| 久久青青草原精品国产软件 | 久久无码人妻精品一区二区三区 | 精品乱码久久久久久夜夜嗨| 亚洲国产精品无码久久一区二区| 久久久久久国产精品免费免费| 99久久精品毛片免费播放| 亚洲AV无码一区东京热久久| 久久精品国产色蜜蜜麻豆| 热久久国产欧美一区二区精品| 97精品伊人久久久大香线蕉 | 日本精品一区二区久久久| 大香网伊人久久综合网2020| 久久精品嫩草影院| 久久国产精品99国产精| 国内精品久久久久影院日本| 日本欧美久久久久免费播放网| 久久精品国产亚洲AV久| 亚洲精品乱码久久久久久按摩| 狠狠色婷婷久久综合频道日韩 | 久久久国产99久久国产一|