久久久精彩视频,亚洲精品中文字幕女同,亚洲午夜女主播在线直播

NLP数据攉��

SunRise_at — Thu, 20 Sep 2012 09:29:00 GMT

目前�|�上可供下蝲�?a title="数据" target="_blank">数据众多�Q�但是内容庞杂，把其中比较有用的数据找了出来�?br />
wiki�p�：
wikipedia大家都不陌生�Q�它的下载地址是：http://dumps.wikimedia.org/ , �q�里有详�l�介�l�：http://en.wikipedia.org/wiki/Wikipedia:Database_download
但是wikipedia只是Wikimedia基金会的一个子��目�Q�wikimedia下面�q�有多个其他的重要项目，包括�Q?br />wiktionary 一�?a title="语义�? target="_blank">语义�?/a>的关联词典，形式上类��g��wordnet
wikiquote 收录各种名�h名言
Wikibooks 免费的教�U�书和手�?br />Wikinews 大量的新��L��?br />Wikiversity 免费的教育材�?br />Wikisource 免费的文本内�?br />上述的这些内容，都可以通过http://dumps.wikimedia.org/ 下蝲到�?br />�q�有一些小型的wiki��目�Q�比如：
http://simple.wikipedia.org   使用Basic English写的wiki�Q�给儿童和初学者看
http://simple.wiktionary.org   使用Basic English写的wiktionary

wikipedia的数据处理有很多方式�Q�我比较推崇�q�两个：
jwpl: http://code.google.com/p/jwpl/
wikipedia-miner:   http://wikipedia-miner.cms.waikato.ac.nz/wiki/

下面我介�l�下另一个商业化的wiki�|�站:http://www.wikia.com  �q�个�|�站�?a title="用户" target="_blank">用户可以创徏单独的维基网站，下面是排名前250位wikia�|�站�Q?br />http://wikis.wikia.com/wiki/List_of_Wikia_wikis
wikia上的资源也可供下载：http://community.wikia.com/wiki/Help:Database_download

Freebase:
freebase是啥��׃��解释了，下面�l�出数据的下载地址�Q?br />http://wiki.freebase.com/wiki/Data_dumps   freebase自��n的数�?br />http://wiki.freebase.com/wiki/WEX   freebase从wikipedia中提取的数据

YAGO2:
http://www.mpi-inf.mpg.de/yago-naga/yago/

dbpedia:
http://www.dbpedia.org

如果要找LinkedData�Q�可以来�q�里�Q?a rel="nofollow" target="_blank">http://www.thedatahub.org   �q�里攉��了很多Linked Data
http://linkeddata.org/   �q�里有一张图�Q�给��Z��各种linkeddata的关�p�d��影响力�?br />
如果要找各种�|�上的api�Q�可以来�q�里�Q?a rel="nofollow" target="_blank">http://www.programmableweb.com
现在外国政府�U�L��对外公开数据�Q�下面是几个政府的开放数据集�Q?br />http://data.gov.au   澛_��利亚
http://data.dc.gov   ��国哥��u比亚州的
http://www.data.gov   ��国
http://data.gov.uk   英国
http://databases.lapl.org/   �z�杉矶地区的开放数据集�Q�知道硅谷�ؓ啥这么牛了吧
http://www.gov.hk/en/theme/psi/welcome   香港政府也公开了很多数�?br />�Ҏ��一下，外国政府做了�q�么多实事，人民大会堂里的那些酒囊饭袋们都在�q�什么？

http://lexsrv3.nlm.nih.gov/LexSysGroup/Projects/lexAccess/current/web/download.html   ��国国家卫生�|�发布的词表
http://www.census.gov/genealogy/www/data/2000surnames/index.html   ��国�l�计局的姓名数�?br />https://www.cia.gov/library/publications/download/   ��国中央情报局发布的factbook�Q�介�l�了世界各国情况
�q�卫生��v�Q�统计局和中情局�q�种单位都�ؓ��国的信息徏讑ց��Z��q�么多的贡献�Q�我们应该知道自��p��帝的差距有多大了吧�?br />
叙词表：
http://www.nlm.nih.gov/mesh/filelist.html   mesh,关于��d��的受控词�?br />http://id.loc.gov/download/   ��国国会图书馆发布的叙词�?br />
一些三元组数据�Q?br />http://www.cs.utexas.edu/users/pclark/dart/   采集自BNC�Q�英国国家语料库�Q�和Reuters�Q?300万条
http://reverb.cs.washington.edu/   华盛��大学的��目�Q?500万条
http://www.cs.washington.edu/research/sherlock-hornclauses/   大约�?00-300万条数据
http://www.cs.rochester.edu/research/knext   �?35万条数据�Q�来自BNC和布朗语料库
http://rtw.ml.cmu.edu/rtw/resources   readtheweb��目�Q�数据量较小

��词典�Q?br />http://wordnet.princeton.edu/   ��p��的wordnet
http://nlpwww.nict.go.jp/wn-ja/index.en.html   日语的wordnet
http://alpage.inria.fr/~sagot/wolf-en.html   法语的wordnet
http://wordnet.ru/   俄罗斯的wordnet
http://cl.haifa.ac.il/projects/mwn/index.shtml   希伯来语的wordnet
http://wordnet.dk/dannet/menu?item=2   业w��语的wordnet
http://grial.uab.es/sensem/download?idioma=en   西班牙语的wordnet
http://www.ling.helsinki.fi/en/lt/research/finnwordnet/download.shtml   芬兰语的wordnet
�q�些不同版本的wordnet都是免费下蝲的。可恨中国泱�׃��千年的文明古国，文献典故��如烟�v�Q�竟�q�一份免费且公开的机读词兔R��没有。这是汉语的耻��i�Q�中国的耻��i�Q�也是中华民族的耻��i。特别是中科院计��所和自动化所的�h们，你们觉得呢？�Q�顺��hownet生意兴隆�Q�越卖越好）

http://dico.fj.free.fr/dico.php   日法词典
http://www.csse.monash.edu.au/~jwb/edict.html   日英词典
http://cc-cedict.org/wiki/start   中文到英文的词典�Q�终于出来中文的了，可惜是外国�h搞出来的�?br />https://framenet.icsi.berkeley.edu   ��Z��框架语义学的东东�Q�恐怕不能算词典�Q�不�q�没地儿放了�?br />
语料库：
http://opus.lingfil.uu.se/   开攄��q��语料�?br />http://opus.lingfil.uu.se/OpenSubtitles_v2.php   大量电媄字幕的下载地址
http://www.statmt.org/europarl   �Ƨ洲议会的��^行语料库
http://www.anc.org/OANC/   开攄��国国家语料�?br />
http://snap.stanford.edu/data/   斯坦��大学的SNAP��目�Q�抓了很多数据，不过旉��较早�Q�只有研�I��h�?/p>

SunRise_at 2012-09-20 17:29 发表评论

搭徏wiki镜像��结

SunRise_at — Tue, 21 Aug 2012 01:22:00 GMT

一.Apache�Q�Php5�Q�Mysql不可��，然后下蝲mediawiki软�g�?

之前没有接触�q�这些��Y�Ӟ��so每一个都需要装....

(1)apache配置

　在Debian下，安装完成后，软�g包�ؓ我们提供的配�|�文件位�?etc/apache2目录下：

　　tony@tonybox:/etc/apache2$ ls -l

　　total 72

　　-rw-r--r-- 1 root root 12482 2006-01-16 18:15 apache2.conf

　　drwxr-xr-x 2 root root 4096 2006-06-30 13:56 conf.d

　　-rw-r--r-- 1 root root 748 2006-01-16 18:05 envvars

　　-rw-r--r-- 1 root root 268 2006-06-30 13:56 httpd.conf

　　-rw-r--r-- 1 root root 12441 2006-01-16 18:15 magic

　　drwxr-xr-x 2 root root 4096 2006-06-30 13:56 mods-available

　　drwxr-xr-x 2 root root 4096 2006-06-30 13:56 mods-enabled

　　-rw-r--r-- 1 root root 10 2006-06-30 13:56 ports.conf

　　-rw-r--r-- 1 root root 2266 2006-01-16 18:15 README

　　drwxr-xr-x 2 root root 4096 2006-06-30 13:56 sites-available

　　drwxr-xr-x 2 root root 4096 2006-06-30 13:56 sites-enabled

　　drwxr-xr-x 2 root root 4096 2006-01-16 18:15

　　其中

　　apache2.conf

　　为apache2服务器的主配�|�文�Ӟ�� 查看此配�|�文�Ӟ�� 你会发现以下内容

　　# Include module configuration:

　　Include /etc/apache2/mods-enabled/*.load

　　Include /etc/apache2/mods-enabled/*.conf

　　# Include all the user configurations:

　　Include /etc/apache2/httpd.conf

　　# Include ports listing

　　Include /etc/apache2/ports.conf

　　# Include generic snippets of statements

　　Include /etc/apache2/conf.d/[^.#]*

　　有此可见�Q?apache2 �Ҏ��配置功能的不同，寚w��|�文件进行了分割�Q?�q�样更利于管�?/p>

　　conf.d

　　下�ؓ配置文�g的附加片断，默认情况下，仅提供了 charset 片断�Q?/p>

　　tony@tonybox:/etc/apache2/conf.d$ cat charset

　　AddDefaultCharset UTF-8

　　如有需要我们可以将默认�~�码修改�?GB2312, ��x��件的内容为： AddDefaultCharset GB2312

　　httpd.conf

　　是个�I�文�?/p>

　　magic

　　文�g中包含的是有关mod_mime_magic模块的数据，一般不需要修改它�?/p>

　　ports.conf

　　则�ؓ服务器监听IP和端口设�|�的配置文�g�Q?/p>

　　tony@tonybox:/etc/apache2$ cat ports.conf

　　Listen 80

　　mods-available

　　目录下是一些。conf和。load 文�g�Q?为系�l�中可以使用的加载各�U�模块的配置文�g�Q?而mods-enabled目录下则是指向这些配�|�文件的�W�号�q�接�Q?从配�|�文件apache2.conf 中可以看出， �pȝ��通过mods-enabled目录来加载模块，也就是说�Q?�pȝ��仅通过在此目录下创��Z��W�号�q�接的mods-available 目录下的配置文�g来加载模块。同时系�l�还提供了两个命�?a2enmod �?a2dismod用于�l�护�q�些�W�号�q�接。这两个命��o�?apache2-common 包提供。命令各式也非常��单： a2enmod [module] �?a2dismod [module]

　　sites-available

　　目录下�ؓ配置好的站点的配�|�文�Ӟ�� sites-enabled 目录下则是指向这些配�|�文件的�W�号�q�接�Q?�pȝ��通过�q�些�W�号�q�接来�v用站�?sites-enabled目录下的�W�号�q�接附有一个数字前�~��Q?�?00-default, �q�个数字用于军_��启动��序�Q?数字��小�Q?启动优先�U�越高�?�pȝ��提供了两个命�?a2ensite �?a2dissite 用于�l�护�q�些�W�号�q�接。这两个命��o�?apache2-common 包提供�?/p>

　　/var/www

　　默认情况下将要发布的�|�页文�g应该�|�于/var/www目录下，�q�一默认值可以同�q�主配置文�g中的DocumnetRoot 选项修改�?/p>

　　�?mediawiki直接解压到apache里面(��是解压在var/www路径�?,解压后重名�ؓwiki�Q?/p>

�? 然后�q�主��localhost/wiki�Q�对MediaWiki�q�行安装。去创徏数据库wikidb。里面有41个表。在导入数据之间�Q�要先清除page,revision,text三个表�?/p>

delete from page;

delete from revision;

delete from text;

�?http://dumps.wikimedia.org/backup-index.html 在这里可以下载�Q何语�a�wiki 的数据库xml 文�g。下载的文�g�c�M��于enwiki-20061130-pages-articles.xml.bz2�Q�英文版的）�Q�wiki差不多每两个月更��C��ơ数据�?/p>

�?安装mediawiki。去下蝲mediawiki的源代码�Q�如果其官方�|�站被封的话可以去www.allwiki.com�q�个中文�|�站上去下蝲。下载后解压��C��的apache能找到的一个目录下�Q�将其config目录权限讄��?77�Q�然后在��览器里讉K��?config/index.php�Q�进行一些配�|�后�Q�会在config目录下生成一个LocalSettings.php的文�Ӟ��这个文件拷贝到它的上一�U�目录。最后别忘了��config的目录再改回原来的权限�?/p>

�?把文件导入数据库�Q?nbsp;
命��o�Q?nbsp;
java -Xmx600M -server -jar mwdumper.jar --format=sql:1.5
enwiki-20061130-pages-articles.xml.bz2 | mysql -u wikiuser -p wikidb

参见�Q?a >http://fuhao-987.iteye.com/blog/1044933

http://jgs80.blog.163.com/blog/static/3566265320076177435762/

SunRise_at 2012-08-21 09:22 发表评论

Penn Treebank Tags

SunRise_at — Tue, 31 Jul 2012 05:31:00 GMT

Note: This information comes from "Bracketing Guidelines for Treebank II Style Penn Treebank Project" - part of the documentation that comes with the Penn Treebank.

Contents:

Bracket Labels

Clause Level

Phrase Level

Word Level

Function Tags

Form/function discrepancies

Grammatical role

Adverbials

Miscellaneous

Index of All Tags

Bracket Labels

Clause Level

S - simple declarative clause, i.e. one that is not introduced by a (possible empty) subordinating conjunction or a wh-word and that does not exhibit subject-verb inversion.

SBAR - Clause introduced by a (possibly empty) subordinating conjunction.

SBARQ - Direct question introduced by a wh-word or a wh-phrase. Indirect questions and relative clauses should be bracketed as SBAR, not SBARQ.

SINV - Inverted declarative sentence, i.e. one in which the subject follows the tensed verb or modal.

SQ - Inverted yes/no question, or main clause of a wh-question, following the wh-phrase in SBARQ.

Phrase Level

ADJP - Adjective Phrase.

ADVP - Adverb Phrase.

CONJP - Conjunction Phrase.

FRAG - Fragment.

INTJ - Interjection. Corresponds approximately to the part-of-speech tag UH.

LST - List marker. Includes surrounding punctuation.

NAC - Not a Constituent; used to show the scope of certain prenominal modifiers within an NP.

NP - Noun Phrase.

NX - Used within certain complex NPs to mark the head of the NP. Corresponds very roughly to N-bar level but used quite differently.

PP - Prepositional Phrase.

PRN - Parenthetical.

PRT - Particle. Category for words that should be tagged RP.

QP - Quantifier Phrase (i.e. complex measure/amount phrase); used within NP.

RRC - Reduced Relative Clause.

UCP - Unlike Coordinated Phrase.

VP - Vereb Phrase.

WHADJP - Wh-adjective Phrase. Adjectival phrase containing a wh-adverb, as in how hot.

WHAVP - Wh-adverb Phrase. Introduces a clause with an NP gap. May be null (containing the 0 complementizer) or lexical, containing a wh-adverb such as how or why.

WHNP - Wh-noun Phrase. Introduces a clause with an NP gap. May be null (containing the 0 complementizer) or lexical, containing some wh-word, e.g. who, which book, whose daughter, none of which, or how many leopards.

WHPP - Wh-prepositional Phrase. Prepositional phrase containing a wh-noun phrase (such as of which or by whose authority) that either introduces a PP gap or is contained by a WHNP.

X - Unknown, uncertain, or unbracketable. X is often used for bracketing typos and in bracketing the...the-constructions.

Word level

CC - Coordinating conjunction

CD - Cardinal number

DT - Determiner

EX - Existential there

FW - Foreign word

IN - Preposition or subordinating conjunction

JJ - Adjective

JJR - Adjective, comparative

JJS - Adjective, superlative

LS - List item marker

MD - Modal

NN - Noun, singular or mass

NNS - Noun, plural

NNP - Proper noun, singular

NNPS - Proper noun, plural

PDT - Predeterminer

POS - Possessive ending

PRP - Personal pronoun

PRP$ - Possessive pronoun (prolog version PRP-S)

RB - Adverb

RBR - Adverb, comparative

RBS - Adverb, superlative

RP - Particle

SYM - Symbol

TO - to

UH - Interjection

VB - Verb, base form

VBD - Verb, past tense

VBG - Verb, gerund or present participle

VBN - Verb, past participle

VBP - Verb, non-3rd person singular present

VBZ - Verb, 3rd person singular present

WDT - Wh-determiner

WP - Wh-pronoun

WP$ - Possessive wh-pronoun (prolog version WP-S)

WRB - Wh-adverb

Function tags

Form/function discrepancies

-ADV (adverbial) - marks a constituent other than ADVP or PP when it is used adverbially (e.g. NPs or free ("headless" relatives). However, constituents that themselves are modifying an ADVP generally do not get -ADV. If a more specific tag is available (for example, -TMP) then it is used alone and -ADV is implied. See the Adverbials section.

-NOM (nominal) - marks free ("headless") relatives and gerunds when they act nominally.

Grammatical role

-DTV (dative) - marks the dative object in the unshifted form of the double object construction. If the preposition introducing the "dative" object is for, it is considered benefactive (-BNF). -DTV (and -BNF) is only used after verbs that can undergo dative shift.

-LGS (logical subject) - is used to mark the logical subject in passives. It attaches to the NP object of by and not to the PP node itself.

-PRD (predicate) - marks any predicate that is not VP. In the do so construction, the so is annotated as a predicate.

-PUT - marks the locative complement of put.

-SBJ (surface subject) - marks the structural surface subject of both matrix and embedded clauses, including those with null subjects.

-TPC ("topicalized") - marks elements that appear before the subject in a declarative sentence, but in two cases only:

if the front element is associated with a *T* in the position of the gap.

if the fronted element is left-dislocated (i.e. it is associated with a resumptive pronoun in the position of the gap).

-VOC (vocative) - marks nouns of address, regardless of their position in the sentence. It is not coindexed to the subject and not get -TPC when it is sentence-initial.

Adverbials

Adverbials are generally VP adjuncts.

-BNF (benefactive) - marks the beneficiary of an action (attaches to NP or PP).

This tag is used only when (1) the verb can undergo dative shift and (2) the prepositional variant (with the same meaning) uses for. The prepositional objects of dative-shifting verbs with other prepositions than for (such as to or of) are annotated -DTV.

-DIR (direction) - marks adverbials that answer the questions "from where?" and "to where?" It implies motion, which can be metaphorical as in "...rose 5 pts. to 57-1/2" or "increased 70% to 5.8 billion yen" -DIR is most often used with verbs of motion/transit and financial verbs.

-EXT (extent) - marks adverbial phrases that describe the spatial extent of an activity. -EXT was incorporated primarily for cases of movement in financial space, but is also used in analogous situations elsewhere. Obligatory complements do not receive -EXT. Words such as fully and completely are absolutes and do not receive -EXT.

-LOC (locative) - marks adverbials that indicate place/setting of the event. -LOC may also indicate metaphorical location. There is likely to be some varation in the use of -LOC due to differing annotator interpretations. In cases where the annotator is faced with a choice between -LOC or -TMP, the default is -LOC. In cases involving SBAR, SBAR should not receive -LOC. -LOC has some uses that are not adverbial, such as with place names that are adjoined to other NPs and NAC-LOC premodifiers of NPs. The special tag -PUT is used for the locative argument of put.

-MNR (manner) - marks adverbials that indicate manner, including instrument phrases.

-PRP (purpose or reason) - marks purpose or reason clauses and PPs.

-TMP (temporal) - marks temporal or aspectual adverbials that answer the questions when, how often, or how long. It has some uses that are not strictly adverbial, auch as with dates that modify other NPs at S- or VP-level. In cases of apposition involving SBAR, the SBAR should not be labeled -TMP. Only in "financialspeak," and only when the dominating PP is a PP-DIR, may temporal modifiers be put at PP object level. Note that -TMP is not used in possessive phrases.

Miscellaneous

-CLR (closely related) - marks constituents that occupy some middle ground between arguments and adjunct of the verb phrase. These roughly correspond to "predication adjuncts", prepositional ditransitives, and some "phrasel verbs". Although constituents marked with -CLR are not strictly speaking complements, they are treated as complements whenever it makes a bracketing difference. The precise meaning of -CLR depends somewhat on the category of the phrase.

on S or SBAR - These categories are usually arguments, so the -CLR tag indicates that the clause is more adverbial than normal clausal arguments. The most common case is the infinitival semi-complement of use, but there are a variety of other cases.

on PP, ADVP, SBAR-PRP, etc - On categories that are ordinarily interpreted as (adjunct) adverbials, -CLR indicates a somewhat closer relationship to the verb. For example:

Prepositional Ditransitives

In order to ensure consistency, the Treebank recognizes only a limited class of verbs that take more than one complement (-DTV and -PUT and Small Clauses) Verbs that fall outside these classes (including most of the prepositional ditransitive verbs in class [D2]) are often associated with -CLR.

Phrasal verbs

Phrasal verbs are also annotated with -CLR or a combination of -PRT and PP-CLR. Words that are considered borderline between particle and adverb are often bracketed with ADVP-CLR.

Predication Adjuncts

Many of Quirk's predication adjuncts are annotated with -CLR.

on NP - To the extent that -CLR is used on NPs, it indicates that the NP is part of some kind of "fixed phrase" or expression, such as take care of. Variation is more likely for NPs than for other uses of -CLR.

-CLF (cleft) - marks it-clefts ("true clefts") and may be added to the labels S, SINV, or SQ.

-HLN (headline) - marks headlines and datelines. Note that headlines and datelines always constitute a unit of text that is structurally independent from the following sentence.

-TTL (title) - is attached to the top node of a title when this title appears inside running text. -TTL implies -NOM. The internal structure of the title is bracketed as usual.

Index of All Tags

ADJP

-ADV

ADVP

-BNF

-CLF

-CLR

CONJP

-DIR

-DTV

-EXT

FRAG

-HLN

INTJ

JJR

JJS

-LGS

-LOC

LST

-MNR

NAC

NNS

NNP

NNPS

-NOM

PDT

POS

-PRD

PRN

PRP

-PRP

PRP$ or PRP-S

PRT

-PUT

RBR

RBS

RRC

SBAR

SBARQ

-SBJ

SINV

SYM

-TMP

-TPC

-TTL

UCP

VBD

VBG

VBN

VBP

VBZ

-VOC

WDT

WHADJP

WHADVP

WHNP

WHPP

WP$ or WP-S

WRB

SunRise_at 2012-07-31 13:31 发表评论

SunRise_at — Mon, 23 Jul 2012 01:41:00 GMT

转自�Q?a target="_blank">http://uwei.blogbus.com/logs/11424864.html
外行人做互联�|�，很多概念不懂。就拿最基础�?#8220;召回�?#8221;�?#8220;准确�?#8221;�q�种概念�Q�看看网上资料知道大概，自己用的时候，脑子里绕着弯儿能想明白�Q�可��到别�h�zȝ��的时候，脑子里还是没法一下子反应�q�来�Q�还是要�l�弯想一下。特地找了些资料�Q�将�q�两个概忉|��理一下，希望能更熟练�?

召回率和准确率是搜烦引擎�Q�或其它��索系�l�）的设计中很重要的两个概念和指标�?br />召回率：Recall�Q�又�U?#8220;查全�?#8221;�Q?
准确率：Precision�Q�又�U?#8220;�_�ֺ�”�?#8220;正确�?#8221;�?br />在一个大规模数据集合中检索文档时�Q�可把集合中的所有文档分成四�c�：

	相关	不相�?/div>
��索到	A	B
未检索到	C	D

A�Q�检索到的，相关�?nbsp; �Q�搜到的也想要的�Q?br />B�Q�检索到的，但是不相关的 �Q�搜到的但没用的�Q?br />C�Q�未��索到的，但却是相关的 �Q�没搜到�Q�然而实际上惌��的）
D�Q�未��索到的，也不相关�?nbsp; �Q�没搜到也没用的�Q?/p>

通常我们希望�Q�数据库中相关的文档�Q�被��索到的越多越好，�q�是�q�求“查全�?#8221;�Q�即A/(A+C)�Q�越大越好�?br />同时我们�q�希望：��索到的文档中�Q�相关的��多��好�Q�不相关的越��越好，�q�是�q�求“准确�?#8221;�Q�即A/(A+B)�Q�越大越好�?br />
归纳如下�Q?br />召回率：��索到的相��x��?�?库中所有的相关文档
准确率：��索到的相��x��?�?所有被��索到的文�?br />
“召回�?#8221;�?#8220;准确�?#8221;虽然没有必然的关�p�（从上面公式中可以看到�Q�，然而在大规模数据集合中�Q�这两个指标却是�怺�制约的�?br />�׃��“��索策�?#8221;�q�不完美�Q�希望更多相关的文档被检索到�Ӟ��攑֮�“��索策�?#8221;�Ӟ��往往也会伴随出现一些不相关的结果，从而��准确率受到媄响�?br />而希望去除检索结果中的不相关文档�Ӟ��务必要将“��索策�?#8221;定的更加严格�Q�这样也会��有一些相关的文档不再能被��索到�Q�从而��召回率受到媄响�?/p>

凡是设计到大规模数据集合的检索和选取�Q�都涉及�?#8220;召回�?#8221;�?#8220;准确�?#8221;�q�两个指标。而由于两个指标相互制�U�，我们通常也会�Ҏ��需要�ؓ“��索策�?#8221;选择一个合适的度，不能太严��g��不能太松�Q�寻求在召回率和准确率中间的一个��^衡点。这个��^衡点由具体需求决定�?/p>

其实�Q�准��率�Q�precision�Q�精度）比较好理解。往往难以�q�速反应的�?#8220;召回�?#8221;。我惌��与字面意思也有关�p�，�?#8220;召回”的字面意思不能直接看到其意义�?br />我觉�?#8220;召回�?#8221;�q�个词翻译的不够好�?#8220;召回”在中文的意思是�Q�把xx调回来。比如sony甉|��有问题，厂家召回�?br />既然说翻译的不好�Q�我们回头看“召回�?#8221;对应的英�?#8220;recall”�Q�recall除了有上面说到的“order sth to return”的意思之外，�q�有“remember”的意思�?/p>

Recall�Q�the ability to remember sth. that you have learned or sth. that has happened in the past.

�q�里�Q�recall应该是这个意思，�q�样��更�Ҏ��理解“召回�?#8221;的意思了�?br />当我们问��索系�l�某一件事的所有细节时�Q�输入检索query�Q�，Recall��是指：��索系�l�能“回忆”起那些事的多��细节，通俗来讲��是“回忆的能�?#8221;。能回忆��h��的细节数除以 �pȝ��知道�q��g事的所有细节，��是“记忆�?#8221;�Q�也��是recall——召回率�?br />
�q�样惻I��要容易的多了�?/p>

SunRise_at 2012-07-23 09:41 发表评论

《数学之��》－�Q�马��可夫链

SunRise_at — Thu, 12 Jul 2012 00:54:00 GMT

最�q�在�ȝ��一本书《数学之��》，�׃��自己寚w��可夫链�~�Z��相关的知识背景，故学习了一下。对于N久没有看�q�概率论的�h来说�Q�重拾�v来也��p��了一�Ҏ��间。比如：P(A|B)是指在B的条件下A的概率，诸如此类�Q�都需要重新复习一下，正所谓温故而知新。知道这个了�Q�也��׃��隄��解马��可夫链的性质�Q�即每一步可以移动到��M��一个相�ȝ��点，在这里移动到每一个点的概率都是相同的�?br />

关于马尔可夫铄��定义�Q?nbsp;http://zh.wikipedia.org/wiki/%E9%A6%AC%E5%8F%AF%E5%A4%AB%E9%8F%88

隐含马尔可夫模型是上�q�马��可夫链的一个扩展：��M��一个时刻t的状态St是不可见的。隐含马��可夫模型在每一个时刻t会输��Z��个符��P��而且�q�个�W�合和st相关�Q�而且仅和st相关�Q�这个被�U�Cؓ独立输出假设。关于隐含马��可夫模型的成功应用可以参见吴军的《数学之��》第5章的内容�?br /> 额，快到上班旉��了，��ȝ��一下。��l�码农中......

SunRise_at 2012-07-12 08:54 发表评论

SunRise_at — Fri, 01 Jun 2012 05:06:00 GMT

今天六一�Q?font face="Times New Roman">C��加不在�w�边�Q��؜球啊。�Q务需要在看曼宁的《统计自然语�a�处理基础》。然后用��C��信息�Q�每�ơ我觉得好高��q��名字�Q�做下去的时候就发现没有那么难�?/font>

搭配

搭配由有限的复合构词法所描述�?/span>

识别搭配对的�Ҏ��有三�U�：1.使用频率信息的搭配识别�?/font>2.��Z��含义和主词搭配词之间的距��识别�?/font>3.��Z��假设��试和互信息的识别�?/font>

1.频率

��语料过滤后得到的动词，名词�Q�之间进行两两配对，�l�计每个词语在一个句子，或在一个段落中出现的次敎ͼ�即�ؓ频率�?/span>

2.均值和方差

�׃��两个词之间的距离是可以变化的�Q�计��两个词之间的偏�U�量的均值和方差�?/span>

均值就是简单的�q�_��偏移量�?/span>

方差衡量的是单独的偏�U�量偏离均值的距离�Q?/span>

是同�?font face="Times New Roman">i的偏�U�量�Q?/font>表示的是��h��偏移量的均倹{�?nbsp;

我们可以通过使用�q�个信息来发现搭配。具体的�Ҏ��是通过��L��带有低偏差的词对。一个低的偏差值意味着�q�两个词通常大致相同距离出现。零偏差意味着�q�两个词��L��以相同的距离出现�?/span>

方差是关于一个相对于其他词分布峰值情�늚�度量�?/span>

关于互信�?/span>

互信息的计算公式是这��L��Q?/span>

MI(a,b) = log( p(ab) / (p(a)*p(b)) )

其中log的底数是2�Q?/font>p(x)表示x出现的概率�?/font>

好吧�Q�好��_��好简单。。着手写代码了�?/span>

SunRise_at 2012-06-01 13:06 发表评论

��Z��HowNet语义�怼�度的�Q�Ａ�Q�q��研究

SunRise_at — Thu, 12 Apr 2012 08:19:00 GMT

    �l�徏立同义词库后的新��d��Q�读文献�Q�然后找出问题的解决�Ҏ��。几��文献都是研�I�句子与句子的相似度计算�Q�我们的关键是词语与句子的相似度计算。据说FAQ是自然语�a�处理领域研究的热炏V��看了几��论文，感觉都是大同��异�?br />   因�ؓ是第一�ơ接触这些东西，所以有很多陌生的词汇，��p��己动手查了查�?br />   关于HowNet,�?a >http://www.keenage.com/zhiwang/c_zhiwang.html
   FAQ自动问答�pȝ��的核心问题是如何快速地��客��h��提问题与FAQ数据库的问题比较�Q�进而确定与其最�怼�的问题，如果有，则将对应的答案作为结果回复给客户�?br />

                                                                                    FAQ�pȝ��l�构�?br />     �怼�度流�E�的计算��是先计��义原相似度�Q�然后是概念�怼�度，接着词语�怼�度，最后就是句子相似度�?br />     /Files/sunrise/�怼��?doc�q�里公式不能昄��q��似度计算��插在附件中了�?br />     FAQ差不多就�q�行到这里了。程序小白的��白文章�Q�小白将�l�箋��白下去�?/p>

SunRise_at 2012-04-12 16:19 发表评论

SunRise_at — Wed, 28 Mar 2012 01:35:00 GMT

特别推荐�Q?br />1�?a target="_blank" style="color: #ca0000; text-decoration: none; ">HMM学习最佌��?/a>全文文档
2�?a target="_blank" style="color: #ca0000; text-decoration: none; ">无约束最优化全文文档

一、书�c�：
1�?a target="_blank" style="color: #ca0000; text-decoration: none; ">《自然语�a�处理�l�D��》英文版�W�二�?/a>
2�?a target="_blank" style="color: #ca0000; text-decoration: none; ">《统计自然语�a�处理基础》英文版
3�?a target="_blank" style="color: #ca0000; text-decoration: none; ">《用Python�q�行自然语言处理》，NLTK配套�?/a>
4�?a target="_blank" style="color: #ca0000; text-decoration: none; ">《Learning Python�W�三版�?/a>�Q�Python入门�l�典书籍�Q�详�l�而不厌其�?br />5�?a target="_blank" style="color: #ca0000; text-decoration: none; ">《自然语�a�处理中的模式识别�?/a>
6�?a target="_blank" style="color: #ca0000; text-decoration: none; ">《EM��法及其扩展�?/a>
7�?a target="_blank" style="color: #ca0000; text-decoration: none; ">《统计学习基��?/a>
8、�?a target="_blank" style="color: #ca0000; text-decoration: none; ">自然语言理解》英文版�Q�似乎只有前9章）
9�?a target="_blank" style="color: #ca0000; text-decoration: none; ">《Fundamentals of Speech Recognition�?/a>�Q�质量不太好�Q�不�q�第6章关于HMM的部分比较详�l�，作者之一便是Lawrence Rabiner�Q?br />10、概率统计经典入门书�Q�《概率论及其应用》（英文版，威廉*费勒著）
　　�W�一�?/a>　　�W�二�?/a>　　DjVuLibre阅读�?/a>�Q�阅��d��两卷书需要）
11、一本利用Perl和Prolog�q�行自然语言处理的介�l�书�c�：�?a target="_blank" style="color: #ca0000; text-decoration: none; ">An Introduction to Language Processing with Perl and Prolog�?br />12、国外机器学习书�c�之�Q?br />　1) “Programming Collective Intelligence“�Q�中文译名《集体智慧编�E�》，机器学习&数据挖掘领域”�q�年出的入门好书�Q�培��d��是最重要的一环，一上来看大部头很容易被吓走�?#8221;
　2) “Machine Learning“,机器学习领域无可争议的经�怹��c�，下蝲完毕��后�~�改�ؓpdf卛_��。豆瓣评�?by王宁�Q�：老书�Q�牛人。现在看来内容�ƈ不算深，很多章节有点��Cؓ止的感觉�Q�但是很适合新手�Q�当�Ӟ��不能”�?#8221;到连��法和概率都不知道）入门。比如决�{�树部分��很�_�ֽ��Q��ƈ且这几年没有特别大的�q�展�Q�所以�ƈ不过时。另外，�q�本书算是对97�q�前数十�q�机器学习工作的大综�q�ͼ�参考文献列表极有�h倹{��国内有��译和媄印版�Q�不知道�l�版否�?br />　3) “Introduction to Machine Learning”
13、国外数据挖掘书�c�之�Q?br />　1) “Data.Mining.Concepts.and.Techniques.2nd“�Q�数据挖掘经�怹��c?作�?: Jiawei Han/Micheline Kamber 出版�C?: Morgan Kaufmann 评语 : 华裔�U�学家写的书�Q�相当深入浅出�?br />　2) Data Mining:Practical Machine Learning Tools and Techniques
　3) Beautiful Data: The Stories Behind Elegant Data Solutions�Q?Toby Segaran, Jeff Hammerbacher�Q?br />14、国外模式识别书�c�之�Q?br />　1�Q?#8220;Pattern Recognition”
　2�Q?#8220;Pattern Recongnition Technologies and Applications”
　3�Q?#8220;An Introduction to Pattern Recognition”
　4�Q?#8220;Introduction to Statistical Pattern Recognition”
　5�Q?#8220;Statistical Pattern Recognition 2nd Edition”
　6�Q?#8220;Supervised and Unsupervised Pattern Recognition”
　7�Q?#8220;Support Vector Machines for Pattern Classification”
15、国外�h工智能书�c�之�Q?br />　1�Q?a target="_blank" style="color: #ca0000; text-decoration: none; ">Artificial Intelligence: A Modern Approach (2nd Edition) 人工��领域无争议的�l�典�?br />　2�Q?#8220;Paradigms of Artificial Intelligence Programming: Case Studies in Common LISP”
16、其他相关书�c�：
　1�Q?a target="_blank" style="color: #ca0000; text-decoration: none; ">Programming the Semantic Web�Q�Toby Segaran , Colin Evans, Jamie Taylor
　2�Q?a target="_blank" style="color: #ca0000; text-decoration: none; ">Learning.Python�W�四�?/a>�Q�英�?/p>

二、课�Ӟ��
1、哈工大刘挺老师�?#8220;�l�计自然语言处理”课�g�Q?br />2、哈工大刘秉权老师�?#8220;自然语言处理”课�g�Q?br />3、中�U�院计算所刘群老师�?#8220;计算语言学讲�?/a>“课�g�Q?br />4、中�U�院自动化所宗成庆老师�?#8220;自然语言理解”课�g�Q?br />5、北大常宝宝老师�?#8220;计算语言�?/a>”课�g�Q?br />6、北大詹卫东老师�?#8220;中文信息处理基础”的课件及相关代码�Q?br />7、MIT Regina Barzilay教授�?#8220;自然语言处理”课�g�Q?2nlp上翻译了�?章；
8、MIT大牛Michael Collins�?#8220;Machine Learning Approaches for Natural Language Processing(面向自然语言处理的机器学习方�?”课�g�Q?br />9、Michael Collins�?#8220;Machine Learning �Q�机器学习）”课�g�Q?br />10、SMT牛�hPhilipp Koehn “Advanced Natural Language Processing�Q�高�U�自然语�a�处理�Q?#8221;课�g�Q?br />11、Philipp Koehn “Empirical Methods in Natural Language Processing”课�g�Q?br />12、Philipp Koehn“Machine Translation�Q�机器翻译）”课�g�Q?/p>

三、语�a�资源和开源工��P��
1、Brown语料库：
　a) XML格式的brown语料�?/a>�Q�带词性标注；
　b) 普通文本格式的brown语料�?/a>�Q�带词性标注；
　c) 合�ƈ�q�去除空行、行首空��|��用于词性标注训�l�：browntest.zip
2�?a target="_blank" style="color: #ca0000; text-decoration: none; ">NLTK官方提供的语料库资源列表
3�?a target="_blank" style="color: #ca0000; text-decoration: none; ">OpenNLP上的开源自然语�a�处理工具列表
4、斯坦福大学自然语言处理�l�维护的“�l�计自然语言处理及基于语料库的计��语�a�学资源列�?/a>”
5�?a target="_blank" style="color: #ca0000; text-decoration: none; ">LDC上免费的中文信息处理资源
6、中文分词相兛_��P��
　1�Q�Java版本的MMSEG�Q?a target="_blank" style="color: #ca0000; text-decoration: none; ">mmseg-v0.3.zip�Q�作者�ؓsolol�Q�详情可参见�Q��?a target="_blank" style="color: #ca0000; text-decoration: none; ">中文分词入门之篇�?/a>�?br />　2�Q�张华��^老师的ICTCLAS2010�Q�该版本非商用免费一�q�_��下蝲地址�Q?br />http://cid-51de2738d3ea0fdd.skydrive.live.com/self.aspx/.Public/ICTCLAS2010-packet-release.rar
7、热心读�?#8220;finallyliuyu”提供的一�Ҏ��闻语料库�Q�包括腾讯，新浪�Q�网易，凤凰�{�，目前攑֜�CSDN上：http://finallyliuyu.download.csdn.net/
　　另外finalllyliuyu�?010�q?月又提供了一�Ҏ��本文�c�语料，详情见：献给热衷于自然语�a�处理的业余爱好者的中文新闻分类语料库之�?/a>

四、文献：
1、ACL-IJCNLP 2009论文全集�Q?br />　a) 大会论文Full Paper�W�一�?/a>
　b) 大会论文Full Paper�W�二�?/a>
　c) 大会论文Short Paper合集
　d) ACL09之EMNLP-2009合集
　e) ACL09 所有workshop论文合集

SunRise_at 2012-03-28 09:35 发表评论

久久久精彩视频,亚洲精品中文字幕女同,亚洲午夜女主播在线直播

NLP数据攉���

搭徏wiki镜像���结

Penn Treebank Tags

《数学之���》－�Q�马���可夫链

��Z��HowNet语义�怼�度的�Q�Ａ�Q�q��研究

NLP数据攉��

搭徏wiki镜像��结

《数学之��》－�Q�马��可夫链