chatler — Thu, 18 Feb 2010 13:54:00 GMT

�q�里是维基百�U�对�|�络爬虫的词条页面。网�l�爬虫以叫网�l�蜘蛛，�|�络机器人，�q�是一个程序，其会(x��)自动的通过�|�络抓取互联�|�上的网��，�q�种技术一般可能用来检查你的站点上所有的链接是否是都是有效的。当�?d��ng)��更��?f��)高��的技术是把网��中的相��x��据保存下来，可以成�ؓ(f��)搜烦(ch��)引擎�?/p>

从技相来��_(d��)��实现抓取�|�页可能�q�不是一件很困难的事情，困难的事情是对网��늚�分析和整理，那是一仉��要有轻量��Q�需要大量数学计��的�E�序才能做的事情。下面一个简单的��程�Q?/p>

在这里，我们只是说一下如何写一个网��|��取程序�?/p>

首先我们先看一下，如何使用命��o(h��)行的方式来找开�|�页�?/p>

telnet somesite.com 80
GET /index.html HTTP/1.0
按回车两��?/p>

使用telnet��是告诉你其实这是一个socket的技术，�q�且使用HTTP的协议，�?GET�Ҏ(gu��)��来获得网��，当然�Q�接下来的事你就需要解析HTML文法�Q�甚臌��需要解析Javascript�Q�因为现在的�|�页使用Ajax的越来越多了�Q�而很多网��内定w��是通过Ajax技术加载的�Q�因为，只是��单地解析HTML文�g在未来会(x��)�q�远不够。当�?d��ng)��在这里，只是展示一个非常简单的抓取�Q�简单到只能做�ؓ(f��)一个例子，下面�q�个�C�Z��的伪代码�Q?/p>

取网��?
for each 链接 in 当前�|�页所有的链接
{
if(如果本链接是我们惌����?|| �q�个链接从未讉K���q?
{
处理�Ҏ(gu��)��链接
把本链接讄���为已讉K��
}
}

require “rubygems”
require “mechanize”
class Crawler < WWW::Mechanize
attr_accessor :callback
INDEX = 0
DOWNLOAD = 1
PASS = 2
def initialize
super
init
@first = true
self.user_agent_alias = “Windows IE 6″
end
def init
@visited = []
end
def remember(link)
@visited << link
end
def perform_index(link)
self.get(link)
if(self.page.class.to_s == “WWW::Mechanize::Page”)
links = self.page.links.map {|link| link.href } - @visited
links.each do |alink|
start(alink)
end
end
end
def start(link)
return if link.nil?
if(!@visited.include?(link))
action = @callback.call(link)
if(@first)
@first = false
perform_index(link)
end
case action
when INDEX
perform_index(link)
when DOWNLOAD
self.get(link).save_as(File.basename(link))
when PASS
puts “passing on #{link}”
end
end
end
def get(site)
begin
puts “getting #{site}”
@visited << site
super(site)
rescue
puts “error getting #{site}”
end
end
end

上面的代码就不必多说了，大家可以去试试。下面是如何使用上面的代码：(x��)

require “crawler”
x = Crawler.new
callback = lambda do |link|
if(link =~/\\.(zip|rar|gz|pdf|doc)
x.remember(link)
return Crawler::PASS
elsif(link =~/\\.(jpg|jpeg)/)
return Crawler::DOWNLOAD
end
return Crawler::INDEX;
end
x.callback = callback
x.start(”http://somesite.com”)

下面是一些和�|�络爬虫相关的开源网�l�项�?/p>

arachnode.net is a .NET crawler written in C# using SQL 2005 and Lucene and is released under the GNU General Public License.
DataparkSearch is a crawler and search engine released under the GNU General Public License.
GNU Wget is a command-line-operated crawler written in C and released under the GPL. It is typically used to mirror Web and FTP sites.
GRUB is an open source distributed search crawler that Wikia Search ( http://wikiasearch.com ) uses to crawl the web.
Heritrix is the Internet Archive’s archival-quality crawler, designed for archiving periodic snapshots of a large portion of the Web. It was written in Java.
ht://Dig includes a Web crawler in its indexing engine.
HTTrack uses a Web crawler to create a mirror of a web site for off-line viewing. It is written in C and released under the GPL.
ICDL Crawler is a cross-platform web crawler written in C++ and intended to crawl Web sites based on

from:
http://coolshell.cn/?p=27

chatler 2010-02-18 21:54 发表评论

亚洲人成人一区二区在线观看,欧美日韩无遮挡,欧美激情视频在线播放