scrapy分布式爬虫-续

此教程是上个教程的后续，此教程中没有原理解释，只对上个教程中出现的错误进行修正

---

跟着上个教程走的同志们应该多少都会遇到这个警报信息。

经过百度，多方面讨论，询问老师，查询scrapy-redis的github项目帮助文档，甚至分析源码得出结论：

爬取不到信息跟这个无关，因为就算你爬取到了信息它还是会显示的有这个警报。

有的同学能爬取到信息，有的同学爬取不到，有的时能爬时不能爬，原因是：

豆瓣网的反爬虫机制发力了。

解决方案：换个网站爬

这里我找到了一个专门让你练习爬虫的网站，没有任何反爬限制，可以随便爬： http://books.toscrape.com/

那么开始构建scrapy分布式爬虫项目：

1.先创建项目：scrapy startproject book

2.创建实例爬虫文件：scrapy genspider books toscrape.com

记得先要 cd book 进入项目文件夹内

3.修改settings.py文件，不废话讲了，直接贴完整代码

# Scrapy settings for book project
#
# For simplicity, this file contains only settings considered important or
# commonly used. You can find more settings consulting the documentation:
#
#     https://docs.scrapy.org/en/latest/topics/settings.html
#     https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
#     https://docs.scrapy.org/en/latest/topics/spider-middleware.html

BOT_NAME = "book"

SPIDER_MODULES = ["book.spiders"]
NEWSPIDER_MODULE = "book.spiders"

# Crawl responsibly by identifying yourself (and your website) on the user-agent
#USER_AGENT = "book (+http://www.yourdomain.com)"

LOG_LEVEL = "WARNING"
# Obey robots.txt rules
ROBOTSTXT_OBEY = False

# Configure maximum concurrent requests performed by Scrapy (default: 16)
#CONCURRENT_REQUESTS = 32

# Configure a delay for requests for the same website (default: 0)
# See https://docs.scrapy.org/en/latest/topics/settings.html#download-delay
# See also autothrottle settings and docs
#DOWNLOAD_DELAY = 3
# The download delay setting will honor only one of:
#CONCURRENT_REQUESTS_PER_DOMAIN = 16
#CONCURRENT_REQUESTS_PER_IP = 16

# Disable cookies (enabled by default)
#COOKIES_ENABLED = False

# Disable Telnet Console (enabled by default)
#TELNETCONSOLE_ENABLED = False

# Override the default request headers:
#DEFAULT_REQUEST_HEADERS = {
#    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
#    "Accept-Language": "en",
#}

# Enable or disable spider middlewares
# See https://docs.scrapy.org/en/latest/topics/spider-middleware.html
#SPIDER_MIDDLEWARES = {
#    "book.middlewares.BookSpiderMiddleware": 543,
#}

# Enable or disable downloader middlewares
# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
#DOWNLOADER_MIDDLEWARES = {
#    "book.middlewares.BookDownloaderMiddleware": 543,
#}

# Enable or disable extensions
# See https://docs.scrapy.org/en/latest/topics/extensions.html
#EXTENSIONS = {
#    "scrapy.extensions.telnet.TelnetConsole": None,
#}

# Configure item pipelines
# See https://docs.scrapy.org/en/latest/topics/item-pipeline.html
ITEM_PIPELINES = {
   'scrapy_redis.pipelines.RedisPipeline': 301,
   "book.pipelines.BookPipeline": 300,
}

# redis相关配置
REDIS_HOST = '127.0.0.1'
REDIS_PORT = 6379
REDIS_DB = 4
# REDIS_PASSWORD = ''

# scrapy_redis相关配置
SCHEDULER = 'scrapy_redis.scheduler.Scheduler'
SCHEDULER_PERSIST = True  # 如果为真，则在关闭时自动保存请求信息
DUPEFILTER_CLASS = 'scrapy_redis.dupefilter.RFPDupeFilter'  # 去重的逻辑,要使用redis的

# Enable and configure the AutoThrottle extension (disabled by default)
# See https://docs.scrapy.org/en/latest/topics/autothrottle.html
#AUTOTHROTTLE_ENABLED = True
# The initial download delay
#AUTOTHROTTLE_START_DELAY = 5
# The maximum download delay to be set in case of high latencies
#AUTOTHROTTLE_MAX_DELAY = 60
# The average number of requests Scrapy should be sending in parallel to
# each remote server
#AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
# Enable showing throttling stats for every response received:
#AUTOTHROTTLE_DEBUG = False

# Enable and configure HTTP caching (disabled by default)
# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
#HTTPCACHE_ENABLED = True
#HTTPCACHE_EXPIRATION_SECS = 0
#HTTPCACHE_DIR = "httpcache"
#HTTPCACHE_IGNORE_HTTP_CODES = []
#HTTPCACHE_STORAGE = "scrapy.extensions.httpcache.FilesystemCacheStorage"

# Set settings whose default value is deprecated to a future-proof value
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"
FEED_EXPORT_ENCODING = "utf-8"

4.爬虫文件books.py的完整代码

import scrapy
from scrapy_redis.spiders import RedisSpider

class BooksSpider(RedisSpider):
    name = "books"
    allowed_domains = ["toscrape.com"]
    # start_urls = ["http://books.toscrape.com/"]
    redis_key = "books_start_url"  # 开始链接换成redis_key

    def parse(self, response, *args, **kwargs):
        next_page_url2 = response.xpath(
            '//*[@id="default"]/div/div/div/div/section/div[2]/div/ul/li[2]/a/@href').extract_first()
        next_page_url3 = response.xpath(
            '//*[@id="default"]/div/div/div/div/section/div[2]/div/ul/li[3]/a/@href').extract_first()
        if next_page_url2 is not None:
            yield scrapy.Request(url=response.urljoin(next_page_url2), callback=self.parse)
        elif next_page_url3 is not None:
            yield scrapy.Request(url=response.urljoin(next_page_url3), callback=self.parse)

        lis = response.xpath('//*[@id="default"]/div/div/div/div/section/div[2]/ol/li')
        for li in lis:
            href = li.xpath('./article/h3/a/@href').extract_first()
            yield scrapy.Request(url=response.urljoin(href), callback=self.parse_detail)

    def parse_detail(self, response, *args, **kwargs):
        name = response.xpath('//*[@id="content_inner"]/article/div[1]/div[2]/h1/text()').extract_first()
        jiage = response.xpath('//*[@id="content_inner"]/article/div[1]/div[2]/p[1]/text()').extract_first()

        dic = {
            "书籍名称": name,
            "书籍价格": jiage
        }
        yield dic

5.管道pipelines.py文件完整代码

# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html

# useful for handling different item types with a single interface
from itemadapter import ItemAdapter

class BookPipeline:
    def process_item(self, item, spider):
        print(item)
        return item

这次为了简便，item数据类型都不用了，查看这次的配置文件可以发现，这次没整反爬，请求头user-agent以及爬取延时都没做，对了，数据库换成了redis的4号数据库

多个终端进入项目文件夹后使用命令：

scrapy crawl books

启动项目

然后在数据库连接软件中切换到4号数据库，将初始网页链接添加进去

键：books_start_url

元素：http://books.toscrape.com/

然后可以在软件中查看爬取到的信息

轻轻松松的爬取到了全部的1000条数据（2分钟）

---

这次就应该没问题了。

有问题可以在下方评论区反馈，不用注册登录也能评论的。