账号密码登录
微信安全登录
微信扫描二维码登录

登录后绑定QQ、微信即可实现信息互通

手机验证码登录
找回密码返回
邮箱找回 手机找回
注册账号返回
其他登录方式
分享
  • 收藏
    X
    scrapy 在爬取部分内容后自动关闭,为什么?
    34
    0
    2018-09-19 11:58:25 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.zhihu.com/question/265749263>; (referer: https://www.zhihu.com/question/265749263/answer/298529974)
    2018-09-19 11:58:25 [scrapy.extensions.logstats] INFO: Crawled 20 pages (at 5 pages/min), scraped 10 items (at 5 items/min)
    2018-09-19 11:58:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.zhihu.com/question/24252236>;
    None
    2018-09-19 11:58:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.zhihu.com/question/24252236>;
    None
    2018-09-19 11:58:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.zhihu.com/question/264756735>;
    None
    2018-09-19 11:58:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.zhihu.com/question/264756735>;
    None
    2018-09-19 11:58:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.zhihu.com/question/264756735>;
    None
    2018-09-19 11:58:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.zhihu.com/question/264756735>;
    None
    2018-09-19 11:58:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.zhihu.com/question/23846254>;
    None
    2018-09-19 11:58:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.zhihu.com/question/23846254>;
    None
    2018-09-19 11:58:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.zhihu.com/question/23846254>;
    None
    2018-09-19 11:58:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.zhihu.com/question/60410308>;
    None
    2018-09-19 11:58:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.zhihu.com/question/264756735>;
    None
    2018-09-19 11:58:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.zhihu.com/question/23846254>;
    None
    2018-09-19 11:58:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.zhihu.com/question/23846254>;
    None
    2018-09-19 11:58:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.zhihu.com/question/23846254>;
    None
    2018-09-19 11:58:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.zhihu.com/question/265749263>;
    None
    2018-09-19 11:58:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.zhihu.com/question/265749263>;
    None
    2018-09-19 11:58:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.zhihu.com/question/265749263>;
    None
    2018-09-19 11:58:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.zhihu.com/question/265749263>;
    None
    2018-09-19 11:58:25 [scrapy.core.engine] INFO: Closing spider (finished)
    2018-09-19 11:58:25 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
    {'downloader/request_bytes': 7823,
     'downloader/request_count': 20,
     'downloader/request_method_count/GET': 20,
     'downloader/response_bytes': 7113,
     'downloader/response_count': 20,
     'downloader/response_status_count/400': 20,
     'dupefilter/filtered': 9,
     'finish_reason': 'finished',
     'finish_time': datetime.datetime(2018, 9, 19, 3, 58, 25, 849646),
     'item_scraped_count': 28,
     'log_count/DEBUG': 202,
     'log_count/INFO': 10,
     'request_depth_max': 3,
     'response_received_count': 20,
     'scheduler/dequeued': 20,
     'scheduler/dequeued/memory': 20,
     'scheduler/enqueued': 20,
     'scheduler/enqueued/memory': 20,
     'start_time': datetime.datetime(2018, 9, 19, 3, 55, 14, 766276)}
    2018-09-19 11:58:25 [scrapy.core.engine] INFO: Spider closed (finished)### 问题描述
    
    爬虫在运行一段时间后自动关闭,爬取了一部分的数据,但是start_urls(start_url
    s中有多条链接,)中的链接并未爬取完的情况下爬虫就自动关闭了
    
    请问  
    2018-09-19 11:58:25 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.zhihu.com/question/265749263>;  None
    这个反馈是在表达什么,为何会自动关闭
    
    
    0
    打赏
    收藏
    点击回答
        全部回答
    • 0
    更多回答
    扫一扫访问手机版
    • 回到顶部
    • 回到顶部