账号密码登录
微信安全登录
微信扫描二维码登录

登录后绑定QQ、微信即可实现信息互通

手机验证码登录
找回密码返回
邮箱找回 手机找回
注册账号返回
其他登录方式
分享
  • 收藏
    X
    Scrapy使用自定义Pipeline无法下载图片
    41
    0

    Scrapy使用自定义继承ImagesPipeline的Pipeline类后爬虫无法下载图片

    使用Python 3.7环境和Scrapy爬虫框架爬取网页上的图片并下载,使用内置的ImagesPipeline可以正常下载,但是使用自定义的Pipeline类后只在命令行输出图片链接地址,而无法下载图片到本地

    相关代码

    items.py

    class ImageItem(scrapy.Item):
        #文件名
        #image_names = scrapy.Field()
        #文件夹名
        #fold = scrapy.Field()
        #图片路径
        #image_paths = scrapy.Field()
        #图片链接地址
        image_urls = scrapy.Field()

    pipelines.py

    class PicPipeline(ImagesPipeline):
        def process_item(self, item, spider):
            return item
    
        def get_media_requests(self, item, info):
            for image_url in item['image_urls']:
                yield scrapy.Request(image_url)
    
        def item_completed(self, results, item, info):
            image_paths = [x['path'] for ok, x in results if ok]
            if not image_paths:
                raise DropItem("Item contains no images")
            item['image_paths'] = image_paths
            return item

    settings.py

    ITEM_PIPELINES = {
        'Pic.pipelines.PicPipeline': 300
    }
    IMAGE_STORES= 'D:\Pic'
    IMAGES_URLS_FIELD = 'image_urls'

    PicSpider.py

    class PicSpider(scrapy.Spider):
        name = 'picspider'
        allowed_domains = ['guokr.com']
        start_urls = ['https://www.guokr.com/']
    
        def parse(self, response):
            images = response.xpath('//img/@src').extract()
            item = ImageItem()
            item['image_urls'] = images
            yield item
    

    就只是想写个小Demo练习一下爬取图片,使用内置的ImagesPipeline能够正常下载图片,但是自定义的Pipeline就无法下载,在命令行输出图片链接地址后就没有了,请各位指教。

    命令行输出如下

    2019-02-19 12:11:06 [scrapy.core.engine] INFO: Spider opened
    2019-02-19 12:11:06 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
    2019-02-19 12:11:06 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
    2019-02-19 12:11:06 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.guokr.com/robots.txt>; (referer: None)
    2019-02-19 12:11:06 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.guokr.com/>; (referer: None)
    2019-02-19 12:11:06 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.guokr.com/>;
    {'image_urls': ['https://3-im.guokr.com/vXVhDq_6nindVo2LqIloosK-2bHrkYpU8DEXP75DpnZKAQAA6wAAAEpQ.jpg',
                    'https://2-im.guokr.com/hD7RoVC8IpQGnc2humofXMGyex-iSZH1VDaWLq2VWCE2BAAA7gMAAEpQ.jpg?imageView2/1/w/330/h/235',
                    'https://2-im.guokr.com/AU-Q8pTYY_OffTqWyKfXTC5NV0RmarK_QJ9m6A6_7qhKAQAA6wAAAEpQ.jpg',
                    'https://1-im.guokr.com/IIlEodManGB8jos3eP7KcrMhu3l8dtG6F5nrJczcrTiAAwAAUwIAAEpQ.jpg?imageView2/1/w/330/h/235',
                    'https://1-im.guokr.com/klfXUFzwXV_jz42yk497oZ-RkLAJEc03spAKMg9AeIw4BAAADQMAAEpQ.jpg?imageView2/1/w/330/h/235',
                    'https://1-im.guokr.com/BZ7R7bpcrwjOyFJ5kajc0tVHlOF8BUyEs3IpWB0l6Q4sAgAA2AEAAEpQ.jpg?imageView2/1/w/135/h/90',
                    'https://1-im.guokr.com/1CJgQkib1ePSCpLBARUhOyMdf6THL2BGrkDj6WDc5eiGAQAABAEAAEpQ.jpg?imageView2/1/w/135/h/90',
                    'https://1-im.guokr.com/4prMeIXxsaF2y6OTfpCB2IiI7udvwK8f_lsTcqbFcaeHAAAAWgAAAEpQ.jpg',
                    'https://1-im.guokr.com/WPrAHjwbKwXNYqiYZgkaYEyh9i2R8zm9noog_AxfpHiaAgAAvAEAAEpQ.jpg?imageView2/1/w/135/h/90',
                    'https://2-im.guokr.com/TNpsKxaaNGuIDTJWTpy2P5wfji_oG66rHUWGa8L7zFhKAQAAtQAAAFBO.png?imageView2/1/w/135/h/90',
                    'https://2-im.guokr.com/gLbC7ix6NWlx3bz6ihFyOxsl_fWqwtB554NswEOmACFKAQAA8AAAAEpQ.jpg?imageView2/1/w/135/h/90',
                    'https://2-im.guokr.com/Rx9MyfI6hndQBTyoGWvfOyb469BZ7ruf0w0k7V0aJ1pKAQAA6wAAAEpQ.jpg?imageView2/1/w/135/h/90',
                    'https://2-im.guokr.com/-OmYOzUa0Nhm9vKimCFn2c2ZR9pHmgxqMiMxijD5KwkLAQAAngAAAFBO.png?imageView2/1/w/135/h/90',
                    'https://1-im.guokr.com/mysULQspmaLPEMu-MQFZGHwaccTPPs9msjtLrYoDtGcsAQAAagAAAEpQ.jpg',
                    'https://2-im.guokr.com/fSvqlLJ6wcRv8cCCc5Ehm5pgqZWg7TyiLZdEba34NTKgAAAAoAAAAEpQ.jpg?imageView2/1/w/48/h/48',
                    'https://3-im.guokr.com/F9IifzSeB9OoKKIP-_2i3SnWHnUceIpmGyOMuwgRvgGgAAAAoAAAAEpQ.jpg?imageView2/1/w/48/h/48',
                    'https://sslstatic.guokr.com/skin/imgs/dimensions-code.jpg?v=unknown',
                    'https://3-im.guokr.com/0Al5wQUv5IAuo87evbERy190Y83ENmP9OpIs8Stm2lMUAAAAFAAAAFBO.png']}
    2019-02-19 12:11:06 [scrapy.core.engine] INFO: Closing spider (finished)
    2019-02-19 12:11:06 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
    {'downloader/request_bytes': 434,
     'downloader/request_count': 2,
     'downloader/request_method_count/GET': 2,
     'downloader/response_bytes': 12316,
     'downloader/response_count': 2,
     'downloader/response_status_count/200': 2,
     'finish_reason': 'finished',
     'finish_time': datetime.datetime(2019, 2, 19, 4, 11, 6, 755334),
     'item_scraped_count': 1,
     'log_count/DEBUG': 3,
     'log_count/INFO': 9,
     'response_received_count': 2,
     'robotstxt/request_count': 1,
     'robotstxt/response_count': 1,
     'robotstxt/response_status_count/200': 1,
     'scheduler/dequeued': 1,
     'scheduler/dequeued/memory': 1,
     'scheduler/enqueued': 1,
     'scheduler/enqueued/memory': 1,
     'start_time': datetime.datetime(2019, 2, 19, 4, 11, 6, 142378)}
    2019-02-19 12:11:06 [scrapy.core.engine] INFO: Spider closed (finished)
    
    1
    打赏
    收藏
    点击回答
        全部回答
    • 0
    • 清风不解语 普通会员 1楼

      Scrapy使用自定义Pipeline可以下载图片,但可能需要进行一些调整。以下是一个简单的例子:

      首先,你需要在你的Scrapy项目中创建一个自定义的Pipeline。在Pipeline.py文件中,你可以定义你的管道逻辑,例如:

      python class CustomPipeline(object): def process_item(self, item, spider): # 在这里处理你的图片下载逻辑 if 'image_url' in item: yield {'image_url': item['image_url']} else: yield item

      在这个例子中,我们定义了一个名为CustomPipeline的自定义Pipeline。这个Pipeline定义了一个处理器,当这个处理器被调用时,它会检查item中的image_url属性是否存在。如果存在,它就会将这个URL返回给Scrapy。如果没有,它就会返回原始的item

      然后,你需要在你的Scrapy项目中配置这个Pipeline。在settings.py文件中,你可以添加一个新的线程池:

      python ITEM_PIPELINES = { 'myproject.pipelines.CustomPipeline': 300, }

      在这个例子中,CustomPipeline的管道ID被设置为300。

      最后,你需要在你的爬虫中使用这个Pipeline。在你的爬虫中,你可以使用Pipeline来处理你的图片。例如,你可以在你的start_requests函数中添加一个CustomPipeline的处理器:

      ```python import scrapy from myproject.pipelines import CustomPipeline

      class MySpider(scrapy.Spider): name = 'myspider' pipeline = CustomPipeline() start_urls = ['http://example.com'] ```

      在这个例子中,我们定义了一个名为MySpider的爬虫,它使用CustomPipeline的处理器来处理图片。

    更多回答
    扫一扫访问手机版
    • 回到顶部
    • 回到顶部