python异步编程 | Python

python异步编程

内容摘自崔庆才《网络爬虫开发实战第2版》，推荐购买

对asyncio的错误使用

只有支持异步操作的请求方式才可以实现真正的异步！

错误示例

import asyncio
import requests
import time

start = time.time()


async def get(url):
    return requests.get(url)


async def request():
    url = 'http://www.httpbin.org/delay/2'
    response = await get(url)
    print(f'请求的链接：{url}， 请求的响应：{response}')


tasks = [asyncio.ensure_future(request()) for _ in range(10)]

loop = asyncio.get_event_loop()
loop.run_until_complete(asyncio.wait(tasks))

end = time.time()

print(end - start)

"""
执行结果：
请求的链接：http://www.httpbin.org/delay/2， 请求的响应：<Response [200]>
请求的链接：http://www.httpbin.org/delay/2， 请求的响应：<Response [200]>
请求的链接：http://www.httpbin.org/delay/2， 请求的响应：<Response [200]>
请求的链接：http://www.httpbin.org/delay/2， 请求的响应：<Response [200]>
请求的链接：http://www.httpbin.org/delay/2， 请求的响应：<Response [200]>
请求的链接：http://www.httpbin.org/delay/2， 请求的响应：<Response [200]>
请求的链接：http://www.httpbin.org/delay/2， 请求的响应：<Response [200]>
请求的链接：http://www.httpbin.org/delay/2， 请求的响应：<Response [200]>
请求的链接：http://www.httpbin.org/delay/2， 请求的响应：<Response [200]>
请求的链接：http://www.httpbin.org/delay/2， 请求的响应：<Response [200]>
26.44580578804016
"""

正确示例
配合异步请求库 aiohttp 使用

import asyncio
import aiohttp
import time

start = time.time()


async def get(url):
    session = aiohttp.ClientSession()
    response = await session.get(url)

    text = await response.text()
    print(text)
    await session.close()

    return response


async def request():
    url = 'https://www.httpbin.org/delay/2'
    response = await get(url)

    print(f'链接：{url} 的响应结果：{response}')


tasks = [asyncio.ensure_future(request()) for _ in range(10)]

loop = asyncio.get_event_loop()
loop.run_until_complete(asyncio.wait(tasks))

# 循环tasks，调用result()来接收结果

end = time.time()

print(end - start)

"""
{
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate", 
    "Host": "www.httpbin.org", 
    "User-Agent": "Python/3.9 aiohttp/3.7.4.post0", 
    "X-Amzn-Trace-Id": "Root=1-61af62f8-3142bec0361f2c6f7dff1c72"
  }, 
  "origin": "36.27.36.217", 
  "url": "https://www.httpbin.org/delay/2"
}

链接：https://www.httpbin.org/delay/2 的响应结果：<ClientResponse(https://www.httpbin.org/delay/2) [200 OK]>
<CIMultiDictProxy('Date': 'Tue, 07 Dec 2021 13:34:50 GMT', 'Content-Type': 'application/json', 'Content-Length': '372', 'Connection': 'keep-alive', 'Server': 'gunicorn/19.9.0', 'Access-Control-Allow-Origin': '*', 'Access-Control-Allow-Credentials': 'true')>

4.411431074142456
"""

asyncio、aiohttp模块的使用方式参考官方网站