#StackBounty: #python #python-3.x #concurrency #pool Parallel network requests: Pool vs Futures vs Asyncio

Bounty: 400

I would like to request 100 urls at a time and am currently doing this:

responses = list(PoolExecutor(max_workers=NUM_PARALLEL).map(
                 lambda xml: requests.post(URL, headers=HEADERS, data={'message': xml}), 
                 xmls))

A few questions on this:

  1. Is list the best way to ‘evaluate’ the actual generator object/expression? Unless I do list I just get something like: <generator object Executor.map.<locals>.result_iterator at 0x10ecf9888>
  2. Is PoolExecutor used frequently to do parallel network requests in python3, or are there are other methods that are more preferable?
  3. What are the differences between PoolExecutor, AsyncIO and concurrent.futures to do something like this?


Get this bounty!!!

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.