#StackBounty: #python-3.x #google-cloud-functions #python-asyncio asyncio seems to be finishing before HTTP request complete

Bounty: 300

I am attempting to run many parallel calls to a web service (formatted by a GET querystring) using asyncio and the calls all immediately return a value of 0. The web service is a physics simulation that returns an integer value with how well it performed. However, I would expect the service to run for approximately 2 minutes and then return the value, however printing out the values from asyncio display immediate 0 values.

This code is part of a genetic algorithm (running DEAP), so what I want to do is to have an outer (generational) loop that runs, with each individual (constructed URL) running in parallel and performing the evaluation. I’m not sure if this has any impact, but it is a Google Cloud Function. The maximum execution time is set well within the bounds of the expected evaluation time, and the maximum memory is also valid.

Here are my asyncio functions, where I’d expect to see some valid integer value if the response returns OK, or -999 if an error was generated:

# AsyncIO functions for interacting with Google Cloud Functions
async def fetchCF(indv: str, session: ClientSession, **kwargs) -> str:
  resp = await session.request(method="GET", url=indv, **kwargs)
  resp.raise_for_status()
  html = await resp.text()
  return html

# Dispatch the CFs and return fitness
async def callCF(indv: str, session: ClientSession, **kwargs) -> int:#float:
  try:
    html = await fetchCF(indv=indv, session=session, **kwargs)
  except (
    aiohttp.ClientError,
    aiohttp.http_exceptions.HttpProcessingError,
  ) as e:
    print(indv,e)
    return -9999,
  else:
    return int(html),

# Called via GA
async def evalAsync(pop: set, **kwargs) -> None:
  async with ClientSession(read_timeout=None) as session:
    tasks = []
    for p in pop:
      #print(p)
      indv_url = '%s?seed=%d&x0=%d&y0=%d&x1=%d&y1=%d&x2=%d&y2=%d&x3=%d&y3=%d&emitX=%d&emitY=%d' % 
                 (url,args.seed,int(p[0]),int(p[1]),int(p[2]),int(p[3]),int(p[4]),int(p[5]),int(p[6]),int(p[7]),int(p[8]),int(p[9]))
      tasks.append(
        callCF(indv=indv_url,session=session,**kwargs)
      )
return await asyncio.gather(*tasks)

And here is how I call them within the generational loop:

for g in generations:
  ...
  fitnesses = asyncio.run(evalAsync(population))

For reference, the code works fine with a local physics simulation where I replace the call to asyncio.run with a pool.map call to my local physics driver.


Get this bounty!!!

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.