I am attempting to run many parallel calls to a web service (formatted by a GET querystring) using
asyncio and the calls all immediately return a value of 0. The web service is a physics simulation that returns an integer value with how well it performed. However, I would expect the service to run for approximately 2 minutes and then return the value, however printing out the values from
asyncio display immediate 0 values.
This code is part of a genetic algorithm (running DEAP), so what I want to do is to have an outer (generational) loop that runs, with each individual (constructed URL) running in parallel and performing the evaluation. I’m not sure if this has any impact, but it is a Google Cloud Function. The maximum execution time is set well within the bounds of the expected evaluation time, and the maximum memory is also valid.
Here are my
asyncio functions, where I’d expect to see some valid integer value if the response returns OK, or -999 if an error was generated:
# AsyncIO functions for interacting with Google Cloud Functions async def fetchCF(indv: str, session: ClientSession, **kwargs) -> str: resp = await session.request(method="GET", url=indv, **kwargs) resp.raise_for_status() html = await resp.text() return html # Dispatch the CFs and return fitness async def callCF(indv: str, session: ClientSession, **kwargs) -> int:#float: try: html = await fetchCF(indv=indv, session=session, **kwargs) except ( aiohttp.ClientError, aiohttp.http_exceptions.HttpProcessingError, ) as e: print(indv,e) return -9999, else: return int(html), # Called via GA async def evalAsync(pop: set, **kwargs) -> None: async with ClientSession(read_timeout=None) as session: tasks =  for p in pop: #print(p) indv_url = '%s?seed=%d&x0=%d&y0=%d&x1=%d&y1=%d&x2=%d&y2=%d&x3=%d&y3=%d&emitX=%d&emitY=%d' % (url,args.seed,int(p),int(p),int(p),int(p),int(p),int(p),int(p),int(p),int(p),int(p)) tasks.append( callCF(indv=indv_url,session=session,**kwargs) ) return await asyncio.gather(*tasks)
And here is how I call them within the generational loop:
for g in generations: ... fitnesses = asyncio.run(evalAsync(population))
For reference, the code works fine with a local physics simulation where I replace the call to
asyncio.run with a
pool.map call to my local physics driver.