[llvm-dev] RFC: Adding background workers to LNT

[Chris Matthews via llvm-dev]

One limitation LNT has is that all processing must take place within a request. This limits our ability to do anything really compute heavy, we are always bound by page load time.

I’d like to add a background processing feature to LNT.  This would allow a request to queue some work to be processed later by a subprocess.  Things I’d like to use this feature on: generating caches to speed up page load times, calculating derived metrics (like geomean and SPEC score) during submission, calculating field changes during submission, running much more compute heavy submission processing to correlate detected regressions.

The defacto standard for doing background processing in Flask is Celery.  I don’t think that is a great option for LNT. It is a big package dependency, and requires Redis/Rabbitmq or another database as a communication mechansim. We want LNT to still be easy to use from the command line.  I propose we use Python’s built in multiprocessing package, to setup a pool of workers.  This adds no dependencies; though it will lack some of the cool features of Celery like debugging and monitoring.

Are there other opportunities for background work anyone can think of?  Are there other task queuing solutions that might be more appropriate?  Thanks