<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Menlo;
panose-1:0 0 0 0 0 0 0 0 0 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"Plain Text Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Courier New";
mso-fareast-language:EN-US;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.PlainTextChar
{mso-style-name:"Plain Text Char";
mso-style-priority:99;
mso-style-link:"Plain Text";
font-family:"Courier New";}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-GB link=blue vlink=purple><div class=WordSection1><p class=MsoNormal>Daniel, Tobias, Renato and myself have been looking a little bit at the potential underlying reason<br>for why <a href="http://llvm.org/perf/">http://llvm.org/perf/</a> is instable, and have found some clues. I want to share them here<br>to give people with more experience in the frameworks used by LNT (flask, sqlalchemy, wsgi, …)<br>a chance to check if our reasoning below seems plausible.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Daniel noticed the following backtrace in the log after <a href="http://llvm.org/perf">http://llvm.org/perf</a> started giving “Internal Server Error”<br>again:<o:p></o:p></p><p style='margin:0cm;margin-bottom:.0001pt'><span style='font-size:8.5pt;font-family:"Menlo","serif"'>2015-05-08 22:57:05,309 ERROR: Exception on /db_default/v4/nts/287/graph [GET] [in /opt/venv/perf/lib/python2.7/site-packages/Flask-0.10.1-py2.7.egg/flask/app.py:1423]<o:p></o:p></span></p><p style='margin:0cm;margin-bottom:.0001pt'><span style='font-size:8.5pt;font-family:"Menlo","serif"'>Traceback (most recent call last):<o:p></o:p></span></p><p style='margin:0cm;margin-bottom:.0001pt'><span style='font-size:8.5pt;font-family:"Menlo","serif"'> File "/opt/venv/perf/lib/python2.7/site-packages/Flask-0.10.1-py2.7.egg/flask/app.py", line 1817, in wsgi_app<o:p></o:p></span></p><p style='margin:0cm;margin-bottom:.0001pt'><span style='font-size:8.5pt;font-family:"Menlo","serif"'> response = self.full_dispatch_request()<o:p></o:p></span></p><p style='margin:0cm;margin-bottom:.0001pt'><span style='font-size:8.5pt;font-family:"Menlo","serif"'> File "/opt/venv/perf/lib/python2.7/site-packages/Flask-0.10.1-py2.7.egg/flask/app.py", line 1477, in full_dispatch_request<o:p></o:p></span></p><p style='margin:0cm;margin-bottom:.0001pt'><span style='font-size:8.5pt;font-family:"Menlo","serif"'> rv = self.handle_user_exception(e)<o:p></o:p></span></p><p style='margin:0cm;margin-bottom:.0001pt'><span style='font-size:8.5pt;font-family:"Menlo","serif"'> File "/opt/venv/perf/lib/python2.7/site-packages/Flask-0.10.1-py2.7.egg/flask/app.py", line 1381, in handle_user_exception<o:p></o:p></span></p><p style='margin:0cm;margin-bottom:.0001pt'><span style='font-size:8.5pt;font-family:"Menlo","serif"'> reraise(exc_type, exc_value, tb)<o:p></o:p></span></p><p style='margin:0cm;margin-bottom:.0001pt'><span style='font-size:8.5pt;font-family:"Menlo","serif"'> File "/opt/venv/perf/lib/python2.7/site-packages/Flask-0.10.1-py2.7.egg/flask/app.py", line 1475, in full_dispatch_request<o:p></o:p></span></p><p style='margin:0cm;margin-bottom:.0001pt'><span style='font-size:8.5pt;font-family:"Menlo","serif"'> rv = self.dispatch_request()<o:p></o:p></span></p><p style='margin:0cm;margin-bottom:.0001pt'><span style='font-size:8.5pt;font-family:"Menlo","serif"'> File "/opt/venv/perf/lib/python2.7/site-packages/Flask-0.10.1-py2.7.egg/flask/app.py", line 1461, in dispatch_request<o:p></o:p></span></p><p style='margin:0cm;margin-bottom:.0001pt'><span style='font-size:8.5pt;font-family:"Menlo","serif"'> return self.view_functions[rule.endpoint](**req.view_args)<o:p></o:p></span></p><p style='margin:0cm;margin-bottom:.0001pt'><span style='font-size:8.5pt;font-family:"Menlo","serif"'> File "/opt/venv/perf/lib/python2.7/site-packages/LNT-0.4.1dev-py2.7.egg/lnt/server/ui/decorators.py", line 67, in wrap<o:p></o:p></span></p><p style='margin:0cm;margin-bottom:.0001pt'><span style='font-size:8.5pt;font-family:"Menlo","serif"'> result = f(**args)<o:p></o:p></span></p><p style='margin:0cm;margin-bottom:.0001pt'><span style='font-size:8.5pt;font-family:"Menlo","serif"'> File "/opt/venv/perf/lib/python2.7/site-packages/LNT-0.4.1dev-py2.7.egg/lnt/server/ui/views.py", line 385, in v4_run_graph<o:p></o:p></span></p><p style='margin:0cm;margin-bottom:.0001pt'><span style='font-size:8.5pt;font-family:"Menlo","serif"'> ts = request.get_testsuite()<o:p></o:p></span></p><p style='margin:0cm;margin-bottom:.0001pt'><span style='font-size:8.5pt;font-family:"Menlo","serif"'> File "/opt/venv/perf/lib/python2.7/site-packages/LNT-0.4.1dev-py2.7.egg/lnt/server/ui/app.py", line 76, in get_testsuite<o:p></o:p></span></p><p style='margin:0cm;margin-bottom:.0001pt'><span style='font-size:8.5pt;font-family:"Menlo","serif"'> testsuites = self.get_db().testsuite<o:p></o:p></span></p><p style='margin:0cm;margin-bottom:.0001pt'><span style='font-size:8.5pt;font-family:"Menlo","serif"'> File "/opt/venv/perf/lib/python2.7/site-packages/LNT-0.4.1dev-py2.7.egg/lnt/server/ui/app.py", line 55, in get_db<o:p></o:p></span></p><p style='margin:0cm;margin-bottom:.0001pt'><span style='font-size:8.5pt;font-family:"Menlo","serif"'> self.db = current_app.old_config.get_database(g.db_name, echo=echo)<o:p></o:p></span></p><p style='margin:0cm;margin-bottom:.0001pt'><span style='font-size:8.5pt;font-family:"Menlo","serif"'> File "/opt/venv/perf/lib/python2.7/site-packages/LNT-0.4.1dev-py2.7.egg/lnt/server/config.py", line 148, in get_database<o:p></o:p></span></p><p style='margin:0cm;margin-bottom:.0001pt'><span style='font-size:8.5pt;font-family:"Menlo","serif"'> return lnt.server.db.v4db.V4DB(db_entry.path, self, echo=echo)<o:p></o:p></span></p><p style='margin:0cm;margin-bottom:.0001pt'><span style='font-size:8.5pt;font-family:"Menlo","serif"'> File "/opt/venv/perf/lib/python2.7/site-packages/LNT-0.4.1dev-py2.7.egg/lnt/server/db/v4db.py", line 108, in __init__<o:p></o:p></span></p><p style='margin:0cm;margin-bottom:.0001pt'><span style='font-size:8.5pt;font-family:"Menlo","serif"'> .filter_by(id = lnt.testing.PASS).first()<o:p></o:p></span></p><p style='margin:0cm;margin-bottom:.0001pt'><span style='font-size:8.5pt;font-family:"Menlo","serif"'> File "/opt/venv/perf/lib/python2.7/site-packages/SQLAlchemy-0.9.6-py2.7.egg/sqlalchemy/orm/query.py", line 2334, in first<o:p></o:p></span></p><p style='margin:0cm;margin-bottom:.0001pt'><span style='font-size:8.5pt;font-family:"Menlo","serif"'> ret = list(self[0:1])<o:p></o:p></span></p><p style='margin:0cm;margin-bottom:.0001pt'><span style='font-size:8.5pt;font-family:"Menlo","serif"'> File "/opt/venv/perf/lib/python2.7/site-packages/SQLAlchemy-0.9.6-py2.7.egg/sqlalchemy/orm/query.py", line 2201, in __getitem__<o:p></o:p></span></p><p style='margin:0cm;margin-bottom:.0001pt'><span style='font-size:8.5pt;font-family:"Menlo","serif"'> return list(res)<o:p></o:p></span></p><p style='margin:0cm;margin-bottom:.0001pt'><span style='font-size:8.5pt;font-family:"Menlo","serif"'> File "/opt/venv/perf/lib/python2.7/site-packages/SQLAlchemy-0.9.6-py2.7.egg/sqlalchemy/orm/query.py", line 2405, in __iter__<o:p></o:p></span></p><p style='margin:0cm;margin-bottom:.0001pt'><span style='font-size:8.5pt;font-family:"Menlo","serif"'> return self._execute_and_instances(context)<o:p></o:p></span></p><p style='margin:0cm;margin-bottom:.0001pt'><span style='font-size:8.5pt;font-family:"Menlo","serif"'> File "/opt/venv/perf/lib/python2.7/site-packages/SQLAlchemy-0.9.6-py2.7.egg/sqlalchemy/orm/query.py", line 2418, in _execute_and_instances<o:p></o:p></span></p><p style='margin:0cm;margin-bottom:.0001pt'><span style='font-size:8.5pt;font-family:"Menlo","serif"'> close_with_result=True)<o:p></o:p></span></p><p style='margin:0cm;margin-bottom:.0001pt'><span style='font-size:8.5pt;font-family:"Menlo","serif"'> File "/opt/venv/perf/lib/python2.7/site-packages/SQLAlchemy-0.9.6-py2.7.egg/sqlalchemy/orm/query.py", line 2409, in _connection_from_session<o:p></o:p></span></p><p style='margin:0cm;margin-bottom:.0001pt'><span style='font-size:8.5pt;font-family:"Menlo","serif"'> **kw)<o:p></o:p></span></p><p style='margin:0cm;margin-bottom:.0001pt'><span style='font-size:8.5pt;font-family:"Menlo","serif"'> File "/opt/venv/perf/lib/python2.7/site-packages/SQLAlchemy-0.9.6-py2.7.egg/sqlalchemy/orm/session.py", line 846, in connection<o:p></o:p></span></p><p style='margin:0cm;margin-bottom:.0001pt'><span style='font-size:8.5pt;font-family:"Menlo","serif"'> close_with_result=close_with_result)<o:p></o:p></span></p><p style='margin:0cm;margin-bottom:.0001pt'><span style='font-size:8.5pt;font-family:"Menlo","serif"'> File "/opt/venv/perf/lib/python2.7/site-packages/SQLAlchemy-0.9.6-py2.7.egg/sqlalchemy/orm/session.py", line 850, in _connection_for_bind<o:p></o:p></span></p><p style='margin:0cm;margin-bottom:.0001pt'><span style='font-size:8.5pt;font-family:"Menlo","serif"'> return self.transaction._connection_for_bind(engine)<o:p></o:p></span></p><p style='margin:0cm;margin-bottom:.0001pt'><span style='font-size:8.5pt;font-family:"Menlo","serif"'> File "/opt/venv/perf/lib/python2.7/site-packages/SQLAlchemy-0.9.6-py2.7.egg/sqlalchemy/orm/session.py", line 315, in _connection_for_bind<o:p></o:p></span></p><p style='margin:0cm;margin-bottom:.0001pt'><span style='font-size:8.5pt;font-family:"Menlo","serif"'> conn = bind.contextual_connect()<o:p></o:p></span></p><p style='margin:0cm;margin-bottom:.0001pt'><span style='font-size:8.5pt;font-family:"Menlo","serif"'> File "/opt/venv/perf/lib/python2.7/site-packages/SQLAlchemy-0.9.6-py2.7.egg/sqlalchemy/engine/base.py", line 1737, in contextual_connect<o:p></o:p></span></p><p style='margin:0cm;margin-bottom:.0001pt'><span style='font-size:8.5pt;font-family:"Menlo","serif"'> self.pool.connect(),<o:p></o:p></span></p><p style='margin:0cm;margin-bottom:.0001pt'><span style='font-size:8.5pt;font-family:"Menlo","serif"'> File "/opt/venv/perf/lib/python2.7/site-packages/SQLAlchemy-0.9.6-py2.7.egg/sqlalchemy/pool.py", line 332, in connect<o:p></o:p></span></p><p style='margin:0cm;margin-bottom:.0001pt'><span style='font-size:8.5pt;font-family:"Menlo","serif"'> return _ConnectionFairy._checkout(self)<o:p></o:p></span></p><p style='margin:0cm;margin-bottom:.0001pt'><span style='font-size:8.5pt;font-family:"Menlo","serif"'> File "/opt/venv/perf/lib/python2.7/site-packages/SQLAlchemy-0.9.6-py2.7.egg/sqlalchemy/pool.py", line 630, in _checkout<o:p></o:p></span></p><p style='margin:0cm;margin-bottom:.0001pt'><span style='font-size:8.5pt;font-family:"Menlo","serif"'> fairy = _ConnectionRecord.checkout(pool)<o:p></o:p></span></p><p style='margin:0cm;margin-bottom:.0001pt'><span style='font-size:8.5pt;font-family:"Menlo","serif"'> File "/opt/venv/perf/lib/python2.7/site-packages/SQLAlchemy-0.9.6-py2.7.egg/sqlalchemy/pool.py", line 433, in checkout<o:p></o:p></span></p><p style='margin:0cm;margin-bottom:.0001pt'><span style='font-size:8.5pt;font-family:"Menlo","serif"'> rec = pool._do_get()<o:p></o:p></span></p><p style='margin:0cm;margin-bottom:.0001pt'><span style='font-size:8.5pt;font-family:"Menlo","serif"'> File "/opt/venv/perf/lib/python2.7/site-packages/SQLAlchemy-0.9.6-py2.7.egg/sqlalchemy/pool.py", line 945, in _do_get<o:p></o:p></span></p><p style='margin:0cm;margin-bottom:.0001pt'><span style='font-size:8.5pt;font-family:"Menlo","serif"'> (self.size(), self.overflow(), self._timeout))<o:p></o:p></span></p><p style='margin:0cm;margin-bottom:.0001pt'><span style='font-size:8.5pt;font-family:"Menlo","serif"'>TimeoutError: QueuePool limit of size 5 overflow 10 reached, connection timed out, timeout 30<o:p></o:p></span></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoPlainText>After browsing through the sqlalchemy documentation and bits of the LNT implementation,<br>it seems so far that the following pieces may be the key parts that cause the problem<br>shown in the log.<o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText>The SQLAlchemy documentation seems to recommend to have a sqlalchemy session per web<br>request. Looking at the following pieces of LNT, I got the impression that instead a<br>session is shared between many or all requests:<o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText>From ui/app.py, it shows Request.get_db() basically caches get_database from "config":<o:p></o:p></p><p class=MsoPlainText>...<o:p></o:p></p><p class=MsoPlainText>class Request(flask.Request):<o:p></o:p></p><p class=MsoPlainText>...<o:p></o:p></p><p class=MsoPlainText> def get_db(self):<o:p></o:p></p><p class=MsoPlainText>...<o:p></o:p></p><p class=MsoPlainText> if self.db is None:<o:p></o:p></p><p class=MsoPlainText> echo = bool(self.args.get('db_log') or self.form.get('db_log'))<o:p></o:p></p><p class=MsoPlainText> self.db = current_app.old_config.get_database(g.db_name, echo=echo)<br>...<o:p></o:p></p><p class=MsoPlainText> return self.db<o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText>in config.py, it is shown that get_database returns a V4DB object by calling a constructor:<o:p></o:p></p><p class=MsoPlainText>...<o:p></o:p></p><p class=MsoPlainText> def get_database(self, name, echo=False):<o:p></o:p></p><p class=MsoPlainText>...<o:p></o:p></p><p class=MsoPlainText> # Instantiate the appropriate database version.<o:p></o:p></p><p class=MsoPlainText> if db_entry.db_version == '0.4':<o:p></o:p></p><p class=MsoPlainText> return lnt.server.db.v4db.V4DB(db_entry.path, self,<o:p></o:p></p><p class=MsoPlainText> db_entry.baseline_revision,<o:p></o:p></p><p class=MsoPlainText> echo)<br>...<o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText>This constructor is in db/v4db.py:<o:p></o:p></p><p class=MsoPlainText>...<o:p></o:p></p><p class=MsoPlainText>class V4DB(object):<o:p></o:p></p><p class=MsoPlainText>...<o:p></o:p></p><p class=MsoPlainText> def __init__(self, path, config, baseline_revision=0, echo=False):<o:p></o:p></p><p class=MsoPlainText>...<o:p></o:p></p><p class=MsoPlainText> self.session = sqlalchemy.orm.sessionmaker(self.engine)()<o:p></o:p></p><p class=MsoPlainText>...<o:p></o:p></p><p class=MsoPlainText> # Add several shortcut aliases.<o:p></o:p></p><p class=MsoPlainText> self.add = self.session.add<o:p></o:p></p><p class=MsoPlainText> self.commit = self.session.commit<o:p></o:p></p><p class=MsoPlainText> self.query = self.session.query<o:p></o:p></p><p class=MsoPlainText> self.rollback = self.session.rollback<br>...<o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText>It seems like a single session object is created in this constructor that will ultimately<br>be shared across all Requests. It seems that instead, the request.get_db method should<br>create a new session for each request. And close that session when the request is finalized<br>which probably needs to be done by hooking into something Flask-specific.<o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText>The self.add and following lines in the constructor show that it probably will be<br>non-trivial to refactor code so that there will not be a single session per v4db object.<o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText>We’re not sure if making separate sessions per Request is going to solve the <a href="http://llvm.org/perf">http://llvm.org/perf</a><br>instability problems; but that’s the best idea we’ve got so far. <o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText>Thanks,<o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText>Kristof<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p></div></body></html>