[PATCH] Make LNT compatible with PostgreSQL

[Yi Kong]

Hi Chris,

Sorry for the delay, I've been occupied by some other work.

Can you try this test and see if it works on your Jenkins? I will
expand the test coverage if it proves to be working.

Cheers,

Yi

On 24 July 2014 00:29, Chris Matthews <chris.matthews at apple.com> wrote:
> The lit test suite supports a requires clause.  This allows you to do this
>
> # REQUIRES: postgres
>
> In a test, then that test is only run if a —param postgres=True is passed to the lit invocation.  This allows us to keep the tests disabled for all except people who want to run the extra postgres tests.
>
> I have a Jenkins bot, which I could install postgres on, so we would have coverage on postgres there.
>
> Would it be possible to start the lit script off with a psql command which drop/creates a database first?  so the database name and URL are known ahead of time.
>
>> On Jul 23, 2014, at 7:41 AM, Yi Kong <kongy.dev at gmail.com> wrote:
>>
>> On 21 July 2014 23:49, Chris Matthews <chris.matthews at apple.com> wrote:
>>> Okay. That is fine.  I think it is okay to tell people to start fresh if they are going to switch.
>>>
>>> Go ahead and commit your patch to add the Postgres creation capability.
>>>
>>> Can you add a little blurb to docs/tools.rst explaining how to create for Postgres.
>>>
>>> Could you also add a Postgres requires directive to the LNT test suite (off by default),
>>
>> I'm not sure what do you mean. Are you suggesting adding 'psycopg2' to
>> install_requires?
>>
>>> and make a new test in lnttool to test this code path?
>>
>> It's not easily testable as Postgres doesn't support in-memory scratch database.
>>
>> Thanks for reviewing!
>>
>> -Yi
>
-------------- next part --------------
Index: docs/intro.rst
===================================================================
--- docs/intro.rst	(revision 214081)
+++ docs/intro.rst	(working copy)
@@ -34,13 +34,13 @@
 If you want to run an LNT server, you will need to perform the following
 additional steps:
 
- 2. Create a new LNT installation:
+ 2. Create a new LNT installation::
 
       lnt create path/to/install-dir
 
     This will create the LNT configuration file, the default database, and a
     .wsgi wrapper to create the application. You can execute the generated app
-    directly to run with the builtin web server, or use 
+    directly to run with the builtin web server, or use::
 
       lnt runserver path/to/install-dir
 
@@ -62,7 +62,22 @@
     If running in a virtualenv you will need to configure that as well; see the
     `modwsgi wiki <http://code.google.com/p/modwsgi/wiki/VirtualEnvironments>`_.
 
+For production servers, you should consider using a full DBMS like PostgreSQL.
+To create an LNT instance with PostgreSQL backend, you need to do this instead:
 
+ 1. Create an LNT database in PostgreSQL, also make sure the user has
+    write permission to the database::
+
+      CREATE DATABASE "lnt.db"
+
+ 2. Then create LNT installation::
+
+      lnt create path/to/install-dir --db-dir postgresql://user@host
+
+ 3. Run server normally::
+
+      lnt runserver path/to/install-dir
+
 Development
 -----------
 
@@ -86,7 +101,6 @@
 if you prefer 'unittest' style output (this still requires that 'lit' be
 installed).
 
-
 Architecture
 ------------
 
Index: docs/tools.rst
===================================================================
--- docs/tools.rst	(revision 214081)
+++ docs/tools.rst	(working copy)
@@ -51,7 +51,8 @@
     to tweak the generated server, but they can all be modified after the fact
     in the LNT configuration file.
 
-    The default server will have one database named *default*.
+    The default server will have a sqlite3 database named *default*. You can
+    specify to use PostgreSQL using ``--db-dir postgresql://user@hostname``.
 
   ``lnt createdb <path>``
     Creates a new LNT sqlite3 database at the specified path.
Index: lnt/lnttool/create.py
===================================================================
--- lnt/lnttool/create.py	(revision 214081)
+++ lnt/lnttool/create.py	(working copy)
@@ -151,9 +151,7 @@
     hosturl = "http://%s/%s" % (hostname, hostsuffix)
 
     python_executable = sys.executable
-    db_dir_path = os.path.join(basepath, db_dir)
     cfg_path = os.path.join(basepath, config)
-    db_path = os.path.join(db_dir_path, default_db)
     tmp_path = os.path.join(basepath, tmp_dir)
     wsgi_path = os.path.join(basepath, wsgi)
     secret_key = (opts.secret_key or
@@ -160,9 +158,16 @@
                   hashlib.sha1(str(random.getrandbits(256))).hexdigest())
 
     os.mkdir(path)
-    os.mkdir(db_dir_path)
     os.mkdir(tmp_path)
 
+    # If the path does not contain database type, assume relative path.
+    if "://" not in db_dir:
+        db_dir_path = os.path.join(basepath, db_dir)
+        db_path = os.path.join(db_dir_path, default_db)
+        os.mkdir(db_dir_path)
+    else:
+        db_path = os.path.join(db_dir, default_db)
+
     cfg_version = kConfigVersion
     cfg_file = open(cfg_path, 'w')
     cfg_file.write(kConfigTemplate % locals())
Index: lnt/server/config.py
===================================================================
--- lnt/server/config.py	(revision 214081)
+++ lnt/server/config.py	(working copy)
@@ -90,8 +90,10 @@
             default_email_config = EmailConfig(False, '', '', [])
 
         dbDir = data.get('db_dir', '.')
-        dbDirPath = os.path.join(baseDir, dbDir)
 
+        # If the path does not contain database type, assume relative path.
+        dbDirPath = dbDir if "://" in dbDir else os.path.join(baseDir, dbDir)
+
         # FIXME: Remove this default.
         tempDir = data.get('tmp_dir', 'viewer/resources/graphs')
 
Index: tests/lit.cfg
===================================================================
--- tests/lit.cfg	(revision 214081)
+++ tests/lit.cfg	(working copy)
@@ -38,6 +38,9 @@
 if lit_config.params.get('long', None):
     config.available_features.add('long')
 
+if lit_config.params.get('postgres', None):
+    config.available_features.add('postgres')
+
 # Enable coverage.py reporting, assuming the coverage module has been installed
 # and sitecustomize.py in the virtualenv has been modified appropriately.
 if lit_config.params.get('check-coverage', None):
Index: tests/lnttool/PostgresDB.py
===================================================================
--- tests/lnttool/PostgresDB.py	(revision 0)
+++ tests/lnttool/PostgresDB.py	(working copy)
@@ -0,0 +1,12 @@
+# RUN: rm -rf %t.install
+# RUN: createdb "lnt.db"
+
+# RUN: lnt create %t.install --db-dir postgresql://localhost/
+
+# Import a test set.
+# RUN: lnt import %t.install %{shared_inputs}/sample-a-small.plist \
+# RUN:     --commit=1 --show-sample-count
+
+# RUN: dropdb "lnt.db"
+
+# REQUIRES: postgres