[LNT] r263881 - [profile] Add a rule hook to update the profile statistics

[Chris Matthews via llvm-commits]

Hi James, 

When submissions happen on our server we get a call process error from this du invocation:

Subprocess failed with:Traceback (most recent call last):
  File "/opt/llvm/lnt/lnt/util/async_ops.py", line 128, in async_wrapper
    nothing = job(ts, **func_args)
  File "/opt/llvm/lnt/lnt/server/db/fieldchange.py", line 18, in post_submit_tasks
    regenerate_fieldchanges_for_run(ts, run_id)
  File "/opt/llvm/lnt/lnt/testing/util/commands.py", line 27, in timed
    result = func(*args, **kw)
  File "/opt/llvm/lnt/lnt/server/db/fieldchange.py", line 136, in regenerate_fieldchanges_for_run
    rules.post_submission_hooks(ts, regressions)
  File "/opt/llvm/lnt/lnt/server/db/rules_manager.py", line 69, in post_submission_hooks
    func(ts, run_id)
  File "/opt/llvm/lnt/lnt/server/db/rules/rule_update_profile_stats.py", line 22, in update_profile_stats
    shell=True).split('\t')[0]
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 575, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
CalledProcessError: Command 'du -s -B 1024 /etc/lnt/data/profiles' returned non-zero exit status 64


I think -B is an invalid option on darwin?

> On Mar 19, 2016, at 7:46 AM, James Molloy via llvm-commits <llvm-commits at lists.llvm.org> wrote:
> 
> Author: jamesm
> Date: Sat Mar 19 09:46:56 2016
> New Revision: 263881
> 
> URL: http://llvm.org/viewvc/llvm-project?rev=263881&view=rev
> Log:
> [profile] Add a rule hook to update the profile statistics
> 
> This rule runs after submission and inspects the profile directory, discovering
> the total size of all profiles and the age distribution of the profiles. It stores
> these in two JSON files that are read by /profile/admin
> 
> Added:
>    lnt/trunk/lnt/server/db/rules/rule_update_profile_stats.py
> 
> Added: lnt/trunk/lnt/server/db/rules/rule_update_profile_stats.py
> URL: http://llvm.org/viewvc/llvm-project/lnt/trunk/lnt/server/db/rules/rule_update_profile_stats.py?rev=263881&view=auto
> ==============================================================================
> --- lnt/trunk/lnt/server/db/rules/rule_update_profile_stats.py (added)
> +++ lnt/trunk/lnt/server/db/rules/rule_update_profile_stats.py Sat Mar 19 09:46:56 2016
> @@ -0,0 +1,35 @@
> +"""
> +Post submission hook to write the current state of the profiles directory. This
> +gets fed into the profile/admin page.
> +"""
> +import json, datetime, os, subprocess, glob, time
> +
> +def update_profile_stats(ts, run_id):
> +    config = ts.v4db.config
> +
> +    history_path = os.path.join(config.profileDir, '_profile-history.json')
> +    age_path = os.path.join(config.profileDir, '_profile-age.json')
> +    profile_path = config.profileDir
> +
> +    try:
> +        history = json.loads(open(history_path).read())
> +    except:
> +        history = []
> +    age = []
> +
> +    dt = time.time()
> +    blocks = subprocess.check_output("du -s -B 1024 %s" % profile_path,
> +                                     shell=True).split('\t')[0]
> +    kb = float(blocks) # 1024 byte blocks.
> +
> +    history.append((dt, kb))
> +
> +    for f in glob.glob('%s/*.lntprof' % profile_path):
> +        mtime = os.stat(f).st_mtime
> +        sz = os.stat(f).st_size / 1000
> +        age.append([mtime, sz])
> +    
> +    open(history_path, 'w').write(json.dumps(history))
> +    open(age_path, 'w').write(json.dumps(age))
> +
> +post_submission_hook = update_profile_stats
> 
> 
> _______________________________________________
> llvm-commits mailing list
> llvm-commits at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits