[LNT] r263881 - [profile] Add a rule hook to update the profile statistics

[Chris Matthews via llvm-commits]

Thanks. No rush, I added a simple work around on our server.

> On Apr 7, 2016, at 3:23 PM, James Molloy <James.Molloy at arm.com> wrote:
> 
> Hi Chris,
> 
> Thanks for the bug report! That should be simple to fix and its benign in the meantime.
> 
> Ccing Kristof as I'm heading to Nepal with no internet access for two weeks tomorrow!
> 
> James
> 
>> On 7 Apr 2016, at 18:57, Chris Matthews <chris4000 at gmail.com> wrote:
>> 
>> Hi James,
>> 
>> When submissions happen on our server we get a call process error from this du invocation:
>> 
>> Subprocess failed with:Traceback (most recent call last):
>> File "/opt/llvm/lnt/lnt/util/async_ops.py", line 128, in async_wrapper
>>   nothing = job(ts, **func_args)
>> File "/opt/llvm/lnt/lnt/server/db/fieldchange.py", line 18, in post_submit_tasks
>>   regenerate_fieldchanges_for_run(ts, run_id)
>> File "/opt/llvm/lnt/lnt/testing/util/commands.py", line 27, in timed
>>   result = func(*args, **kw)
>> File "/opt/llvm/lnt/lnt/server/db/fieldchange.py", line 136, in regenerate_fieldchanges_for_run
>>   rules.post_submission_hooks(ts, regressions)
>> File "/opt/llvm/lnt/lnt/server/db/rules_manager.py", line 69, in post_submission_hooks
>>   func(ts, run_id)
>> File "/opt/llvm/lnt/lnt/server/db/rules/rule_update_profile_stats.py", line 22, in update_profile_stats
>>   shell=True).split('\t')[0]
>> File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 575, in check_output
>>   raise CalledProcessError(retcode, cmd, output=output)
>> CalledProcessError: Command 'du -s -B 1024 /etc/lnt/data/profiles' returned non-zero exit status 64
>> 
>> 
>> I think -B is an invalid option on darwin?
>> 
>>> On Mar 19, 2016, at 7:46 AM, James Molloy via llvm-commits <llvm-commits at lists.llvm.org> wrote:
>>> 
>>> Author: jamesm
>>> Date: Sat Mar 19 09:46:56 2016
>>> New Revision: 263881
>>> 
>>> URL: http://llvm.org/viewvc/llvm-project?rev=263881&view=rev
>>> Log:
>>> [profile] Add a rule hook to update the profile statistics
>>> 
>>> This rule runs after submission and inspects the profile directory, discovering
>>> the total size of all profiles and the age distribution of the profiles. It stores
>>> these in two JSON files that are read by /profile/admin
>>> 
>>> Added:
>>>  lnt/trunk/lnt/server/db/rules/rule_update_profile_stats.py
>>> 
>>> Added: lnt/trunk/lnt/server/db/rules/rule_update_profile_stats.py
>>> URL: http://llvm.org/viewvc/llvm-project/lnt/trunk/lnt/server/db/rules/rule_update_profile_stats.py?rev=263881&view=auto
>>> ==============================================================================
>>> --- lnt/trunk/lnt/server/db/rules/rule_update_profile_stats.py (added)
>>> +++ lnt/trunk/lnt/server/db/rules/rule_update_profile_stats.py Sat Mar 19 09:46:56 2016
>>> @@ -0,0 +1,35 @@
>>> +"""
>>> +Post submission hook to write the current state of the profiles directory. This
>>> +gets fed into the profile/admin page.
>>> +"""
>>> +import json, datetime, os, subprocess, glob, time
>>> +
>>> +def update_profile_stats(ts, run_id):
>>> +    config = ts.v4db.config
>>> +
>>> +    history_path = os.path.join(config.profileDir, '_profile-history.json')
>>> +    age_path = os.path.join(config.profileDir, '_profile-age.json')
>>> +    profile_path = config.profileDir
>>> +
>>> +    try:
>>> +        history = json.loads(open(history_path).read())
>>> +    except:
>>> +        history = []
>>> +    age = []
>>> +
>>> +    dt = time.time()
>>> +    blocks = subprocess.check_output("du -s -B 1024 %s" % profile_path,
>>> +                                     shell=True).split('\t')[0]
>>> +    kb = float(blocks) # 1024 byte blocks.
>>> +
>>> +    history.append((dt, kb))
>>> +
>>> +    for f in glob.glob('%s/*.lntprof' % profile_path):
>>> +        mtime = os.stat(f).st_mtime
>>> +        sz = os.stat(f).st_size / 1000
>>> +        age.append([mtime, sz])
>>> +
>>> +    open(history_path, 'w').write(json.dumps(history))
>>> +    open(age_path, 'w').write(json.dumps(age))
>>> +
>>> +post_submission_hook = update_profile_stats
>>> 
>>> 
>>> _______________________________________________
>>> llvm-commits mailing list
>>> llvm-commits at lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits
>> 
> IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
>