<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body>
<div dir="auto" style="direction: ltr; margin: 0; padding: 0; font-family: sans-serif; font-size: 11pt; color: black; ">
Hi Yahav, <br>
<br>
<br>
</div>
<div dir="auto" style="direction: ltr; margin: 0; padding: 0; font-family: sans-serif; font-size: 11pt; color: black; ">
I would suggest using clang-tidy and it's checks as inspiration. However for the purposes of what you need it would be best to make a new libtooling tool that still registers matchers and callbacks but doesn't have all the boilerplate that clang tidy needs.
This will make it easier to implement the hooks you need for altering a students grade as well as prevent things like `//NOLINT` comments.<br>
<br>
</div>
<div dir="auto" style="direction: ltr; margin: 0; padding: 0; font-family: sans-serif; font-size: 11pt; color: black; ">
As for the issue with compiler errors, that shouldn't be a huge problem as the AST will get generated even with small compiler errors that the compiler can infer the fix for.
<br>
<br>
</div>
<div dir="auto" style="direction: ltr; margin: 0; padding: 0; font-family: sans-serif; font-size: 11pt; color: black; ">
I do have another suggestion if you want to speed up marking the submissions is a testing framework. Just a tool that will compile and run the students submissions with specified inputs and verify their code has the desired output, after all their is no point
in pretty code bug produces bad outputs. <br>
<br>
</div>
<div dir="auto" style="direction: ltr; margin: 0; padding: 0; font-family: sans-serif; font-size: 11pt; color: black; ">
Regards, <br>
</div>
<div dir="auto" style="direction: ltr; margin: 0; padding: 0; font-family: sans-serif; font-size: 11pt; color: black; ">
Nathan James<br>
<br>
</div>
<div dir="auto" style="direction: ltr; margin: 0; padding: 0; font-family: sans-serif; font-size: 11pt; color: black; ">
<span id="OutlookSignature">
<div dir="auto" style="direction: ltr; margin: 0; padding: 0; font-family: sans-serif; font-size: 11pt; color: black; ">
Get <a href="https://aka.ms/ghei36">Outlook for Android</a></div>
</span><br>
</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Yahav Bar <yahav.bar@mail.huji.ac.il><br>
<b>Sent:</b> Saturday, February 29, 2020 7:11:54 PM<br>
<b>To:</b> Nathan James <n.james93@hotmail.co.uk><br>
<b>Cc:</b> cfe-dev@lists.llvm.org <cfe-dev@lists.llvm.org><br>
<b>Subject:</b> Re: [cfe-dev] Using clang static analyser / clang-tidy for assignments grading?</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">Hi Nathan! How're you?</div>
<div dir="ltr">Thanks for your kind reply!
<div><br>
</div>
<div>I started to dive into Clang and while it does require a learning curve, I think that it'll be perfect. </div>
<div><br>
</div>
<div>About manual grading: we do have a manual grading process, but we do want to automate simple tasks as checking if a forbidden function was used, as it's really easy to miss it when grading an exercise. Of course, when the system will report on such things,
we will check it manually as well, but the first check, ideally, be automatic.</div>
<div><br>
</div>
<div>My main wonder now is if I should write a clang-tidy library, or create an entirely new tool, from scratch, using Clang and Libtooling. I'd love to hear your thoughts: On the one hand, clang-tidy is very mature, got modules support and even contain some
tests that we need. But on the other hand, I have a few concerns that I'm worried about:</div>
<div>
<ul>
<li>We want to perform actions with the diagnostic results. For example, if a student used a public class variable, we want to reduce points on that. Our faculty has an internal file format that you can use to automatically reduce (or add) points from a student.
So, we need some kind of a hook that allows us to get a report from "NonPrivateMemberVariablesInClassesCheck" so that we can write on that file at that point.<br>
As far as I understand, clang-tidy only allows us to print the diagnostics to the screen. Is there any way I can achieve that behaviour using clang-tidy?</li><li>clang-tidy allows disabling linting using "// NOLINT" comment. We want to disable this feature, so that students won't be able to cancel our checks. If I understand correctly, it can be done using CommentHandler but I want to make sure, as otherwise, we
won't be able to use clang-tidy.</li><li>We want to perform an API comparison between the expected API definition (provided as JSON/YAML etc.) and the actual implementation API. Based on that, we want to write a new .h file that contains macro for every implemented function and feature. For example,
if we asked the students to implement HashMap<K, V>::at(const K&), we will write on the output .h file a macro named HASHMAP_METHOD_AT if the method exists, and in addition, we will write on it HASHMAP_METHOD_AT_RETURN_LVALUE if the function returns an lvalue.
We need that as we write some C++ tests to check the student's code, and we noticed that they sometimes mistake between lvalue and rvalue, so then our tests won't compile and everything falls for them. In addition, sometimes they even forget to implement some
methods (for example, in the HashMap exercise we gave them, some forgot to implement the entire set of methods iterator requires) - and that killed the compilation process as well. <br>
<br>
That's just a concept idea as I'm not sure how good it'll get implemented, but our end goal is to be able to write C++ tests that won't get compilation errors if the students didn't implement correctly the entire API.<br>
</li></ul>
<div>So my question is, do you think that clang-tidy is suitable for such things? or I should write a new Clang and Libtooling based tool that can do that?</div>
</div>
<div>I really appreciate your time and efforts to help us!</div>
<div><br>
</div>
<div>Best regards,</div>
<div>Yahav.</div>
<div><br>
</div>
</div>
</div>
</div>
<br>
<div class="x_gmail_quote">
<div dir="ltr" class="x_gmail_attr">On Sat, Feb 29, 2020 at 4:57 AM Nathan James <<a href="mailto:n.james93@hotmail.co.uk" target="_blank">n.james93@hotmail.co.uk</a>> wrote:<br>
</div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left-width:1px; border-left-style:solid; border-left-color:rgb(204,204,204); padding-left:1ex">
<div>
<div dir="auto" style="direction:ltr; margin:0px; padding:0px; font-family:sans-serif; font-size:11pt; color:black">
Hi Yahav, <br>
</div>
<div dir="auto" style="direction:ltr; margin:0px; padding:0px; font-family:sans-serif; font-size:11pt; color:black">
<br>
</div>
<div dir="auto" style="direction:ltr; margin:0px; padding:0px; font-family:sans-serif; font-size:11pt; color:black">
It is certainly possible using the static analyser and clang tidy frameworks to implement most of that checking. In clang tidy there are dedicated checks for some of the restrictions you need. Others would be vary easy to write AST matches or preprocessor callbacks
for. <br>
<br>
</div>
<div dir="auto" style="direction:ltr; margin:0px; padding:0px; font-family:sans-serif; font-size:11pt; color:black; text-align:left">
However it's my personal opinion that this should not be used as a replacement for a human grading but more as an assistant. <span style="font-size:11pt">When patches are submitted for review here we have automatic checks that check validity and can reject
obvious flaws but it is always up to a reviewer to have a look over before a go ahead is given. </span></div>
<div dir="auto" style="direction:ltr; margin:0px; padding:0px; font-family:sans-serif; font-size:11pt; color:black; text-align:left">
<br>
</div>
<div dir="auto" style="direction:ltr; margin:0px; padding:0px; font-family:sans-serif; font-size:11pt; color:black; text-align:left">
If you do decide to go down the route of using clang to help out then definitely check out the code in the clang-tidy checks(static analyser not so much), have a look at the AST matches docs
<a href="https://clang.llvm.org/docs/LibASTMatchersReference.html" target="_blank">
https://clang.llvm.org/docs/LibASTMatchersReference.html</a> and have a play with clang-query. </div>
<div dir="auto" style="direction:ltr; margin:0px; padding:0px; font-family:sans-serif; font-size:11pt; color:black; text-align:left">
<br>
</div>
<div dir="auto" style="direction:ltr; margin:0px; padding:0px; font-family:sans-serif; font-size:11pt; color:black; text-align:left">
Kind regards, </div>
<div dir="auto" style="direction:ltr; margin:0px; padding:0px; font-family:sans-serif; font-size:11pt; color:black; text-align:left">
Nathan James. </div>
<div id="x_gmail-m_3769326466940666063gmail-m_-2449567757242470922id-4416d62a-f228-4f62-b057-0c51f86c7952">
<div style="font-family:sans-serif; font-size:12pt; color:rgb(0,0,0)"><br>
</div>
<hr style="display:inline-block; width:98%">
<div id="x_gmail-m_3769326466940666063gmail-m_-2449567757242470922divRplyFwdMsg">
<strong>From:</strong> cfe-dev <<a href="mailto:cfe-dev-bounces@lists.llvm.org" target="_blank">cfe-dev-bounces@lists.llvm.org</a>> on behalf of Yahav Bar via cfe-dev <<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a>><br>
<strong>Sent:</strong> Friday, 28 February 2020, 16:29<br>
<strong>To:</strong> <a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a><br>
<strong>Subject:</strong> [cfe-dev] Using clang static analyser / clang-tidy for assignments grading?<br>
</div>
<br>
<div dir="ltr">Hi everyone!
<div>I hope that I'm querying the correct mailing list about my question.</div>
<div><br>
</div>
<div>I work as a teaching assistant at the Hebrew University, teaching C and C++. As part of our course, we ask the students to submit C and C++ exercises which we grade (both manually and automatically).</div>
<div><br>
</div>
<div>When grading students exercises, we check the validity of their code. For example,</div>
<div>
<ul>
<li>We check if the students didn't forget to use include safe-guard;</li><li>If the students used non-safe functions, which we consider forbidden to use (in addition, sometimes we explicitly tell students not to use set of predefined functions or C++ classes, as the exercises ask them to implement this set of functions).</li><li>If the students didn't include a forbidden header;</li><li>If the students didn't use a #pragma statement to bypass our compilation instructions;</li><li>C++: If the students remembered to use "const" when required, and to pass parameters by reference when needed;</li><li>C++: that the students returned lvalue when needed and rvalue when needed etc.</li><li>C++: When writing an iterator, if it was implemented correctly (a.k.a, according to Input/Output/Forward/Bidirectional/Random Access iterator rules + using iterator traits).</li></ul>
<div>Right now these tests are being done by a human. As our classes formed from 300 to 600 students (next semester we'll have 650...) it'll be really hard and non-efficient to do it by hand. Thus I thought it might be a good idea to automate these checks too.</div>
</div>
<div><br>
</div>
<div>Initially, I thought to write that with Python using ANTLr (creating an AST for both the Preprocessing stage and the C stage and just iterating over them), but at the middle of programming, I came across the Static Analyser API of Clang and thought I should
switch to it, as it seems very mature and well fit for our needs.</div>
<div><br>
</div>
<div>Before diving deeply into clang, I'd love to hear from you, who have experience in clang and LLVM dev, if I'm on the right track, and can actually achieve my goal using clang, or I should stick with my previous attempt. Our end goal is a program that we
can run, send the path to the student exercise, and get the errors that she had so that we can reduce points accordingly. </div>
<div><br>
</div>
<div>Thank you very much!</div>
<div>Yahav.</div>
</div>
<br>
</div>
</div>
</blockquote>
</div>
</div>
</body>
</html>