Experimental Valgrind coverage tool (fwd)
|
View:
New views
6 Messages
—
Rating Filter:
Alert me
|
 |
Experimental Valgrind coverage tool (fwd)
by njn
::
Rate this Message:
[Forwarding this from the valgrind-developers list in response to a query from Robert Lieske about coverage tools]] ---------- Forwarded message ---------- Date: Mon, 4 Feb 2008 10:06:34 +1100 (EST) From: Nicholas Nethercote <njn@...> To: Valgrind Developers <valgrind-developers@...> Subject: [Valgrind-developers] Experimental Valgrind coverage tool Hi, I've written an experimental Valgrind coverage tool, called VCov. To try it out, do this: svn co svn://svn.valgrind.org/valgrind/branches/VCOV Then build in the usual way (see the README file). To run it, use 'valgrind --tool=exp-vcov <program>'. It's pretty simple. It just records, for each line of source code, how many instructions that were derived from that line were executed. It aggregates data from multiple executions. It all goes in a file called "vcov.out". To interpret the data, run this command: perl exp-vcov/vc_annotate vcov.out The output gives total, per-file and per-line coverage. An example: -------------------------------------------------------------------------------- Total coverage -------------------------------------------------------------------------------- 100.0% ( 9 of 9 lines) -------------------------------------------------------------------------------- Per-file coverage (files with the most unexecuted lines are shown first) -------------------------------------------------------------------------------- 100.0% ( 3 of 3 lines): /home/njn/grind/trunk9/b.h 100.0% ( 6 of 6 lines): /home/njn/grind/trunk9/a.c -------------------------------------------------------------------------------- /home/njn/grind/trunk9/b.h -------------------------------------------------------------------------------- -: 1:__inline__ int foo(void) 4: 2:{ 2: 3: return 0; 4: 4:} -------------------------------------------------------------------------------- /home/njn/grind/trunk9/a.c -------------------------------------------------------------------------------- -: 1:#include <stdio.h> -: 2:#include "b.h" -: 3: -: 4:int main(int argc, char* argv[]) 20: 5:{ 4: 6: if (argc < 2) 5: 7: printf("hello, world\n"); -: 8: else 4: 9: printf("hello, extended world\n"); -: 10: 2: 11: return foo(); 4: 12:} -------------------------------------------------------------------------------- Total coverage -------------------------------------------------------------------------------- 88.9% ( 8 of 9 lines) -------------------------------------------------------------------------------- Per-file coverage (files with the most unexecuted lines are shown first) -------------------------------------------------------------------------------- 83.3% ( 5 of 6 lines): /home/njn/grind/trunk9/a.c 100.0% ( 3 of 3 lines): /home/njn/grind/trunk9/b.h -------------------------------------------------------------------------------- /home/njn/grind/trunk9/b.h -------------------------------------------------------------------------------- -: 1:__inline__ int foo(void) 2: 2:{ 1: 3: return 0; 2: 4:} -------------------------------------------------------------------------------- /home/njn/grind/trunk9/a.c -------------------------------------------------------------------------------- -: 1:#include <stdio.h> -: 2:#include "b.h" -: 3: -: 4:int main(int argc, char* argv[]) 10: 5:{ 2: 6: if (argc < 2) 5: 7: printf("hello, world\n"); -: 8: else #####: 9: printf("hello, extended world\n"); -: 10: 1: 11: return foo(); 2: 12:} Non-executable lines are preceded with '-'. Lines that are executable but not executed are preceded with '######', so they are obvious. There are lots of shortcomings, but it gets the general idea across. Some things worth noting: - Files not compiled with -g won't have any results. - You can use the --fresh=yes option to overwrite any prior coverage data. - I plan to add the ability to change the name of the output file. - Results with --trace-children will be suspect, as there is no file locking done yet on the output file. - I hope to add some kind of branch coverage info, at least for the encountered branches. - If your program has a file for which no code is executed, it won't get included in the output. - I plan to add the ability to be more selective about which files get annotated (currently every file that has been executed and that can be found is annotated). Any comments are welcome. Nick ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ Valgrind-developers mailing list Valgrind-developers@... https://lists.sourceforge.net/lists/listinfo/valgrind-developers ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ Valgrind-users mailing list Valgrind-users@... https://lists.sourceforge.net/lists/listinfo/valgrind-users |
|
|
Re: Experimental Valgrind coverage tool (fwd)
by Bugzilla from olig9@gmx.de
::
Rate this Message:
Nicholas Nethercote schrieb:
> [Forwarding this from the valgrind-developers list in response to a query > from Robert Lieske about coverage tools]] > > ---------- Forwarded message ---------- > Date: Mon, 4 Feb 2008 10:06:34 +1100 (EST) > From: Nicholas Nethercote <njn@...> > To: Valgrind Developers <valgrind-developers@...> > Subject: [Valgrind-developers] Experimental Valgrind coverage tool > > Hi, > > I've written an experimental Valgrind coverage tool, called VCov. To try it > out, do this: > > svn co svn://svn.valgrind.org/valgrind/branches/VCOV > > Then build in the usual way (see the README file). To run it, use 'valgrind > --tool=exp-vcov <program>'. > > It's pretty simple. It just records, for each line of source code, how many > instructions that were derived from that line were executed. It aggregates > data from multiple executions. It all goes in a file called "vcov.out". Hi, that sounds quite like I was looking for. But one thing that I've wondered about in particular when using Valgrind for coverage is how it works with C++ templates? If a template is never instantiated, how can a coverage tool detect that there is source code that is never executed? Is that possible somehow? Regards, Oliver Gerlich ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ Valgrind-users mailing list Valgrind-users@... https://lists.sourceforge.net/lists/listinfo/valgrind-users |
|
|
Re: Experimental Valgrind coverage tool (fwd)
by njn
::
Rate this Message:
On Thu, 7 Feb 2008, Oliver Gerlich wrote:
> that sounds quite like I was looking for. But one thing that I've wondered > about in particular when using Valgrind for coverage is how it works with C++ > templates? If a template is never instantiated, how can a coverage tool > detect that there is source code that is never executed? Is that possible > somehow? My tool just records which instructions get executed, and then maps that back to the source code lines according to the debug info. It also identifies source lines mentioned in the debug info which weren't executed. So I don't know the answer to your question, it depends entirely how GCC produces its debug info for templates, but I suspect the lines of code within the template would end up marked as unexecuted. Nick ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ Valgrind-users mailing list Valgrind-users@... https://lists.sourceforge.net/lists/listinfo/valgrind-users |
|
|
Re: Experimental Valgrind coverage tool (fwd)
by Bugzilla from olig9@gmx.de
::
Rate this Message:
Nicholas Nethercote wrote:
> On Thu, 7 Feb 2008, Oliver Gerlich wrote: > >> that sounds quite like I was looking for. But one thing that I've >> wondered about in particular when using Valgrind for coverage is how >> it works with C++ templates? If a template is never instantiated, how >> can a coverage tool detect that there is source code that is never >> executed? Is that possible somehow? > > My tool just records which instructions get executed, and then maps that > back to the source code lines according to the debug info. It also > identifies source lines mentioned in the debug info which weren't > executed. So I don't know the answer to your question, it depends > entirely how GCC produces its debug info for templates, but I suspect > the lines of code within the template would end up marked as unexecuted. > > Nick which parses callgrind/cachegrind output files; but I have not found a way to get it working correctly for this "not-instantiated template" case... That's why I'm interested whether someone has found solution for this. Attached is an example where my coverage tool fails: while it explicitly marks templateFunction3 as not-executed, it doesn't mark templateFunction2 like that. You can see at http://img509.imageshack.us/img509/9682/cgctemplateexample1na6.png how it looks. Also, because of this, the "68% executed" number is wrong as well. Regards, Oliver Gerlich /* Test whether a coverage tool detects unused templates as "not covered". Compile with: g++ -o test -Wall -Wextra -g coverage-template-test.C and run with parameter "10". Correct coverage result would be: - in templateFunction1 and normalFunction, the "else" branch is executed, and the "if" code is not covered. - in templateFunction2 and templateFunction3, no code is not covered. */ #include <iostream> using std::cout; using std::cerr; using std::endl; template <typename T> bool templateFunction1 (T input) { if (input == 23) return true; else return false; } template <typename T> bool templateFunction2 (T input) { if (input == 24) return true; else return false; } template <typename T> bool templateFunction3 (T input) { if (input == 25) return true; else return false; } bool normalFunction (int input) { if (input == 42) return templateFunction3(input); else return false; } int main (int argc, char* argv[]) { if (argc != 2) { cerr << "Usage: " << argv[0] << " <number>" << endl; return 1; } int number = atoi(argv[1]); cout << "number: " << number << endl; cout << "normalFunction: " << normalFunction(number) << endl; cout << "templateFunction1: " << templateFunction1(number) << endl; cout << "templateFunction2: " << /*templateFunction2(number)*/ "NOT EXECUTED" << endl; return 0; } ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ Valgrind-users mailing list Valgrind-users@... https://lists.sourceforge.net/lists/listinfo/valgrind-users |
|
|
Re: Experimental Valgrind coverage tool (fwd)
by Bugzilla from josef.weidendorfer@gmx.de
::
Rate this Message:
On Thursday 07 February 2008, Oliver Gerlich wrote:
> way to get it working correctly for this "not-instantiated template" > case... That's why I'm interested whether someone has found solution for > this. When the compiler does not instantiate a template, there is no code generated for that template, and there can not be any debug info for not existing code. Thus, any coverage tool purely based on the binary and its debug info has no way to know about the existance of template code without instantiation. To make this work, you need additional information from the compiler. Does gcov cover this issue? Josef ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ Valgrind-users mailing list Valgrind-users@... https://lists.sourceforge.net/lists/listinfo/valgrind-users |
|
|
Re: Experimental Valgrind coverage tool (fwd)
by Bugzilla from tom@compton.nu
::
Rate this Message:
In message <200802071350.08204.Josef.Weidendorfer@...>
Josef Weidendorfer <Josef.Weidendorfer@...> wrote: > On Thursday 07 February 2008, Oliver Gerlich wrote: >> way to get it working correctly for this "not-instantiated template" >> case... That's why I'm interested whether someone has found solution for >> this. > > When the compiler does not instantiate a template, there is > no code generated for that template, and there can not be any debug info > for not existing code. > > Thus, any coverage tool purely based on the binary and its debug info > has no way to know about the existance of template code without instantiation. But equally the code can never be executed because it doesn't exist. So you know what percentage of the code that actually exists in your shipping executable is tested, you just don't know about hypothetical code that might be created some day if somebody instantiates a template differently. Tom -- Tom Hughes (tom@...) http://www.compton.nu/ ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ Valgrind-users mailing list Valgrind-users@... https://lists.sourceforge.net/lists/listinfo/valgrind-users |
| Free embeddable forum powered by Nabble | Forum Help |
