[PATCH] Junit report for Subversion tests
|
View:
New views
20 Messages
—
Rating Filter:
Alert me
|
 |
[PATCH] Junit report for Subversion tests
by Bhuvaneswaran Arumugam
::
Rate this Message:
Hello,
Please find attached a script to generate the Junit report based on Subversion test results. The junit reports can be used to integrate the Subversion build process with continuous integration tools like Hudson. [[ New script to generate the junit reports for Subversion test cases. The junit reports can be used by continuous integration tools like Hudson for building and testing Subversion. * tools/dev/gen_junit_report.py The junit reports are generated by parsing the tests log file, tests.log. The test results, PASS, FAIL and SKIP are marked as appropriate in the junit reports. The XFAIL tests are considered as PASS. This script can take options: --log-file and --output-dir. The value for --log-file option denotes the log file to parse for generating the junit reports. The value for --output-dir option denotes the directory to store the junit reports. A separate file is created in this directory for each test case. ]] -- Bhuvaneswaran A CollabNet Software P Ltd. | www.collab.net ------------------------------------------------------ http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2410275 [gen-junit.diff] Index: tools/dev/gen_junit_report.py =================================================================== --- tools/dev/gen_junit_report.py (revision 0) +++ tools/dev/gen_junit_report.py (revision 0) @@ -0,0 +1,232 @@ +#!/bin/env python + +# $Id$ +""" +gen_junit_report.py -- The script is to generate the junit report for +Subversion tests. The script uses the log file, tests.log created by +"make check" process. It parses the log file and generate the junit +files for each test separately in the specified output directory. The +script can take --log-file and --output-dir arguments. +""" + +import sys +import os +import getopt + +def xml_encode(data): + """encode the xml characters in the data""" + encode = { + '&': '&', + '<': '<', + '>': '>', + '"': '"', + "'": ''' + } + for char in encode.keys(): + data = data.replace(char, encode[char]) + return data + +def start_junit(): + """define the beginning of xml document""" + head = """<?xml version="1.0" encoding="UTF-8"?>""" + return head + +def start_testsuite(test_name): + """start testsuite. The value for the attributes are replaced later + when the junit file handling is concluded""" + sub_test_name = test_name.replace('.', '-') + start = """<testsuite time="ELAPSED_%s" tests="TOTAL_%s" name="%s" + failures="FAIL_%s" errors="FAIL_%s" skipped="SKIP_%s">""" % \ + (test_name, test_name, sub_test_name, test_name, test_name, test_name) + return start + +def junit_testcase_ok(test_name, casename): + """mark the test case as PASSED""" + casename = xml_encode(casename) + sub_test_name = test_name.replace('.', '-') + case = """<testcase time="ELAPSED_CASE_%s" name="%s" classname="%s"/>""" % \ + (test_name, casename, sub_test_name) + return case + +def junit_testcase_fail(test_name, casename, reason=None): + """mark the test case as FAILED""" + casename = xml_encode(casename) + sub_test_name = test_name.replace('.', '-') + case = """<testcase time="ELAPSED_CASE_%s" name="%s" classname="%s"> + <failure type="Failed"><![CDATA[%s]]></failure> + </testcase>""" % (test_name, casename, sub_test_name, reason) + return case + +def junit_testcase_xfail(test_name, casename, reason=None): + """mark the test case as XFAILED""" + casename = xml_encode(casename) + sub_test_name = test_name.replace('.', '-') + case = """<testcase time="ELAPSED_CASE_%s" name="%s" classname="%s"> + <system-out><![CDATA[%s]]></system-out> + </testcase>""" % (test_name, casename, sub_test_name, reason) + return case + +def junit_testcase_skip(test_name, casename): + """mark the test case as SKIPPED""" + casename = xml_encode(casename) + sub_test_name = test_name.replace('.', '-') + case = """<testcase time="ELAPSED_CASE_%s" name="%s" classname="%s"> + <skipped message="Skipped"/> + </testcase>""" % (test_name, casename, sub_test_name) + return case + +def end_testsuite(): + """mark the end of testsuite""" + end = """</testsuite>""" + return end + +def update_stat(test_name, junit, count): + """update the test statistics in the junit string""" + junit_str = '\n'.join(junit) + t_count = count[test_name] + total = t_count['pass'] + t_count['fail'] + t_count['skip'] + elapsed = t_count['elapsed'] + case_time = 0 + if total > 0: # there are tests with no test cases + case_time = '%.2f' % float(elapsed/total) + + total_patt = 'TOTAL_%s' % test_name + fail_patt = 'FAIL_%s' % test_name + skip_patt = 'SKIP_%s' % test_name + elapsed_patt = 'ELAPSED_%s' % test_name + elapsed_case_patt = 'ELAPSED_CASE_%s' % test_name + + # replace the pattern in junit string with actual statistics + junit_str = junit_str.replace(total_patt, "%s" % total) + junit_str = junit_str.replace(fail_patt, "%s" % t_count['fail']) + junit_str = junit_str.replace(skip_patt, "%s" % t_count['skip']) + junit_str = junit_str.replace(elapsed_patt, "%d" % elapsed) + junit_str = junit_str.replace(elapsed_case_patt, "%s" % case_time) + return junit_str + +def main(): + """main method""" + try: + opts, args = getopt.getopt(sys.argv[1:], 'l:d:h', + ['log-file=', 'output-dir=', 'help']) + except getopt.GetoptError, err: + usage(err) + + log_file = None + output_dir = None + for opt, value in opts: + if (opt in ('-h', '--help')): + usage() + elif (opt in ('-l', '--log-file')): + log_file = value + elif (opt in ('-d', '--output-dir')): + output_dir = value + else: + usage('Unable to recognize option') + + if not log_file or not output_dir: + usage("The options --log-file and --output-dir are mandatory") + + # create junit output directory, if not exists + if not os.path.exists(output_dir): + print "Directory '%s' not exists, creating ..." % output_dir + try: + os.makedirs(output_dir) + except OSError, err: + print >> sys.stderr, "ERROR: %s" % err + sys.exit(1) + patterns = { + 'start' : 'START:', + 'end' : 'END:', + 'pass' : 'PASS:', + 'skip' : 'SKIP:', + 'fail' : 'FAIL:', + 'xfail' : 'XFAIL:', + 'elapsed' : 'ELAPSED:' + } + + junit = [] + junit.append(start_junit()) + reason = None + count = {} + fp = None + try: + fp = open(log_file, 'r') + except IOError, err: + print >> sys.stderr, "ERROR: %s" % err + sys.exit(1) + + for line in fp.readlines(): + line = line.strip() + if line.startswith(patterns['start']): + reason = "" + test_name = line.split(' ')[1] + # replace '.' in test name with '_' to avoid confusing class + # name in test result displayed in the CI user interface + test_name.replace('.', '_') + count[test_name] = { + 'pass' : 0, + 'skip' : 0, + 'fail' : 0, + 'xfail' : 0, + 'elapsed' : 0, + 'total' : 0 + } + junit.append(start_testsuite(test_name)) + elif line.startswith(patterns['end']): + junit.append(end_testsuite()) + elif line.startswith(patterns['pass']): + reason = "" + casename = line.strip(patterns['pass']).strip() + junit.append(junit_testcase_ok(test_name, casename)) + count[test_name]['pass'] += 1 + elif line.startswith(patterns['skip']): + reason = "" + casename = line.strip(patterns['skip']).strip() + junit.append(junit_testcase_skip(test_name, casename)) + count[test_name]['skip'] += 1 + elif line.startswith(patterns['fail']): + casename = line.strip(patterns['fail']).strip() + junit.append(junit_testcase_fail(test_name, casename, reason)) + count[test_name]['fail'] += 1 + reason = "" + elif line.startswith(patterns['xfail']): + casename = line.strip(patterns['xfail']).strip() + junit.append(junit_testcase_xfail(test_name, casename, reason)) + count[test_name]['pass'] += 1 + reason = "" + elif line.startswith(patterns['elapsed']): + reason = "" + elapsed = line.split(' ')[2].strip() + (hrs, mins, secs) = elapsed.split(':') + secs_taken = int(hrs)*24 + int(mins)*60 + int(secs) + count[test_name]['elapsed'] = int(secs_taken) + + junit_str = update_stat(test_name, junit, count) + test_junit_file = os.path.join(output_dir, + "%s.junit.xml" % test_name) + w_fp = open (test_junit_file, 'w') + w_fp.writelines(junit_str) + w_fp.close() + junit = [] + elif len(line): + reason = "%s\n%s" % (reason, line) + fp.close() + +def usage(errorMsg=None): + script_name = os.path.basename(sys.argv[0]) + sys.stdout.write("""USAGE: %s: [--help|h] --log-file|l --output-dir|d + +Options: + --help|-h Display help message + --log-file|l The log file to parse for generating junit xml files + --output-dir|d The directory to create the junit xml file for each + test +""" % script_name) + if errorMsg is not None: + sys.stderr.write("\nERROR: %s\n" % errorMsg) + sys.exit(1) + sys.exit(0) + +if __name__ == '__main__': + main() Property changes on: tools/dev/gen_junit_report.py ___________________________________________________________________ Added: svn:executable + * Added: svn:keywords + Id |
|
|
Re: [PATCH] Junit report for Subversion tests
by Hyrum K. Wright-4
::
Rate this Message:
|
|
|
Re: [PATCH] Junit report for Subversion tests
by Bhuvaneswaran Arumugam
::
Rate this Message:
On Thu, 2009-10-22 at 10:31 -0500, Hyrum K. Wright wrote:
> We've been using this to test a sample Hudson instance for > Subversion > trunk and 1.6.x. I asked Bhuvan to send the patch here so I could > +1 > it to live in the repo. Version controlled, open source, useful for > posterity, and all that. > > +1 to commit. Thank you, Hyrum. Committed this script in r40187. May I: (a) nominate this script for 1.6.x branch. (b) include your vote, in addition to mine in 1.6.x/STATUS? -- Bhuvaneswaran A CollabNet Software P Ltd. | www.collab.net ------------------------------------------------------ http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2410325 |
|
|
Re: [PATCH] Junit report for Subversion tests
by Mark Phippard-3
::
Rate this Message:
|
|
|
Re: [PATCH] Junit report for Subversion tests
by Bhuvaneswaran Arumugam
::
Rate this Message:
On Thu, 2009-10-22 at 13:49 -0400, Mark Phippard wrote:
> > Before we backport it, let's be sure we have finished fine-tuning it > on trunk. For example, I do not think the times it is reporting is > right. 12 minutes for all the tests? I do not think so. Also see > this page that has times: > > http://orac.ece.utexas.edu:8080/job/svn-trunk/294/testReport/(root)/ > > Being able to see trends in the test times is one of the things I am > hoping to get out of this so it would be good to be sure it is right. with 4GB RAM having *4* processors, each running "Intel(R) Core(TM)2 Quad CPU @ 2.66GHz". Hyrum, correct me if i'm wrong. The sum of all >1 seconds test cases sum up to 12.6 minutes. -- Bhuvaneswaran A CollabNet Software P Ltd. | www.collab.net ------------------------------------------------------ http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2410342 |
|
|
Re: [PATCH] Junit report for Subversion tests
by Mark Phippard-3
::
Rate this Message:
|
|
|
Re: [PATCH] Junit report for Subversion tests
by Hyrum K. Wright-4
::
Rate this Message:
|
|
|
Re: [PATCH] Junit report for Subversion tests
by Branko Cibej
::
Rate this Message:
|
|
|
Re: [PATCH] Junit report for Subversion tests
by Bhuvaneswaran Arumugam
::
Rate this Message:
On Thu, 2009-10-22 at 22:09 +0200, Branko Čibej wrote:
> > Upon further consideration, I think that we should be recording > milliseconds for elapsed times, too. Yes, I agree. I'll attach a patch for this. Thank you. -- Bhuvaneswaran A CollabNet Software P Ltd. | www.collab.net ------------------------------------------------------ http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2411327 |
|
|
Re: [PATCH] Junit report for Subversion tests
by Bhuvaneswaran Arumugam
::
Rate this Message:
On Thu, 2009-10-22 at 14:11 -0500, Hyrum K. Wright wrote:
> > That does seem rather quick. I just ran 'make check' on a separate > working copy on the same machine, and got something different: > real 32m51.756s > user 6m40.321s > sys 6m27.504s > > I don't know that this is a problem with the junit converter script, > or perhaps the test running reporting bogus ELAPSED times. This seem to be a problem with combination of both scripts. As there is no way to calculate the time taken for EACH TEST, we calculate it based on total time taken. However it is not done correctly. For instance, consider a test suite that has 10 tests taking 15 seconds to complete. We don't know the time taken for 10 tests individually. We only know that it took 15 seconds for all 10 tests together. We calculate the time taken for individual tests as 1 instead of 1.5. It's due to a bug in gen_junit_report.py script. On the other hand, since we do not account for milliseconds, those tests are recorded as 0 ms for individual tests. The Hudson as such, does not recognise the time specified for each test suite in the junit report. Instead it calculates the time taken for each test suite based on time taken for each test individually. Due to this bug, in the above use case, the Hudson would report 10 seconds, because according to the junit report, each test has taken 1 second. Bottom line, I see two issues to be fixed here: a) Include milliseconds in the tests log file, tests.log. b) Do not round the result while calculating the time taken for each test. Attached a patch for issue {a} in another thread. > Another problem: on some of the test information pages, I see stuff > like: > lock_tests-py.lock_tests.py 1: lock a file and verify that it's > locked > The html entity looks a bit out-of-place. We do escape the xml entities {<,>,',",&}. They seem to work well with the test names when they are displayed in the report. In the final screen where we see the status of the test, it does not play along well. I'll see to fix this. Thank you. -- Bhuvaneswaran A CollabNet Software P Ltd. | www.collab.net ------------------------------------------------------ http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2411335 |
|
|
Re: [PATCH] Junit report for Subversion tests
by Mark Phippard-3
::
Rate this Message:
|
|
|
Re: [PATCH] Junit report for Subversion tests
by Bhuvaneswaran Arumugam
::
Rate this Message:
On Mon, 2009-10-26 at 08:54 -0400, Mark Phippard wrote:
> > I do not understand how this accounts for such a big discrepancy in > the time reported? For example, as I pointed out, the entire > commit_tests.py reports as taking 0ms. How is including and not > rounding the milliseconds going to suddenly make that reports as the > several minutes that it in fact does take? I'll explain the issue using the commit_tests.py test suite. Originally, the commit_tests.py test suite has taken 39 seconds to complete. In total, there are 62 tests comprising this test suite. The time taken by each test is set as 0 seconds in the junit file instead of 0.62 seconds (39/62). Due to this fact, the Hudson set the total time taken for this test suite as 0 (0*62). It is irrespective of the fact that the actual time for this test suite is defined as 39 seconds in the junit file, globally. > As someone else speculated, is it possible that the test setup and > teardown is not being included in the times? The ELAPSED: line in tests.log prints the *exact* time taken to execute each test suite. If the setup/teardown part is not performed within START/END of test suite, then yes, it's not included. I'll look into this, once we are done with the known bug. Thanks! -- Bhuvaneswaran A CollabNet Software P Ltd. | www.collab.net ------------------------------------------------------ http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2411393 |
|
|
Re: [PATCH] Junit report for Subversion tests
by Mark Phippard-3
::
Rate this Message:
|
|
|
Re: [PATCH] Junit report for Subversion tests
by Hyrum K. Wright-4
::
Rate this Message:
|
|
|
Re: [PATCH] Junit report for Subversion tests
by Bhuvaneswaran Arumugam
::
Rate this Message:
On Mon, 2009-10-26 at 22:09 +0100, Hyrum K. Wright wrote:
> > On Oct 26, 2009, at 3:02 PM, Bhuvaneswaran A wrote: > > > On Mon, 2009-10-26 at 08:54 -0400, Mark Phippard wrote: > >> > >> I do not understand how this accounts for such a big discrepancy in > >> the time reported? For example, as I pointed out, the entire > >> commit_tests.py reports as taking 0ms. How is including and not > >> rounding the milliseconds going to suddenly make that reports as > the > >> several minutes that it in fact does take? > > > > I'll explain the issue using the commit_tests.py test suite. > > > > Originally, the commit_tests.py test suite has taken 39 seconds to > > complete. In total, there are 62 tests comprising this test suite. > The > > I find it really hard to believe that the commit test suite only > takes > 39 seconds to run. says: ELAPSED: commit_tests.py 00:00:39 > (In other news, the admin of orac tells me it had a disk failure, so > it may be offline for a bit.) Sounds like it's not up even after 4hrs. -- Bhuvaneswaran A CollabNet Software P Ltd. | www.collab.net ------------------------------------------------------ http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2411568 |
|
|
Re: [PATCH] Junit report for Subversion tests
by Branko Cibej
::
Rate this Message:
|
|
|
Re: [PATCH] Junit report for Subversion tests
by Gavin Baumanis-4
::
Rate this Message:
|
|
|
Re: [PATCH] Junit report for Subversion tests
by Gavin Baumanis-4
::
Rate this Message:
|
|
|
Re: [PATCH] Junit report for Subversion tests
by Bhuvaneswaran Arumugam
::
Rate this Message:
On Sat, 2009-11-07 at 20:30 +1100, Gavin 'Beau' Baumanis wrote:
> I have logged issue #3521 > http://subversion.tigris.org/issues/show_bug.cgi?id=3521 > > Gavin. Gavin, The script along with follow-up fixes are already committed to trunk. See r40187 and r40228 for details. > On 03/11/2009, at 10:18 , Gavin Baumanis wrote: > > > Hi All, > > > > Although there is a patch submission - there was a quite a bit of > > discussion too. > > Though there doesn't seem to have been an agreement or a subsequent > > patch submission (if at all needed). > > But since it has been a week now without activity I thought I would > > bring it back up the list - in case it had fallen off people's radar. > > > > Gavin. > > > > On 27/10/2009, at 14:55 , Branko Cibej wrote: > > > >> Hyrum K. Wright wrote: > >>> On Oct 22, 2009, at 1:17 PM, Bhuvaneswaran A wrote: > >>> > >>> > >>>> The sum of all >1 seconds test cases sum up to 12.6 minutes. > >>>> > >>> > >>> That does seem rather quick. I just ran 'make check' on a separate > >>> working copy on the same machine, and got something different: > >>> real 32m51.756s > >>> user 6m40.321s > >>> sys 6m27.504s > >>> > >>> I don't know that this is a problem with the junit converter script, > >>> or perhaps the test running reporting bogus ELAPSED times. > >>> > >> > >> Apparently it's quite reasonable. As I posted elsewhere, I got a > >> similar > >> "real" time on my laptop (with tmpfs and cleanups and parallel test > >> executions), but my user time was almost 22 minutes. And I have an > >> elapsed time for commit-tests.py under one minute. While it's easy to > >> imagine this being twice as fast on a real machine with something > >> more > >> than this trundle for a CPU, I can hardly imagine how that would be > >> truncated to 0 seconds. > >> > >> -- Brane > >> > >> ------------------------------------------------------ > >> http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2411585 > > > > ------------------------------------------------------ > > http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2413974 > -- Bhuvaneswaran A CollabNet Software P Ltd. | www.collab.net ------------------------------------------------------ http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2415681 |
|
|
Re: [PATCH] Junit report for Subversion tests
by Gavin Baumanis-4
::
Rate this Message:
|
signature.asc (204 bytes) | Free embeddable forum powered by Nabble | Forum Help |
