[rvm-research] Automating test case generation?
|
View:
New views
6 Messages
—
Rating Filter:
Alert me
|
 |
[rvm-research] Automating test case generation?
by Suriya Subramanian
::
Rate this Message:
Hi all,
Quite often, I encounter Jikes RVM bugs that show up when running complex applications. Most of the time, I have been able to construct small test programs that demonstrate buggy behavior. Starting with the application jEdit I was able to find this bug RVM-868 http://jira.codehaus.org/browse/RVM-868 and a test case http://jira.codehaus.org/secure/attachment/45362/ResolverTestCase.java that is about 20 lines long. I was lucky today and it took me about 15 minutes to get to here. This is an exception. Usually, this process has been difficult and time-consuming. Are there ways to automate this process, i.e. removing parts of an application that do not contribute to some behavior we are looking at? I am sure there should be a lot of research done in this area. Can someone give me pointers. Are there tools that are readily usable? Thanks, Suriya ------------------------------------------------------------------------------ Come build with us! The BlackBerry(R) Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9 - 12, 2009. Register now! http://p.sf.net/sfu/devconference _______________________________________________ Jikesrvm-researchers mailing list Jikesrvm-researchers@... https://lists.sourceforge.net/lists/listinfo/jikesrvm-researchers |
|
|
Re: [rvm-research] Automating test case generation?
by Filip Pizlo-2
::
Rate this Message:
That's a tough one ... whether or not people have researched this, I
don't know. But often, the really interesting bugs are caused precisely by the complexity of the application. It seems to me that I and the other team members should suck it up and do a stability drive... you've found a lot of bugs, many of which are outside the scope of the benchmarks we usually test. As well, we're not yet 100% stable on Dacapo (or even SPEC, I think), which is at least slightly worrisome. I would be able to make a limited time commitment on this in January ... anyone else have thoughts on being able to do this? My thinking is that RVM should move forward with support for OpenJDK at some point - but *before* we do that I'd like to make sure that the RVM as it is today doesn't routinely crash on stuff we care about, OpenJDK or not. -Filip On Oct 20, 2009, at 1:35 PM, Suriya Subramanian wrote: > Hi all, > > Quite often, I encounter Jikes RVM bugs that show up when running > complex > applications. Most of the time, I have been able to construct small > test > programs that demonstrate buggy behavior. > > Starting with the application jEdit I was able to find this bug > RVM-868 > http://jira.codehaus.org/browse/RVM-868 and a test case > http://jira.codehaus.org/secure/attachment/45362/ > ResolverTestCase.java that > is about 20 lines long. I was lucky today and it took me about 15 > minutes > to get to here. This is an exception. Usually, this process has been > difficult and time-consuming. > > Are there ways to automate this process, i.e. removing parts of an > application that do not contribute to some behavior we are looking > at? I am > sure there should be a lot of research done in this area. Can > someone give > me pointers. Are there tools that are readily usable? > > Thanks, > Suriya > > ------------------------------------------------------------------------------ > Come build with us! The BlackBerry(R) Developer Conference in SF, CA > is the only developer event you need to attend this year. Jumpstart > your > developing skills, take BlackBerry mobile applications to market and > stay > ahead of the curve. Join us from November 9 - 12, 2009. Register now! > http://p.sf.net/sfu/devconference > _______________________________________________ > Jikesrvm-researchers mailing list > Jikesrvm-researchers@... > https://lists.sourceforge.net/lists/listinfo/jikesrvm-researchers ------------------------------------------------------------------------------ Come build with us! The BlackBerry(R) Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9 - 12, 2009. Register now! http://p.sf.net/sfu/devconference _______________________________________________ Jikesrvm-researchers mailing list Jikesrvm-researchers@... https://lists.sourceforge.net/lists/listinfo/jikesrvm-researchers |
|
|
Re: [rvm-research] Automating test case generation?
by Kenneth Hoste
::
Rate this Message:
On Oct 20, 2009, at 7:35 PM, Suriya Subramanian wrote: > Hi all, > > Quite often, I encounter Jikes RVM bugs that show up when running > complex > applications. Most of the time, I have been able to construct small > test > programs that demonstrate buggy behavior. > > Starting with the application jEdit I was able to find this bug > RVM-868 > http://jira.codehaus.org/browse/RVM-868 and a test case > http://jira.codehaus.org/secure/attachment/45362/ > ResolverTestCase.java that > is about 20 lines long. I was lucky today and it took me about 15 > minutes > to get to here. This is an exception. Usually, this process has been > difficult and time-consuming. > > Are there ways to automate this process, i.e. removing parts of an > application that do not contribute to some behavior we are looking > at? I am > sure there should be a lot of research done in this area. Can > someone give > me pointers. Are there tools that are readily usable? You should look into delta (http://delta.tigris.org/), and especially the more recent multidelta tool. Although it's targeted towards GCC mainly, I think you should be able to get it to work with other stuff too with a fair amount of work. I've used it to reduce testcases for GCC bugs, with spectacular results. greetings, Kenneth ------------------------------------------------------------------------------ Come build with us! The BlackBerry(R) Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9 - 12, 2009. Register now! http://p.sf.net/sfu/devconference _______________________________________________ Jikesrvm-researchers mailing list Jikesrvm-researchers@... https://lists.sourceforge.net/lists/listinfo/jikesrvm-researchers |
|
|
Re: [rvm-research] Automating test case generation?
by Suriya Subramanian
::
Rate this Message:
On further thought, the problem seems to be conceptually simple. Given a
1. Grammar (in this case valid Java programs) 2. function that maps strings to { Buggy, Not-Buggy } (in this case, running the program on Jikes RVM to see if it triggers the bug), 3. string/parse-tree that is valid in that grammar (in this case, a specific Java application) and maps to Buggy find, a sub-string/sub-parse-tree that also maps to Buggy. Our goal is to find the smallest such string. Obviously, we could use various search strategies to find the smallest input string. If we could engineer a general solution, this would be useful in various contexts. "I have a large .java/.doc/.pdf/.png/.html file that crashes JikesRVM/Word/Acrobat/libpng/Firefox. Can I get the smallest sub-file that does the same?" Any comments on how easy/hard/worthy this problem is? Suriya On Tue, Oct 20, 2009 at 01:43:40PM -0400, Filip Pizlo wrote: > From: Filip Pizlo <pizlo@...> > To: Suriya Subramanian <suriya@...>, > "General discussion of Jikes RVM design, implementation, issues, and plans" <jikesrvm-researchers@...> > Subject: Re: [rvm-research] Automating test case generation? > Date: Tue, 20 Oct 2009 13:43:40 -0400 > > That's a tough one ... whether or not people have researched this, I > don't know. But often, the really interesting bugs are caused > precisely by the complexity of the application. > > It seems to me that I and the other team members should suck it up and > do a stability drive... you've found a lot of bugs, many of which are > outside the scope of the benchmarks we usually test. As well, we're > not yet 100% stable on Dacapo (or even SPEC, I think), which is at > least slightly worrisome. > > I would be able to make a limited time commitment on this in > January ... anyone else have thoughts on being able to do this? My > thinking is that RVM should move forward with support for OpenJDK at > some point - but *before* we do that I'd like to make sure that the > RVM as it is today doesn't routinely crash on stuff we care about, > OpenJDK or not. > > -Filip > > > > On Oct 20, 2009, at 1:35 PM, Suriya Subramanian wrote: > > >Hi all, > > > >Quite often, I encounter Jikes RVM bugs that show up when running > >complex > >applications. Most of the time, I have been able to construct small > >test > >programs that demonstrate buggy behavior. > > > >Starting with the application jEdit I was able to find this bug > >RVM-868 > >http://jira.codehaus.org/browse/RVM-868 and a test case > >http://jira.codehaus.org/secure/attachment/45362/ > >ResolverTestCase.java that > >is about 20 lines long. I was lucky today and it took me about 15 > >minutes > >to get to here. This is an exception. Usually, this process has been > >difficult and time-consuming. > > > >Are there ways to automate this process, i.e. removing parts of an > >application that do not contribute to some behavior we are looking > >at? I am > >sure there should be a lot of research done in this area. Can > >someone give > >me pointers. Are there tools that are readily usable? > > > >Thanks, > >Suriya > > > >------------------------------------------------------------------------------ > >Come build with us! The BlackBerry(R) Developer Conference in SF, CA > >is the only developer event you need to attend this year. Jumpstart > >your > >developing skills, take BlackBerry mobile applications to market and > >stay > >ahead of the curve. Join us from November 9 - 12, 2009. Register now! > >http://p.sf.net/sfu/devconference > >_______________________________________________ > >Jikesrvm-researchers mailing list > >Jikesrvm-researchers@... > >https://lists.sourceforge.net/lists/listinfo/jikesrvm-researchers ------------------------------------------------------------------------------ Come build with us! The BlackBerry(R) Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9 - 12, 2009. Register now! http://p.sf.net/sfu/devconference _______________________________________________ Jikesrvm-researchers mailing list Jikesrvm-researchers@... https://lists.sourceforge.net/lists/listinfo/jikesrvm-researchers |
|
|
Re: [rvm-research] Automating test case generation?
by Eliot Moss
::
Rate this Message:
I've not read it, but just saw an announcement of this book,
which might be relevant to what you are talking about: Why Programs Fail: A Guide to Systematic Debugging, Second Edition by Andreas Zeller Morgan Kaufmann Publishers © 2009 (425 pages) ISBN: 9780123745156 Covering a wide range of tools and techniques ranging from hands-on observation to fully automated diagnoses, this book discusses bugs in computer programs, how to find them, how to reproduce them, and how to fix them so they do not occur anymore. Best wishes -- Eliot ------------------------------------------------------------------------------ Come build with us! The BlackBerry(R) Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9 - 12, 2009. Register now! http://p.sf.net/sfu/devconference _______________________________________________ Jikesrvm-researchers mailing list Jikesrvm-researchers@... https://lists.sourceforge.net/lists/listinfo/jikesrvm-researchers |
|
|
Re: [rvm-research] Automating test case generation?
by Emery Berger
::
Rate this Message:
I have the book (which is a good read). Zeller invented "delta debugging", an implementation of which was referred to by a previous poster.
-- Emery Berger Associate Professor Dept. of Computer Science University of Massachusetts www.cs.umass.edu/~emery On Wed, Oct 21, 2009 at 10:39 AM, Eliot Moss <moss@...> wrote: I've not read it, but just saw an announcement of this book, ------------------------------------------------------------------------------ Come build with us! The BlackBerry(R) Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9 - 12, 2009. Register now! http://p.sf.net/sfu/devconference _______________________________________________ Jikesrvm-researchers mailing list Jikesrvm-researchers@... https://lists.sourceforge.net/lists/listinfo/jikesrvm-researchers |
| Free embeddable forum powered by Nabble | Forum Help |
