VLDB 2008 Experiments & Analyses Topic

[Volker Markl]

Dear Database Community Member,

I am sure many you have followed the discussions about the repeatability
of results for SIGMOD 2008 and the Panel on performance evaluation at VLDB 2007.

VLDB 2008 will have a special topic on "Experiments and Analysis", which
solicits papers containing solid experimenal or analytical comparisons of
previously published work. The special topic will have one session at the
VLDB conference; the accepted papers will be published in the VLDB proceedings.

Our motivation for the experimental and analyses special topic
stems from the need for consolidation in our field. We have a multitude
of spatial indexing methods, clustering algorithms, etc. without any
clarity about which one to use in which situation.

Consolidation can help to guide practitioners through the multitude of
methods; therefore consolidation papers have research value in themselves,
even without proposing new algorithms.

In addition, the special topic is an alternative approach to deal with the
repeatability requirement pioneered in SIGMOD 2008. It will provide a world-class
venue for independent validation of prior art. The program committee therefore is
interested in papers that provide insight by means of experimental or analytical evaluation
and comparison of existing methods, for example by consolidating a large set of methods
and giving guidance about what method to use in a particular setting, or by identifying
important incorrect results from papers published at major conferences.

Overall, careful, systematic experimental or theoretical analysis of
algorithms, e.g., following Jim Gray's book on benchmarking, will help to organize
our field better, and identify relevant work. This is particularly true if an analysis
and comparison is conducted by an independent third party, and not by any inventor of
a particularalgorithm.

This special topic is work in progress. We want to raise awareness for the
need of consolidation, by giving consolidation papers a venue at a leading
conference. In 2008, the papers will be presented in one special session.
We may experiment with other ways of presenting the results in future conferences,
e.g., a poster sessions or short presentations, depending on how VLDB 2008 goes.
We may also consider making algorithm impementations used for E&A papers available
publicly and thereby creating a library of algorithms.

I see high-quality submissions as a challenge, as the acceptance and institutionalization
of this effort for future VLDB conferences will greatly depend on a good start with strong
first papers following systematic analsyis and providing relevant insights.

I would see the special topic as an interesting venue for publishing the results of a group
project of 1st/2nd year PhD students together with their advisor, who could ensure the
systematic approach.

Please submit your world-class consolidation papers to this VLDB 2008 topic in order to help
starting a culture change in our community that accepts consolidation papers and  papers that
provide insight by validating past results.

Thanks,

Volker Markl
VLDB 2008 Experiments and Analyses PC Chair

>
>                                       Call for Papers
>
>                           Experiments and Analyses Papers
>
>                              A VLDB 2008 Special Topic
>
>  Database management has been an active area of research for
> several  decades. This special topic aims to meet needs for
> consolidation of a  maturing research area by providing a
> prestigious forum for in-depth  analytical or empirical
> studies and comparisons of existing  techniques.
>
>  The expected contribution of an Experiments and Analyses
> (E&A) paper  is new, independent, comprehensive and
> reproducible evaluations and  comparisons of existing data
> management techniques. Thus, the intended  contribution of an
> E&A paper is not a new algorithm or technique but  rather
> further insight into the state-of-the-art by means of
> careful,  systematic, and scientific evaluation. Comparisons
> of algorithmic  techniques must either use best-effort
> re-implementations based on the  original papers, or use
> existing implementations from the original  authors, if
> publicly available.
>
>  The program committee will create a shortlist of E&A
> submissions that  will be accepted tentatively for
> publication. If controversial, these papers may then enter into a
> rebuttal phase with the authors and inventors or acknowledged
> experts of the techniques being evaluated, moderated by one of the
> reviewers. The program committee may require the authors to
> include feedback from the rebuttal phase as a condition of
> final  acceptance.
>
>  Submissions should be 3-12 pages long; authors are
> encouraged to keep  their submissions short and concise.
> Formatting requirements are the  same as for other research papers.
>
>  This special topic is a trial for VLDB 2008 and may
> significantly  change for future VLDB conferences.

> -- Experiments and Analyses Program Committee
> Chair: Volker Markl, IBM Research Almaden, USA
> - Gustavo Alonso, ETH Zurich
> - Christoph Freytag, Humboldt Universitaet Berlin
> - Nick Koudas, University of Toronto
> - Paul Larson, Microsoft Research
> - Hamid Pirahesh, IBM Almaden Research Center
> - Gerhard Weikum, MPI Saarbruecken
> (further PC members to be announced on the website).
_______________________________________________
Please do not post msgs that are not relevant to the database community at large.  Go to www.cs.wisc.edu/dbworld for guidelines and posting forms.
To unsubscribe, go to https://lists.cs.wisc.edu/mailman/listinfo/dbworld