Next-gen modules

[Chris Fields-5]

On Jun 17, 2009, at 12:49 PM, Elia Stupka wrote:

> I would suggest developing the "standard" version first, then moving  
> onto potential optimizations.

Yes, agreed.

> When we went through a similar argument in Ensembl about 8 years ago  
> we ended up dropping Bio::Root completely...

They (strangely enough) still use it in a few modules and require  
bioperl 1.2.3, but (in my experience) the latest bioperl works just  
fine.  I asked about that and never got a response.

> If one is truly after performance for these large next-gen projects,  
> it'd be down to pure piping, shell, and worrying about location and  
> copying of files, sticking to systems-level as much as possible, and  
> quite far from Bioperl altogether, so I think it's a whole different  
> level of optimization issues, probably outside the scope of Bioperl.
>
> Elia

In the end I don't think we can run it using perl alone, no, and I  
believe using BioPerl by itself will not be the optimal solution, but  
it can probably interface with something that is.

chris
_______________________________________________
Bioperl-l mailing list
Bioperl-l@...
http://lists.open-bio.org/mailman/listinfo/bioperl-l

[Elia Stupka]

Excellent, I was thinking of working on Maq and BowTie as priorities.

Elia

On 17 Jun 2009, at 14:28, John Marshall wrote:

> On 17 Jun 2009, at 12:29, Elia Stupka wrote:
>> Similarly, there seems to be little in bioperl-run to support tools  
>> that have been developed in this area, such as Maq, BowTie, TopHat,  
>> etc?
>
> FYI I have a Bio::Tools::Run::Velvet wrapper [1] that I plan to  
> submit in the not too distant future.  (First it needs some "blah  
> blah" replaced with actual documentation and a test suite.)
>
> Cheers,
>
>    John
>
> [1] http://www.ebi.ac.uk/~zerbino/velvet/
>
>
> --
> The Wellcome Trust Sanger Institute is operated by Genome  
> ResearchLimited, a charity registered in England with number 1021457  
> and acompany registered in England with number 2742969, whose  
> registeredoffice is 215 Euston Road, London, NW1  
> 2BE._______________________________________________
> Bioperl-l mailing list
> Bioperl-l@...
> http://lists.open-bio.org/mailman/listinfo/bioperl-l

---
Senior Lecturer, Bioinformatics
UCL Cancer Institute
Paul O' Gorman Building
University College London
Gower Street
WC1E 6BT
London
UK

Office (UCL): [phone]
Office (ICMS): [phone]

Mobile: [phone]
Mobile (Italy): [phone]

_______________________________________________
Bioperl-l mailing list
Bioperl-l@...
http://lists.open-bio.org/mailman/listinfo/bioperl-l

[michael watson (IAH-C)]

In answer to your question, yes!  We have 6 illumina datasets which we have searched against sequence databases using fasta, and I used SearchIO to parse the results.  This is where BioPerl comes into its own - wrapped around fast, optimised solutions written in C or Java.  Sure, I could have written something in sed/awk/pure perl/C etc to parse out the information I needed faster, but the SearchIO solution only took a few minutes to parse a huge fasta results file, and for me (and many others, I suspect) a few minutes is not a problem.

 
________________________________

From: bioperl-l-bounces@... on behalf of Sendu Bala
Sent: Wed 17/06/2009 7:20 PM
To: tristan.lefebure@...
Cc: bioperl-l@...
Subject: Re: [Bioperl-l] Next-gen modules



Tristan Lefebure wrote:
> Hello,
> Regarding next-gen sequences and bioperl, following my
> experience, another issue is bioperl speed. For example, if
> you want to trim bad quality bases at ends of 1E6 Solexa
> reads using Bio::SeqIO::fastq and some methods in
> Bio::Seq::Quality, well, you've got to be patient (but may
> be I missed some shortcuts...).

This is my concern as well. Or, rather, is there actually a significant
set of users out there who are dealing with next-gen sequencing and
would consider using BioPerl for their work?

I'm working with all the 1000-genomes data at the Sanger, and we at
least are probably never going to use BioPerl for the work.


> A pure perl solution will be between 100 to 1000x faster...
> Would it be possible to have an ultra-light quality object
> with few simple methods for next-gen reads?

The fastq parser itself already seems pretty fast. The way to get the
speedup is to not create any Bio::Seq* objects but just return the data
directly. At that point it's not taking much advantage of BioPerl. But
certainly it could be done...
_______________________________________________
Bioperl-l mailing list
Bioperl-l@...
http://lists.open-bio.org/mailman/listinfo/bioperl-l



_______________________________________________
Bioperl-l mailing list
Bioperl-l@...
http://lists.open-bio.org/mailman/listinfo/bioperl-l

[Chris Fields-5]

On Jun 17, 2009, at 1:20 PM, Sendu Bala wrote:

> Tristan Lefebure wrote:
>> Hello,
>> Regarding next-gen sequences and bioperl, following my experience,  
>> another issue is bioperl speed. For example, if you want to trim  
>> bad quality bases at ends of 1E6 Solexa reads using  
>> Bio::SeqIO::fastq and some methods in Bio::Seq::Quality, well,  
>> you've got to be patient (but may be I missed some shortcuts...).
>
> This is my concern as well. Or, rather, is there actually a  
> significant set of users out there who are dealing with next-gen  
> sequencing and would consider using BioPerl for their work?
>
> I'm working with all the 1000-genomes data at the Sanger, and we at  
> least are probably never going to use BioPerl for the work.

Are you using pure perl or (gasp) something else?  ;>

Judging by the feedback there are definitely a set of users who would  
like to integrate nextgen into bioperl somehow, probably to take  
advantage of other aspects of bioperl.

>> A pure perl solution will be between 100 to 1000x faster... Would  
>> it be possible to have an ultra-light quality object with few  
>> simple methods for next-gen reads?
>
> The fastq parser itself already seems pretty fast. The way to get  
> the speedup is to not create any Bio::Seq* objects but just return  
> the data directly. At that point it's not taking much advantage of  
> BioPerl. But certainly it could be done...


I suppose the best way to assess what needs to be done is come up with  
a set of 'use cases' specifying what users want so we can design  
around them, otherwise we're shooting in the dark.

I'm personally wondering if this could be done as a sequence database,  
something similar in theme to Lincoln's SeqFeature::Store, but  
sequence only, and returns quality objects in a similar manner (ala  
Storable)?  Not sure whether that's feasible, but it's appears at  
least scalable.

chris


_______________________________________________
Bioperl-l mailing list
Bioperl-l@...
http://lists.open-bio.org/mailman/listinfo/bioperl-l

[Elia Stupka]

Dear all,

I tried to summarize today's discussion with what seems to be the  
"shaping consensus" on the Wiki page:

http://www.bioperl.org/wiki/Nextgen_in_Bioperl

good night,

Elia


On 17 Jun 2009, at 13:19, Mark A. Jensen wrote:

> [ and here's a new wikipage: http://www.bioperl.org/wiki/Nextgen_in_Bioperl 
>  ]
> ----- Original Message ----- From: "Elia Stupka" <e.stupka@...>
> To: <bioperl-l@...>
> Sent: Wednesday, June 17, 2009 7:29 AM
> Subject: [Bioperl-l] Next-gen modules
>
>
>> Dear all,
>> after several years of absence I am slowly coming back to Bioperl,  
>> and  hope to contribute again to its development.
>> One area that I was thinking of starting from, since we are  
>> actively  involved with it, is to improve BIoperl's support fo next-
>> gen  sequencing data, tools, etc. Since I am sure I have missed out  
>> on a  lot of recent developments, do let me know if/what is useful.
>> One example that comes to mind is that the conversion of various  
>> formats to/from FASTQ does not seem to be supported. Some code can  
>> be  found within Li Heng's script: http://maq.sourceforge.net/ 
>> fq_all2std.pl but it would be good if it could make its way into  
>> SeqIO? And similarly, potentially, for other next-gen sequence  
>> formats?
>> Similarly, there seems to be little in bioperl-run to support  
>> tools  that have been developed in this area, such as Maq, BowTie,  
>> TopHat, etc?
>> Do let me know if there is a past thread on this, or other people  
>> actively developing, etc. so that I can find out what priorities are.
>> thanks and best regards to all (old friends and new),
>> Elia
>> ---
>> Senior Lecturer, Bioinformatics
>> UCL Cancer Institute
>> Paul O' Gorman Building
>> University College London
>> Gower Street
>> WC1E 6BT
>> London
>> UK
>> Office (UCL): [phone]
>> Office (ICMS): [phone]
>> Mobile: [phone]
>> Mobile (Italy): [phone]
>> _______________________________________________
>> Bioperl-l mailing list
>> Bioperl-l@...
>> http://lists.open-bio.org/mailman/listinfo/bioperl-l
>>

---
Senior Lecturer, Bioinformatics
UCL Cancer Institute
Paul O' Gorman Building
University College London
Gower Street
WC1E 6BT
London
UK

Office (UCL): [phone]
Office (ICMS): [phone]

Mobile: [phone]
Mobile (Italy): [phone]

_______________________________________________
Bioperl-l mailing list
Bioperl-l@...
http://lists.open-bio.org/mailman/listinfo/bioperl-l

[Elia Stupka]

Interesting that you mention the database issue. We found that for  
specific memory/CPU intenstive things we also switch to using dbs. For  
example, after many years of loyal use of disconnected_ranges we  
switched to a simple SQL implementation of it, because of the large  
performance gains it would give us.  Similarly in Ensembl as well as  
in the old days of bioperl-db we opted for doing subseq within SQL  
where possible.

Some lean way of SQL'izing specific components could be less  
"disruptive" than avoiding object creation and provide significant  
gains in performance. Could be set as an optional flag, and could use  
temporary ad hoc SQL databases?

Still, priority now is to make SeqIO compliant with all those formats,  
than we can worry about performance :)

Elia

On 17 Jun 2009, at 20:30, Chris Fields wrote:

> On Jun 17, 2009, at 1:20 PM, Sendu Bala wrote:
>
>> Tristan Lefebure wrote:
>>> Hello,
>>> Regarding next-gen sequences and bioperl, following my experience,  
>>> another issue is bioperl speed. For example, if you want to trim  
>>> bad quality bases at ends of 1E6 Solexa reads using  
>>> Bio::SeqIO::fastq and some methods in Bio::Seq::Quality, well,  
>>> you've got to be patient (but may be I missed some shortcuts...).
>>
>> This is my concern as well. Or, rather, is there actually a  
>> significant set of users out there who are dealing with next-gen  
>> sequencing and would consider using BioPerl for their work?
>>
>> I'm working with all the 1000-genomes data at the Sanger, and we at  
>> least are probably never going to use BioPerl for the work.
>
> Are you using pure perl or (gasp) something else?  ;>
>
> Judging by the feedback there are definitely a set of users who  
> would like to integrate nextgen into bioperl somehow, probably to  
> take advantage of other aspects of bioperl.
>
>>> A pure perl solution will be between 100 to 1000x faster... Would  
>>> it be possible to have an ultra-light quality object with few  
>>> simple methods for next-gen reads?
>>
>> The fastq parser itself already seems pretty fast. The way to get  
>> the speedup is to not create any Bio::Seq* objects but just return  
>> the data directly. At that point it's not taking much advantage of  
>> BioPerl. But certainly it could be done...
>
>
> I suppose the best way to assess what needs to be done is come up  
> with a set of 'use cases' specifying what users want so we can  
> design around them, otherwise we're shooting in the dark.
>
> I'm personally wondering if this could be done as a sequence  
> database, something similar in theme to Lincoln's SeqFeature::Store,  
> but sequence only, and returns quality objects in a similar manner  
> (ala Storable)?  Not sure whether that's feasible, but it's appears  
> at least scalable.
>
> chris
>
>
> _______________________________________________
> Bioperl-l mailing list
> Bioperl-l@...
> http://lists.open-bio.org/mailman/listinfo/bioperl-l

---
Senior Lecturer, Bioinformatics
UCL Cancer Institute
Paul O' Gorman Building
University College London
Gower Street
WC1E 6BT
London
UK

Office (UCL): [phone]
Office (ICMS): [phone]

Mobile: [phone]
Mobile (Italy): [phone]

_______________________________________________
Bioperl-l mailing list
Bioperl-l@...
http://lists.open-bio.org/mailman/listinfo/bioperl-l

[Mark A. Jensen]

Thanks Elia for those wiki notes--
[I would say you received an enthusiatic 'welcome back'!]
cheers,
Mark
----- Original Message -----
From: "Elia Stupka" <e.stupka@...>
To: "Mark A. Jensen" <maj@...>
Cc: <bioperl-l@...>
Sent: Wednesday, June 17, 2009 3:37 PM
Subject: Re: [Bioperl-l] Next-gen modules

> Dear all,
>
> I tried to summarize today's discussion with what seems to be the  
> "shaping consensus" on the Wiki page:
>
> http://www.bioperl.org/wiki/Nextgen_in_Bioperl
>
> good night,
>
> Elia
>
>
> On 17 Jun 2009, at 13:19, Mark A. Jensen wrote:
>
>> [ and here's a new wikipage: http://www.bioperl.org/wiki/Nextgen_in_Bioperl 
>>  ]
>> ----- Original Message ----- From: "Elia Stupka" <e.stupka@...>
>> To: <bioperl-l@...>
>> Sent: Wednesday, June 17, 2009 7:29 AM
>> Subject: [Bioperl-l] Next-gen modules
>>
>>
>>> Dear all,
>>> after several years of absence I am slowly coming back to Bioperl,  
>>> and  hope to contribute again to its development.
>>> One area that I was thinking of starting from, since we are  
>>> actively  involved with it, is to improve BIoperl's support fo next-
>>> gen  sequencing data, tools, etc. Since I am sure I have missed out  
>>> on a  lot of recent developments, do let me know if/what is useful.
>>> One example that comes to mind is that the conversion of various  
>>> formats to/from FASTQ does not seem to be supported. Some code can  
>>> be  found within Li Heng's script: http://maq.sourceforge.net/ 
>>> fq_all2std.pl but it would be good if it could make its way into  
>>> SeqIO? And similarly, potentially, for other next-gen sequence  
>>> formats?
>>> Similarly, there seems to be little in bioperl-run to support  
>>> tools  that have been developed in this area, such as Maq, BowTie,  
>>> TopHat, etc?
>>> Do let me know if there is a past thread on this, or other people  
>>> actively developing, etc. so that I can find out what priorities are.
>>> thanks and best regards to all (old friends and new),
>>> Elia
>>> ---
>>> Senior Lecturer, Bioinformatics
>>> UCL Cancer Institute
>>> Paul O' Gorman Building
>>> University College London
>>> Gower Street
>>> WC1E 6BT
>>> London
>>> UK
>>> Office (UCL): [phone]
>>> Office (ICMS): [phone]
>>> Mobile: [phone]
>>> Mobile (Italy): [phone]
>>> _______________________________________________
>>> Bioperl-l mailing list
>>> Bioperl-l@...
>>> http://lists.open-bio.org/mailman/listinfo/bioperl-l
>>>
>
> ---
> Senior Lecturer, Bioinformatics
> UCL Cancer Institute
> Paul O' Gorman Building
> University College London
> Gower Street
> WC1E 6BT
> London
> UK
>
> Office (UCL): [phone]
> Office (ICMS): [phone]
>
> Mobile: [phone]
> Mobile (Italy): [phone]
>
> _______________________________________________
> Bioperl-l mailing list
> Bioperl-l@...
> http://lists.open-bio.org/mailman/listinfo/bioperl-l
>
>

_______________________________________________
Bioperl-l mailing list
Bioperl-l@...
http://lists.open-bio.org/mailman/listinfo/bioperl-l

[Chris Fields-5]

So, #1 priority is to get fastq up-to-speed, then maybe assess other  
options.

Illuminating discussion, thanks Elia!

urgh, excuse unintended bad pun above...

chris

On Jun 17, 2009, at 3:06 PM, Elia Stupka wrote:

> Interesting that you mention the database issue. We found that for  
> specific memory/CPU intenstive things we also switch to using dbs.  
> For example, after many years of loyal use of disconnected_ranges we  
> switched to a simple SQL implementation of it, because of the large  
> performance gains it would give us.  Similarly in Ensembl as well as  
> in the old days of bioperl-db we opted for doing subseq within SQL  
> where possible.
>
> Some lean way of SQL'izing specific components could be less  
> "disruptive" than avoiding object creation and provide significant  
> gains in performance. Could be set as an optional flag, and could  
> use temporary ad hoc SQL databases?
>
> Still, priority now is to make SeqIO compliant with all those  
> formats, than we can worry about performance :)
>
> Elia
>
> On 17 Jun 2009, at 20:30, Chris Fields wrote:
>
>> On Jun 17, 2009, at 1:20 PM, Sendu Bala wrote:
>>
>>> Tristan Lefebure wrote:
>>>> Hello,
>>>> Regarding next-gen sequences and bioperl, following my  
>>>> experience, another issue is bioperl speed. For example, if you  
>>>> want to trim bad quality bases at ends of 1E6 Solexa reads using  
>>>> Bio::SeqIO::fastq and some methods in Bio::Seq::Quality, well,  
>>>> you've got to be patient (but may be I missed some shortcuts...).
>>>
>>> This is my concern as well. Or, rather, is there actually a  
>>> significant set of users out there who are dealing with next-gen  
>>> sequencing and would consider using BioPerl for their work?
>>>
>>> I'm working with all the 1000-genomes data at the Sanger, and we  
>>> at least are probably never going to use BioPerl for the work.
>>
>> Are you using pure perl or (gasp) something else?  ;>
>>
>> Judging by the feedback there are definitely a set of users who  
>> would like to integrate nextgen into bioperl somehow, probably to  
>> take advantage of other aspects of bioperl.
>>
>>>> A pure perl solution will be between 100 to 1000x faster... Would  
>>>> it be possible to have an ultra-light quality object with few  
>>>> simple methods for next-gen reads?
>>>
>>> The fastq parser itself already seems pretty fast. The way to get  
>>> the speedup is to not create any Bio::Seq* objects but just return  
>>> the data directly. At that point it's not taking much advantage of  
>>> BioPerl. But certainly it could be done...
>>
>>
>> I suppose the best way to assess what needs to be done is come up  
>> with a set of 'use cases' specifying what users want so we can  
>> design around them, otherwise we're shooting in the dark.
>>
>> I'm personally wondering if this could be done as a sequence  
>> database, something similar in theme to Lincoln's  
>> SeqFeature::Store, but sequence only, and returns quality objects  
>> in a similar manner (ala Storable)?  Not sure whether that's  
>> feasible, but it's appears at least scalable.
>>
>> chris
>>
>>
>> _______________________________________________
>> Bioperl-l mailing list
>> Bioperl-l@...
>> http://lists.open-bio.org/mailman/listinfo/bioperl-l
>
> ---
> Senior Lecturer, Bioinformatics
> UCL Cancer Institute
> Paul O' Gorman Building
> University College London
> Gower Street
> WC1E 6BT
> London
> UK
>
> Office (UCL): [phone]
> Office (ICMS): [phone]
>
> Mobile: [phone]
> Mobile (Italy): [phone]
>
> _______________________________________________
> Bioperl-l mailing list
> Bioperl-l@...
> http://lists.open-bio.org/mailman/listinfo/bioperl-l

_______________________________________________
Bioperl-l mailing list
Bioperl-l@...
http://lists.open-bio.org/mailman/listinfo/bioperl-l

[Elia Stupka]

Better than colorspaced discussions for sure ;)

Elia

On 17 Jun 2009, at 21:35, Chris Fields wrote:

> So, #1 priority is to get fastq up-to-speed, then maybe assess other  
> options.
>
> Illuminating discussion, thanks Elia!
>
> urgh, excuse unintended bad pun above...
>
> chris
>
> On Jun 17, 2009, at 3:06 PM, Elia Stupka wrote:
>
>> Interesting that you mention the database issue. We found that for  
>> specific memory/CPU intenstive things we also switch to using dbs.  
>> For example, after many years of loyal use of disconnected_ranges  
>> we switched to a simple SQL implementation of it, because of the  
>> large performance gains it would give us.  Similarly in Ensembl as  
>> well as in the old days of bioperl-db we opted for doing subseq  
>> within SQL where possible.
>>
>> Some lean way of SQL'izing specific components could be less  
>> "disruptive" than avoiding object creation and provide significant  
>> gains in performance. Could be set as an optional flag, and could  
>> use temporary ad hoc SQL databases?
>>
>> Still, priority now is to make SeqIO compliant with all those  
>> formats, than we can worry about performance :)
>>
>> Elia
>>
>> On 17 Jun 2009, at 20:30, Chris Fields wrote:
>>
>>> On Jun 17, 2009, at 1:20 PM, Sendu Bala wrote:
>>>
>>>> Tristan Lefebure wrote:
>>>>> Hello,
>>>>> Regarding next-gen sequences and bioperl, following my  
>>>>> experience, another issue is bioperl speed. For example, if you  
>>>>> want to trim bad quality bases at ends of 1E6 Solexa reads using  
>>>>> Bio::SeqIO::fastq and some methods in Bio::Seq::Quality, well,  
>>>>> you've got to be patient (but may be I missed some shortcuts...).
>>>>
>>>> This is my concern as well. Or, rather, is there actually a  
>>>> significant set of users out there who are dealing with next-gen  
>>>> sequencing and would consider using BioPerl for their work?
>>>>
>>>> I'm working with all the 1000-genomes data at the Sanger, and we  
>>>> at least are probably never going to use BioPerl for the work.
>>>
>>> Are you using pure perl or (gasp) something else?  ;>
>>>
>>> Judging by the feedback there are definitely a set of users who  
>>> would like to integrate nextgen into bioperl somehow, probably to  
>>> take advantage of other aspects of bioperl.
>>>
>>>>> A pure perl solution will be between 100 to 1000x faster...  
>>>>> Would it be possible to have an ultra-light quality object with  
>>>>> few simple methods for next-gen reads?
>>>>
>>>> The fastq parser itself already seems pretty fast. The way to get  
>>>> the speedup is to not create any Bio::Seq* objects but just  
>>>> return the data directly. At that point it's not taking much  
>>>> advantage of BioPerl. But certainly it could be done...
>>>
>>>
>>> I suppose the best way to assess what needs to be done is come up  
>>> with a set of 'use cases' specifying what users want so we can  
>>> design around them, otherwise we're shooting in the dark.
>>>
>>> I'm personally wondering if this could be done as a sequence  
>>> database, something similar in theme to Lincoln's  
>>> SeqFeature::Store, but sequence only, and returns quality objects  
>>> in a similar manner (ala Storable)?  Not sure whether that's  
>>> feasible, but it's appears at least scalable.
>>>
>>> chris
>>>
>>>
>>> _______________________________________________
>>> Bioperl-l mailing list
>>> Bioperl-l@...
>>> http://lists.open-bio.org/mailman/listinfo/bioperl-l
>>
>> ---
>> Senior Lecturer, Bioinformatics
>> UCL Cancer Institute
>> Paul O' Gorman Building
>> University College London
>> Gower Street
>> WC1E 6BT
>> London
>> UK
>>
>> Office (UCL): [phone]
>> Office (ICMS): [phone]
>>
>> Mobile: [phone]
>> Mobile (Italy): [phone]
>>
>> _______________________________________________
>> Bioperl-l mailing list
>> Bioperl-l@...
>> http://lists.open-bio.org/mailman/listinfo/bioperl-l
>

---
Senior Lecturer, Bioinformatics
UCL Cancer Institute
Paul O' Gorman Building
University College London
Gower Street
WC1E 6BT
London
UK

Office (UCL): [phone]
Office (ICMS): [phone]

Mobile: [phone]
Mobile (Italy): [phone]

_______________________________________________
Bioperl-l mailing list
Bioperl-l@...
http://lists.open-bio.org/mailman/listinfo/bioperl-l

[George Hartzell-2]

Sendu Bala writes:
 > Tristan Lefebure wrote:
 > > Hello,
 > > Regarding next-gen sequences and bioperl, following my
 > > experience, another issue is bioperl speed. For example, if
 > > you want to trim bad quality bases at ends of 1E6 Solexa
 > > reads using Bio::SeqIO::fastq and some methods in
 > > Bio::Seq::Quality, well, you've got to be patient (but may
 > > be I missed some shortcuts...).
 >
 > This is my concern as well. Or, rather, is there actually a significant
 > set of users out there who are dealing with next-gen sequencing and
 > would consider using BioPerl for their work?
 >
 > I'm working with all the 1000-genomes data at the Sanger, and we at
 > least are probably never going to use BioPerl for the work.
 > [...]

Is it purely a speed issue, or are there other issues (e.g. stability,
correctness, compatibility) that are contributing to your decision?

What *are* you using?

g.

_______________________________________________
Bioperl-l mailing list
Bioperl-l@...
http://lists.open-bio.org/mailman/listinfo/bioperl-l

[Mark A. Jensen]

unintended! Does that mean your delete key's broke...?
----- Original Message -----
From: "Chris Fields" <cjfields@...>
To: "Elia Stupka" <e.stupka@...>
Cc: <bioperl-l@...>; <tristan.lefebure@...>
Sent: Wednesday, June 17, 2009 4:35 PM
Subject: Re: [Bioperl-l] Next-gen modules

> So, #1 priority is to get fastq up-to-speed, then maybe assess other  
> options.
>
> Illuminating discussion, thanks Elia!
>
> urgh, excuse unintended bad pun above...
>
> chris
>
> On Jun 17, 2009, at 3:06 PM, Elia Stupka wrote:
>
>> Interesting that you mention the database issue. We found that for  
>> specific memory/CPU intenstive things we also switch to using dbs.  
>> For example, after many years of loyal use of disconnected_ranges we  
>> switched to a simple SQL implementation of it, because of the large  
>> performance gains it would give us.  Similarly in Ensembl as well as  
>> in the old days of bioperl-db we opted for doing subseq within SQL  
>> where possible.
>>
>> Some lean way of SQL'izing specific components could be less  
>> "disruptive" than avoiding object creation and provide significant  
>> gains in performance. Could be set as an optional flag, and could  
>> use temporary ad hoc SQL databases?
>>
>> Still, priority now is to make SeqIO compliant with all those  
>> formats, than we can worry about performance :)
>>
>> Elia
>>
>> On 17 Jun 2009, at 20:30, Chris Fields wrote:
>>
>>> On Jun 17, 2009, at 1:20 PM, Sendu Bala wrote:
>>>
>>>> Tristan Lefebure wrote:
>>>>> Hello,
>>>>> Regarding next-gen sequences and bioperl, following my  
>>>>> experience, another issue is bioperl speed. For example, if you  
>>>>> want to trim bad quality bases at ends of 1E6 Solexa reads using  
>>>>> Bio::SeqIO::fastq and some methods in Bio::Seq::Quality, well,  
>>>>> you've got to be patient (but may be I missed some shortcuts...).
>>>>
>>>> This is my concern as well. Or, rather, is there actually a  
>>>> significant set of users out there who are dealing with next-gen  
>>>> sequencing and would consider using BioPerl for their work?
>>>>
>>>> I'm working with all the 1000-genomes data at the Sanger, and we  
>>>> at least are probably never going to use BioPerl for the work.
>>>
>>> Are you using pure perl or (gasp) something else?  ;>
>>>
>>> Judging by the feedback there are definitely a set of users who  
>>> would like to integrate nextgen into bioperl somehow, probably to  
>>> take advantage of other aspects of bioperl.
>>>
>>>>> A pure perl solution will be between 100 to 1000x faster... Would  
>>>>> it be possible to have an ultra-light quality object with few  
>>>>> simple methods for next-gen reads?
>>>>
>>>> The fastq parser itself already seems pretty fast. The way to get  
>>>> the speedup is to not create any Bio::Seq* objects but just return  
>>>> the data directly. At that point it's not taking much advantage of  
>>>> BioPerl. But certainly it could be done...
>>>
>>>
>>> I suppose the best way to assess what needs to be done is come up  
>>> with a set of 'use cases' specifying what users want so we can  
>>> design around them, otherwise we're shooting in the dark.
>>>
>>> I'm personally wondering if this could be done as a sequence  
>>> database, something similar in theme to Lincoln's  
>>> SeqFeature::Store, but sequence only, and returns quality objects  
>>> in a similar manner (ala Storable)?  Not sure whether that's  
>>> feasible, but it's appears at least scalable.
>>>
>>> chris
>>>
>>>
>>> _______________________________________________
>>> Bioperl-l mailing list
>>> Bioperl-l@...
>>> http://lists.open-bio.org/mailman/listinfo/bioperl-l
>>
>> ---
>> Senior Lecturer, Bioinformatics
>> UCL Cancer Institute
>> Paul O' Gorman Building
>> University College London
>> Gower Street
>> WC1E 6BT
>> London
>> UK
>>
>> Office (UCL): [phone]
>> Office (ICMS): [phone]
>>
>> Mobile: [phone]
>> Mobile (Italy): [phone]
>>
>> _______________________________________________
>> Bioperl-l mailing list
>> Bioperl-l@...
>> http://lists.open-bio.org/mailman/listinfo/bioperl-l
>
> _______________________________________________
> Bioperl-l mailing list
> Bioperl-l@...
> http://lists.open-bio.org/mailman/listinfo/bioperl-l
>
>

_______________________________________________
Bioperl-l mailing list
Bioperl-l@...
http://lists.open-bio.org/mailman/listinfo/bioperl-l

[Sendu Bala-2]

Chris Fields wrote:

> On Jun 17, 2009, at 1:20 PM, Sendu Bala wrote:
>
>> Tristan Lefebure wrote:
>>> Hello,
>>> Regarding next-gen sequences and bioperl, following my experience,
>>> another issue is bioperl speed. For example, if you want to trim bad
>>> quality bases at ends of 1E6 Solexa reads using Bio::SeqIO::fastq and
>>> some methods in Bio::Seq::Quality, well, you've got to be patient
>>> (but may be I missed some shortcuts...).
>>
>> This is my concern as well. Or, rather, is there actually a
>> significant set of users out there who are dealing with next-gen
>> sequencing and would consider using BioPerl for their work?
>>
>> I'm working with all the 1000-genomes data at the Sanger, and we at
>> least are probably never going to use BioPerl for the work.
>
> Are you using pure perl or (gasp) something else?  ;>

We use some perl stuff, some C stuff. My own stuff is OO perl, but much
lighter weight than BioPerl. Absolute minimal object creation.

>>> A pure perl solution will be between 100 to 1000x faster... Would it
>>> be possible to have an ultra-light quality object with few simple
>>> methods for next-gen reads?
>>
>> The fastq parser itself already seems pretty fast. The way to get the
>> speedup is to not create any Bio::Seq* objects but just return the
>> data directly. At that point it's not taking much advantage of
>> BioPerl. But certainly it could be done...
>
> I suppose the best way to assess what needs to be done is come up with a
> set of 'use cases' specifying what users want so we can design around
> them, otherwise we're shooting in the dark.

Indeed. Though at least I think we can all agree it would be nice to
have the functionality there even if it's slow. There will always be at
least some use-cases where the run speed doesn't matter.


> I'm personally wondering if this could be done as a sequence database,
> something similar in theme to Lincoln's SeqFeature::Store, but sequence
> only, and returns quality objects in a similar manner (ala Storable)?  
> Not sure whether that's feasible, but it's appears at least scalable.

I think not. Well, at least SeqFeature::Store doesn't scale. Try storing
millions of features in a database and watch it crawl to complete
unusability. I can't imagine a db scaling to holding hundreds of TB of
data either. I'm also not sure what the benefit is. There are already
high-speed ways of indexing your fastq or bam files.
_______________________________________________
Bioperl-l mailing list
Bioperl-l@...
http://lists.open-bio.org/mailman/listinfo/bioperl-l

[Sendu Bala-2]

George Hartzell wrote:

> Sendu Bala writes:
>  > Tristan Lefebure wrote:
>  > > Hello,
>  > > Regarding next-gen sequences and bioperl, following my
>  > > experience, another issue is bioperl speed. For example, if
>  > > you want to trim bad quality bases at ends of 1E6 Solexa
>  > > reads using Bio::SeqIO::fastq and some methods in
>  > > Bio::Seq::Quality, well, you've got to be patient (but may
>  > > be I missed some shortcuts...).
>  >
>  > This is my concern as well. Or, rather, is there actually a significant
>  > set of users out there who are dealing with next-gen sequencing and
>  > would consider using BioPerl for their work?
>  >
>  > I'm working with all the 1000-genomes data at the Sanger, and we at
>  > least are probably never going to use BioPerl for the work.
>  > [...]
>
> Is it purely a speed issue, or are there other issues (e.g. stability,
> correctness, compatibility) that are contributing to your decision?

Too heavy-weight, too slow, too memory intensive, missing too much
functionality in any case. If I have to write new parsers and wrappers,
I may as well make them fast (which means they don't "fit" into BioPerl).


> What *are* you using?

There are already great tools written in C that do all the heavy lifting
and the rest is done in perl written for speed and low memory.
_______________________________________________
Bioperl-l mailing list
Bioperl-l@...
http://lists.open-bio.org/mailman/listinfo/bioperl-l

[Chris Fields-5]

On Jun 17, 2009, at 5:10 PM, Sendu Bala wrote:

> Chris Fields wrote:
>> On Jun 17, 2009, at 1:20 PM, Sendu Bala wrote:
>>> Tristan Lefebure wrote:
>>>> Hello,
>>>> Regarding next-gen sequences and bioperl, following my  
>>>> experience, another issue is bioperl speed. For example, if you  
>>>> want to trim bad quality bases at ends of 1E6 Solexa reads using  
>>>> Bio::SeqIO::fastq and some methods in Bio::Seq::Quality, well,  
>>>> you've got to be patient (but may be I missed some shortcuts...).
>>>
>>> This is my concern as well. Or, rather, is there actually a  
>>> significant set of users out there who are dealing with next-gen  
>>> sequencing and would consider using BioPerl for their work?
>>>
>>> I'm working with all the 1000-genomes data at the Sanger, and we  
>>> at least are probably never going to use BioPerl for the work.
>> Are you using pure perl or (gasp) something else?  ;>
>
> We use some perl stuff, some C stuff. My own stuff is OO perl, but  
> much lighter weight than BioPerl. Absolute minimal object creation.

Makes sense.

>>>> A pure perl solution will be between 100 to 1000x faster... Would  
>>>> it be possible to have an ultra-light quality object with few  
>>>> simple methods for next-gen reads?
>>>
>>> The fastq parser itself already seems pretty fast. The way to get  
>>> the speedup is to not create any Bio::Seq* objects but just return  
>>> the data directly. At that point it's not taking much advantage of  
>>> BioPerl. But certainly it could be done...
>> I suppose the best way to assess what needs to be done is come up  
>> with a set of 'use cases' specifying what users want so we can  
>> design around them, otherwise we're shooting in the dark.
>
> Indeed. Though at least I think we can all agree it would be nice to  
> have the functionality there even if it's slow. There will always be  
> at least some use-cases where the run speed doesn't matter.

Agreed.

>> I'm personally wondering if this could be done as a sequence  
>> database, something similar in theme to Lincoln's  
>> SeqFeature::Store, but sequence only, and returns quality objects  
>> in a similar manner (ala Storable)?  Not sure whether that's  
>> feasible, but it's appears at least scalable.
>
> I think not. Well, at least SeqFeature::Store doesn't scale. Try  
> storing millions of features in a database and watch it crawl to  
> complete unusability. I can't imagine a db scaling to holding  
> hundreds of TB of data either. I'm also not sure what the benefit  
> is. There are already high-speed ways of indexing your fastq or bam  
> files.

Interesting that you ran into issues with SF::Store; wonder if object  
storage is the limiting factor there, or if it is something else.  
Anyone else having this issue?

chris



_______________________________________________
Bioperl-l mailing list
Bioperl-l@...
http://lists.open-bio.org/mailman/listinfo/bioperl-l

[Chris Fields-5]

On Jun 17, 2009, at 5:24 PM, Sendu Bala wrote:

> George Hartzell wrote:
>> Sendu Bala writes:
>> > Tristan Lefebure wrote:
>> > > Hello,
>> > > Regarding next-gen sequences and bioperl, following my  > >  
>> experience, another issue is bioperl speed. For example, if  > >  
>> you want to trim bad quality bases at ends of 1E6 Solexa  > > reads  
>> using Bio::SeqIO::fastq and some methods in  > > Bio::Seq::Quality,  
>> well, you've got to be patient (but may  > > be I missed some  
>> shortcuts...).
>> >  > This is my concern as well. Or, rather, is there actually a  
>> significant  > set of users out there who are dealing with next-gen  
>> sequencing and  > would consider using BioPerl for their work?
>> >  > I'm working with all the 1000-genomes data at the Sanger, and  
>> we at  > least are probably never going to use BioPerl for the work.
>> > [...]
>> Is it purely a speed issue, or are there other issues (e.g.  
>> stability,
>> correctness, compatibility) that are contributing to your decision?
>
> Too heavy-weight, too slow, too memory intensive, missing too much  
> functionality in any case. If I have to write new parsers and  
> wrappers, I may as well make them fast (which means they don't "fit"  
> into BioPerl).

That's (unfortunately) true.  It may be easy to whip up something that  
works, but it probably won't be fast.

>> What *are* you using?
>
> There are already great tools written in C that do all the heavy  
> lifting and the rest is done in perl written for speed and low memory.

Like this one?

http://www.sanger.ac.uk/Users/lh3/parsefastq.shtml

I suppose if one were inclined, this could be wrapped with SWIG in  
BioLib, but would it be worth it (maybe beyond grabbing the file  
indices)?

chris
_______________________________________________
Bioperl-l mailing list
Bioperl-l@...
http://lists.open-bio.org/mailman/listinfo/bioperl-l

[Hilmar Lapp]

On Jun 17, 2009, at 3:06 PM, Elia Stupka wrote:

> Similarly in Ensembl as well as in the old days of bioperl-db we  
> opted for doing subseq within SQL where possible.


BTW Bioperl-db still lazy-loads sequences, and does subseq in SQL,  
unless you manipulate the sequence, or make it a non-persistent object.

        -hilmar
--
===========================================================
: Hilmar Lapp  -:-  Durham, NC  -:-  hlapp at gmx dot net :
===========================================================



_______________________________________________
Bioperl-l mailing list
Bioperl-l@...
http://lists.open-bio.org/mailman/listinfo/bioperl-l

[Sendu Bala-2]

Chris Fields wrote:
> On Jun 17, 2009, at 5:10 PM, Sendu Bala wrote:
 >

>>> I'm personally wondering if this could be done as a sequence
>>> database, something similar in theme to Lincoln's SeqFeature::Store,
>>> but sequence only, and returns quality objects in a similar manner
>>> (ala Storable)?  Not sure whether that's feasible, but it's appears
>>> at least scalable.
>>
>> I think not. Well, at least SeqFeature::Store doesn't scale. Try
>> storing millions of features in a database and watch it crawl to
>> complete unusability. I can't imagine a db scaling to holding hundreds
>> of TB of data either. I'm also not sure what the benefit is. There are
>> already high-speed ways of indexing your fastq or bam files.
>
> Interesting that you ran into issues with SF::Store; wonder if object
> storage is the limiting factor there, or if it is something else.

Object storage certainly was an issue, which is why I patched it to
(optionally) not store objects. That helped a great deal, but ultimately
only increased the number of features you could store before it slowed
down; it didn't solve the problem completely.
_______________________________________________
Bioperl-l mailing list
Bioperl-l@...
http://lists.open-bio.org/mailman/listinfo/bioperl-l

[Chris Fields-5]

So, to follow up (and make sure we don't have any overlapping tuits)  
we should probably determine who wants to work on what (i.e. fastq  
updating, etc). I think it's possible to quickly add in Solexa/
Illumina/Sanger fastq similar to BioPython, just don't want to step on  
anyone's toes if they are halfway through doing this.

chris

On Jun 17, 2009, at 3:36 PM, Elia Stupka wrote:

> Better than colorspaced discussions for sure ;)
>
> Elia
>
> On 17 Jun 2009, at 21:35, Chris Fields wrote:
>
>> So, #1 priority is to get fastq up-to-speed, then maybe assess  
>> other options.
>>
>> Illuminating discussion, thanks Elia!
>>
>> urgh, excuse unintended bad pun above...
>>
>> chris
>>
>> On Jun 17, 2009, at 3:06 PM, Elia Stupka wrote:
>>
>>> Interesting that you mention the database issue. We found that for  
>>> specific memory/CPU intenstive things we also switch to using dbs.  
>>> For example, after many years of loyal use of disconnected_ranges  
>>> we switched to a simple SQL implementation of it, because of the  
>>> large performance gains it would give us.  Similarly in Ensembl as  
>>> well as in the old days of bioperl-db we opted for doing subseq  
>>> within SQL where possible.
>>>
>>> Some lean way of SQL'izing specific components could be less  
>>> "disruptive" than avoiding object creation and provide significant  
>>> gains in performance. Could be set as an optional flag, and could  
>>> use temporary ad hoc SQL databases?
>>>
>>> Still, priority now is to make SeqIO compliant with all those  
>>> formats, than we can worry about performance :)
>>>
>>> Elia
>>>
>>> On 17 Jun 2009, at 20:30, Chris Fields wrote:
>>>
>>>> On Jun 17, 2009, at 1:20 PM, Sendu Bala wrote:
>>>>
>>>>> Tristan Lefebure wrote:
>>>>>> Hello,
>>>>>> Regarding next-gen sequences and bioperl, following my  
>>>>>> experience, another issue is bioperl speed. For example, if you  
>>>>>> want to trim bad quality bases at ends of 1E6 Solexa reads  
>>>>>> using Bio::SeqIO::fastq and some methods in Bio::Seq::Quality,  
>>>>>> well, you've got to be patient (but may be I missed some  
>>>>>> shortcuts...).
>>>>>
>>>>> This is my concern as well. Or, rather, is there actually a  
>>>>> significant set of users out there who are dealing with next-gen  
>>>>> sequencing and would consider using BioPerl for their work?
>>>>>
>>>>> I'm working with all the 1000-genomes data at the Sanger, and we  
>>>>> at least are probably never going to use BioPerl for the work.
>>>>
>>>> Are you using pure perl or (gasp) something else?  ;>
>>>>
>>>> Judging by the feedback there are definitely a set of users who  
>>>> would like to integrate nextgen into bioperl somehow, probably to  
>>>> take advantage of other aspects of bioperl.
>>>>
>>>>>> A pure perl solution will be between 100 to 1000x faster...  
>>>>>> Would it be possible to have an ultra-light quality object with  
>>>>>> few simple methods for next-gen reads?
>>>>>
>>>>> The fastq parser itself already seems pretty fast. The way to  
>>>>> get the speedup is to not create any Bio::Seq* objects but just  
>>>>> return the data directly. At that point it's not taking much  
>>>>> advantage of BioPerl. But certainly it could be done...
>>>>
>>>>
>>>> I suppose the best way to assess what needs to be done is come up  
>>>> with a set of 'use cases' specifying what users want so we can  
>>>> design around them, otherwise we're shooting in the dark.
>>>>
>>>> I'm personally wondering if this could be done as a sequence  
>>>> database, something similar in theme to Lincoln's  
>>>> SeqFeature::Store, but sequence only, and returns quality objects  
>>>> in a similar manner (ala Storable)?  Not sure whether that's  
>>>> feasible, but it's appears at least scalable.
>>>>
>>>> chris
>>>>
>>>>
>>>> _______________________________________________
>>>> Bioperl-l mailing list
>>>> Bioperl-l@...
>>>> http://lists.open-bio.org/mailman/listinfo/bioperl-l
>>>
>>> ---
>>> Senior Lecturer, Bioinformatics
>>> UCL Cancer Institute
>>> Paul O' Gorman Building
>>> University College London
>>> Gower Street
>>> WC1E 6BT
>>> London
>>> UK
>>>
>>> Office (UCL): [phone]
>>> Office (ICMS): [phone]
>>>
>>> Mobile: [phone]
>>> Mobile (Italy): [phone]
>>>
>>> _______________________________________________
>>> Bioperl-l mailing list
>>> Bioperl-l@...
>>> http://lists.open-bio.org/mailman/listinfo/bioperl-l
>>
>
> ---
> Senior Lecturer, Bioinformatics
> UCL Cancer Institute
> Paul O' Gorman Building
> University College London
> Gower Street
> WC1E 6BT
> London
> UK
>
> Office (UCL): [phone]
> Office (ICMS): [phone]
>
> Mobile: [phone]
> Mobile (Italy): [phone]
>
> _______________________________________________
> Bioperl-l mailing list
> Bioperl-l@...
> http://lists.open-bio.org/mailman/listinfo/bioperl-l

_______________________________________________
Bioperl-l mailing list
Bioperl-l@...
http://lists.open-bio.org/mailman/listinfo/bioperl-l

[Peter-329]

On Wed, Jun 17, 2009 at 6:06 PM, Chris Fields<cjfields@...> wrote:

>
>
> On Jun 17, 2009, at 8:25 AM, Peter wrote:
>
>>> Peter's suggestions also are reasonable, though does biopython have a
>>> separate module for each of these variations?  Our version (I believe)
>>> mainly varied the conversion within Bio::SeqIO::fastq itself based on the
>>> fastq variant passed in as a separate named argument.
>>
>> Biopython's SeqIO gives the three FASTQ variants their own unique
>> names. This format name is a required argument for parsing/writing
>> (we don't try and guess the file format from the data contents).
>> Internally we have three separate FASTQ parsers/writers although
>> they do share code.
>
> We could easily do the same if others agree.  Actually, if we specified that
> shorthand for a variant on a format would be designated as -format =>
> 'format-variant', I think we could easily hack SeqIO to deal with that by
> splitting on '-' and passing everything to the constructor as (-format =>
> 'format', -variant => 'variant').  Very little repeated code in this case,
> just an additional named parameter indicating the format variant (and the
> SeqIO class can do the type checking on that within the constructor).

Yes, when I started using names like "fastq-solexa" I did have in mind
"main-variant" naming convention, and potentially Biopython may one
day actually use this structure when allocating a Bio.SeqIO job to the
appropriate parser or writer.

For now, the Biopython list of formats is fairly short (and there are
relatively few of these sub-formats) so to keep things simple we just
have a flat mapping from the format name (e.g. "fasta", "fastq",
"fastq-solexa") to the parser/write code.

Peter

_______________________________________________
Bioperl-l mailing list
Bioperl-l@...
http://lists.open-bio.org/mailman/listinfo/bioperl-l

[Elia Stupka]

Hi Chris,

I agree. I have not written a single line of code so far, while Heikki  
has some (but has been silent for a while) and you have perhaps some  
code ready to roll. I am happy to help where needed, just let me know  
what you'd like me to focus on. If you want to go ahead and implement  
the fastq staff discussed I can focus on bioperl-run.

cheers

Elia



On 19 Jun 2009, at 21:57, Chris Fields wrote:

> So, to follow up (and make sure we don't have any overlapping tuits)  
> we should probably determine who wants to work on what (i.e. fastq  
> updating, etc). I think it's possible to quickly add in Solexa/
> Illumina/Sanger fastq similar to BioPython, just don't want to step  
> on anyone's toes if they are halfway through doing this.
>
> chris
>
> On Jun 17, 2009, at 3:36 PM, Elia Stupka wrote:
>
>> Better than colorspaced discussions for sure ;)
>>
>> Elia
>>
>> On 17 Jun 2009, at 21:35, Chris Fields wrote:
>>
>>> So, #1 priority is to get fastq up-to-speed, then maybe assess  
>>> other options.
>>>
>>> Illuminating discussion, thanks Elia!
>>>
>>> urgh, excuse unintended bad pun above...
>>>
>>> chris
>>>
>>> On Jun 17, 2009, at 3:06 PM, Elia Stupka wrote:
>>>
>>>> Interesting that you mention the database issue. We found that  
>>>> for specific memory/CPU intenstive things we also switch to using  
>>>> dbs. For example, after many years of loyal use of  
>>>> disconnected_ranges we switched to a simple SQL implementation of  
>>>> it, because of the large performance gains it would give us.  
>>>> Similarly in Ensembl as well as in the old days of bioperl-db we  
>>>> opted for doing subseq within SQL where possible.
>>>>
>>>> Some lean way of SQL'izing specific components could be less  
>>>> "disruptive" than avoiding object creation and provide  
>>>> significant gains in performance. Could be set as an optional  
>>>> flag, and could use temporary ad hoc SQL databases?
>>>>
>>>> Still, priority now is to make SeqIO compliant with all those  
>>>> formats, than we can worry about performance :)
>>>>
>>>> Elia
>>>>
>>>> On 17 Jun 2009, at 20:30, Chris Fields wrote:
>>>>
>>>>> On Jun 17, 2009, at 1:20 PM, Sendu Bala wrote:
>>>>>
>>>>>> Tristan Lefebure wrote:
>>>>>>> Hello,
>>>>>>> Regarding next-gen sequences and bioperl, following my  
>>>>>>> experience, another issue is bioperl speed. For example, if  
>>>>>>> you want to trim bad quality bases at ends of 1E6 Solexa reads  
>>>>>>> using Bio::SeqIO::fastq and some methods in Bio::Seq::Quality,  
>>>>>>> well, you've got to be patient (but may be I missed some  
>>>>>>> shortcuts...).
>>>>>>
>>>>>> This is my concern as well. Or, rather, is there actually a  
>>>>>> significant set of users out there who are dealing with next-
>>>>>> gen sequencing and would consider using BioPerl for their work?
>>>>>>
>>>>>> I'm working with all the 1000-genomes data at the Sanger, and  
>>>>>> we at least are probably never going to use BioPerl for the work.
>>>>>
>>>>> Are you using pure perl or (gasp) something else?  ;>
>>>>>
>>>>> Judging by the feedback there are definitely a set of users who  
>>>>> would like to integrate nextgen into bioperl somehow, probably  
>>>>> to take advantage of other aspects of bioperl.
>>>>>
>>>>>>> A pure perl solution will be between 100 to 1000x faster...  
>>>>>>> Would it be possible to have an ultra-light quality object  
>>>>>>> with few simple methods for next-gen reads?
>>>>>>
>>>>>> The fastq parser itself already seems pretty fast. The way to  
>>>>>> get the speedup is to not create any Bio::Seq* objects but just  
>>>>>> return the data directly. At that point it's not taking much  
>>>>>> advantage of BioPerl. But certainly it could be done...
>>>>>
>>>>>
>>>>> I suppose the best way to assess what needs to be done is come  
>>>>> up with a set of 'use cases' specifying what users want so we  
>>>>> can design around them, otherwise we're shooting in the dark.
>>>>>
>>>>> I'm personally wondering if this could be done as a sequence  
>>>>> database, something similar in theme to Lincoln's  
>>>>> SeqFeature::Store, but sequence only, and returns quality  
>>>>> objects in a similar manner (ala Storable)?  Not sure whether  
>>>>> that's feasible, but it's appears at least scalable.
>>>>>
>>>>> chris
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Bioperl-l mailing list
>>>>> Bioperl-l@...
>>>>> http://lists.open-bio.org/mailman/listinfo/bioperl-l
>>>>
>>>> ---
>>>> Senior Lecturer, Bioinformatics
>>>> UCL Cancer Institute
>>>> Paul O' Gorman Building
>>>> University College London
>>>> Gower Street
>>>> WC1E 6BT
>>>> London
>>>> UK
>>>>
>>>> Office (UCL): [phone]
>>>> Office (ICMS): [phone]
>>>>
>>>> Mobile: [phone]
>>>> Mobile (Italy): [phone]
>>>>
>>>> _______________________________________________
>>>> Bioperl-l mailing list
>>>> Bioperl-l@...
>>>> http://lists.open-bio.org/mailman/listinfo/bioperl-l
>>>
>>
>> ---
>> Senior Lecturer, Bioinformatics
>> UCL Cancer Institute
>> Paul O' Gorman Building
>> University College London
>> Gower Street
>> WC1E 6BT
>> London
>> UK
>>
>> Office (UCL): [phone]
>> Office (ICMS): [phone]
>>
>> Mobile: [phone]
>> Mobile (Italy): [phone]
>>
>> _______________________________________________
>> Bioperl-l mailing list
>> Bioperl-l@...
>> http://lists.open-bio.org/mailman/listinfo/bioperl-l
>

---
Senior Lecturer, Bioinformatics
UCL Cancer Institute
Paul O' Gorman Building
University College London
Gower Street
WC1E 6BT
London
UK

Office (UCL): [phone]
Office (ICMS): [phone]

Mobile: [phone]
Mobile (Italy): [phone]

_______________________________________________
Bioperl-l mailing list
Bioperl-l@...
http://lists.open-bio.org/mailman/listinfo/bioperl-l