Nabble - Bio.net - Genbankb

GenBank Release 174.0 Now Available

2009-10-19T13:38:18Z

Greetings GenBank Users,

GenBank Release 174.0 is now available via FTP from the National
Center for Biotechnology Information (NCBI):

Ftp Site Directory Contents
---------------- --------- ---------------------------------------
ftp.ncbi.nih.gov genbank GenBank Release 174.0 flatfiles
ncbi-asn1 ASN.1 data used to create Release 174.0

Close-of-data for GenBank 174.0 occurred on 10/16/2009. Uncompressed,
the Release 174.0 flatfiles require roughly 416 GB (sequence files only)
or 445 GB (including the 'short directory', 'index' and the *.txt
files). The ASN.1 data require approximately 376 GB.

Recent statistics for non-WGS, non-CON sequences:

Release Date Base Pairs Entries

173 Aug 2009 106533156756 108431692
174 Oct 2009 108560236506 110946879

Recent statistics for WGS sequences:

Release Date Base Pairs Entries

173 Aug 2009 148165117763 48443067
174 Oct 2009 149348923035 48119301

During the 56 days between the close dates for GenBank Releases 173.0
and 174.0, the non-WGS/non-CON portion of GenBank grew by 2,027,079,750
basepairs and by 2,515,187 sequence records. During that same period,
467,858 records were updated. An average of about 53,269 non-WGS/non-CON
records were added and/or updated per day.

Between releases 173.0 and 174.0, the WGS component of GenBank grew by
1,183,805,272 basepairs, while the number of sequence records decreased
by 323,766 (due to the re-assembly of some WGS projects into fewer, but
larger, records).

For additional release information, see the README files in either of
the directories mentioned above, and the release notes (gbrel.txt) in
the genbank directory. Sections 1.3 and 1.4 of the release notes
(Changes in Release 174.0 and Upcoming Changes) have been appended
below for your convenience.

** Important Notes **

* GenBank 'index' files are now provided without any EST content, and
without most GSS content. See Section 1.3.4 of the release notes for
further details.

NCBI is considering ceasing support for the index files, so we
encourage affected users to review that section and provide feedback.

Release 174.0 data, and subsequent updates, are available now via
NCBI's Entrez and Blast services.

As a general guideline, we suggest first transferring the GenBank
release notes (gbrel.txt) whenever a release is being obtained. Check
to make sure that the date and release number in the header of the
release notes are current (eg: August 15 2009, 174.0). If they are
not, interrupt the remaining transfers and then request assistance from
the NCBI Service Desk.

A comprehensive check of the headers of all release files after your
transfers are complete is also suggested. Here's how one might go about
this on a unix platform, using csh/tcsh :

set files = `ls gb*.*`
foreach i ($files)
head -10 $i | grep Release
end

Or, if the files are compressed, perhaps:

gzcat $i | head -10 | grep Release

If you encounter problems while ftp'ing or uncompressing Release
174.0, please send email outlining your difficulties to:

info@...

Mark Cavanaugh, Michael Kimelman, Ilya Dondoshansky, Sergey Zhdanov
GenBank
NCBI/NLM/NIH/HHS

1.3 Important Changes in Release 174.0

1.3.1 Organizational changes

The total number of sequence data files increased by 31 with this
release:

- the CON division is now composed of 132 files (+1)
- the ENV division is now composed of 22 files (+5)
- the EST division is now composed of 892 files (+6)
- the GSS division is now composed of 350 files (+10)
- the INV division is now composed of 19 files (+1)
- the PAT division is now composed of 85 files (+6)
- the VRL division is now composed of 13 files (+1)
- the VRT division is now composed of 20 files (+1)

The total number of 'index' files increased by 3 with this release:

- the AUT (author) index is now composed of 69 files (+3)

1.3.2 New class of /exception value

As of this October 2009 release, a new class of /exception is
available
for use on coding region features:

/exception="annotated by transcript or proteomic data"

This exception can be used if: a) the protein sequence (presented via
a
coding region's /translation qualifier) differs from the conceptual
translation; b) the quality of the DNA sequencing is high; and c) there
is evidence at the transcript or proteome level that the presented
protein
*is* actually expressed by the organism.

An inference qualifier of type "similar to" should be used in
conjunction
with this new type of exception, to indicate the supporting
EST/cDNA/protein
sequence.

An update to the definition of the /exception qualifier which
incorporates
the new class will be provided via the GenBank newsgroup within a few
weeks.

1.3.3 /haplogroup qualifier introduced

A haplotype is a combination of alleles at multiple loci that are
transmitted together on the same chromosome. A haplogroup is a group of
similar haplotypes that share a common ancestor with a single nucleotide
polymorphism mutation. The majority of submitters of complete human
mitochondrial genomes provide information about their haplogroup rather
than
their haplotype. Stable mtDNA polymorphic variants clustered together in
specific combination form a haplogroup.

To accommodate this need, a new /haplogroup qualifier has been
introduced
as of this October 2009 GenBank Release.

A formal definition of /haplogroup will be provided via the GenBank
newsgroup within a few weeks.

1.3.4 Changes in the content of index files

As described in the GB 153 release notes, the 'index' files which
accompany
GenBank releases (see Section 3.3) are considered to be a legacy data
product by
NCBI, generated mostly for historical reasons. FTP statistics of January
2005
seem to support this: the index files were transferred only half as
frequently as
the files of sequence records. The inherent inefficiencies of the index
file
format also lead us to suspect that they have little serious use by the
user
community, particularly for EST and GSS records.

The software that generated the index file products received little
attention over the years, and finally reached its limitations in
February 2006 (Release 152.0). The required multi-server queries which
obtained and sorted many millions of rows of terms from several
different
databases simply outgrew the capacity of the hardware used for GenBank
Release generation.

Our short-term solution is to cease generating some index-file content
for all EST sequence records, and for GSS sequence records that
originate
via direct submission to NCBI.

The three gbacc*.idx index files continue to reflect the entirety of
the
release, including all EST and GSS records, however the file contents
are
unsorted.

These 'solutions' are really just stop-gaps, and we will likely pursue
one of two options:

a) Cease support of the 'index' file products altogether.

b) Provide new products that present some of the most useful data from
the legacy 'index' files, and cease support for other types of index
data.

If you are a user of the 'index' files associated with GenBank
releases, we
encourage you to make your wishes known, either via the GenBank
newsgroup,
or via email to NCBI's Service Desk:

info@...

Our apologies for any inconvenience that these changes may cause.

1.3.5 GSS File Header Problem

GSS sequences at GenBank are maintained in two different systems,
depending
on their origin, and the dumps from those systems occur in parallel.
Because
the second dump (for example) has no prior knowledge of exactly how many
GSS
files will be dumped by the first, it does not know how to number its
own
output files.

There is thus a discrepancy between the filenames and file headers for
seventy-eight of the GSS flatfiles in Release 174.0. Consider
gbgss273.seq :

GBGSS1.SEQ Genetic Sequence Data Bank
October 15 2009

NCBI-GenBank Flat File Release 174.0

GSS Sequences (Part 1)

87177 loci, 64203737 bases, from 87177 reported sequences

Here, the filename and part number in the header is "1", though the
file
has been renamed as "273" based on the number of files dumped from the
other
system. We hope to resolve this discrepancy at some point, but the
priority
is certainly much lower than many other tasks.

1.4 Upcoming Changes

1.4.1 /artificial_location qualifier introduced

A new qualifier, intended for use in limited genome-scale annotation
contexts, will be introduced as of GenBank Release 175.0 in December
2009:

Qualifier /artificial_location

Definition indicates that location of the CDS or mRNA is modified
to
adjust for the presence of a frameshift or internal stop
codon and not because of biological processing between
the
regions. This is expected to be used only for
genome-scale
annotation, either because a heterogeneous population
was
sequenced, or because the feature is in a region of
low-quality sequence.

1.4.2 New /pseudogene and /non_functional qualifiers

The GenBank 173.0 release notes described an anticipated conversion
of the /pseudo qualifier to /non_functional, based on the results of
the May 2009 INSDC annual meeting:

"Because the term "pseudo" is often assumed to mean 'pseudogene',
the /pseudo qualifier will be renamed as /non_functional, to
better reflect its actual usage in the sequence databases."

During follow-up discussions, the INSDC members decided that existing
uses of /pseudo can include both of the possible meanings of the term,
and that a more conservative course would be to introduce two new
qualifiers:

/pseudogene
/non_functional

Sequence submission tools will be updated to utilize these, and
the ambiguous /pseudo qualifier will be deprecated. If it is
possible, existing instances of /pseudo would then be converted
to one of the two new qualifiers.

/pseudogene and /non_functional will become legal for the Feature
Table as of the April 2010 GenBank Release.

_______________________________________________
Genbankb mailing list
Genbankb@...
http://www.bio.net/biomail/listinfo/genbankb

GenBank 174.0 Close-Of-Data

2009-10-19T08:28:49Z

Greetings GenBank Users,

Close-of-data for the upcoming GenBank Release 174.0 occurred on
Friday October 16 2009 at approximately 1:30am EDT.

The subsequently generated GenBank Incremental Update files
nc1016.aso, nc1016.flat, etc. contain data through the close.

Note: Release processing often does not begin until sometime during
business hours on the close date. As a result, a number of sequence
records processed *after* 1:30am are likely to be present in the
GenBank 174.0 release files, even though they are "post-close" .

Similarly, the first GenBank Incremental Update that is generated
after the close date is likely to contain a number of sequence
records that are unchanged, compared to their appearance in the
release files.

We expect to make the GenBank 174.0 data files available sometime
later today.

Our apologies for the lack of advanced notice about the close date.

Mark Cavanaugh
GenBank
NCBI/NLM/NIH/HHS

_______________________________________________
Genbankb mailing list
Genbankb@...
http://www.bio.net/biomail/listinfo/genbankb

GenBank Release 173.0 Now Available

2009-08-26T19:24:45Z

Greetings GenBank Users,

GenBank Release 173.0 is now available via FTP from the National
Center for Biotechnology Information (NCBI):

Ftp Site Directory Contents
---------------- --------- ---------------------------------------
ftp.ncbi.nih.gov genbank GenBank Release 173.0 flatfiles
ncbi-asn1 ASN.1 data used to create Release 173.0

Close-of-data for GenBank 173.0 occured on 08/21/2009. Uncompressed,
the Release 173.0 flatfiles require roughly 408 GB (sequence files only)
or 437 GB (including the 'short directory', 'index' and the *.txt
files). The ASN.1 data require approximately 370 GB.

Recent statistics for non-WGS, non-CON sequences:

Release Date Base Pairs Entries

172 Jun 2009 105277306080 106073709
173 Aug 2009 106533156756 108431692

Recent statistics for WGS sequences:

Release Date Base Pairs Entries

172 Jun 2009 145959997864 49063546
173 Aug 2009 148165117763 48443067

During the 72 days between the close dates for GenBank Releases 172.0
and 173.0, the non-WGS/non-CON portion of GenBank grew by 1,255,850,676
basepairs and by 2,357,983 sequence records. During that same period,
654,396 records were updated. An average of about 41,838
non-WGS/non-CON records were added and/or updated per day.

Between releases 172.0 and 173.0, the WGS component of GenBank grew
by 2,205,119,899 basepairs, while the number of sequence records decreased
by 620,479 (due to some WGS projects being re-assembled into fewer,
but larger, records).

For additional release information, see the README files in either of
the directories mentioned above, and the release notes (gbrel.txt) in
the genbank directory. Sections 1.3 and 1.4 of the release notes
(Changes in Release 173.0 and Upcoming Changes) have been appended
below for your convenience.

** Important Notes **

* GenBank 'index' files are now provided without any EST content, and
without most GSS content. See Section 1.3.5 of the release notes for
further details.

NCBI is considering ceasing support for the index files, so we
encourage affected users to review that section and provide feedback.

Release 173.0 data, and subsequent updates, are available now via
NCBI's Entrez and Blast services.

As a general guideline, we suggest first transferring the GenBank
release notes (gbrel.txt) whenever a release is being obtained. Check
to make sure that the date and release number in the header of the
release notes are current (eg: August 15 2009, 173.0). If they are
not, interrupt the remaining transfers and then request assistance from
the NCBI Service Desk.

A comprehensive check of the headers of all release files after your
transfers are complete is also suggested. Here's how one might go about
this on a unix platform, using csh/tcsh :

set files = `ls gb*.*`
foreach i ($files)
head -10 $i | grep Release
end

Or, if the files are compressed, perhaps:

gzcat $i | head -10 | grep Release

If you encounter problems while ftp'ing or uncompressing Release
173.0, please send email outlining your difficulties to:

info@...

Mark Cavanaugh, Michael Kimelman, Ilya Dondoshansky, Sergey Zhdanov
GenBank
NCBI/NLM/NIH/HHS

1.3 Important Changes in Release 173.0

1.3.1 Organizational changes

The total number of sequence data files increased by 28 with this release:

- the BCT division is now composed of 48 files (+3)
- the ENV division is now composed of 17 files (+1)
- the EST division is now composed of 886 files (+11)
- the GSS division is now composed of 340 files (+3)
- the HTC division is now composed of 15 files (+1)
- the HTG division is now composed of 134 files (+1)
- the PAT division is now composed of 79 files (+6)
- the PLN division is now composed of 38 files (-1)
- the PRI division is now composed of 40 files (+1)
- the TSA division is now composed of 2 files (+1)
- the VRT division is now composed of 19 files (+1)

Note: The decline in the number of PLN division files is due to the fact that
twelve records representing the chromosomes of Oryza sativa Japonica Group
have been converted to a CON-division representation. See section 1.3.2 for
more details.

The total number of 'index' files increased by 1 with this release:

- the AUT (author) index is now composed of 66 files (+1)

1.3.2 Twelve Oryza sativa Japonica Group records moved to CON division.

Prior to Release 173.0, the sequence records for 12 Oryza sativa Japonica
Group chromosomes had been 'traditional' records, totalling over 382 Mbp of
sequence data.

However, on August 7 2009, the records were converted to a CON-division
representation, with CONTIG-line join() statements that describe how the
chromosomes are constructed from underlying PAC and fosmid sequences. The
records involved are:

LOCUS AP008207 45064769 bp DNA linear CON 07-AUG-2009
LOCUS AP008208 36823111 bp DNA linear CON 07-AUG-2009
LOCUS AP008209 37257345 bp DNA linear CON 07-AUG-2009
LOCUS AP008210 35863200 bp DNA linear CON 07-AUG-2009
LOCUS AP008211 30039014 bp DNA linear CON 07-AUG-2009
LOCUS AP008212 32124789 bp DNA linear CON 07-AUG-2009
LOCUS AP008213 30357780 bp DNA linear CON 07-AUG-2009
LOCUS AP008214 28530027 bp DNA linear CON 07-AUG-2009
LOCUS AP008215 23843360 bp DNA linear CON 07-AUG-2009
LOCUS AP008216 23661561 bp DNA linear CON 07-AUG-2009
LOCUS AP008217 30828668 bp DNA linear CON 07-AUG-2009
LOCUS AP008218 27757321 bp DNA linear CON 07-AUG-2009

Due to this change, these chromosomal records are now present in file
gbcon1.seq of the CON division. In addition, the total number of Oryza sativa
Japonica Group basepairs listed in the table in Section 2.2.7 has declined
by 370,461,808 bases, compared to Release 172.0. The table intentionally
excludes CON-division records, to avoid double-counting of underlying sequence
records and any larger-scale objects that are constructed from them.

1.3.3 File header problem for EST and GSS files

A new method of generating the EST and GSS sequence files has been
developed, which has reduced the time required to generate a GenBank
release by one day. However, a minor problem in the formatting of the
header of the sequence files was inadvertently introduced : a leading
space exists before the filename on the very first line. For example:

GBGSS100.SEQ Genetic Sequence Data Bank
August 15 2009
It should be:

GBGSS100.SEQ Genetic Sequence Data Bank
August 15 2009

The problem effects all EST files and most GSS files. We had hoped to
repair this formatting issue for Release 173.0, but the code changes
which were expected to fix the problem did not perform correctly. It
is hoped that this issue will be resolved for Release 174.0 .

1.3.4 Changes in the content of index files

As described in the GB 153 release notes, the 'index' files which accompany
GenBank releases (see Section 3.3) are considered to be a legacy data product by
NCBI, generated mostly for historical reasons. FTP statistics of January 2005
seem to support this: the index files were transferred only half as frequently as
the files of sequence records. The inherent inefficiencies of the index file
format also lead us to suspect that they have little serious use by the user
community, particularly for EST and GSS records.

The software that generated the index file products received little
attention over the years, and finally reached its limitations in
February 2006 (Release 152.0). The required multi-server queries which
obtained and sorted many millions of rows of terms from several different
databases simply outgrew the capacity of the hardware used for GenBank
Release generation.

Our short-term solution is to cease generating some index-file content
for all EST sequence records, and for GSS sequence records that originate
via direct submission to NCBI.

The three gbacc*.idx index files continue to reflect the entirety of the
release, including all EST and GSS records, however the file contents are
unsorted.

These 'solutions' are really just stop-gaps, and we will likely pursue
one of two options:

a) Cease support of the 'index' file products altogether.

b) Provide new products that present some of the most useful data from
the legacy 'index' files, and cease support for other types of index data.

If you are a user of the 'index' files associated with GenBank releases, we
encourage you to make your wishes known, either via the GenBank newsgroup,
or via email to NCBI's Service Desk:

info@...

Our apologies for any inconvenience that these changes may cause.

1.3.5 GSS File Header Problem

GSS sequences at GenBank are maintained in two different systems, depending
on their origin, and the dumps from those systems occur in parallel. Because
the second dump (for example) has no prior knowledge of exactly how many GSS
files will be dumped by the first, it does not know how to number its own
output files.

There is thus a discrepancy between the filenames and file headers for
seventy-two of the GSS flatfiles in Release 173.0. Consider gbgss269.seq :

GBGSS1.SEQ Genetic Sequence Data Bank
August 15 2009

NCBI-GenBank Flat File Release 173.0

GSS Sequences (Part 1)

87186 loci, 64231458 bases, from 87186 reported sequences

Here, the filename and part number in the header is "1", though the file
has been renamed as "269" based on the number of files dumped from the other
system. We hope to resolve this discrepancy at some point, but the priority
is certainly much lower than many other tasks.

1.4 Upcoming Changes

1.4.1 New class of /exception value

As of October 2009, a new class of /exception will be available for use
on coding region features:

/exception="annotated by transcript or proteomic data"

This exception will be used for situations in which: the protein sequence
(presented via the coding region's /translation qualifier) differs from the
conceptual translation; the quality of the DNA sequencing is high; and there
is evidence at the transcript or proteome level that the presented protein
*is* actually expressed by the organism.

An inference qualifier of type "similar to" should be used in conjunction
with this new type of exception, to indicate the supporting EST/cDNA/protein
sequence.

An updated definition of /exception, with examples for this new type
of exception value, will be provided via the GenBank newsgroup prior to
the October release.

1.4.2 /haplogroup qualifier introduced

A haplotype is a combination of alleles at multiple loci that are transmitted
together on the same chromosome. A haplogroup is a group of similar haplotypes
that share a common ancestor with a single nucleotide polymorphism mutation.
The majority of submitters of complete human mitochondrial genomes provide
information about their haplogroup rather than their haplotype. Stable mtDNA
polymorphic variants clustered together in specific combination form a haplogroup.

To accomodate this need, a new /haplogroup qualifier will be introduced as
of GenBank Release 174.0 in October 2009.

A formal definition of /haplogroup was not available as of the creation
of these release notes, so it will be provided via the GenBank newsgroup
prior to the October release.

1.4.3 /artificial_location qualifier introduced

A new qualifier, intended for use in limited genome-scale annotation
contexts, will be introduced as of GenBank Release 175.0 in December 2009:

Qualifier /artificial_location

Definition indicates that location of the CDS or mRNA is modified to
adjust for the presence of a frameshift or internal stop
codon and not because of biological processing between the
regions. This is expected to be used only for genome-scale
annotation, either because a heterogeneous population was
sequenced, or because the feature is in a region of
low-quality sequence.

1.4.4 /pseudo qualifier renamed as /non_functional

Because the term "pseudo" is often assumed to mean "pseudogene", the
/pseudo qualifier will be renamed as /non_functional, to better reflect
its actual usage in the sequence databases. This change will take place
as of the April 2010 GenBank Release.

_______________________________________________
Genbankb mailing list
Genbankb@...
http://www.bio.net/biomail/listinfo/genbankb

GenBank 173.0 Close-Of-Data

2009-08-25T10:32:47Z

Greetings GenBank Users,

Close-of-data for the upcoming GenBank Release 173.0 occurred on
Friday August 21 2009 at approximately 1:30am EDT.

The subsequently generated GenBank Incremental Update files
nc0821.aso, nc0821.flat, etc. contain data through the close.

Note: Release processing often does not begin until sometime during
business hours on the close date. As a result, a number of sequence
records processed *after* 1:30am are likely to be present in the
GenBank 173.0 release files, even though they are "post-close" .

Similarly, the first GenBank Incremental Update that is generated
after the close date is likely to contain a number of sequence
records that are unchanged, compared to their appearance in the
release files.

We expect to make the GenBank 173.0 data files available sometime
tomorrow.

Our apologies for the lack of advanced notice about the close date,
and for delays that have affected the delivery of the GB 173 product.

Mark Cavanaugh
GenBank
NCBI/NLM/NIH/HHS

_______________________________________________
Genbankb mailing list
Genbankb@...
http://www.bio.net/biomail/listinfo/genbankb

GenBank Updates : Three day outage for GenBank Incremental Updates : 0613-0615

2009-06-16T10:24:05Z

Due to an error in a configuration file, attempts to generate the 0613,
0614,
and 0615 GenBank Incremental Update (GIU) products failed.

This problem was resolved on Tuesday June 16 at approximately 1:00pm
EDT,
and a set of 0616 data files were made available at the NCBI FTP site at
approximately 1:19pm EDT.

The 0616 GIU contains all GenBank records new/modified since 1:33am EDT
on June 12.

Our apologies for the inconvenience that this outage caused.

Mark Cavanaugh
GenBank
NCBI/NLM/NIH/HHS

_______________________________________________
Genbankb mailing list
Genbankb@...
http://www.bio.net/biomail/listinfo/genbankb

GenBank Release 172.0 Now Available

2009-06-12T12:41:53Z

Greetings GenBank Users,

GenBank Release 172.0 is now available via FTP from the National
Center for Biotechnology Information (NCBI):

Ftp Site Directory Contents
---------------- --------- ---------------------------------------
ftp.ncbi.nih.gov genbank GenBank Release 172.0 flatfiles
ncbi-asn1 ASN.1 data used to create Release 172.0

Close-of-data for GenBank 172.0 occured on 06/10/2009. Uncompressed,
the Release 172.0 flatfiles require roughly 403 GB (sequence files only)
or 431 GB (including the 'short directory', 'index' and the *.txt
files). The ASN.1 data require approximately 366 GB.

Recent statistics for non-WGS, non-CON sequences:

Release Date Base Pairs Entries

171 Apr 2009 102980268709 103335421
172 Jun 2009 105277306080 106073709

Recent statistics for WGS sequences:

Release Date Base Pairs Entries

171 Apr 2009 144522542010 48948309
172 Jun 2009 145959997864 49063546

During the 60 days between the close dates for GenBank Releases 171.0
and 172.0, the non-WGS/non-CON portion of GenBank grew by 2,297,037,371
basepairs and by 2,738,288 sequence records. During that same period,
3,680,844 records were updated. An average of about 106,985
non-WGS/non-CON records were added and/or updated per day.

Between releases 171.0 and 172.0, the WGS component of GenBank grew
by 1,437,455,854 basepairs and by 115,237 sequence records.

For additional release information, see the README files in either of
the directories mentioned above, and the release notes (gbrel.txt) in
the genbank directory. Sections 1.3 and 1.4 of the release notes
(Changes in Release 172.0 and Upcoming Changes) have been appended
below for your convenience.

** Important Notes **

* GenBank 'index' files are now provided without any EST content, and
without most GSS content. See Section 1.3.5 of the release notes for
further details.

NCBI is considering ceasing support for the index files, so we
encourage affected users to review that section and provide feedback.

Release 172.0 data, and subsequent updates, are available now via
NCBI's Entrez and Blast services.

As a general guideline, we suggest first transferring the GenBank
release notes (gbrel.txt) whenever a release is being obtained. Check
to make sure that the date and release number in the header of the
release notes are current (eg: April 15 2009, 172.0). If they are
not, interrupt the remaining transfers and then request assistance from
the NCBI Service Desk.

A comprehensive check of the headers of all release files after your
transfers are complete is also suggested. Here's how one might go about
this on a unix platform, using csh/tcsh :

set files = `ls gb*.*`
foreach i ($files)
head -10 $i | grep Release
end

Or, if the files are compressed, perhaps:

gzcat $i | head -10 | grep Release

If you encounter problems while ftp'ing or uncompressing Release
172.0, please send email outlining your difficulties to:

info@...

Mark Cavanaugh, Michael Kimelman, Ilya Dondoshansky, Sergey Zhdanov
GenBank
NCBI/NLM/NIH/HHS

1.3 Important Changes in Release 172.0

1.3.1 PROJECT linetype has been replaced by DBLINK

The DBLINK linetype was introduced as of the February 2009
GenBank Release 170.0, to accomodate links to Project IDs and
the NCBI Trace Assembly Archive, and new types of links that
will arise in the future.

DBLINK co-existed with its predecessor linetype (PROJECT) for GenBank
releases 170.0 and 171.0 . With Release 172.0, however, the PROJECT
line has been completely removed, as this record illustrates:

LOCUS CP000964 5641239 bp DNA circular BCT
24-SEP-2008
DEFINITION Klebsiella pneumoniae 342, complete genome.
ACCESSION CP000964
VERSION CP000964.1 GI:206564770
DBLINK Project:28471

1.3.2 Organizational changes

The total number of sequence data files increased by 36 with this
release:

- the BCT division is now composed of 45 files (+5)
- the ENV division is now composed of 16 files (+3)
- the EST division is now composed of 875 files (+15)
- the GSS division is now composed of 337 files (+2)
- the INV division is now composed of 18 files (+3)
- the PAT division is now composed of 73 files (+6)
- the PLN division is now composed of 39 files (+1)
- the VRL division is now composed of 12 files (+1)

The total number of 'index' files increased by 2 with this release:

- the JOU (journal) index is now composed of 7 files (+1)
- the KEY (keyword) index is now composed of 4 files (+1)

1.3.3 File header problem for EST and GSS files

A new method of generating the EST and GSS sequence files has been
developed, which has reduced the time required to generate a GenBank
release by one day. However, a minor problem in the formatting of the
header of the sequence files was inadvertently introduced : a leading
space exists before the filename on the very first line. For example:

GBGSS100.SEQ Genetic Sequence Data Bank

June 15 2009

It should be:

GBGSS100.SEQ Genetic Sequence Data Bank

June 15 2009

The problem effects all EST files and most GSS files. We had hoped to
repair this formatting issue for Release 172.0, but the code changes
just missed the cut-off for release generation. The problem will
definitely be resolved for Release 173.0 .

1.3.4 Changes in the content of index files

As described in the GB 153 release notes, the 'index' files which
accompany
GenBank releases (see Section 3.3) are considered to be a legacy data
product by
NCBI, generated mostly for historical reasons. FTP statistics of January
2005
seem to support this: the index files were transferred only half as
frequently as
the files of sequence records. The inherent inefficiencies of the index
file
format also lead us to suspect that they have little serious use by the
user
community, particularly for EST and GSS records.

The software that generated the index file products received little
attention over the years, and finally reached its limitations in
February 2006 (Release 152.0). The required multi-server queries which
obtained and sorted many millions of rows of terms from several
different
databases simply outgrew the capacity of the hardware used for GenBank
Release generation.

Our short-term solution is to cease generating some index-file content
for all EST sequence records, and for GSS sequence records that
originate
via direct submission to NCBI.

The three gbacc*.idx index files continue to reflect the entirety of
the
release, including all EST and GSS records, however the file contents
are
unsorted.

These 'solutions' are really just stop-gaps, and we will likely pursue
one of two options:

a) Cease support of the 'index' file products altogether.

b) Provide new products that present some of the most useful data from
the legacy 'index' files, and cease support for other types of index
data.

If you are a user of the 'index' files associated with GenBank
releases, we
encourage you to make your wishes known, either via the GenBank
newsgroup,
or via email to NCBI's Service Desk:

info@...

Our apologies for any inconvenience that these changes may cause.

1.3.5 GSS File Header Problem

GSS sequences at GenBank are maintained in two different systems,
depending
on their origin, and the dumps from those systems occur in parallel.
Because
the second dump (for example) has no prior knowledge of exactly how many
GSS
files will be dumped by the first, it does not know how to number its
own
output files.

There is thus a discrepancy between the filenames and file headers for
seventy-two of the GSS flatfiles in Release 172.0. Consider gbgss266.seq
:

GBGSS1.SEQ Genetic Sequence Data Bank
June 15 2009

NCBI-GenBank Flat File Release 172.0

GSS Sequences (Part 1)

87198 loci, 64267715 bases, from 87198 reported sequences

Here, the filename and part number in the header is "1", though the
file
has been renamed as "266" based on the number of files dumped from the
other
system. We hope to resolve this discrepancy at some point, but the
priority
is certainly much lower than many other tasks.

1.4 Upcoming Changes

1.4.1 Qualifier changes from INSDC 2009

Several qualifier changes for the Feature Table were agreed to at the
annual INSDC meeting in May 2009. Complete details and implementation
timelines will be made available in the August GenBank Release Notes.
In the meantime, here is an early preview of the changes that were
approved:

New value for /exception:

/exception="annotated by transcript or proteomic data"

/pseudo qualifier to be re-named as /non_functional

Because the term "pseudo" is often equated with "pseudogene", the
/pseudo qualifier will be renamed as /non_functional, to better
reflect its actual usage.

New /haplogroup qualifier defined for the source feature

_______________________________________________
Genbankb mailing list
Genbankb@...
http://www.bio.net/biomail/listinfo/genbankb

GenBank 172.0 Close-Of-Data

2009-06-11T14:45:19Z

Greetings GenBank Users,

Close-of-data for the upcoming GenBank Release 172.0 occurred on
Wednesday June 10 2009 at approximately 1:30am EDT.

The subsequently generated GenBank Incremental Update files
nc0610.aso, nc0610.flat, etc. contain data through the close.

Note: Release processing often does not begin until sometime during
business hours on the close date. As a result, a number of sequence
records processed *after* 1:30am are likely to be present in the
GenBank 172.0 release files, even though they are "post-close" .

Similarly, the first GenBank Incremental Update that is generated
after the close date is likely to contain a number of sequence
records that are unchanged, compared to their appearance in the
release files.

We expect to make the GenBank 172.0 data files available sometime
tomorrow.

Our apologies for the lack of advanced notice about the close date.

Mark Cavanaugh
GenBank
NCBI/NLM/NIH/HHS

_______________________________________________
Genbankb mailing list
Genbankb@...
http://www.bio.net/biomail/listinfo/genbankb

GenBank Release 171.0 Now Available

2009-04-14T16:14:06Z

Greetings GenBank Users,

GenBank Release 171.0 is now available via FTP from the National
Center for Biotechnology Information (NCBI):

Ftp Site Directory Contents
---------------- --------- ---------------------------------------
ftp.ncbi.nih.gov genbank GenBank Release 171.0 flatfiles
ncbi-asn1 ASN.1 data used to create Release 171.0

Close-of-data for GenBank 171.0 occured on 04/10/2009. Uncompressed,
the Release 171.0 flatfiles require roughly 395 GB (sequence files only)
or 422 GB (including the 'short directory', 'index' and the *.txt
files). The ASN.1 data require approximately 360 GB.

Recent statistics for non-WGS, non-CON sequences:

Release Date Base Pairs Entries

170 Feb 2009 101467270308 101815678
171 Apr 2009 102980268709 103335421

Recent statistics for WGS sequences:

Release Date Base Pairs Entries

170 Feb 2009 143797800446 49036947
171 Apr 2009 144522542010 48948309

During the 56 days between the close dates for GenBank Releases 170.0
and 171.0, the non-WGS/non-CON portion of GenBank grew by 1,512,998,401
basepairs and by 1,519,743 sequence records. During that same period,
1,040,778 records were updated. An average of about 45,723
non-WGS/non-CON records were added and/or updated per day.

Between releases 170.0 and 171.0, the WGS component of GenBank grew
by 724,741,564 basepairs and the number of records **decreased** by
88,638. A decrease in the overall number of WGS records can
occasionally occur, as a result of genome re-assemblies which yield
larger (but fewer) records, and due to the submission of completed
genomes which supercede WGS projects.

For additional release information, see the README files in either of
the directories mentioned above, and the release notes (gbrel.txt) in
the genbank directory. Sections 1.3 and 1.4 of the release notes
(Changes in Release 171.0 and Upcoming Changes) have been appended
below for your convenience.

** Important Notes **

* This is the final release for which the PROJECT linetype
will be present in GenBank flatfiles. The new DBLINK linetype
replaces PROJECT. Post-171.0 GenBank Update files will
have only the new DBLINK linetype within about two weeks,
and the same will be true of all future complete releases.
See Section 1.3.1 for more information.

* GenBank 'index' files are now provided without any EST content, and
without most GSS content. See Section 1.3.5 of the release notes for
further details.

NCBI is considering ceasing support for the index files, so we
encourage affected users to review that section and provide feedback.

Release 171.0 data, and subsequent updates, are available now via
NCBI's Entrez and Blast services.

As a general guideline, we suggest first transferring the GenBank
release notes (gbrel.txt) whenever a release is being obtained. Check
to make sure that the date and release number in the header of the
release notes are current (eg: April 15 2009, 171.0). If they are
not, interrupt the remaining transfers and then request assistance from
the NCBI Service Desk.

A comprehensive check of the headers of all release files after your
transfers are complete is also suggested. Here's how one might go about
this on a unix platform, using csh/tcsh :

set files = `ls gb*.*`
foreach i ($files)
head -10 $i | grep Release
end

Or, if the files are compressed, perhaps:

gzcat $i | head -10 | grep Release

If you encounter problems while ftp'ing or uncompressing Release
171.0, please send email outlining your difficulties to:

info@...

Mark Cavanaugh, Michael Kimelman, Ilya Dondoshansky, Sergey Zhdanov
GenBank
NCBI/NLM/NIH/HHS

1.3 Important Changes in Release 171.0

1.3.1 PROJECT linetype to be replaced by DBLINK (April 2009)

The new DBLINK linetype was introduced as of the February 2009
GenBank Release 170.0.

Genome Project IDs and Trace Assembly Archive IDs are now presented
via DBLINK, in conjunction with the legacy PROJECT linetype, as this
mock-up for CP000964 illustrates:

LOCUS CP000964 5641239 bp DNA circular BCT
24-SEP-2008
DEFINITION Klebsiella pneumoniae 342, complete genome.
ACCESSION CP000964
VERSION CP000964.1 GI:206564770
PROJECT GenomeProject:28471
DBLINK Project:28471
Trace Assembly Archive:123456
....
COMMENT The source for the DNA and/or cells is: Professor Eric W.
Triplett, Chair, Department of Microbiology and Cell
Science,
Institute of Food and Agricultural Sciences, University of
Florida,
P.O. Box 110700, Gainesville, FL 32611-0700, ewt@....

PROJECT and DBLINK have co-existed for GenBank releases 170.0 and 171.0
.
But subsequent to this April release, the PROJECT line will be removed
from the
flatfile format. In its final state, the above mock-up for CP000964
becomes:

LOCUS CP000964 5641239 bp DNA circular BCT
24-SEP-2008
DEFINITION Klebsiella pneumoniae 342, complete genome.
ACCESSION CP000964
VERSION CP000964.1 GI:206564770
DBLINK Project:28471
Trace Assembly Archive:123456
....
COMMENT The source for the DNA and/or cells is: Professor Eric W.
Triplett, Chair, Department of Microbiology and Cell
Science,
Institute of Food and Agricultural Sciences, University of
Florida,
P.O. Box 110700, Gainesville, FL 32611-0700, ewt@....

In summary: The PROJECT linetype will cease to be displayed in
post-171.0
GenBank Update data products within the next two weeks. The same will be
true of all future complete GenBank releases.

1.3.2 Organizational changes

The total number of sequence data files increased by 34 with this
release:

- the BCT division is now composed of 40 files (+2)
- the CON division is now composed of 131 files (+3)
- the ENV division is now composed of 13 files (+1)
- the EST division is now composed of 860 files (+22)
- the GSS division is now composed of 335 files (+13)
- the INV division is now composed of 15 files (+1)
- the MAM division is now composed of 5 files (+1)
- the PAT division is now composed of 67 files (+2)
- the PLN division is now composed of 38 files (+1)

1.3.3 CON-division records for 'segmented sets' restored.

A previously overlooked problem in Release 170.0 processing resulted
in
the exclusion of roughly 14,000 CON-division entries for a type of
record
referred to as a 'segmented set'. Segmented sets consist of small
sequence
fragments of an incompletely sequenced molecule, packaged together, with
a
top-level sequence that specifies the order of the underlying fragments.
That top-level sequence can be displayed as a CON division record.
AH000819
is an example:

ASN.1 for AH000819:

http://www.ncbi.nlm.nih.gov/nuccore/405204?report=asn1&log$=seqview

Flatfile view of the nine sequenced fragments:

http://www.ncbi.nlm.nih.gov/nuccore/405204

These (largely legacy) CON-division records have been restored in
Release 171.0 and can be found in gbcon131.seq .

1.3.4 File header problem for EST and GSS files

A new method of generating the EST and GSS sequence files has been
developed, which has reduced the time required to generate a GenBank
release by one day. However, a minor problem in the formatting of the
header of the sequence files was inadvertently introduced : a leading
space exists before the filename on the very first line. For example:

GBGSS100.SEQ Genetic Sequence Data Bank

April 15 2009

It should be:

GBGSS100.SEQ Genetic Sequence Data Bank

April 15 2009

The problem effects all EST files and most GSS files. We doubt that it
will cause significant problems for users, however the problem will be
corrected for our next release.

1.3.5 Changes in the content of index files

As described in the GB 153 release notes, the 'index' files which
accompany
GenBank releases (see Section 3.3) are considered to be a legacy data
product by
NCBI, generated mostly for historical reasons. FTP statistics of January
2005
seem to support this: the index files were transferred only half as
frequently as
the files of sequence records. The inherent inefficiencies of the index
file
format also lead us to suspect that they have little serious use by the
user
community, particularly for EST and GSS records.

The software that generated the index file products received little
attention over the years, and finally reached its limitations in
February 2006 (Release 152.0). The required multi-server queries which
obtained and sorted many millions of rows of terms from several
different
databases simply outgrew the capacity of the hardware used for GenBank
Release generation.

Our short-term solution is to cease generating some index-file content
for all EST sequence records, and for GSS sequence records that
originate
via direct submission to NCBI.

The three gbacc*.idx index files continue to reflect the entirety of
the
release, including all EST and GSS records, however the file contents
are
unsorted.

These 'solutions' are really just stop-gaps, and we will likely pursue
one of two options:

a) Cease support of the 'index' file products altogether.

b) Provide new products that present some of the most useful data from
the legacy 'index' files, and cease support for other types of index
data.

If you are a user of the 'index' files associated with GenBank
releases, we
encourage you to make your wishes known, either via the GenBank
newsgroup,
or via email to NCBI's Service Desk:

info@...

Our apologies for any inconvenience that these changes may cause.

1.3.6 GSS File Header Problem

GSS sequences at GenBank are maintained in two different systems,
depending
on their origin, and the dumps from those systems occur in parallel.
Because
the second dump (for example) has no prior knowledge of exactly how many
GSS
files will be dumped by the first, it does not know how to number its
own
output files.

There is thus a discrepancy between the filenames and file headers for
seventy-two of the GSS flatfiles in Release 171.0. Consider gbgss264.seq
:

GBGSS1.SEQ Genetic Sequence Data Bank
April 15 2009

NCBI-GenBank Flat File Release 171.0

GSS Sequences (Part 1)

87206 loci, 64290178 bases, from 87206 reported sequences

Here, the filename and part number in the header is "1", though the
file
has been renamed as "254" based on the number of files dumped from the
other
system. We will work to resolve this discrepancy in future releases,
but the
priority is certainly much lower than many other tasks.

1.4 Upcoming Changes

There are no scheduled format changes for GenBank.

_______________________________________________
Genbankb mailing list
Genbankb@...
http://www.bio.net/biomail/listinfo/genbankb

GenBank 171.0 Close-Of-Data

2009-04-14T09:43:10Z

Greetings GenBank Users,

Close-of-data for the upcoming GenBank Release 171.0 occurred on
Friday April 10 2009 at approximately 1:30am EDT.

The subsequently generated GenBank Incremental Update files
nc0410.aso, nc0410.flat, etc. contain data through the close.

Note: Release processing often does not begin until sometime during
business hours on the close date. As a result, a number of sequence
records processed *after* 1:30am are likely to be present in the
GenBank 171.0 release files, even though they are "post-close" .

Similarly, the first GenBank Incremental Update that is generated
after the close date is likely to contain a number of sequence
records that are unchanged, compared to their appearance in the
release files.

We expect to make the GenBank 171.0 data files available later
today.

Our apologies for the lack of advanced notice about the close date.

Mark Cavanaugh
GenBank
NCBI/NLM/NIH/HHS

_______________________________________________
Genbankb mailing list
Genbankb@...
http://www.bio.net/biomail/listinfo/genbankb

RE: enquire

2009-03-30T09:30:47Z

Dear Yang Jiang,

There are many data products at the NCBI FTP site, generated by
many different NCBI groups.

The RefSeq/UniProt mapping file that you mention is not a
GenBank-related product, so it might be better to send your
inquiry to the NCBI Help Desk:

info@...

I hope this helps.

Regards,

Mark Cavanaugh
GenBank
NCBI/NLM/NIH/HHS

> -----Original Message-----
> From: jybackup [mailto:jybackup@...]
> Sent: Friday, March 27, 2009 2:41 AM
> To: genbankb@...
> Subject: [Genbank-bb] enquire
>
> Dear administrator,
> i was not very clearly about the meaning of the word "identical",could
you tell me? in NCBI ftp,there is
> a document which map the ncbi refseq accno to uniprot acc, it refers
that it get the identical taxid and
> sequence,but the length of ncbi doesn't equal to UNIPROT'S.can you
tell me why.identical means equal
> to?waiting for your reply.
> best wishes
> yang jiang

_______________________________________________
Genbankb mailing list
Genbankb@...
http://www.bio.net/biomail/listinfo/genbankb

enquire

2009-03-26T23:41:17Z

Dear administrator,
i was not very clearly about the meaning of the word "identical",could you tell me? in NCBI ftp,there is a document which map the ncbi refseq accno to uniprot acc, it refers that it get the identical taxid and sequence,but the length of ncbi doesn't equal to UNIPROT'S.can you tell me why.identical means equal to?waiting for your reply.
best wishes
yang jiang

网易邮箱，中国第一大电子邮件服务商
_______________________________________________
Genbankb mailing list
Genbankb@...
http://www.bio.net/biomail/listinfo/genbankb

GenBank Release 170.0 Now Available

2009-02-18T17:03:07Z

Greetings GenBank Users,

GenBank Release 170.0 is now available via FTP from the National
Center for Biotechnology Information (NCBI):

Ftp Site Directory Contents
---------------- --------- ---------------------------------------
ftp.ncbi.nih.gov genbank GenBank Release 170.0 flatfiles
ncbi-asn1 ASN.1 data used to create Release 170.0

Close-of-data for GenBank 170.0 occured on 02/13/2009. Uncompressed,
the Release 170.0 flatfiles require roughly 390 GB (sequence files only)
or 417 GB (including the 'short directory', 'index' and the *.txt
files). The ASN.1 data require approximately 356 GB.

Recent statistics for non-WGS, non-CON sequences:

Release Date Base Pairs Entries

169 Dec 2008 99116431942 98868465
170 Feb 2009 101467270308 101815678

Recent statistics for WGS sequences:

Release Date Base Pairs Entries

169 Dec 2008 141374971004 48394838
170 Feb 2009 143797800446 49036947

During the 64 days between the close dates for GenBank Releases 169.0
and 170.0, the non-WGS/non-CON portion of GenBank grew by 2,350,838,366
basepairs and by 2,947,213 sequence records. During that same period,
1,318,594 records were updated. An average of about 66,653
non-WGS/non-CON
records were added and/or updated per day.

Between releases 169.0 and 170.0, the WGS component of GenBank grew by
2,422,829,442 basepairs and by 642,109 records.

For additional release information, see the README files in either of
the directories mentioned above, and the release notes (gbrel.txt) in
the genbank directory. Sections 1.3 and 1.4 of the release notes
(Changes in Release 170.0 and Upcoming Changes) have been appended
below for your convenience.

** Important Notes **

* A new linetype ( DBLINK ) has been implemented with this February
2009
release. See Section 1.3.2 for information. Records with the DBLINK
line will begin to appear via the GenBank Updates shortly after
February 18, 2009.

* GenBank 'index' files are now provided without any EST content, and
without most GSS content. See Section 1.3.3 of the release notes for
further details.

NCBI is considering ceasing support for the index files, so we
encourage affected users to review that section and provide feedback.

Release 170.0 data, and subsequent updates, are available now via
NCBI's Entrez and Blast services.

As a general guideline, we suggest first transferring the GenBank
release notes (gbrel.txt) whenever a release is being obtained. Check
to make sure that the date and release number in the header of the
release notes are current (eg: February 15 2009, 170.0). If they are
not, interrupt the remaining transfers and then request assistance from
the NCBI Service Desk.

A comprehensive check of the headers of all release files after your
transfers are complete is also suggested. Here's how one might go about
this on a unix platform, using csh/tcsh :

set files = `ls gb*.*`
foreach i ($files)
head -10 $i | grep Release
end

Or, if the files are compressed, perhaps:

gzcat $i | head -10 | grep Release

If you encounter problems while ftp'ing or uncompressing Release
170.0, please send email outlining your difficulties to:

info@...

Mark Cavanaugh, Michael Kimelman, Ilya Dondoshansky
GenBank
NCBI/NLM/NIH/HHS

1.3.1 Organizational changes

The total number of sequence data files increased by 34 with this
release:

- the BCT division is now composed of 38 files (+4)
- the ENV division is now composed of 12 files (+1)
- the EST division is now composed of 838 files (+16)
- the GSS division is now composed of 322 files (+3)
- the HTG division is now composed of 133 files (-1)
- the PAT division is now composed of 65 files (+10)
- the VRT division is now composed of 18 files (+1)

Note that the HTG division has one less file than previously.
Occasional
decreases like this are now possible, given a new method of flatfile
processing that was adopted in December 2008. Essentially, records
within
a particular sequence file (eg, gbhtg134.seq) can become absorbed in a
different file, depending on the overall distribution of the number of
records in the files that make up a division. Such decreases are
reviewed,
to ensure that they do not indicate a loss of release content.

In addition, the total number of index files increased by 3 with this
release:

- the AUT index is now composed of 65 files (+3)

1.3.2 New DBLINK linetype legal as of February 2009.

The PROJECT linetype allows a sequence record to be linked to
information
about the sequencing project that generated the data which ultimately
resulted in the record's submission to the International Nucleotide
Sequence
Database ( INSD; see http://www.insdc.org ) .

This complete bacterial GenBank record illustrates the use of the
PROJECT
line:

LOCUS CP000964 5641239 bp DNA circular BCT
24-SEP-2008
DEFINITION Klebsiella pneumoniae 342, complete genome.
ACCESSION CP000964
VERSION CP000964.1 GI:206564770
PROJECT GenomeProject:28471

When viewed on the web in NCBI's Entrez:Nucleotide, the record's
project
identifier (28471) links to an entry in the Genome Project Database
(GPDB) :

http://www.ncbi.nlm.nih.gov/sites/entrez?db=genomeprj&cmd=Retrieve&dopt=
Overview&uid=28471

where information about the sequencing center, the bacterium, and other
GenBank records (eg, plasmids) associated with the sequencing project
can be found.

Since the introduction of PROJECT, the scope of the "Genome Project"
Database has expanded, to include projects that are not necessarily
targetted
to the sequencing of a complete genome.

In addition, there can be other resources which underlie an INSD
sequence
record, such as the Trace Assembly Archive at the NCBI:

http://www.ncbi.nlm.nih.gov/Traces/assembly/assmbrowser.cgi?cmd=show&f=t
ree&m=main&s=tree

Because of the expanded scope of the GPDB, and because we anticipate a
need
to link to more resources than just the GPDB, the PROJECT linetype is
going to
be replaced by a new linetype:

DBLINK

With this release, the new DBLINK linetype is now legal for GenBank
sequence
records, and it will begin to appear in GenBank Update files, soon after
GenBank 170.0 is made available.

The Genome Project ID and the Trace Assembly Archive ID will be
presented
via DBLINK, and the existing PROJECT line will continue to be displayed,
as
illustrated in the below mock-up of CP000964 :

LOCUS CP000964 5641239 bp DNA circular BCT
24-SEP-2008
DEFINITION Klebsiella pneumoniae 342, complete genome.
ACCESSION CP000964
VERSION CP000964.1 GI:206564770
PROJECT GenomeProject:28471
DBLINK Project:28471
Trace Assembly Archive:123456
....
COMMENT The source for the DNA and/or cells is: Professor Eric W.
Triplett, Chair, Department of Microbiology and Cell
Science,
Institute of Food and Agricultural Sciences, University of
Florida,
P.O. Box 110700, Gainesville, FL 32611-0700, ewt@....

Note: Use of the Trace Assembly Archive is still in its early stages, so
only
a few records are expected to have that type of DBLINK in the short
term.

For those who process sequence data in NCBI's ASN.1 format: The
underlying representation for (Genome) Project IDs will remain
unchanged.
There will be no changes to the ASN.1 User-object that is used to store
them:

user {
type
str "GenomeProjectsDB" ,
data {
{
label
str "ProjectID" ,
data
int 28471 } ,
{
label
str "ParentID" ,
data
int 0 } } } ,

However, to support linkages to other resources, such as the Trace
Assembly Archive, a new "DBLink" User-object will be introduced:

user {
type
str "DBLink" ,
data {
{
label
str "Trace Assembly Archive" ,
data
ints { 123456 } } } }

As new types of linkages are established, they will be added to
the DBLink User-object, and displayed via the DBLINK linetype in
the GenBank flatfile format.

There is a possibility that the GenomeProjectsDB User-object
might someday be incorporated into the new DBLink User-object.
But at the moment, there are no firm plans to do so.

1.3.3 Changes in the content of index files

As described in the GB 153 release notes, the 'index' files which
accompany
GenBank releases (see Section 3.3) are considered to be a legacy data
product by
NCBI, generated mostly for historical reasons. FTP statistics of January
2005
seem to support this: the index files were transferred only half as
frequently as
the files of sequence records. The inherent inefficiencies of the index
file
format also lead us to suspect that they have little serious use by the
user
community, particularly for EST and GSS records.

The software that generated the index file products received little
attention over the years, and finally reached its limitations in
February 2006 (Release 152.0). The required multi-server queries which
obtained and sorted many millions of rows of terms from several
different
databases simply outgrew the capacity of the hardware used for GenBank
Release generation.

Our short-term solution is to cease generating some index-file content
for all EST sequence records, and for GSS sequence records that
originate
via direct submission to NCBI.

The three gbacc*.idx index files continue to reflect the entirety of
the
release, including all EST and GSS records, however the file contents
are
unsorted.

These 'solutions' are really just stop-gaps, and we will likely pursue
one of two options:

a) Cease support of the 'index' file products altogether.

b) Provide new products that present some of the most useful data from
the legacy 'index' files, and cease support for other types of index
data.

If you are a user of the 'index' files associated with GenBank
releases, we
encourage you to make your wishes known, either via the GenBank
newsgroup,
or via email to NCBI's Service Desk:

info@...

Our apologies for any inconvenience that these changes may cause.

1.3.4 GSS File Header Problem

GSS sequences at GenBank are maintained in two different systems,
depending
on their origin, and the dumps from those systems occur in parallel.
Because
the second dump (for example) has no prior knowledge of exactly how many
GSS
files will be dumped by the first, it does not know how to number its
own
output files.

There is thus a discrepancy between the filenames and file headers for
sixty-nine of the GSS flatfiles in Release 170.0. Consider gbgss254.seq
:

GBGSS1.SEQ Genetic Sequence Data Bank
February 15 2009

NCBI-GenBank Flat File Release 170.0

GSS Sequences (Part 1)

87215 loci, 64322450 bases, from 87215 reported sequences

Here, the filename and part number in the header is "1", though the
file
has been renamed as "254" based on the number of files dumped from the
other
system. We will work to resolve this discrepancy in future releases,
but the
priority is certainly much lower than many other tasks.

1.4 Upcoming Changes

1.4.1 PROJECT linetype to be replaced by DBLINK (April 2009)

The new DBLINK linetype has been introduced as of the February 2009
GenBank Release 170.0.

Genome Project IDs and Trace Assembly Archive IDs can now be presented
via DBLINK, in conjunction with the legacy PROJECT linetype:

LOCUS CP000964 5641239 bp DNA circular BCT
24-SEP-2008
DEFINITION Klebsiella pneumoniae 342, complete genome.
ACCESSION CP000964
VERSION CP000964.1 GI:206564770
PROJECT GenomeProject:28471
DBLINK Project:28471
Trace Assembly Archive:123456
....
COMMENT The source for the DNA and/or cells is: Professor Eric W.
Triplett, Chair, Department of Microbiology and Cell
Science,
Institute of Food and Agricultural Sciences, University of
Florida,
P.O. Box 110700, Gainesville, FL 32611-0700, ewt@....

PROJECT and DBLINK will co-exist for one GenBank release, until Release
171.0
(April 15, 2009), at which point the PROJECT line will be removed from
the
flatfile format. In its final state, the above mock-up for CP000964
becomes:

LOCUS CP000964 5641239 bp DNA circular BCT
24-SEP-2008
DEFINITION Klebsiella pneumoniae 342, complete genome.
ACCESSION CP000964
VERSION CP000964.1 GI:206564770
DBLINK Project:28471
Trace Assembly Archive:123456
....
COMMENT The source for the DNA and/or cells is: Professor Eric W.
Triplett, Chair, Department of Microbiology and Cell
Science,
Institute of Food and Agricultural Sciences, University of
Florida,
P.O. Box 110700, Gainesville, FL 32611-0700, ewt@....

In summary: The PROJECT linetype will be replaced by DBLINK as of
GenBank
Release 171.0 in April 2009.

_______________________________________________
Genbankb mailing list
Genbankb@...
http://www.bio.net/biomail/listinfo/genbankb

GenBank 170.0 Close-Of-Data

2009-02-18T09:43:51Z

Greetings GenBank Users,

Close-of-data for the upcoming GenBank Release 170.0 occurred on
Friday February 13 2009 at approximately 1:30am EST.

The subsequently generated GenBank Incremental Update files
nc0213.aso, nc0213.flat, etc. contain data through the close.

Note: Release processing often does not begin until sometime during
business hours on the close date. As a result, a number of sequence
records processed *after* 1:30am are likely to be present in the
GenBank 170.0 release files, even though they are "post-close" .

Similarly, the first GenBank Incremental Update that is generated
after the close date is likely to contain a number of sequence
records that are unchanged, compared to their appearance in the
release files.

We expect to make the GenBank 170.0 data files available later
today.

Our apologies for the lack of advanced notice about the close date.

Mark Cavanaugh
GenBank
NCBI/NLM/NIH/HHS

_______________________________________________
Genbankb mailing list
Genbankb@...
http://www.bio.net/biomail/listinfo/genbankb

GenBank WGS projects : WGS-master records provided as of January 12

2009-01-02T08:51:53Z

Greetings GenBank Users,

Starting on January 12th of 2009, a new type of data file will be
made available for GenBank WGS (Whole Genome Shotgun) projects,
in the WGS areas of our FTP site.

Since their inception in 2002, WGS projects have had an associated
'WGS-master' record, which summarizes the content of a project. Here
is a link to the master for project ABRT (Philippine tarsier) :

http://www.ncbi.nlm.nih.gov/entrez/viewer.fcgi?db=nuccore&id=203287470

And here is an excerpt from that master record:

LOCUS ABRT010000000 1201173 rc DNA linear PRI
18-NOV-2008
DEFINITION Tarsius syrichta, whole genome shotgun sequence.
ACCESSION ABRT000000000
VERSION ABRT000000000.1 GI:203287470
PROJECT GenomeProject:20339
KEYWORDS WGS.
SOURCE Tarsius syrichta (Philippine tarsier)
....
WGS ABRT010000001-ABRT011201173
WGS_SCAFLD GG299110-GG500513
//

This flatfile representation of the ABRT WGS-master does *not*
conform to the specifications for normal GenBank flatfiles.
For example:

- It has neither sequence data nor a CONTIG join() statement.

- The 'rc' (record count) value on the LOCUS line represents the
number of sequence-overlap contig records in the project, rather
than a basepair count.

- Undocumented linetypes 'WGS' and 'WGS_SCAFLD' exist, which
provide the ranges of accession numbers for the 1,201,173
sequence-overlap contig sequences in the project, and for
the 201,404 CON-division records that have been constructed
from the ABRT01 contigs.

Nonetheless, a WGS-master record has utility because it provides
an overview of many important characteristics of a WGS project,
in a simple and concise way.

The ASN.1 version of WGS-master records will be placed in:

ftp://ftp.ncbi.nih.gov/ncbi-asn1/wgs

and the file naming convention will be:

wgs.XXXX.mstr.bse.gz

These files will contain a gzip-compressed, binary ASN.1 Seq-entry
value. 'XXXX' represents a four-character WGS Project Code, such as
ABYH.

The GenBank flatfile representation of WGS-master records will be
placed in:

ftp://ftp.ncbi.nih.gov/genbank/wgs

and the file naming convention will be:

wgs.XXXX.mstr.gbff.gz

Here is an example of the filenames that one would encounter for
the ABYH project in the /genbank/wgs area, as of January 12:

wgs.ABYH.1.gbff.gz
wgs.ABYH.1.gnp.gz
wgs.ABYH.1.qscore.gz
wgs.ABYH.mstr.gbff.gz

If you process the GenBank flatfile representation of WGS projects,
and you are *not* interested in WGS-masters, you may need to add
a filtration step to remove the master files from automated FTP
transfers (due to similarities in filename patterns).

Mark Cavanaugh
GenBank
NCBI/NLM/NIH/HHS

_______________________________________________
Genbankb mailing list
Genbankb@...
http://www.bio.net/biomail/listinfo/genbankb

GenBank Release 169.0 Now Available

2008-12-16T01:17:17Z

Greetings GenBank Users,

GenBank Release 169.0 is now available via FTP from the National
Center for Biotechnology Information (NCBI):

Ftp Site Directory Contents
---------------- --------- ---------------------------------------
ftp.ncbi.nih.gov genbank GenBank Release 169.0 flatfiles
ncbi-asn1 ASN.1 data used to create Release 169.0

Close-of-data for GenBank 169.0 occured on 12/11/2008. Uncompressed,
the Release 169.0 flatfiles require roughly 381 GB (sequence files only)
or 407 GB (including the 'short directory', 'index' and the *.txt
files). The ASN.1 data require approximately 349 GB.

Recent statistics for non-WGS, non-CON sequences:

Release Date Base Pairs Entries

168 Oct 2008 97381682336 96400790
169 Dec 2008 99116431942 98868465

Recent statistics for WGS sequences:

Release Date Base Pairs Entries

168 Oct 2008 136085973423 46108952
169 Dec 2008 141374971004 48394838

During the 45 days between the close dates for GenBank Releases 168.0
and 169.0, the non-WGS/non-CON portion of GenBank grew by 1,734,749,606
basepairs and by 2,467,675 sequence records. During that same period,
4,183,486 records were updated. An average of about 147,803 non-WGS/non-CON
records were added and/or updated per day.

Between releases 168.0 and 169.0, the WGS component of GenBank grew by
5,288,997,581 basepairs and by 2,285,886 records.

For additional release information, see the README files in either of
the directories mentioned above, and the release notes (gbrel.txt) in
the genbank directory. Sections 1.3 and 1.4 of the release notes
(Changes in Release 169.0 and Upcoming Changes) have been appended
below for your convenience.

** Important Notes **

* A new linetype ( DBLINK ) will be implemented as of the February 2009
release. See Section 1.4.1 for information.

* GenBank 'index' files are now provided without any EST content, and
without most GSS content. See Section 1.3.12 of the release notes for
further details.

NCBI is considering ceasing support for the index files, so we
encourage affected users to review that section and provide feedback.

Release 169.0 data, and subsequent updates, are available now via
NCBI's Entrez and Blast services.

As a general guideline, we suggest first transferring the GenBank
release notes (gbrel.txt) whenever a release is being obtained. Check
to make sure that the date and release number in the header of the
release notes are current (eg: December 15 2008, 169.0). If they are
not, interrupt the remaining transfers and then request assistance from
the NCBI Service Desk.

A comprehensive check of the headers of all release files after your
transfers are complete is also suggested. Here's how one might go about
this on a unix platform, using csh/tcsh :

set files = `ls gb*.*`
foreach i ($files)
head -10 $i | grep Release
end

Or, if the files are compressed, perhaps:

gzcat $i | head -10 | grep Release

If you encounter problems while ftp'ing or uncompressing Release
169.0, please send email outlining your difficulties to:

info@...

Mark Cavanaugh, Michael Kimelman, Ilya Dondoshansky
GenBank
NCBI/NLM/NIH/HHS

1.3 Important Changes in Release 169.0

1.3.1 Organizational changes

As described in a post to the GenBank Newsgroup on December 11, 2008, the
number of sequence data files associated with this GenBank release has
increased by a larger-than-usual amount, due to changes in the way that data
are processed and stored at NCBI. Thus the per-division increases described
below reflect both the storage-related changes *and* actual database growth.

In addition, GenBank divisions which consist of a single data file now
include a number in their names:

gbtsa.seq -> gbtsa1.seq
gbuna.seq -> gbuna1.seq

The total number of sequence data files increased by 94 with this release:

- the BCT division is now composed of 34 files (+2)
- the CON division is now composed of 128 files (+25)
- the ENV division is now composed of 11 files (+1)
- the EST division is now composed of 822 files (+20)
- the GSS division is now composed of 319 files (+10)
- the HTC division is now composed of 13 files (+1)
- the HTG division is now composed of 134 files (+12)
- the INV division is now composed of 14 files (+1)
- the PAT division is now composed of 55 files (+8)
- the PLN division is now composed of 37 files (+5)
- the PRI division is now composed of 39 files (+3)
- the ROD division is now composed of 28 files (+2)
- the STS division is now composed of 20 files (+2)
- the VRL division is now composed of 11 files (+1)
- the VRT division is now composed of 17 files (+1)

In addition, the total number of index files increased by 3 with this release:

- the AUT index is now composed of 62 files (+3)

1.3.2 Changes in the content of index files

As described in the GB 153 release notes, the 'index' files which accompany
GenBank releases (see Section 3.3) are considered to be a legacy data product by
NCBI, generated mostly for historical reasons. FTP statistics of January 2005
seem to support this: the index files were transferred only half as frequently as
the files of sequence records. The inherent inefficiencies of the index file
format also lead us to suspect that they have little serious use by the user
community, particularly for EST and GSS records.

The software that generated the index file products received little
attention over the years, and finally reached its limitations in
February 2006 (Release 152.0). The required multi-server queries which
obtained and sorted many millions of rows of terms from several different
databases simply outgrew the capacity of the hardware used for GenBank
Release generation.

Our short-term solution is to cease generating some index-file content
for all EST sequence records, and for GSS sequence records that originate
via direct submission to NCBI.

The three gbacc*.idx index files continue to reflect the entirety of the
release, including all EST and GSS records, however the file contents are
unsorted.

These 'solutions' are really just stop-gaps, and we will likely pursue
one of two options:

a) Cease support of the 'index' file products altogether.

b) Provide new products that present some of the most useful data from
the legacy 'index' files, and cease support for other types of index data.

If you are a user of the 'index' files associated with GenBank releases, we
encourage you to make your wishes known, either via the GenBank newsgroup,
or via email to NCBI's Service Desk:

info@...

Our apologies for any inconvenience that these changes may cause.

1.3.3 GSS File Header Problem

GSS sequences at GenBank are maintained in two different systems, depending
on their origin, and the dumps from those systems occur in parallel. Because
the second dump (for example) has no prior knowledge of exactly how many GSS
files will be dumped by the first, it does not know how to number its own
output files.

There is thus a discrepancy between the filenames and file headers for
sixty-nine of the GSS flatfiles in Release 169.0. Consider gbgss251.seq :

GBGSS1.SEQ Genetic Sequence Data Bank
December 15 2008

NCBI-GenBank Flat File Release 169.0

GSS Sequences (Part 1)

87209 loci, 64341123 bases, from 87209 reported sequences

Here, the filename and part number in the header is "1", though the file
has been renamed as "251" based on the number of files dumped from the other
system. We will work to resolve this discrepancy in future releases, but the
priority is certainly much lower than many other tasks.

1.4 Upcoming Changes

1.4.1 PROJECT linetype to be replaced by DBLINK (April 2009)

The PROJECT linetype allows a sequence record to be linked to information
about the sequencing project that generated the data which ultimately
resulted in the record's submission to the International Nucleotide Sequence
Database ( INSD; see http://www.insdc.org ) .

This complete bacterial GenBank record illustrates the use of the PROJECT
line:

LOCUS CP000964 5641239 bp DNA circular BCT 24-SEP-2008
DEFINITION Klebsiella pneumoniae 342, complete genome.
ACCESSION CP000964
VERSION CP000964.1 GI:206564770
PROJECT GenomeProject:28471

When viewed on the web in NCBI's Entrez:Nucleotide, the record's project
identifier (28471) links to an entry in the Genome Project Database (GPDB) :

http://www.ncbi.nlm.nih.gov/sites/entrez?db=genomeprj&cmd=Retrieve&dopt=Overview&uid=28471

where information about the sequencing center, the bacterium, and other
GenBank records (eg, plasmids) associated with the sequencing project
can be found.

Since the introduction of PROJECT, the scope of the "Genome" Project
Database has expanded, to include projects that are not necessarily targetted
to the sequencing of a complete genome.

In addition, there can be other resources which underlie an INSD sequence
record, such as the Trace Assembly Archive at the NCBI:

http://www.ncbi.nlm.nih.gov/Traces/assembly/assmbrowser.cgi?cmd=show&f=tree&m=main&s=tree

Because of the expanded scope of the GPDB, and because we anticipate a need
to link to more resources than just the GPDB, the PROJECT linetype is going to
be replaced by a new linetype:

DBLINK

Modifications to linetypes can be disruptive, so the switch to DBLINK will occur
in several stages. As of October 2008, links to the NCBI Trace Assembly Archive are
displayed via a line of text in the COMMENT section of sequence records. Here is a
mock-up, based on CP000964, to illustrate that initial change:

LOCUS CP000964 5641239 bp DNA circular BCT 24-SEP-2008
DEFINITION Klebsiella pneumoniae 342, complete genome.
ACCESSION CP000964
VERSION CP000964.1 GI:206564770
PROJECT GenomeProject:28471
....
COMMENT Trace Assembly Archive:123456
The source for the DNA and/or cells is: Professor Eric W.
Triplett, Chair, Department of Microbiology and Cell Science,
Institute of Food and Agricultural Sciences, University of Florida,
P.O. Box 110700, Gainesville, FL 32611-0700, ewt@....

Note: Use of the Trace Assembly Archive is still in its early stages, so only
a few records are expected to have these links in the short term.

The new DBLINK linetype will then be introduced with GenBank Release 170.0 ,
on or near February 15, 2009 .

The Genome Project ID and the Trace Assembly Archive ID will be presented
via DBLINK, and the existing PROJECT line will continue to be displayed:

LOCUS CP000964 5641239 bp DNA circular BCT 24-SEP-2008
DEFINITION Klebsiella pneumoniae 342, complete genome.
ACCESSION CP000964
VERSION CP000964.1 GI:206564770
PROJECT GenomeProject:28471
DBLINK Project:28471
Trace Assembly Archive:123456
....
COMMENT The source for the DNA and/or cells is: Professor Eric W.
Triplett, Chair, Department of Microbiology and Cell Science,
Institute of Food and Agricultural Sciences, University of Florida,
P.O. Box 110700, Gainesville, FL 32611-0700, ewt@....

PROJECT and DBLINK will co-exist for one GenBank release, until Release 171.0
(April 15, 2009), at which point the PROJECT line will be removed. In its final
state, our mock-up for CP000964 becomes:

LOCUS CP000964 5641239 bp DNA circular BCT 24-SEP-2008
DEFINITION Klebsiella pneumoniae 342, complete genome.
ACCESSION CP000964
VERSION CP000964.1 GI:206564770
DBLINK Project:28471
Trace Assembly Archive:123456
....
COMMENT The source for the DNA and/or cells is: Professor Eric W.
Triplett, Chair, Department of Microbiology and Cell Science,
Institute of Food and Agricultural Sciences, University of Florida,
P.O. Box 110700, Gainesville, FL 32611-0700, ewt@....

In summary: The PROJECT linetype will be replaced by DBLINK as of GenBank
Release 171.0 in April 2009.

For those who process sequence data in NCBI's ASN.1 format: The
underlying representation for (Genome) Project IDs will remain unchanged.
There will be no changes to the ASN.1 User-object that is used to store them:

user {
type
str "GenomeProjectsDB" ,
data {
{
label
str "ProjectID" ,
data
int 28471 } ,
{
label
str "ParentID" ,
data
int 0 } } } ,

However, to support linkages to other resources, such as the Trace
Assembly Archive, a new "DBLink" User-object will be introduced:

user {
type
str "DBLink" ,
data {
{
label
str "Trace Assembly Archive" ,
data
ints { 123456 } } } }

As new types of linkages are established, they will be added to
the DBLink User-object, and displayed via the DBLINK linetype in
the GenBank flatfile format.

There is a possibility that the GenomeProjectsDB User-object
might someday be incorporated into the new DBLink User-object.
But at the moment, there are no firm plans to do so.

_______________________________________________
Genbankb mailing list
Genbankb@...
http://www.bio.net/biomail/listinfo/genbankb

GenBank 169.0 Close-of-Data

2008-12-11T16:00:19Z

Greetings GenBank Users,

Close-of-data for the upcoming GenBank Release 169.0 occurred on
Thursday December 11 at approximately 1:30am EST.

The subsequently generated GenBank Incremental Update files
nc1211.aso, nc1211.flat, etc. contain data through the close.

Note: Release processing often does not begin until sometime during
business hours on the close date. As a result, a number of sequence
records processed *after* 1:30am are likely to be present in the
GenBank 169.0 release files, even though they are "post-close" .

Similarly, the first GenBank Incremental Update that is generated
after the close date is likely to contain a number of sequence
records that are unchanged, compared to their appearance in the
release files.

Our apologies for the lack of advanced notice about the close date.

Mark Cavanaugh
GenBank
NCBI/NLM/NIH/HHS

_______________________________________________
Genbankb mailing list
Genbankb@...
http://www.bio.net/biomail/listinfo/genbankb

GenBank : Increase in number of release files planned

2008-12-11T15:57:09Z

Greetings GenBank Users,

Due to changes in the method by which the flatfile representations
of GenBank sequence records are stored at NCBI, there will be
about a 10% increase in the overall number of files in GenBank
Release 169.0, compared to Release 168.0 in October. This will be in
addition to the normal increase in files caused by the addition of
new records.

The sizes of the files in each division will also vary more than
previously (although most files will still be in the 250MB to 350MB
size range).

These changes are the costs of an improvement in overall flatfile
processing. Among the benefits: large-scale changes which affect
many records (flatfile format changes; changes to widely
used organism names) can now be handled much more quickly, and
the preparation of certain GB release files now requires much
less time.

These changes, and others that are underway, should ultimately
translate into a much faster release generation process in 2009.

Mark Cavanaugh
GenBank
NCBI/NLM/NIH/HHS

_______________________________________________
Genbankb mailing list
Genbankb@...
http://www.bio.net/biomail/listinfo/genbankb

GenBank Release 168.0 Now Available

2008-10-30T16:57:15Z

Greetings GenBank Users,

GenBank Release 168.0 is now available via FTP from the National
Center for Biotechnology Information (NCBI):

Ftp Site Directory Contents
---------------- --------- ---------------------------------------
ftp.ncbi.nih.gov genbank GenBank Release 168.0 flatfiles
ncbi-asn1 ASN.1 data used to create Release 168.0

Close-of-data for GenBank 168.0 occured on 10/27/2008. Uncompressed,
the
Release 168.0 flatfiles require roughly 371 GB (sequence files only)
or 396 GB (including the 'short directory', 'index' and the *.txt
files).
The ASN.1 data require approximately 338 GB.

Recent statistics for non-WGS, non-CON sequences:

Release Date Base Pairs Entries

167 Aug 2008 95033791652 92748599
168 Oct 2008 97381682336 96400790

Recent statistics for WGS sequences:

Release Date Base Pairs Entries

167 Aug 2008 118593509342 40214247
168 Oct 2008 136085973423 46108952

During the 69 days between the close dates for GenBank Releases 167.0
and
168.0, the non-WGS/non-CON portion of GenBank grew by 2,347,890,684
basepairs
and by 3,652,191 sequence records. During that same period, 1,111,311
records
were updated. An average of about 69,036 non-WGS/non-CON records were
added
and/or updated per day.

Between releases 167.0 and 168.0, the WGS component of GenBank grew by
17,492,464,081 basepairs and by 5,894,705 records.

The combined WGS/non-WGS single-release increase of 19.84 Gbp for
Release 168.0 is the largest that GenBank has experienced, to date.

For additional release information, see the README files in either of
the directories mentioned above, and the release notes (gbrel.txt) in
the genbank directory. Sections 1.3 and 1.4 of the release notes
(Changes in Release 168.0 and Upcoming Changes) have been appended
below for your convenience.

** Important Notes **

* A number of changes have been implemented for the October 2008
GenBank Release. Please see Section 1.3 for a complete list.

* GenBank 'index' files are now provided without any EST content, and
without most GSS content. See Section 1.3.12 of the release notes for
further details.

NCBI is considering ceasing support for the index files, so we
encourage
affected users to review that section and provide feedback.

* A new linetype ( DBLINK ) will be implemented as of the February 2009
release. See Section 1.4.1 for information.

Release 168.0 data, and subsequent updates, are available now via
NCBI's Entrez and Blast services.

As a general guideline, we suggest first transferring the GenBank
release
notes (gbrel.txt) whenever a release is being obtained. Check to make
sure
that the date and release number in the header of the release notes are
current (eg: October 15 2008, 168.0). If they are not, interrupt the
remaining transfers and then request assistance from the NCBI Service
Desk.

A comprehensive check of the headers of all release files after your
transfers are complete is also suggested. Here's how one might go about
this on a unix platform, using csh/tcsh :

set files = `ls gb*.*`
foreach i ($files)
head -10 $i | grep Release
end

Or, if the files are compressed, perhaps:

gzcat $i | head -10 | grep Release

If you encounter problems while ftp'ing or uncompressing Release
168.0, please send email outlining your difficulties to:

info@...

Mark Cavanaugh, Michael Kimelman, Ilya Dondoshansky
GenBank
NCBI/NLM/NIH/HHS

1.3 Important Changes in Release 168.0

1.3.1 Organizational changes

The total number of sequence data files increased by 60 with this
release:

- the BCT division is now comprised of 32 files (+2)
- the CON division is now comprised of 103 files (+6)
- the EST division is now comprised of 802 files (+40)
- the GSS division is now comprised of 309 files (+3)
- the HTG division is now comprised of 122 files (+2)
- the PAT division is now comprised of 47 files (+1)
- the PLN division is now comprised of 32 files (+2)
- the STS division is now comprised of 18 files (+4)

The total number of index files increased by 1 with this release:

- the JOU index is now comprised of 6 files (+1)

1.3.2 Changes related to ncRNA features, /ncRNA_class, and /moltype

The list of allowed values for the /ncRNA_class qualifier, which is
mandatory for all ncRNA features, has been expanded to include:

/ncRNA_class="ribozyme"

Non-coding RNAs which are not yet in the INSDC's controlled vocabulary:

http://www.insdc.org/page.php?page=rna_vocab

previously required /ncRNA_class="other" plus an accompanying /note
qualifer to describes the nature of the ncRNA. This requirement will
be changed, such that *either* a /product or a /note qualifier must
accompany "other" ncRNAs features.

The list of allowed /mol_type qualifiers for the source feature
currently includes:

/mol_type="snoRNA"
/mol_type="snRNA"
/mol_type="scRNA"
/mol_type="tmRNA"

All of these molecule types have been collapsed into a single value:

/mol_type="transcribed RNA"

Sequence records which represent one of these four types of molecules
will thus have:

an ncRNA feature with /ncRNA_class of "snoRNA", "scRNA" or
"snRNA"
a source feature with /mol_type of "transcribed RNA"
or
a tmRNA feature
a source feature with /mol_type of "transcribed RNA"

All of these changes take effect with this October 2008 release.

1.3.3 Merging the satellite and repeat_unit features into repeat_region

Satellites, minisatellites and microsatellites are comprised of
repetitive
units of DNA, with a variety of lengths and repeat patterns. With the
addition of a new qualifier (/satellite), the satellite and repeat_unit
features are now represented by the repeat_region feature.

Qualifier /satellite=

Definition identifier for satellite DNA marker; many tandem
repeats
(identical or related) of a short basic repeating
unit; many
have a base composition or other property different
from the
genome average that allows them to be separated from
the bulk
genomic DNA;

Value format "<satellite_type>[:<class>][ <identifier>]"

where satellite_type is one of the following
"satellite", "microsatellite", "minisatellite"

Example /satellite="satellite: S1a"
/satellite="satellite: alpha"
/satellite="satellite: gamma III"
/satellite="microsatellite: DC130"

As of this October 2008 GenBank release, all satellite and repeat_unit
features have been transformed into repeat_region features with an
appropriate /satellite qualifier.

1.3.4 New /gene_synonym qualifier

Gene symbols are presented via the /gene qualifier. When synonymous or

alternative gene symbols are available, they have often been presented
via
multiple /gene qualifiers.

To distinguish what might be an approved or official gene symbol from
its
synonyms or alternatives, a new /gene_synonym qualifier has been
introduced
for GenBank Release 168.0 .

Qualifier /gene_synonym=

Definition synonymous or alternative symbol for a gene
corresponding to
a sequence region
Value format "text"

Examples /gene="CF"
/gene="ABCC7"

1.3.5 New /mating_type qualifier

Because the /sex qualifier has a free-text value format, is has been
innapropriately utilized for certain organisms, such as bacteria, fungi,
and some insects and worms. In such cases, a more appropriate term would
be 'mating type'.

A new qualifier has been made available for non-sexual reproductive
strategies as of October 2008:

Qualifier /mating_type=
Definition mating type of the organism from which the sequence was
obtained; mating type is used for prokaryotes, and for
eukaryotes that undergo meiosis without sexually
dimorphic
gametes
Value format "text"
Examples /mating_type="MAT-1"
/mating_type="plus"
/mating_type="-"
/mating_type="odd"
/mating_type="even"
Comment /mating_type="male" and /mating_type="female" are
valid in the prokaryotes, but not in the eukaryotes;
for more information, see the entry for /sex.

In light of the above, the definition for the /sex qualifier has been
refined:

Qualifier /sex=
Definition sex of the organism from which the sequence was
obtained;
sex is used for eukaryotic organisms that undergo
meiosis
and have sexually dimorphic gametes
Value format "text"
Examples /sex="female"
/sex="male"
/sex="hermaphrodite"
/sex="unisexual"
/sex="bisexual"
/sex="asexual"
/sex="monoecious" [or monecious]
/sex="dioecious" [or diecious]
Comment /sex should be used (instead of /mating_type)
in the Metazoa, Embryophyta, Rhodophyta & Phaeophyceae;
/mating_type should be used (instead of /sex)
in the Bacteria, Archaea & Fungi;
neither /sex nor /mating_type should be used
in the viruses;
outside of the taxa listed above, /mating_type
should be used unless the value of the qualifier
is taken from the vocabulary given in the examples
above

Records which inappropriately used the /sex qualifier have been
updated,
to utilize the new /mating_type qualifier.

1.3.6 Renaming of /specific_host as /host

The /specific_host qualifier has been renamed as /host for Release
168.0 .
>From the Feature Table document:

Qualifier /host=
Definition natural (as opposed to laboratory) host to the organism
from
which sequenced molecule was obtained
Value format "text"
Example /host="Homo sapiens"
/host="Homo sapiens 12 year old girl"
/host="Rhizobium NGR234"

In contrast:

Qualifier /lab_host=
Definition scientific name of the laboratory host used to propagate
the
source organism from which the sequenced molecule was
obtained
Value format "text"
Example /lab_host="Gallus gallus"
/lab_host="Gallus gallus embryo"
/lab_host="Escherichia coli strain DH5 alpha"
/lab_host="Homo sapiens HeLa cells"
Comment the full binomial scientific name of the host organism
should
be used when known; extra conditional information
relating to
the host may also be included

1.3.7 New value for /organelle

As of October 2008, the list of allowed values for /organelle has been
expanded
to include:

/organelle="chromatophore"

1.3.8 Modification to value format for /frequency

As of October 2008, the definition of /frequency has been expanded to
accomodate both the fraction of a population carrying a variation
expressed
as a decimal value, and as the number of observed instances vs. the
total
number of sequenced isolates:

Qualifier /frequency=
Definition frequency of the occurrence of a feature
Value format text representing the proportion of a population
carrying the
feature expressed as a fraction
Example /frequency="23/108"
/frequency="1 in 12"
/frequency=".85"

1.3.9 /cons_splice qualifier removed

The /cons_splice qualifier has almost no usage within the sequence
database. In addition, it does not account for the variation in splice
signals that might be used by different classes of introns. So this
qualfier has been removed from sequence records, and the Feature Table
document, as of Release 168.0 .

1.3.10 /virion qualifier removed

The intent of /virion was to indicate that a sequenced molecule
originates from an encapsidated viral particle (as opposed to the
proviral form of a virus, integrated into the host's genome). Viral
sequences derived from a blood sample taken from an infected organism
might be flagged with /virion, if it is believed that the sample
contained viral particles.

However, a review of the database revealed that /virion was not
used consistently, and furthermore, submitters are often unable to
conclusively state that a virus sequence derives from the encapsidated
form. So the /virion qualifier has been removed from sequence records,
and the Feature Table document, as of Release 168.0 .

1.3.11 Updated value format for /exception

Only three values for the /exception qualifier have been approved
for use by the INSDC :

"rearrangement required for product"
"RNA editing"
"reasons given in citation"

However, the definition of /exception in the Feature Table document
does not indicate that the contents of /exception are controlled.
This oversight has been corrected, and the definition of the qualifier
is now:

Qualifier /exception=
Definition indicates that the coding region cannot be translated
using
standard biological rules
Value format "RNA editing", "reasons given in citation",
"rearrangement required for product"
Example /exception="RNA editing"
/exception="reasons given in citation"
/exception="rearrangement required for product"
Comment only to be used to describe biological mechanisms such
as RNA editing; where the exception cannot easily be
described
a published citation must be referred to; protein
translation of
/exception CDS will be different from the according
conceptual
translation;
- must not be used where transl_except would be
adequate,
e.g. in case of stop codon completion use:
/transl_except=(pos:6883,aa:TERM)
/note="TAA stop codon is completed by addition of 3' A
residues to
mRNA".
- must not be used for ribosomal slippage, instead use
join operator,
e.g.: CDS join(486..1784,1787..4810)
/note="ribosomal slip on tttt sequence at
1784..1787"

1.3.12 Changes in the content of index files

As described in the GB 153 release notes, the 'index' files which
accompany
GenBank releases (see Section 3.3) are considered to be a legacy data
product by
NCBI, generated mostly for historical reasons. FTP statistics of January
2005
seem to support this: the index files were transferred only half as
frequently as
the files of sequence records. The inherent inefficiencies of the index
file
format also lead us to suspect that they have little serious use by the
user
community, particularly for EST and GSS records.

The software that generated the index file products received little
attention over the years, and finally reached its limitations in
February 2006 (Release 152.0). The required multi-server queries which
obtained and sorted many millions of rows of terms from several
different
databases simply outgrew the capacity of the hardware used for GenBank
Release generation.

Our short-term solution is to cease generating some index-file content
for all EST sequence records, and for GSS sequence records that
originate
via direct submission to NCBI.

The three gbacc*.idx index files continue to reflect the entirety of
the
release, including all EST and GSS records, however the file contents
are
unsorted.

These 'solutions' are really just stop-gaps, and we will likely pursue
one of two options:

a) Cease support of the 'index' file products altogether.

b) Provide new products that present some of the most useful data from
the legacy 'index' files, and cease support for other types of index
data.

If you are a user of the 'index' files associated with GenBank
releases, we
encourage you to make your wishes known, either via the GenBank
newsgroup,
or via email to NCBI's Service Desk:

info@...

Our apologies for any inconvenience that these changes may cause.

1.3.13 GSS File Header Problem

GSS sequences at GenBank are maintained in two different systems,
depending
on their origin, and the dumps from those systems occur in parallel.
Because
the second dump (for example) has no prior knowledge of exactly how many
GSS
files will be dumped by the first, it does not know how to number its
own
output files.

There is thus a discrepancy between the filenames and file headers for
sixty of the GSS flatfiles in Release 168.0. Consider gbgss250.seq :

GBGSS1.SEQ Genetic Sequence Data Bank
October 15 2008

NCBI-GenBank Flat File Release 168.0

GSS Sequences (Part 1)

87217 loci, 64373883 bases, from 87217 reported sequences

Here, the filename and part number in the header is "1", though the
file
has been renamed as "250" based on the number of files dumped from the
other
system. We will work to resolve this discrepancy in future releases,
but the
priority is certainly much lower than many other tasks.

1.4 Upcoming Changes

1.4.1 PROJECT linetype to be replaced by DBLINK (April 2009)

The PROJECT linetype allows a sequence record to be linked to
information
about the sequencing project that generated the data which ultimately
resulted in the record's submission to the International Nucleotide
Sequence
Database ( INSD; see http://www.insdc.org ) .

This complete bacterial GenBank record illustrates the use of the
PROJECT
line:

LOCUS CP000964 5641239 bp DNA circular BCT
24-SEP-2008
DEFINITION Klebsiella pneumoniae 342, complete genome.
ACCESSION CP000964
VERSION CP000964.1 GI:206564770
PROJECT GenomeProject:28471

When viewed on the web in NCBI's Entrez:Nucleotide, the record's
project
identifier (28471) links to an entry in the Genome Project Database
(GPDB) :

http://www.ncbi.nlm.nih.gov/sites/entrez?db=genomeprj&cmd=Retrieve&dopt=
Overview&uid=28471

where information about the sequencing center, the bacterium, and other
GenBank records (eg, plasmids) associated with the sequencing project
can be found.

Since the introduction of PROJECT, the scope of the "Genome" Project
Database has expanded, to include projects that are not necessarily
targetted
to the sequencing of a complete genome.

In addition, there can be other resources which underlie an INSD
sequence
record, such as the Trace Assembly Archive at the NCBI:

http://www.ncbi.nlm.nih.gov/Traces/assembly/assmbrowser.cgi?cmd=show&f=t
ree&m=main&s=tree

Because of the expanded scope of the GPDB, and because we anticipate a
need
to link to more resources than just the GPDB, the PROJECT linetype is
going to
be replaced by a new linetype:

DBLINK

Modifications to linetypes can be disruptive, so the switch to DBLINK
will occur in several stages. Starting in October 2008, links to the
NCBI Trace Assembly Archive will be supported via a line of text in the
COMMENT section of sequence records. Here is a mock-up, based on
CP000964,
to illustrate this change:

LOCUS CP000964 5641239 bp DNA circular BCT
24-SEP-2008
DEFINITION Klebsiella pneumoniae 342, complete genome.
ACCESSION CP000964
VERSION CP000964.1 GI:206564770
PROJECT GenomeProject:28471
....
COMMENT Trace Assembly Archive:123456
The source for the DNA and/or cells is: Professor Eric W.
Triplett, Chair, Department of Microbiology and Cell
Science,
Institute of Food and Agricultural Sciences, University of
Florida,
P.O. Box 110700, Gainesville, FL 32611-0700, ewt@....

Note: Use of the Trace Assembly Archive is still in its early stages, so
only
a few records are expected to have these links in the short term.

The new DBLINK linetype will be introduced as of GenBank Release 170.0
,
on or near February 15, 2009 .

The Genome Project ID and the Trace Assembly Archive ID will be
presented
via DBLINK, and the existing PROJECT line will continue to be displayed:

LOCUS CP000964 5641239 bp DNA circular BCT
24-SEP-2008
DEFINITION Klebsiella pneumoniae 342, complete genome.
ACCESSION CP000964
VERSION CP000964.1 GI:206564770
PROJECT GenomeProject:28471
DBLINK Project:28471
Trace Assembly Archive:123456
....
COMMENT The source for the DNA and/or cells is: Professor Eric W.
Triplett, Chair, Department of Microbiology and Cell
Science,
Institute of Food and Agricultural Sciences, University of
Florida,
P.O. Box 110700, Gainesville, FL 32611-0700, ewt@....

PROJECT and DBLINK will co-exist for one GenBank release, until Release
171.0
(April 15, 2009), at which point the PROJECT line will be removed. In
its final
state, our mock-up for CP000964 becomes:

LOCUS CP000964 5641239 bp DNA circular BCT
24-SEP-2008
DEFINITION Klebsiella pneumoniae 342, complete genome.
ACCESSION CP000964
VERSION CP000964.1 GI:206564770
DBLINK Project:28471
Trace Assembly Archive:123456
....
COMMENT The source for the DNA and/or cells is: Professor Eric W.
Triplett, Chair, Department of Microbiology and Cell
Science,
Institute of Food and Agricultural Sciences, University of
Florida,
P.O. Box 110700, Gainesville, FL 32611-0700, ewt@....

In summary: The PROJECT linetype will be replaced by DBLINK as of
Release 171.0 in April 2009.

For those who process sequence data in NCBI's ASN.1 format: The
underlying representation for (Genome) Project IDs will remain
unchanged.
There will be no changes to the ASN.1 User-object that is used to store
them:

user {
type
str "GenomeProjectsDB" ,
data {
{
label
str "ProjectID" ,
data
int 28471 } ,
{
label
str "ParentID" ,
data
int 0 } } } ,

However, to support linkages to other resources, such as the Trace
Assembly Archive, a new "DBLink" User-object will be introduced:

user {
type
str "DBLink" ,
data {
{
label
str "Trace Assembly Archive" ,
data
ints { 123456 } } } }

As new types of linkages are established, they will be added to
the DBLink User-object, and displayed via the DBLINK linetype in
the GenBank flatfile format.

There is a possibility that the GenomeProjectsDB User-object
might someday be incorporated into the new DBLink User-object.
But at the moment, there are no firm plans to do so.

_______________________________________________
Genbankb mailing list
Genbankb@...
http://www.bio.net/biomail/listinfo/genbankb

GenBank 168.0 Close-of-Data

2008-10-30T14:54:12Z

Greetings GenBank Users,

Close-of-data for the upcoming GenBank Release 168.0 occurred on
Monday October 27 at approximately 1:30am EDT.

The subsequently generated GenBank Incremental Update files
nc1027.aso, nc1027.flat, etc. contain data through the close.

Note: Release processing often does not begin until sometime during
business hours on the close date. As a result, a number of sequence
records processed *after* 1:30am are likely to be present in the
GenBank 168.0 release files, even though they are "post-close" .

Similarly, the first GenBank Incremental Update that is generated
after the close date is likely to contain a number of sequence
records that are unchanged, compared to their appearance in the
release files.

Our apologies for the lack of advanced notice about the close date.

Mark Cavanaugh
GenBank
NCBI/NLM/NIH/HHS

_______________________________________________
Genbankb mailing list
Genbankb@...
http://www.bio.net/biomail/listinfo/genbankb

GenBank 168.0 Status Report

2008-10-30T14:49:45Z

Dear GenBank Users,

October's GenBank Release 168.0 has been significantly delayed
by a multitude of factors, most of which were related to the
Feature Table and other changes scheduled for implementation
this month. An unusually large number of records were affected,
which caused processing difficulties for one of our systems.

The difficulties were finally resolved as of Monday October 27,
and we anticipate that the release files will be made available
later this evening (Thursday, October 30).

Our apologies for both the two-week delay, and for the lack of
prior status reports.

Mark Cavanaugh
GenBank
NCBI/NLM/NIH/HHS

_______________________________________________
Genbankb mailing list
Genbankb@...
http://www.bio.net/biomail/listinfo/genbankb

Re-Send : PROJECT linetype to be replaced by DBLINK

2008-09-26T14:03:42Z

[This listserv seems to impose a fairly short line-wrap for
text messages, which made my previous post difficult to
read. Hence this re-send, with shorter line lengths, where
possible.]

Greetings GenBank Users,

The PROJECT linetype allows a sequence record to be linked to
information about the sequencing project that generated the data
which ultimately resulted in the record's submission to the
International Nucleotide Sequence Database ( INSD; see
http://www.insdc.org ).

This complete bacterial GenBank record illustrates the use of
the PROJECT line:

LOCUS CP000964 5641239 bp DNA circular BCT
24-SEP-2008
DEFINITION Klebsiella pneumoniae 342, complete genome.
ACCESSION CP000964
VERSION CP000964.1 GI:206564770
PROJECT GenomeProject:28471

When viewed on the web in NCBI's Entrez:Nucleotide, the record's
project identifier (28471) links to an entry in the Genome Project
Database (GPDB) :

http://www.ncbi.nlm.nih.gov/sites/entrez?db=genomeprj&cmd=Retrieve&dopt=
Overview&uid=28471

where information about the sequencing center, the bacterium, and
other GenBank records (eg, plasmids) associated with the sequencing
project can be found.

Since the introduction of PROJECT, the scope of the "Genome" Project
Database has expanded, to include projects that are not necessarily
targetted to the sequencing of a complete genome.

In addition, there can be other resources which underlie an INSD
sequence record, such as the Trace Assembly Archive at the NCBI:

http://www.ncbi.nlm.nih.gov/Traces/assembly/assmbrowser.cgi?cmd=show&f=t
ree&m=main&s=tree

Because of the expanded scope of the GPDB, and because we
anticipate a need to link to more resources than just the GPDB,
the PROJECT linetype is going to be replaced by a new linetype:

DBLINK

Further details about this change, and its timetable, follow.

Mark Cavanaugh
GenBank
NCBI/NLM/NIH/HHS

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

Modifications to linetypes can be disruptive, so the switch to
DBLINK will occur in several stages.

Starting in October 2008, links to the NCBI Trace Assembly Archive
will be supported via a line of text in the COMMENT section of
sequence records.

Here is a mock-up, based on CP000964, to illustrate this change:

LOCUS CP000964 5641239 bp DNA circular BCT
24-SEP-2008
DEFINITION Klebsiella pneumoniae 342, complete genome.
ACCESSION CP000964
VERSION CP000964.1 GI:206564770
PROJECT GenomeProject:28471
....
COMMENT Trace Assembly Archive:123456
The source for the DNA and/or cells is: Professor Eric W.
Triplett, Chair, Department of Microbiology and Cell
Science,
Institute of Food and Agricultural Sciences, University of
Florida,
P.O. Box 110700, Gainesville, FL 32611-0700, ewt@....

Note: Use of the Trace Assembly Archive is still in its early
stages, so only a few records are expected to have these links in
the short term.

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

The new DBLINK linetype will be introduced as of GenBank Release
170.0 (February 15, 2009) .

The Genome Project ID and the Trace Assembly Archive ID will be
presented via DBLINK, and the existing PROJECT line will continue
to be displayed:

LOCUS CP000964 5641239 bp DNA circular BCT
24-SEP-2008
DEFINITION Klebsiella pneumoniae 342, complete genome.
ACCESSION CP000964
VERSION CP000964.1 GI:206564770
PROJECT GenomeProject:28471
DBLINK Project:28471
Trace Assembly Archive:123456
....
COMMENT The source for the DNA and/or cells is: Professor Eric W.
Triplett, Chair, Department of Microbiology and Cell
Science,
Institute of Food and Agricultural Sciences, University of
Florida,
P.O. Box 110700, Gainesville, FL 32611-0700, ewt@....

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

PROJECT and DBLINK will co-exist for one GenBank release, until
Release 171.0 (April 15, 2009), at which point the PROJECT line
will be removed.

In its final state, our mock-up for CP000964 becomes:

LOCUS CP000964 5641239 bp DNA circular BCT
24-SEP-2008
DEFINITION Klebsiella pneumoniae 342, complete genome.
ACCESSION CP000964
VERSION CP000964.1 GI:206564770
DBLINK Project:28471
Trace Assembly Archive:123456
....
COMMENT The source for the DNA and/or cells is: Professor Eric W.
Triplett, Chair, Department of Microbiology and Cell
Science,
Institute of Food and Agricultural Sciences, University of
Florida,
P.O. Box 110700, Gainesville, FL 32611-0700, ewt@....

In summary:

PROJECT -> DBLINK

'GenomeProject' -> 'Project'

Additional linkages, such as Trace Assembly, added to DBLINK
as-needed

The PROJECT line will be removed as of April 15 2009.

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

For those who process sequence data in NCBI's ASN.1 format:

The underlying representation for (Genome) Project IDs will remain
unchanged; there will be no changes to the ASN.1 User-object that
is used to store them:

user {
type
str "GenomeProjectsDB" ,
data {
{
label
str "ProjectID" ,
data
int 28471 } ,
{
label
str "ParentID" ,
data
int 0 } } } ,

However, to support linkages to other resources, like the Trace
Assembly Archive, a new "DBLink" User-object will be introduced:

user {
type
str "DBLink" ,
data {
{
label
str "Trace Assembly Archive" ,
data
ints { 123456 } } } }

As new types of linkages are established, they will be added to
the DBLink User-object, and displayed via the DBLINK linetype in
the GenBank flatfile format.

There is a possibility that the GenomeProjectsDB User-object
might someday be incorporated into the new DBLink User-object.
But at the moment, there are no firm plans to do so.

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

_______________________________________________
Genbankb mailing list
Genbankb@...
http://www.bio.net/biomail/listinfo/genbankb

PROJECT linetype to be replaced by DBLINK

2008-09-26T13:41:11Z

Greetings GenBank Users,

The PROJECT linetype allows a sequence record to be linked to
information
about the sequencing project that generated the data which ultimately
resulted in the record's submission to the International Nucleotide
Sequence
Database ( INSD : http://www.insdc.org ).

This complete bacterial GenBank record illustrates the use of PROJECT:

LOCUS CP000964 5641239 bp DNA circular BCT
24-SEP-2008
DEFINITION Klebsiella pneumoniae 342, complete genome.
ACCESSION CP000964
VERSION CP000964.1 GI:206564770
PROJECT GenomeProject:28471

When viewed on the web in NCBI's Entrez:Nucleotide, the record's project
identifier (28471) links to an entry in the Genome Project Database
(GPDB) :

http://www.ncbi.nlm.nih.gov/sites/entrez?db=genomeprj&cmd=Retrieve&dopt=
Overview&uid=28471

where information about the sequencing center, the bacterium, and other
GenBank records (eg, plasmids) associated with the sequencing project
can
be obtained.

Since the introduction of PROJECT, the scope of the "Genome" Project
Database
has expanded, to include projects that are not necessarily targeted to
the sequencing of a complete genome.

In addition, there can be other resources which underlie an INSD
sequence
record, such as the Trace Assembly Archive at the NCBI:

http://www.ncbi.nlm.nih.gov/Traces/assembly/assmbrowser.cgi?cmd=show&f=t
ree&m=main&s=tree

Because of the expanded scope of the GPDB, and because we anticipate a
need
to link to more resources than just the GPDB, the PROJECT linetype is
going
to be replaced by a new linetype:

DBLINK

Further details about this change, and its timetable, follow.

Mark Cavanaugh
GenBank
NCBI/NLM/NIH/HHS

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
=-=

Modifications to linetypes can be disruptive, so the switch to DBLINK
will
occur in several stages.

Starting in October 2008, links to the NCBI Trace Assembly Archive will
be
supported via a line of text in the COMMENT section of sequence records.

Here is a mock-up, based on CP000964, which illustrates this change:

LOCUS CP000964 5641239 bp DNA circular BCT
24-SEP-2008
DEFINITION Klebsiella pneumoniae 342, complete genome.
ACCESSION CP000964
VERSION CP000964.1 GI:206564770
PROJECT GenomeProject:28471
....
COMMENT Trace Assembly Archive:123456
The source for the DNA and/or cells is: Professor Eric W.
Triplett, Chair, Department of Microbiology and Cell
Science,
Institute of Food and Agricultural Sciences, University of
Florida,
P.O. Box 110700, Gainesville, FL 32611-0700, ewt@....

Note: Use of the Trace Assembly Archive is still in its early stages, so
only a few records are expected to have these links in the short term.

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
=-=

The new DBLINK linetype will be introduced as of GenBank Release 170.0
(February 15, 2009) .

The Genome Project ID and the Trace Assembly Archive ID will be
presented
via DBLINK, and the existing PROJECT line will continue to be displayed:

LOCUS CP000964 5641239 bp DNA circular BCT
24-SEP-2008
DEFINITION Klebsiella pneumoniae 342, complete genome.
ACCESSION CP000964
VERSION CP000964.1 GI:206564770
PROJECT GenomeProject:28471
DBLINK Project:28471
Trace Assembly Archive:123456
....
COMMENT The source for the DNA and/or cells is: Professor Eric W.
Triplett, Chair, Department of Microbiology and Cell
Science,
Institute of Food and Agricultural Sciences, University of
Florida,
P.O. Box 110700, Gainesville, FL 32611-0700, ewt@....

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
=-=

PROJECT and DBLINK will co-exist for one GenBank release, until Release
171.0
(April 15, 2009), at which point the PROJECT line will be removed.

In its final state, our mock-up for CP000964 becomes:

LOCUS CP000964 5641239 bp DNA circular BCT
24-SEP-2008
DEFINITION Klebsiella pneumoniae 342, complete genome.
ACCESSION CP000964
VERSION CP000964.1 GI:206564770
DBLINK Project:28471
Trace Assembly Archive:123456
....
COMMENT The source for the DNA and/or cells is: Professor Eric W.
Triplett, Chair, Department of Microbiology and Cell
Science,
Institute of Food and Agricultural Sciences, University of
Florida,
P.O. Box 110700, Gainesville, FL 32611-0700, ewt@....

In summary:

PROJECT -> DBLINK

'GenomeProject' -> 'Project'

Additional linkages, such as Trace Assembly, will be added to
DBLINK as-needed

The PROJECT line will be removed as of April 15 2009.

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
=-=

For those who process sequence data in NCBI's ASN.1 format:

The underlying representation for (Genome) Project IDs will remain
unchanged; there will be no changes to the ASN.1 User-object that
is used to store them:

user {
type
str "GenomeProjectsDB" ,
data {
{
label
str "ProjectID" ,
data
int 28471 } ,
{
label
str "ParentID" ,
data
int 0 } } } ,

However, to support linkages to other resources, like the Trace
Assembly Archive, a new "DBLink" User-object will be introduced:

user {
type
str "DBLink" ,
data {
{
label
str "Trace Assembly Archive" ,
data
ints { 123456 } } } }

As new types of linkages are established, they will be added to
the DBLink User-object, and displayed via the DBLINK linetype in
the GenBank flatfile format.

There is a possibility that the GenomeProjectsDB User-object
might someday be incorporated into the new DBLink User-object.
But at the moment, there are no firm plans to do so.

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
=-=

_______________________________________________
Genbankb mailing list
Genbankb@...
http://www.bio.net/biomail/listinfo/genbankb

GenBank Update Problem : 0912 : Incorrect files between 04:30am and 10:08am

2008-09-12T08:35:21Z

Dear GenBank Users,

Processing for the GenBank Incremental Update (GIU) and for GenBank WGS
data products was moved to new hardware on Thursday, September 11.

Unfortunately, some configuration files that were used during previous
tests of the new hardware were *not* updated with the files from the
production system.

This led to the creation of unnecessarily large GIU files on September
12
(nc0912), containing records that date back to (at least) August 10th.

The affected 0912 GIU files had these timestamps and sizes:

-rw-r--r-- 1 gbupdate gbproces 30892117 Sep 12 04:44
con_nc.0912.flat.gz
-rw-r--r-- 1 gbupdate gbproces 533951491 Sep 12 04:31 nc0912.flat.gz
-rw-r--r-- 1 gbupdate gbproces 317692100 Sep 12 04:37 nc0912.fsa_nt.gz
-rw-r--r-- 1 gbupdate gbproces 34925179 Sep 12 04:10 nc0912.fsa.gz
-rw-r--r-- 1 gbupdate gbproces 60205075 Sep 12 04:10 nc0912.gnp.gz
-rw-r--r-- 1 gbupdate gbproces 92824768 Sep 12 04:14 nc0912.qscore.gz

-rw-r--r-- 1 gbupdate gbproces 33507308 Sep 12 04:44
con_nc.0912.aso.gz
-rw-r--r-- 1 gbupdate gbproces 423510078 Sep 12 04:16 nc0912.aso.gz

Note that the uncompressed size of nc0912.flat.gz is over 2.5 GB :

compressed uncompressed ratio uncompressed_name
533951491 2652522424 79.9% nc0912.flat

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

This problem was discovered on the morning of September 12. The
incorrect
GIU files were removed, a new GIU run was started, and this yielded
corrected 0912 update products at about 10:00am :

-rw-r--r-- 1 gbupdate gbproces 55850950 Sep 12 10:08 nc0912.flat.gz
-rw-r--r-- 1 gbupdate gbproces 36800819 Sep 12 10:08 nc0912.fsa_nt.gz
-rw-r--r-- 1 gbupdate gbproces 1919440 Sep 12 10:08 nc0912.fsa.gz
-rw-r--r-- 1 gbupdate gbproces 3169378 Sep 12 10:08 nc0912.gnp.gz
-rw-r--r-- 1 gbupdate gbproces 2254192 Sep 12 10:08 nc0912.qscore.gz

-rw-r--r-- 1 gbupdate gbproces 38159306 Sep 12 10:08 nc0912.aso.gz

Note that the uncompressed size of the corrected nc0912.flat.gz GIU
is only a tenth of the size of the incorrect version:

compressed uncompressed ratio uncompressed_name
55850950 252578593 77.9% nc0912.flat

Note also that there are no CON-division GIU products for 0912 .

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

The invalid 0912 GIU products were available via FTP for approximately
six hours. If you transferred them between 4:00am ET and 10:08am ET,
please check their sizes to see if you need to obtain new, corrected,
smaller versions of the files.

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

Fortunately, the effect on our WGS project files was very minimal :
the data files for a single project, CABB, were unnecessarily refreshed.

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

Our apologies for the inconvenience that this error has caused.

Mark Cavanaugh
GenBank
NCBI/NLM/NIH/HHS

_______________________________________________
Genbankb mailing list
Genbankb@...
http://www.bio.net/biomail/listinfo/genbankb

GenBank Release 167.0 Now Available

2008-08-22T16:12:51Z

[Apologies, the initial announcement was interpreted as an attachment.
Re-sending because the listserv might not handle it gracefully.]

Greetings GenBank Users,

GenBank Release 167.0 is now available via FTP from the National
Center for Biotechnology Information (NCBI):

Ftp Site Directory Contents
---------------- --------- ---------------------------------------
ftp.ncbi.nih.gov genbank GenBank Release 167.0 flatfiles
ncbi-asn1 ASN.1 data used to create Release 167.0

Close-of-data for GenBank 167.0 occured on 08/19/2008. Uncompressed, the
Release 167.0 flatfiles require roughly 357 GB (sequence files only)
or 381 GB (including the 'short directory', 'index' and the *.txt files).
The ASN.1 data require approximately 326 GB.

Recent statistics for non-WGS, non-CON sequences:

Release Date Base Pairs Entries

166 Jun 2008 92008611867 88554578
167 Aug 2008 95033791652 92748599

Recent statistics for WGS sequences:

Release Date Base Pairs Entries

166 Jun 2008 113639291344 39163548
167 Aug 2008 118593509342 40214247

During the 69 days between the close dates for GenBank Releases 166.0 and
167.0, the non-WGS/non-CON portion of GenBank grew by 3,025,179,785 basepairs
and by 4,194,021 sequence records. During that same period, 939,305 records
were updated. An average of about 74,396 non-WGS/non-CON records were added
and/or updated per day.

Between releases 166.0 and 167.0, the WGS component of GenBank grew by
4,954,217,998 basepairs and by 1,050,699.

For additional release information, see the README files in either of
the directories mentioned above, and the release notes (gbrel.txt) in
the genbank directory. Sections 1.3 and 1.4 of the release notes
(Changes in Release 167.0 and Upcoming Changes) have been appended
below for your convenience.

** Important Notes **

* Support for the single, comprehensive protein FASTA file which accompanies
GenBank releases ceased as of the June 2008 release. See Section 1.3.3 of
the release notes for details.

* A number of changes are expected for October's GenBank Release 168.0 .
Please see Section 1.4 for a complete list.

* GenBank 'index' files are now provided without any EST content, and without
most GSS content. See Section 1.3.4 of the release notes for further details.
NCBI is considering ceasing support for the index files, so we encourage
affected users to review that section and provide feedback.

Release 167.0 data, and subsequent updates, are available now via
NCBI's Entrez and Blast services.

As a general guideline, we suggest first transferring the GenBank release
notes (gbrel.txt) whenever a release is being obtained. Check to make sure
that the date and release number in the header of the release notes are
current (eg: August 15 2008, 167.0). If they are not, interrupt the
remaining transfers and then request assistance from the NCBI Service Desk.

A comprehensive check of the headers of all release files after your
transfers are complete is also suggested. Here's how one might go about
this on a unix platform, using csh/tcsh :

set files = `ls gb*.*`
foreach i ($files)
head -10 $i | grep Release
end

Or, if the files are compressed, perhaps:

gzcat $i | head -10 | grep Release

If you encounter problems while ftp'ing or uncompressing Release
167.0, please send email outlining your difficulties to:

info@...

Mark Cavanaugh, Vladimir Alekseyev, Michael Kimelman
GenBank
NCBI/NLM/NIH/HHS

1.3 Important Changes in Release 167.0

1.3.1 Announcements for upcoming changes absent from Release 166.0

An oversight during GenBank Release 166.0 processing resulted in the
exclusion of several announcements for changes that will be implemented
as of GenBank Release 168.0 in October 2008 (see section 1.4, below).

This means that only two months advance notice can be provided for those
changes, rather than the customary four months. Our apologies for this
oversight and any inconvenience that it may cause.

1.3.2 Organizational changes

The total number of sequence data files increased by 58 with this release:

- the BCT division is now comprised of 30 files (+1)
- the CON division is now comprised of 97 files (+6)
- the ENV division is now comprised of 10 files (+1)
- the EST division is now comprised of 762 files (+24)
- the GSS division is now comprised of 306 files (+16)
- the HTG division is now comprised of 120 files (+3)
- the INV division is now comprised of 13 files (+1)
- the PAT division is now comprised of 46 files (+4)
- the PLN division is now comprised of 30 files (+1)
- the VRL division is now comprised of 10 files (+1)
- the VRT division is now comprised of 16 files (+1)

The total number of index files increased by 5 with this release:

- the AUT index is now comprised of 59 files (+5)

1.3.3 Comprehensive protein FASTA file has been discontinued

'Divisional' protein FASTA files :

ftp://ftp.ncbi.nih.gov/ncbi-asn1/gbXXX.fsa_aa.gz

where 'XXX' represents an alphanumeric GenBank division code (such as
pri10) were made available starting with GenBank Release 164.0 . Given
their availability, support for the single, comprehensive protein FASTA
file:

ftp://ftp.ncbi.nih.gov/genbank/relNNN.fsa_aa.gz

(where 'NNN' represents a three-digit GenBank release number) was
discontinued as of GenBank Release 166.0 in June 2008. The size of the
comprehensive file had exceeded 4GB, which was unmanageable for many users.

The final comprehensive protein FASTA file was: rel166.fsa_aa .

1.3.4 Changes in the content of index files

As described in the GB 153 release notes, the 'index' files which accompany
GenBank releases (see Section 3.3) are considered to be a legacy data product by
NCBI, generated mostly for historical reasons. FTP statistics of January 2005
seem to support this: the index files were transferred only half as frequently as
the files of sequence records. The inherent inefficiencies of the index file
format also lead us to suspect that they have little serious use by the user
community, particularly for EST and GSS records.

The software that generated the index file products received little
attention over the years, and finally reached its limitations in
February 2006 (Release 152.0). The required multi-server queries which
obtained and sorted many millions of rows of terms from several different
databases simply outgrew the capacity of the hardware used for GenBank
Release generation.

Our short-term solution is to cease generating some index-file content
for all EST sequence records, and for GSS sequence records that originate
via direct submission to NCBI.

The three gbacc*.idx index files continue to reflect the entirety of the
release, including all EST and GSS records, however the file contents are
unsorted.

These 'solutions' are really just stop-gaps, and we will likely pursue
one of two options:

a) Cease support of the 'index' file products altogether.

b) Provide new products that present some of the most useful data from
the legacy 'index' files, and cease support for other types of index data.

If you are a user of the 'index' files associated with GenBank releases, we
encourage you to make your wishes known, either via the GenBank newsgroup,
or via email to NCBI's Service Desk:

info@...

Our apologies for any inconvenience that these changes may cause.

1.3.5 GSS File Header Problem

GSS sequences at GenBank are maintained in two different systems, depending
on their origin, and the dumps from those systems occur in parallel. Because
the second dump (for example) has no prior knowledge of exactly how many GSS
files will be dumped by the first, it does not know how to number its own
output files.

There is thus a discrepancy between the filenames and file headers for
fifty-nine of the GSS flatfiles in Release 167.0. Consider gbgss248.seq :

GBGSS1.SEQ Genetic Sequence Data Bank
August 15 2008

NCBI-GenBank Flat File Release 167.0

GSS Sequences (Part 1)

87184 loci, 64446495 bases, from 87184 reported sequences

Here, the filename and part number in the header is "1", though the file
has been renamed as "248" based on the number of files dumped from the other
system. We will work to resolve this discrepancy in future releases, but the
priority is certainly much lower than many other tasks.

1.4 Upcoming Changes

1.4.1 Changes related to ncRNA features, /ncRNA_class, and /moltype

The list of allowed values for the /ncRNA_class qualifier, which is
mandatory for all ncRNA features, will be expanded to include:

/ncRNA_class="ribozyme"

Non-coding RNAs which are not yet in the INSDC's controlled vocabulary:

http://www.insdc.org/page.php?page=rna_vocab

previously required /ncRNA_class="other" plus an accompanying /note
qualifer to describes the nature of the ncRNA. This requirement will
be changed, such that *either* a /product or a /note qualifier must
accompany "other" ncRNAs features.

The list of allowed /mol_type qualifiers for the source feature
currently includes:

/mol_type="snoRNA"
/mol_type="snRNA"
/mol_type="scRNA"
/mol_type="tmRNA"

All of these molecule types will be collapsed into a single value:

/mol_type="transcribed RNA"

Sequence records which represent one of these four types of molecules
will thus have:

an ncRNA feature with /ncRNA_class of "snoRNA", "scRNA" or "snRNA"
a source feature with /mol_type of "transcribed RNA"
or
a tmRNA feature
a source feature with /mol_type of "transcribed RNA"

All of these changes will take effect as of Release 169.0 in October 2008.

1.4.2 Merging the satellite and repeat_unit features into repeat_region

Satellites, minisatellites and microsatellites are comprised of repetitive
units of DNA, with a variety of lengths and repeat patterns. With the
addition of a new qualifier (/satellite), the satellite and repeat_unit
features can be represented by the repeat_region feature.

Qualifier /satellite=

Definition identifier for satellite DNA marker; many tandem repeats
(identical or related) of a short basic repeating unit; many
have a base composition or other property different from the
genome average that allows them to be separated from the bulk
genomic DNA;

Value format "<satellite_type>[:<class>][ <identifier>]"

where satellite_type is one of the following
"satellite", "microsatellite", "minisatellite"

Example /satellite="satellite: S1a"
/satellite="satellite: alpha"
/satellite="satellite: gamma III"
/satellite="microsatellite: DC130"

As of the October 2008 GenBank release, all satellite features will be
transformed into repeat_region features with /satellite qualifiers of
type "satellite", and all repeat_unit features will be transformed into
repeat_region features.

1.4.3 New /gene_synonym qualifier

Gene symbols are presented via the /gene qualifier. When synonymous or
alternative gene symbols are available, they have often been presented via
multiple /gene qualifiers.

To distinguish what might be an approved or official gene symbol from its
synonyms or alternatives, a new /gene_synonym qualifier will be introduced
for GenBank Release 168.0 .

Qualifier /gene_synonym=

Definition synonymous or alternative symbol for a gene corresponding to
a sequence region
Value format "text"

Examples /gene="CF"
/gene="ABCC7"

1.4.4 New /mating_type qualifier

Because the /sex qualifier has a free-text value format, is has been
innapropriately utilized for certain organisms, such as bacteria, fungi,
and some insects and worms. In such cases, a more appropriate term would
be 'mating type'.

In October 2008, a new qualifier will be made available for non-sexual
reproductive strategies:

Qualifier /mating_type=

Definition mating type of the organism from which the sequence was obtained
Value format "text"

Example /mating_type="Mating type A"

1.4.5 Renaming of /specific_host as /host

The definition of /specific_host in the Feature Table document is as follows:

Qualifier /specific_host=

Definition natural host from which the sequence was obtained
Value format "text"

Example /specific_host="Rhizobium NGR234"

The usage of /specific_host, and particularly the distinction between it and
/lab_host, is not made clear with this definition. So the qualifier is going
to be renamed and redefined, starting with Release 168.0 in October 2008 :

Qualifier /host=

Definition Natural (as opposed to laboratory) host to the organism from
which the sequenced molecule was obtained
Value format "text"

Example /host="Homo sapiens"
/host="Homo sapiens 12 year old girl"

1.4.6 New value for /organelle

In October 2008, the list of allowed values for /organelle will be expanded
to include:

/organelle="chromatophore"

1.4.7 Modification to value format for /frequency

The current definition of /frequency is :

Qualifier /frequency=

Definition frequency of the occurrence of a feature

Value format text representing the fraction of population carrying the
variation expressed as a decimal fraction

Example /frequency=".85"

Although this format is appropriate and useful in some contexts (for
example, within a global population of individuals), it does not convey
the number of sequences that might have been included in a variation
study. As of Release 168.0, the format of /frequency will be expanded:

Value format text representing the fraction of population carrying the
variation expressed as a decimal fraction, or the number
observed instances vs the total number of sequenced isolates

Example /frequency=".85"
/frequency="23/108"
/frequency="1 in 12"

1.4.8 /cons_splice qualifier to be removed

The /cons_splice qualifier has almost no usage within the sequence
database. In addition, it does not account for the variation in splice
signals that might be used by different classes of introns. So this
qualfier will be removed from sequence records, and the Feature Table
document, as of Release 168.0 in October 2008.

1.4.9 /virion qualifier to be removed

The intent of /virion is to indicate that a sequenced molecule
is from an encapsidated viral particle (as opposed to the proviral
form of a virus, integrated into the host's genome). Viral sequences
derived from a blood sample taken from an infected organism might
typically be flagged with /virion, if it is believed that the sample
contained viral particles.

However, a review of the database reveals that this qualifier is
not used consistently, and furthermore, submitters often are unable
to conclusively state that a virus sequence derives from the
encapsidated form. So the /virion qualifier will be removed from
sequence records, and the Feature Table document, as of Release 168.0
in October 2008.

1.4.10 Updated value format for /exception

Only three values for the /exception qualifier have been approved
for use by the INSDC :

"rearrangement required for product"
"RNA editing"
"reasons given in citation"

However, the definition of /exception in the Feature Table document
does not indicate that the contents of /exception are controlled.
This oversight will be corrected in October 2008.

_______________________________________________
Genbankb mailing list
Genbankb@...
http://www.bio.net/biomail/listinfo/genbankb

GenBank Release 167.0 Now Available

2008-08-22T16:00:06Z

_______________________________________________
Genbankb mailing list
Genbankb@...
http://www.bio.net/biomail/listinfo/genbankb

attachment0 (15K) Download Attachment

GenBank 167.0 Close-of-Data

2008-08-21T11:58:21Z

Greetings GenBank Users,

Close-of-data for the upcoming GenBank Release 167.0 occurred on
Tuesday August 19 at approximately 1:30am ET.

The subsequently generated GenBank Incremental Update files
nc0819.aso, nc0819,flat, etc. contain data through the close.

Note: Release processing often does not begin until sometime during
business hours on the close date. As a result, a number of sequence
records processed *after* 1:30am are likely to be present in the
GenBank 167.0 release files, even though they are "post-close" .

Similarly, the first GenBank Incremental Update that is generated
after the close date is likely to contain a number of sequence
records that are unchanged, compared to their appearance in the
release files.

Our apologies for the lack of advanced notice about the close date.

Mark Cavanaugh
GenBank
NCBI/NLM/NIH/HHS

_______________________________________________
Genbankb mailing list
Genbankb@...
http://www.bio.net/biomail/listinfo/genbankb

Re: GenBank Release 166.0 Now Available

2008-06-14T08:19:14Z

> From: Mark Cavanaugh <cavanaug@...>
>
> GenBank has surpassed the 200-billion basepair threshold, with a total
> of 205,647,903,211 bases as of this June 2008 release.

Wow 200 billion nucleotides! I remember when we celebrated 1 Billion nucleotides!

Good work all the folks at the NCBI - good way to celebrate 25 years!

Cheers,

f.

--
B.F. Francis Ouellette http://oicr.on.ca/research/ouellette/
We are hiring a "Manager, Web Development": http://tinyurl.com/4dgyuk
NextGen sequencing Bioinformatics Workshop: http://tinyurl.com/29ulqj

_______________________________________________
Genbankb mailing list
Genbankb@...
http://www.bio.net/biomail/listinfo/genbankb

GenBank Release 166.0 Now Available

2008-06-13T20:03:16Z

[Apologies for the double-post : The contents of my original post
about Release 166.0 availability will be interpreted as an
attachment under some circumstances...]

Greetings GenBank Users,

GenBank Release 166.0 is now available via FTP from the National
Center for Biotechnology Information (NCBI):

Ftp Site Directory Contents
---------------- --------- ---------------------------------------
ftp.ncbi.nih.gov genbank GenBank Release 166.0 flatfiles
ncbi-asn1 ASN.1 data used to create Release 166.0

Close-of-data for GenBank 166.0 occured on 06/11/2008. Uncompressed, the
Release 166.0 flatfiles require roughly 343 GB (sequence files only)
or 366 GB (including the 'short directory', 'index' and the *.txt files).
The ASN.1 data require approximately 314 GB.

Recent statistics for non-WGS, non-CON sequences:

Release Date Base Pairs Entries

165 Apr 2008 89172350468 85500730
166 Jun 2008 92008611867 88554578

Recent statistics for WGS sequences:

Release Date Base Pairs Entries

165 Apr 2008 110500961400 26931049
166 Jun 2008 113639291344 39163548

During the 57 days between the close dates for GenBank Releases 165.0 and
166.0, the non-WGS/non-CON portion of GenBank grew by 2,836,261,399 basepairs
and by 3,053,848 sequence records. During that same period, 376,881 records
were updated. An average of about 60,188 non-WGS/non-CON records were added
and/or updated per day.

Between releases 165.0 and 166.0, the WGS component of GenBank grew by
3,138,329,944 basepairs and by 12,232,499 records.

GenBank has surpassed the 200-billion basepair threshold, with a total
of 205,647,903,211 bases as of this June 2008 release.

For additional release information, see the README files in either of
the directories mentioned above, and the release notes (gbrel.txt) in
the genbank directory. Sections 1.3 and 1.4 of the release notes
(Changes in Release 166.0 and Upcoming Changes) have been appended
below.
** Important Notes **

* A new GenBank division appears with Release 166.0 : the Transcriptome Shotgun
Assembly, or TSA, division. Please see Section 1.3.2 of the release notes for
more information about TSA and the records that it contains.

* Support for the single, comprehensive protein FASTA file which accompanies
GenBank releases ceases with this June 2008 release. See Section 1.4.1 of
the release notes for details.

* GenBank 'index' files are now provided without any EST content, and without
most GSS content. See Section 1.3.3 of the release notes for further details.
NCBI is considering ceasing support for the index files, so we encourage
affected users to review that section and provide feedback.

Release 166.0 data, and subsequent updates, are available now via
NCBI's Entrez and Blast services.

As a general guideline, we suggest first transferring the GenBank release
notes (gbrel.txt) whenever a release is being obtained. Check to make sure
that the date and release number in the header of the release notes are
current (eg: June 15 2008, 166.0). If they are not, interrupt the
remaining transfers and then request assistance from the NCBI Service Desk.

A comprehensive check of the headers of all release files after your
transfers are complete is also suggested. Here's how one might go about
this on a unix platform, using csh/tcsh :

set files = `ls gb*.*`
foreach i ($files)
head -10 $i | grep Release
end

Or, if the files are compressed, perhaps:

gzcat $i | head -10 | grep Release

If you encounter problems while ftp'ing or uncompressing Release
166.0, please send email outlining your difficulties to:

info@...

Mark Cavanaugh, Vladimir Alekseyev, Michael Kimelman
GenBank
NCBI/NLM/NIH/HHS

1.3 Important Changes in Release 166.0

1.3.1 Organizational changes

The total number of sequence data files increased by 47 with this release:

- the TSA division is now comprised of 1 file (+1)
- the BCT division is now comprised of 29 files (+1)
- the CON division is now comprised of 91 files (+6)
- the EST division is now comprised of 738 files (+25)
- the ENV division is now comprised of 9 files (+1)
- the GSS division is now comprised of 290 files (+5)
- the HTG division is now comprised of 117 files (+3)
- the PAT division is now comprised of 42 files (+4)
- the PRI division is now comprised of 36 files (+1)

The total number of index files increased by 4 with this release:

- the AUT index is now comprised of 54 files (+4)

1.3.2 New file for the Transcriptome Shotgun Assembly (TSA) division

As announced with the previous release, Release 166.0 contains a new
divisional file (gbtsa.seq) for Transcriptome Shotgun Assembly (TSA)
mRNA sequences.

TSA sequences are shotgun assemblies of primary sequences deposited in
dbEST, the Trace Archive (TA) or the Short-Read Archive (SRA). Keywords
"TSA" and "Transcriptome Shotgun Assembly" are present on all TSA
records, in addition to a division code value of "TSA" on the LOCUS line.

No format changes (new or changed line types, features, or qualifiers)
are anticipated for this new class of GenBank record.

However, note that TSA records make use of the same PRIMARY block that
is utilized for Third-Party Annotation (TPA) records. The PRIMARY block
contains references to the underlying reads/transcripts that were assembled
to construct a TSA record.

It might be helpful to review the content of TSA record EZ000001 and its
use of the PRIMARY block:

http://www.ncbi.nlm.nih.gov/entrez/viewer.fcgi?db=nuccore&id=189498984

Requirements for the new Transcriptome Shotgun Assembly division include:

1. Submission of primary transcipt sequence data to dbEST, the Trace Archive,
or the Short-Read archive (SRA).

2. Registration of an associated transcriptome project with the International
Nucleotide Sequence Database Collaboration (INSDC).

For information about submitting projects via NCBI/GenBank, see:

http://www.ncbi.nlm.nih.gov/genomes/mpfsubmission.cgi

3. Submission of TSA sequence records to GenBank, including an assembly file
(.ace format)

Note that TSA records and the primary transcript sequences that they are
built from must be provided by the same submitter or collaborative group.

1.3.3 Changes in the content of index files

As described in the GB 153 release notes, the 'index' files which accompany
GenBank releases (see Section 3.3) are considered to be a legacy data product by
NCBI, generated mostly for historical reasons. FTP statistics of January 2005
seem to support this: the index files were transferred only half as frequently as
the files of sequence records. The inherent inefficiencies of the index file
format also lead us to suspect that they have little serious use by the user
community, particularly for EST and GSS records.

The software that generated the index file products received little
attention over the years, and finally reached its limitations in
February 2006 (Release 152.0). The required multi-server queries which
obtained and sorted many millions of rows of terms from several different
databases simply outgrew the capacity of the hardware used for GenBank
Release generation.

Our short-term solution is to cease generating some index-file content
for all EST sequence records, and for GSS sequence records that originate
via direct submission to NCBI.

The three gbacc*.idx index files continue to reflect the entirety of the
release, including all EST and GSS records, however the file contents are
unsorted.

These 'solutions' are really just stop-gaps, and we will likely pursue
one of two options:

a) Cease support of the 'index' file products altogether.

b) Provide new products that present some of the most useful data from
the legacy 'index' files, and cease support for other types of index data.

If you are a user of the 'index' files associated with GenBank releases, we
encourage you to make your wishes known, either via the GenBank newsgroup,
or via email to NCBI's Service Desk:

info@...

Our apologies for any inconvenience that these changes may cause.

1.3.4 GSS File Header Problem

GSS sequences at GenBank are maintained in two different systems, depending
on their origin, and the dumps from those systems occur in parallel. Because
the second dump (for example) has no prior knowledge of exactly how many GSS
files will be dumped by the first, it does not know how to number its own
output files.

There is thus a discrepancy between the filenames and file headers for
fifty-nine of the GSS flatfiles in Release 166.0. Consider gbgss232.seq :

GBGSS1.SEQ Genetic Sequence Data Bank
June 15 2008

NCBI-GenBank Flat File Release 166.0

GSS Sequences (Part 1)

87182 loci, 64465152 bases, from 87182 reported sequences

Here, the filename and part number in the header is "1", though the file
has been renamed as "232" based on the number of files dumped from the other
system. We will work to resolve this discrepancy in future releases, but the
priority is certainly much lower than many other tasks.

1.4 Upcoming Changes

1.4.1 Comprehensive protein FASTA file to be discontinued

Given the availability of divisional protein FASTA files as of GenBank
Release 164.0, support for the single, large, comprehensive protein FASTA
file:

ftp://ftp.ncbi.nih.gov/genbank/relNNN.fsa_aa.gz

(where 'NNN' represents a three-digit GenBank release number) will be
discontinued after GenBank Release 166.0 in June of 2008. The size
of this file has grown to exceed 4GB, which is unmanageable for many users.

Users are advised to make plans to utilize the new divisional files by
August of 2008. The divisional protein FASTA files are located at:

ftp://ftp.ncbi.nih.gov/ncbi-asn1/gbXXX.fsa_aa.gz

_______________________________________________
Genbankb mailing list
Genbankb@...
http://www.bio.net/biomail/listinfo/genbankb

GenBank Release 166.0 Now Available

2008-06-13T19:51:40Z

_______________________________________________
Genbankb mailing list
Genbankb@...
http://www.bio.net/biomail/listinfo/genbankb

attachment0 (9K) Download Attachment

GenBank 166.0 Close-of-Data

2008-06-12T08:51:59Z

Greetings GenBank Users,

Close-of-data for the upcoming GenBank Release 166.0 occurred on
Wednesday June 11 at approximately 1:30am ET.

The subsequently generated GenBank Incremental Update files
nc0611.aso, nc0611,flat, etc. contain data through the close.

Note: Release processing often does not begin until sometime during
business hours on the close date. As a result, a number of sequence
records processed *after* 1:30am are likely to be present in the
GenBank 166.0 release files, even though they are "post-close" .

Similarly, the first GenBank Incremental Update that is generated
after the close date is likely to contain a number of sequence
records that are unchanged, compared to their appearance in the
release files.

Our apologies for the lack of advanced notice about the close date.

Mark Cavanaugh
GenBank
NCBI/NLM/NIH/HHS

_______________________________________________
Genbankb mailing list
Genbankb@...
http://www.bio.net/biomail/listinfo/genbankb

GenBank Release 165.0 Now Aailable

2008-04-18T13:52:46Z

Greetings GenBank Users,

GenBank Release 165.0 is now available via FTP from the National
Center for Biotechnology Information (NCBI):

Ftp Site Directory Contents
---------------- --------- ---------------------------------------
ftp.ncbi.nih.gov genbank GenBank Release 165.0 flatfiles
ncbi-asn1 ASN.1 data used to create Release 165.0

Close-of-data for GenBank 165.0 occured on 04/15/2008. Uncompressed, the
Release 165.0 flatfiles require roughly 332 GB (sequence files only)
or 353 GB (including the 'short directory', 'index' and the *.txt files).
The ASN.1 data require approximately 305 GB.

Recent statistics for non-WGS, non-CON sequences:

Release Date Base Pairs Entries

164 Feb 2008 85759586764 82853685
165 Apr 2008 89172350468 85500730

Recent statistics for WGS sequences:

Release Date Base Pairs Entries

164 Feb 2008 108635736141 27439206
165 Apr 2008 110500961400 26931049

During the 62 days between the close dates for GenBank Releases 164.0 and
165.0, the non-WGS/non-CON portion of GenBank grew by 3,412,763,704 basepairs
and by 2,647,045 sequence records. During that same period, 1,590,201 records
were updated. An average of about 68,340 non-WGS/non-CON records were added
and/or updated per day.

Between releases 164.0 and 165.0, the WGS component of GenBank grew by
1,865,225,259 basepairs. The number of records decreased by 508,157 due to
the re-assembly of WGS project AAKN, into far fewer (but larger) contigs.

For additional release information, see the README files in either of
the directories mentioned above, and the release notes (gbrel.txt) in
the genbank directory. Sections 1.3 and 1.4 of the release notes
(Changes in Release 165.0 and Upcoming Changes) have been appended
below.
** Important Notes **

* A new GenBank division has become legal with this April's Release 165.0 :
the Transcriptome Shotgun Assembly, or TSA, division. Please see
Section 1.3.2 of the release notes for more information about TSA and the
records that it will contain.

* Support for the single, comprehensive protein FASTA file which accompanies
GenBank releases will be ceased as of the June 2008 release. See Section
1.4.1 of the release notes for details.

* GenBank 'index' files are now provided without any EST content, and without
most GSS content. See Section 1.3.3 of the release notes for further details.
NCBI is considering ceasing support for the index files, so we encourage
affected users to review that section and provide feedback.

Release 165.0 data, and subsequent updates, are available now via
NCBI's Entrez and Blast services.

As a general guideline, we suggest first transferring the GenBank release
notes (gbrel.txt) whenever a release is being obtained. Check to make sure
that the date and release number in the header of the release notes are
current (eg: April 15 2008, 165.0). If they are not, interrupt the
remaining transfers and then request assistance from the NCBI Service Desk.

A comprehensive check of the headers of all release files after your
transfers are complete is also suggested. Here's how one might go about
this on a unix platform, using csh/tcsh :

set files = `ls gb*.*`
foreach i ($files)
head -10 $i | grep Release
end

Or, if the files are compressed, perhaps:

gzcat $i | head -10 | grep Release

If you encounter problems while ftp'ing or uncompressing Release
165.0, please send email outlining your difficulties to:

info@...

Mark Cavanaugh, Vladimir Alekseyev, Michael Kimelman
GenBank
NCBI/NLM/NIH/HHS

1.3 Important Changes in Release 165.0

1.3.1 Organizational changes

The total number of sequence data files increased by 41 with this release:

- the BCT division is now comprised of 28 files (+2)
- the EST division is now comprised of 713 files (+19)
- the GSS division is now comprised of 285 files (+8)
- the HTG division is now comprised of 114 files (+7)
- the PAT division is now comprised of 38 files (+3)
- the PLN division is now comprised of 29 files (+1)
- the VRL division is now comprised of 9 files (+1)

1.3.2 New Transcriptome Shotgun Assembly (TSA) division now legal

A new GenBank division for assembled mRNA sequences, Transcriptome Shotgun
Assembly (TSA), can now appear in GenBank releases, as of this April 2008
Release 165.0. The date of first appearance of a TSA record will depend
on the status of TSA submission processing, but it *is* expected that they will
begin to appear in the GenBank Incremental Updates (GIU) within the next
month, and that Release 166.0 will include a divisional TSA file.

Files in this new division will have filenames of:

gbtsaNN.aso.gz (ASN.1 format)
gbtsaNN.seq.gz (GenBank flatfile format)

where 'NN' represents an integer file-number within the TSA division.

TSA sequences are shotgun assemblies of primary sequences deposited in
dbEST, the Trace Archive (TA) or the Short-Read Archive (SRA). Keywords
"TSA" and "Transcriptome Shotgun Assembly" are present on all TSA
records, in addition to a division code value of "TSA" on the LOCUS line.

No format changes (new or changed line types, features, or qualifiers)
are anticipated for this new class of GenBank record.

However, note that TSA records make use of the same PRIMARY block that
is utilized for Third-Party Annotation (TPA) records. The PRIMARY block
contains references to the underlying reads/transcripts that were assembled
to construct a TSA record.

It might be helpful to review Third Party Annotation record BK005658,
which provides a good example of PRIMARY block usage:

http://www.ncbi.nlm.nih.gov/entrez/viewer.fcgi?db=nuccore&id=83843278

Requirements for the new Transcriptome Shotgun Assembly division include:

1. Submission of primary transcipt sequence data to dbEST, the Trace Archive,
or the Short-Read archive (SRA).

2. Registration of an associated transcriptome project with the International
Nucleotide Sequence Database Collaboration (INSDC).

For information about submitting projects via NCBI/GenBank, see:

http://www.ncbi.nlm.nih.gov/genomes/mpfsubmission.cgi

3. Submission of TSA sequence records to GenBank, including an assembly file
(.ace format)

Note that TSA records and the primary transcript sequences that they are
built from must be provided by the same submitter or collaborative group.

Examples of TSA records and more information about how to submit them
will be provided in future editions of these release notes, and via the
GenBank newsgroup.

1.3.3 Changes in the content of index files

As described in the GB 153 release notes, the 'index' files which accompany
GenBank releases (see Section 3.3) are considered to be a legacy data product by
NCBI, generated mostly for historical reasons. FTP statistics of January 2005
seem to support this: the index files were transferred only half as frequently as
the files of sequence records. The inherent inefficiencies of the index file
format also lead us to suspect that they have little serious use by the user
community, particularly for EST and GSS records.

The software that generated the index file products received little
attention over the years, and finally reached its limitations in
February 2006 (Release 152.0). The required multi-server queries which
obtained and sorted many millions of rows of terms from several different
databases simply outgrew the capacity of the hardware used for GenBank
Release generation.

Our short-term solution is to cease generating some index-file content
for all EST sequence records, and for GSS sequence records that originate
via direct submission to NCBI.

The three gbacc*.idx index files continue to reflect the entirety of the
release, including all EST and GSS records, however the file contents are
unsorted.

These 'solutions' are really just stop-gaps, and we will likely pursue
one of two options:

a) Cease support of the 'index' file products altogether.

b) Provide new products that present some of the most useful data from
the legacy 'index' files, and cease support for other types of index data.

If you are a user of the 'index' files associated with GenBank releases, we
encourage you to make your wishes known, either via the GenBank newsgroup,
or via email to NCBI's Service Desk:

info@...

Our apologies for any inconvenience that these changes may cause.

1.3.4 GSS File Header Problem

GSS sequences at GenBank are maintained in two different systems, depending
on their origin, and the dumps from those systems occur in parallel. Because
the second dump (for example) has no prior knowledge of exactly how many GSS
files will be dumped by the first, it does not know how to number its own
output files.

There is thus a discrepancy between the filenames and file headers for
fifty-four of the GSS flatfiles in Release 165.0. Consider gbgss232.seq :

GBGSS1.SEQ Genetic Sequence Data Bank
April 15 2008

NCBI-GenBank Flat File Release 165.0

GSS Sequences (Part 1)

87177 loci, 64476488 bases, from 87177 reported sequences

Here, the filename and part number in the header is "1", though the file
has been renamed as "232" based on the number of files dumped from the other
system. We will work to resolve this discrepancy in future releases, but the
priority is certainly much lower than many other tasks.

1.4 Upcoming Changes

1.4.1 Comprehensive protein FASTA file to be discontinued

With the availability of divisional protein FASTA files as of GenBank
Release 164.0, support for the single, large, comprehensive protein FASTA
file:

ftp://ftp.ncbi.nih.gov/genbank/relNNN.fsa_aa.gz

(where 'NNN' represents a three-digit GenBank release number) will be
discontinued after GenBank Release 166.0 in June of 2008. The size
of this file has grown to exceed 4GB, which is unmanageable for many users.

Users are advised to make plans to utilize the new divisional files by
August of 2008. If this timetable poses problems, please let us know at the
NCBI Service Desk:

info@...

_______________________________________________
Genbankb mailing list
Genbankb@...
http://www.bio.net/biomail/listinfo/genbankb

GenBank 165.0 Close-of-Data

2008-04-17T11:51:18Z

Greetings GenBank Users,

Close-of-data for the upcoming GenBank Release 165.0 occurred on
Monday April 14 at approximately 8:00pm EDT.

The subsequently generated GenBank Incremental Update files
nc0415.aso, nc0415,flat, etc. contain data through the close.

Our apologies for the lack of advanced notice about the close date.

Mark Cavanaugh
GenBank
NCBI/NLM/NIH/HHS

_______________________________________________
Genbankb mailing list
Genbankb@...
http://www.bio.net/biomail/listinfo/genbankb

GenBank Release 164.0 Now Aailable

2008-02-18T22:17:39Z

Greetings GenBank Users,

GenBank Release 164.0 is now available via FTP from the National
Center for Biotechnology Information (NCBI):

Ftp Site Directory Contents
---------------- --------- ---------------------------------------
ftp.ncbi.nih.gov genbank GenBank Release 164.0 flatfiles
ncbi-asn1 ASN.1 data used to create Release 164.0

Close-of-data for GenBank 164.0 occured on 02/12/2008. Uncompressed, the
Release 164.0 flatfiles require roughly 321 GB (sequence files only)
or 342 GB (including the 'short directory', 'index' and the *.txt files).
The ASN.1 data require approximately 295 GB.

Recent statistics for non-WGS, non-CON sequences:

Release Date Base Pairs Entries

163 Dec 2007 83874179730 80388382
164 Feb 2008 85759586764 82853685

Recent statistics for WGS sequences:

Release Date Base Pairs Entries

163 Dec 2007 106505691578 26177471
164 Feb 2008 108635736141 27439206

During the 56 days between the close dates for GenBank Releases 163.0 and
164.0, the non-WGS/non-CON portion of GenBank grew by 1,885,407,034 basepairs
and by 2,465,303 sequence records. During that same period, 1,750,703 records
were updated. An average of about 75,286 non-WGS/non-CON records were added
and/or updated per day.

Between releases 163.0 and 164.0, the WGS component of GenBank grew by
2,130,044,563 basepairs and by 1,261,735 sequence records.

For additional release information, see the README files in either of
the directories mentioned above, and the release notes (gbrel.txt) in
the genbank directory. Sections 1.3 and 1.4 of the release notes
(Changes in Release 164.0 and Upcoming Changes) have been appended
below.
** Important Notes **

A new GenBank division will become legal as of Release 165.0 in April 2008:
the Transcriptome Shotgun Assembly, or TSA, division. Please see Section 1.4.1
of the release notes for more information about TSA and the records that it
will contain.

GenBank 'index' files are now provided without any EST content, and without
most GSS content. See Section 1.3.3 of the release notes for further details.
NCBI is considering ceasing support for the index files, so we encourage
affected users to review that section and provide feedback.

Release 164.0 data, and subsequent updates, are available now via
NCBI's Entrez and Blast services.

As a general guideline, we suggest first transferring the GenBank release
notes (gbrel.txt) whenever a release is being obtained. Check to make sure
that the date and release number in the header of the release notes are
current (eg: February 15 2008, 164.0). If they are not, interrupt the
remaining transfers and then request assistance from the NCBI Service Desk.

A comprehensive check of the headers of all release files after your
transfers are complete is also suggested. Here's how one might go about
this on a unix platform, using csh/tcsh :

set files = `ls gb*.*`
foreach i ($files)
head -10 $i | grep Release
end

Or, if the files are compressed, perhaps:

gzcat $i | head -10 | grep Release

If you encounter problems while ftp'ing or uncompressing Release
164.0, please send email outlining your difficulties to:

info@...

Mark Cavanaugh, Vladimir Alekseyev, Michael Kimelman
GenBank
NCBI/NLM/NIH/HHS

1.3 Important Changes in Release 164.0

1.3.1 Organizational changes

The total number of sequence data files increased by 10 with this release:

- the CON division is now comprised of 85 files (+1)
- the ENV division is now comprised of 8 files (+1)
- the HTC division is now comprised of 13 files (+1)
- the EST division is now comprised of 694 files (+19)
- the GSS division is now comprised of 277 files (-18) (see Note below)
- the HTG division is now comprised of 107 files (+2)
- the INV division is now comprised of 12 files (+1)
- the PAT division is now comprised of 35 files (+1)
- the PLN division is now comprised of 28 files (+1)
- the PRI division is now comprised of 35 files (+1)

The total number of index files increased by 2 with this release:

- the AUT (AUTHOR Name) index is now comprised of 50 files (+2)

NOTE:

A configuration setting that determines the average size for many
of the GSS division GenBank flatfiles was mistakenly changed for
Release 163.0. This resulted in a filesize decrease for a large
number of GSS flatfiles, from 230 MB to 210 MB. Consequently, the
total number of GSS flatfiles underwent an artificially-large
increase of 26 files.

The configuration setting was restored for GenBank Release 164.0.
As as a result, there is now an apparent net decrease of 18 GSS files.
Our apologies for any confusion that this may have caused.

1.3.2 Divisional protein FASTA files now available

Individual protein FASTA data files are now being made available for
GenBank releases, in the ASN.1 area of the NCBI FTP site:

ftp://ftp.ncbi.nih.gov/ncbi-asn1/protein_fasta

Each protein FASTA file reflects the protein data content of the ASN.1
data file bearing the same division code in its name. For example, these
two "pri12" divisional files:

gbpri12.aso.gz
gbpri12.fsa_aa.gz

are 'equivalent', in that the proteins annotated on the DNA sequences
of gbpri12.aso are all present in gbpri12.fsa_aa.gz . For further
information, please see this README:

ftp://ftp.ncbi.nih.gov/ncbi-asn1/protein_fasta/README.protein_fasta

These divisional files are a replacement (see Section 1.4.2) for the single
protein FASTA file that has been provided in conjunction with GenBank releases:

ftp://ftp.ncbi.nih.gov/genbank/relNNN.fsa_aa.gz

where 'NNN' represents a three-digit GenBank release number.

1.3.3 Changes in the content of index files

As described in the GB 153 release notes, the 'index' files which accompany
GenBank releases (see Section 3.3) are considered to be a legacy data product by
NCBI, generated mostly for historical reasons. FTP statistics of January 2005
seem to support this: the index files were transferred only half as frequently as
the files of sequence records. The inherent inefficiencies of the index file
format also lead us to suspect that they have little serious use by the user
community, particularly for EST and GSS records.

The software that generated the index file products received little
attention over the years, and finally reached its limitations in
February 2006 (Release 152.0). The required multi-server queries which
obtained and sorted many millions of rows of terms from several different
databases simply outgrew the capacity of the hardware used for GenBank
Release generation.

Our short-term solution is to cease generating some index-file content
for all EST sequence records, and for GSS sequence records that originate
via direct submission to NCBI.

The three gbacc*.idx index files continue to reflect the entirety of the
release, including all EST and GSS records, however the file contents are
unsorted.

These 'solutions' are really just stop-gaps, and we will likely pursue
one of two options:

a) Cease support of the 'index' file products altogether.

b) Provide new products that present some of the most useful data from
the legacy 'index' files, and cease support for other types of index data.

If you are a user of the 'index' files associated with GenBank releases, we
encourage you to make your wishes known, either via the GenBank newsgroup,
or via email to NCBI's Service Desk:

info@...

Our apologies for any inconvenience that these changes may cause.

1.3.4 GSS File Header Problem

GSS sequences at GenBank are maintained in two different systems, depending
on their origin, and the dumps from those systems occur in parallel. Because
the second dump (for example) has no prior knowledge of exactly how many GSS
files will be dumped by the first, it does not know how to number its own
output files.

There is thus a discrepancy between the filenames and file headers for
fifty-one of the GSS flatfiles in Release 164.0. Consider gbgss227.seq :

GBGSS1.SEQ Genetic Sequence Data Bank
February 15 2008

NCBI-GenBank Flat File Release 164.0

GSS Sequences (Part 1)

86927 loci, 64285938 bases, from 86927 reported sequences

Here, the filename and part number in the header is "1", though the file
has been renamed as "227" based on the number of files dumped from the other
system. We will work to resolve this discrepancy in future releases, but the
priority is certainly much lower than many other tasks.

1.4 Upcoming Changes

1.4.1 New GenBank TSA division: Transcriptome Shotgun Assembly

A new GenBank division for assembled mRNA sequences, Transcriptome Shotgun
Assembly (TSA), will be included in GenBank releases on or after Release 165.0
in April of 2008.

Files in this new division will have filenames of:

gbtsaNN.aso.gz (ASN.1 format)
gbtsaNN.seq.gz (GenBank flatfile format)

where 'NN' represents an integer file-number within the TSA division.

TSA sequences are shotgun assemblies of primary sequences deposited in
dbEST, the Trace Archive (TA) or the Short-Read Archive (SRA). Keywords
"TSA" and "Transcriptome Shotgun Assembly" will be present for all TSA
records, in addition to a division code value of "TSA" on the LOCUS line.

No format changes (new or changed line types, features, or qualifiers)
are anticipated for this new class of GenBank record.

TSA records make use of the same PRIMARY block that is utilized for
Third-Party Annotation (TPA) records. The PRIMARY block will contain
references to the underlying reads/transcripts that were assembled to
construct the TSA record.

It might be helpful to review Third Party Annotation record BK005658,
which provides a good example of PRIMARY block usage:

http://www.ncbi.nlm.nih.gov/entrez/viewer.fcgi?db=nuccore&id=83843278

Requirements for the new Transcriptome Shotgun Assembly division include:

1. Submission of primary transcipt sequence data to dbEST, the Trace Archive,
or the Short-Read archive (SRA).

2. Registration of an associated transcriptome project with the International
Nucleotide Sequence Database Collaboration (INSDC).

For information about submitting projects via NCBI/GenBank, see:

http://www.ncbi.nlm.nih.gov/genomes/mpfsubmission.cgi

3. Submission of TSA sequence records to GenBank, including an assembly file
(.ace format)

Note that TSA records and the primary transcript sequences that they are
built from must be provided by the same submitter or collaborative group.

Examples of TSA records and more information about how to submit them
will be provided in future editions of these release notes, and via the
GenBank newsgroup.

1.4.2 Comprehensive protein FASTA file to be discontinued

With the availability of divisional protein FASTA files as of GenBank
Release 164.0, support for the single, large, comprehensive protein FASTA
file:

ftp://ftp.ncbi.nih.gov/genbank/relNNN.fsa_aa.gz

(where 'NNN' represents a three-digit GenBank release number) will be
discontinued after GenBank Release 166.0 in June of 2008. The size
of this file has grown to exceed 4GB, which is unmanageable for many users.

Users are advised to make plans to utilize the new divisional files by
August of 2008. If this timetable poses problems, please let us know at the
NCBI Service Desk:

info@...

_______________________________________________
Genbankb mailing list
Genbankb@...
http://www.bio.net/biomail/listinfo/genbankb

Protein FASTA files for GenBank releases : new location and file convention

2008-02-15T09:34:02Z

Prior to February 2008, a FASTA product for protein sequences from
coding regions annotated on the DNA sequences in GenBank has been
provided as a single large file:

ftp://ftp.ncbi.nih.gov/genbank/relNNN.fsa_aa.gz

where 'NNN' represents a 3-digit GenBank release number.

The uncompressed size of this file has grown to exceed 4GB,
which is unmanageable for many users. So as of GenBank Release
164.0, individual protein FASTA files will be provided on
a per-division basis, in a new subdirectory of the NCBI FTP site:

ftp://ftp.ncbi.nih.gov/ncbi-asn1/protein_fasta

One such file would be named:

gbpri1.fsa_aa.gz

Further information will be available via a README file in the
new ncbi-asn1/protein_fasta directory, when the Release 164.0
files are made installed (possibly by Saturday February 16).

Note that the location of the protein FASTA files is within the
/ncbi-asn1 area, not the /genbank area. Since the protein FASTA
files have a 1-to-1 correspondence with NCBI's ASN.1 files, this
is a more natural location for them.

[In fact, the quality-score data files currently located
in /genbank/quality_scores would *also* be located more
naturally in the /ncbi-asn1 area. They may be relocated
at a future date.]

The old single-file protein FASTA product will be supported
for two more GenBank releases, through Release 166.0 in June
of 2008. But after that release, the relNNN.fsa_aa.gz file
will be discontinued.

Mark Cavanaugh
GenBank
NCBI/NLM/NIH/HHS

_______________________________________________
Genbankb mailing list
Genbankb@...
http://www.bio.net/biomail/listinfo/genbankb

GenBank 164.0 Close-of-Data

2008-02-14T08:56:36Z

Greetings GenBank Users,

Close-of-data for the upcoming GenBank Release 164.0 occurred on
Tuesday February 12 at approximately 1:30am EST.

The subsequently generated GenBank Incremental Update files
nc0212.aso, nc0212,flat, etc. contain data through the close.

Note: Release processing often does not begin until sometime during
business hours on the close date. As a result, a number of sequence
records processed *after* 1:30am are likely to be present in the
GenBank 164.0 release files, even though they are "post-close" .

Similarly, the first GenBank Incremental Update that is generated
after the close date is likely to contain a number of sequence
records that are unchanged, compared to their appearance in the
release files.

Our apologies for the lack of advanced notice about the close date.

Mark Cavanaugh
GenBank
NCBI/NLM/NIH/HHS

_______________________________________________
Genbankb mailing list
Genbankb@...
http://www.bio.net/biomail/listinfo/genbankb