420 lines
14 KiB
Text
420 lines
14 KiB
Text
Notes on library handling
|
|
-------------------------
|
|
|
|
Contents of this document:
|
|
|
|
I) Introduction
|
|
II) Details of file organisation and use
|
|
III) Options currently available
|
|
IV) Installation guide
|
|
V) New feature table handling routines
|
|
VI) Indexing the sequence libraries
|
|
|
|
|
|
Section I Introduction
|
|
----------------------
|
|
|
|
Available sequence libraries
|
|
|
|
There are a number of different sequence libraries for nucleotide and protein:
|
|
PIR, GenBank, EMBL, Swissprot, and the Japanese Databank. Even after all the
|
|
years of their existence they still use different formats for their data. This
|
|
provides tedious and unrewarding work for software developers. Recently EMBL
|
|
and GenBank agreed a new and common way of writing their feature tables, which
|
|
is great help, although the rest of their format is different. Swissprot still
|
|
uses the old embl style feature table format and PIR yet another.
|
|
|
|
All the libraries distribute their data on magnetic tapes and EMBL and GenBank
|
|
have started to distribute on cdrom. The EMBL cdrom also contains Swissprot.
|
|
The GenBank and EMBL cdroms use different formats and have different contents.
|
|
The EMBL cdrom has useful indexes sorted alphabetically: those for entry name
|
|
and accession number, brief descriptions, keywords and freetext indexes are
|
|
already available and others are expected. These indexes point to the data for
|
|
each entry, and can be used to extract the data for any entry quickly.
|
|
|
|
Moving to unix
|
|
|
|
The VAX version of our package used PIR format which meant reformatting all
|
|
libraries other than PIR into that format. This required, at least
|
|
temporarily, having space for two copies of the libraries, and quite a lot of
|
|
cpu time. The software for doing this was provided by PIR, and is very VAX
|
|
specific and hence will not run under unix. For the unix version of our package
|
|
I have decided to use the EMBL cdrom format and its indexes as the primary
|
|
format. The current programs also support the use of PIR format libraries
|
|
without indexes - ie just the sequence and annotation files.
|
|
|
|
Indexing GenBank, EMBL updates, PIR and NRL3D
|
|
|
|
We include programs to create indexes for the above libraries. See below and
|
|
the README file in indexseqlibs. The programs can read all the above libraries
|
|
once the indexes are created. The indexing programs index the data in its
|
|
distributed form: WE DO NOT REFORMAT OR COPY THE LIBRARIES but simply create
|
|
indexes to the original files. Obviously this saves a lot of disk space, and
|
|
for those content to use only embl and swissprot from the cdrom, almost no disk
|
|
space is required. We havent tried it yet, but for genbank on cdrom, the only
|
|
extra disk space required would be for the indexes.
|
|
|
|
---------------------------------------------------------------------------
|
|
|
|
Section II Details of file organisation and use
|
|
-----------------------------------------------
|
|
|
|
The following strategy has been used to try to deal with alternate
|
|
and changing sequence library formats.
|
|
|
|
1) libraries are described at several levels:
|
|
|
|
a) the top level file is a list of available libraries which contains:
|
|
the library type, the name of the file containing the name of
|
|
each libraries individual files, and the prompt to appear on
|
|
the users screen: LTYPE LOGNAM PROMPT
|
|
|
|
b) the file containing the names of the libraries individual files
|
|
contains flags to define the file types: FTYPE LOGNAM
|
|
|
|
c) the individual library files
|
|
|
|
|
|
|
|
2) libary types handled:
|
|
|
|
a) EMBL/SWISSPROT in distributed format with cdrom index format
|
|
LTYPE = 'A'
|
|
b) GenBank in distributed format with cdrom index format LTYPE = 'C'
|
|
c) PIR/NRL3D in CODATA format with cdrom index format LTYPE = 'B'
|
|
d) PIR/NBRF .seq files can be read sequentially as "personal files
|
|
in PIR format" and do not appear in the list of available libraries.
|
|
e) FASTA format files can be read sequentially as "personal files
|
|
in FASTA format" and do not appear in the list of available
|
|
libraries.
|
|
|
|
3) EMBL, SWISSPROT and other libraries for which EMBL-style indexes have been
|
|
created
|
|
|
|
current file types:
|
|
|
|
A division.lookup
|
|
B entryname.index
|
|
C accession.target
|
|
D accession.hits
|
|
E brief description
|
|
F freetext.target
|
|
G freetext.hits
|
|
H author.target
|
|
I author.hits
|
|
|
|
|
|
Library list
|
|
level 1
|
|
|
|
|
|
|
|
-----------------------------------------------------------
|
|
| | |
|
|
lib 1 file list lib 2 file list lib 3 file list
|
|
level 2
|
|
| |
|
|
-------- ---------
|
|
level 3
|
|
file 1 file 1
|
|
file 2 file 2
|
|
. .
|
|
file n file n
|
|
|
|
---------------------------------------------------------------------------
|
|
|
|
|
|
Example
|
|
-------
|
|
|
|
Level 1
|
|
|
|
File name: sequence.libs
|
|
Environment variable: SEQUENCELIBRARIES
|
|
Contents:
|
|
|
|
A EMBLFILES EMBL nucleotide library ! in cdrom format
|
|
C GENBFILES GenBank nucleotide library!
|
|
A SWISSFILES SWISSPROT protein library! in cdrom format
|
|
B PIRFILES PIR protein library!
|
|
B NRL3DFILES NRL3D protein library!
|
|
|
|
Notes:
|
|
|
|
The libraries have types A,B,C. The logical names are EMBLLIBDESCRP and
|
|
SWISSLIBDESCRP, etc and the prompts are 'EMBL nucleotide library' and
|
|
'SWISSPROT protein library', etc. Anything to the right of a ! is a comment.
|
|
|
|
Level 2: the list of library files (using embl as an example)
|
|
|
|
File name: embl.files
|
|
Environment variable: EMBLFILES
|
|
Contents:
|
|
|
|
A EMBLDIVPATH/embl_div.lkp
|
|
B EMBLINDPATH/entrynam.idx
|
|
C EMBLINDPATH/acnum.trg
|
|
D EMBLINDPATH/acnum.hit
|
|
E EMBLINDPATH/brief.idx
|
|
F EMBLINDPATH/freetext.trg
|
|
G EMBLINDPATH/freetext.hit
|
|
H EMBLINDPATH/author.trg
|
|
I EMBLINDPATH/author.hit
|
|
|
|
|
|
Level 3: the sequence and annotation files (eg 15 for embl, 1 for swissprot).
|
|
|
|
Paths and file names:
|
|
|
|
EMBLPATH/bb.dat
|
|
EMBLPATH/fun.dat
|
|
EMBLPATH/inv.dat
|
|
EMBLPATH/mam.dat
|
|
EMBLPATH/org.dat
|
|
EMBLPATH/patent.dat
|
|
EMBLPATH/phg.dat
|
|
EMBLPATH/pln.dat
|
|
EMBLPATH/pri.dat
|
|
EMBLPATH/pro.dat
|
|
EMBLPATH/rod.dat
|
|
EMBLPATH/syn.dat
|
|
EMBLPATH/una.dat
|
|
EMBLPATH/vrl.dat
|
|
EMBLPATH/vrt.dat
|
|
|
|
All files from the division lookup file down are exactly as they appear on the
|
|
cdrom. The division lookup file relates numbers stored in the indexes to
|
|
actual division (or data) files stored on the disk. We rewrite it so the
|
|
directory structure and file names can be chosen locally. Its format is
|
|
I6,1x,A. An example is given below.
|
|
|
|
Division lookup file
|
|
|
|
File name: STADTABL/embl_div.lkp
|
|
Environment variable path EMBLDIVPATH
|
|
Contents:
|
|
|
|
1 EMBLPATH/bb.dat
|
|
2 EMBLPATH/fun.dat
|
|
3 EMBLPATH/inv.dat
|
|
4 EMBLPATH/mam.dat
|
|
5 EMBLPATH/org.dat
|
|
6 EMBLPATH/patent.dat
|
|
7 EMBLPATH/phg.dat
|
|
8 EMBLPATH/pln.dat
|
|
9 EMBLPATH/pri.dat
|
|
10 EMBLPATH/pro.dat
|
|
11 EMBLPATH/rod.dat
|
|
12 EMBLPATH/syn.dat
|
|
13 EMBLPATH/una.dat
|
|
14 EMBLPATH/vrl.dat
|
|
15 EMBLPATH/vrt.dat
|
|
---------------------------------------------------------------------------
|
|
|
|
|
|
Section III Options currently available
|
|
---------------------------------------
|
|
|
|
Facilities currently offered in nip,pip,sip,nipl,pipl,sipl:
|
|
|
|
Get a sequence by knowing its entry name
|
|
Get a sequences' annotation by knowing its entry name
|
|
Get an entry name by knowing its accession number
|
|
Search the freetext index
|
|
Search the author index
|
|
|
|
Facilities currently offered in nipl,pipl,sipl:
|
|
|
|
Search whole library
|
|
Search only a list of entry names
|
|
Search all but a list of entry names
|
|
|
|
Outline of each type of operation
|
|
|
|
Looking for an entry by name: the programs will open the library description
|
|
file and read the names of its files and their file types. Then they will open
|
|
the entrynam.idx file, and find the sequence offset, annotation offset and
|
|
division number. Then open the division lookup file, find the file name for the
|
|
division required, open that file, seek to the required byte and get the data.
|
|
|
|
Looking for an entry by accession number: the programs will open the library
|
|
description file and read the names of its files and their file types. Then
|
|
they open the acnum.trg and acnum.hit files. The acnum.trg file is read to find
|
|
the accession number and a pointer to the acnum.hit file and the number of
|
|
hits. That file is read and the corresponding entry names displayed. At
|
|
present no further action is performed, although I expect to list out the
|
|
titles for the entries found.
|
|
|
|
Searching the whole of a library: the programs will open the library
|
|
description file and read the names of its files and their file types. Then
|
|
they open the division lookup file, read the names and numbers of the sequence
|
|
files, open all of them, then open the entryname file. Then the library is
|
|
processed sequentially by reading the entry names, their sequence offsets and
|
|
division numbers from the entry names file, and then the sequence from the
|
|
appropriate data file.
|
|
|
|
Searching the whole of a library using a list of entry names to include: the
|
|
programs will open the library description file and read the names of its files
|
|
and their file types. Then they open the division lookup file, read the names
|
|
and numbers of the sequence files, open all of them, then open the entryname
|
|
file. Then the library is processed by reading the list of entry names and
|
|
finding the names in the entry names file to get their sequence offsets and
|
|
division numbers, and then the sequence from the appropriate data file. It will
|
|
stop when it reaches the end of the list of entry names. The list of entry
|
|
names can be in any order.
|
|
|
|
Searching the whole of a library using a list of entry names to exclude: the
|
|
programs will open the library description file and read the names of its files
|
|
and their file types. Then they open the division lookup file, read the names
|
|
and numbers of the sequence files, open all of them, then open the entryname
|
|
file. Then the library is processed sequentially by reading the list of entry
|
|
names, reading the next entry in the entry names file to make sure it does not
|
|
match, then getting the sequence offsets and division numbers, and then the
|
|
sequence from the appropriate data file. If a the next name matches the name on
|
|
the list of entry names, it will be skipped, and the next name to exclude read.
|
|
If the list of excluded names is finished the rest of the library is searched
|
|
sequentially. The list of entry names must be in the same order as those in the
|
|
library (ie sorted alphabetically).
|
|
|
|
Searching a whole library using a PIR format file is performed by reading it
|
|
sequentially. If as list of entry names is used it must be in the same order as
|
|
the entries in the library file.
|
|
---------------------------------------------------------------------------
|
|
|
|
|
|
|
|
|
|
Section IV Installation guide
|
|
-----------------------------
|
|
|
|
EMBL CDROM
|
|
|
|
The data can be left on the cdrom or copied to hard disk. The files
|
|
staden.login and staden.profile source the file $STADTABL/libraries.config.csh
|
|
and $STADTABL/libraries.config.sh respectively. Refer to this file to see what
|
|
is required to install, add or move a sequence library that you want to be used
|
|
by the programs.
|
|
|
|
Other libraries (PIR, Genbank, EMBL updates)
|
|
|
|
Create the indexes then edit the files that tell the programs where the data is
|
|
stored. The files staden.login and staden.profile source the file
|
|
$STADTABL/libraries.config Refer to this file to see what is required to
|
|
install, add or move a sequence library that you want to be used by the
|
|
programs.
|
|
|
|
|
|
------------------------------------------------------------------------------
|
|
|
|
|
|
Section V New feature table handling facilities
|
|
-----------------------------------------------
|
|
|
|
As mentioned above EMBL and GenBank have recently introduced new feature tables
|
|
for annotating the sequences. They are a great improvement on the previous ones
|
|
and, among other things, now permit correct translation of spliced genes.
|
|
Various options within nip have been added or modified to take advantage of
|
|
these changes. The routine to translate DNA to protein and write the protein
|
|
to disk now gives correct results for spliced genes. The routine to translate
|
|
DNA to protein and display the two together now gives correct translations
|
|
except for the amino acids spanning intron/exon junctions. The routine to plot
|
|
maps from feature tables can use the new style. The open reading frame finding
|
|
routine writes out its results in the new style. The routine that finds open
|
|
reading frames and writes their translations to disk also writes a title in the
|
|
form of a new style feature table entry. The feature table format output from
|
|
the pattern searches in nip also uses the new style.
|
|
|
|
|
|
|
|
----------------------------------------------------------------------------
|
|
|
|
Section VI Indexing the sequence libraries
|
|
--------------------------------------------
|
|
|
|
We handle EMBL, SwissProt, and GenBank in their distributed format, plus
|
|
PIR and NRL3D in codata format. All programs and scripts are in directory
|
|
indexseqlibs.
|
|
|
|
Currently we produce entryname index, accession number index freetext index,
|
|
and brief index (brief index contains the entry name the primary accession
|
|
number the sequence length and an 80 character description).
|
|
|
|
To produce any of the indexes requires the creation of several intermediate
|
|
files and the indexing programs are written so that the intermediate files
|
|
are the same for all libraries. This means that only the programs that read
|
|
the distributed form of each library need to be unique to that library, and
|
|
all the other processing programs can be used for all libraries.
|
|
|
|
|
|
However even the though the indexes have the same format, programs (like nip)
|
|
that read the libraries need to treat each library separately because their
|
|
actual contents are written differently.
|
|
|
|
Making the entry name index
|
|
---------------------------
|
|
|
|
Common program entryname2
|
|
|
|
EMBL emblentryname1
|
|
SwissProt emblentryname1
|
|
|
|
GenBank genbentryname1
|
|
|
|
PIR pirentryname1
|
|
NRL3D pirentryname1
|
|
|
|
|
|
Making the accession number index
|
|
---------------------------------
|
|
|
|
Common programs access2 access3 access4
|
|
|
|
EMBL emblaccess1
|
|
SwissProt emblaccess1
|
|
|
|
GenBank genbaccess1
|
|
|
|
PIR piraccess1 piraccess2
|
|
NRL3D No accession numbers
|
|
|
|
Making the brief index
|
|
----------------------
|
|
|
|
Common program title2
|
|
|
|
EMBL embltitle1
|
|
SwissProt embltitle1
|
|
|
|
GenBank genbtitle1
|
|
|
|
PIR pirtitle1 pirtitle2 (pir3 has no accession numbers)
|
|
NRL3D pirtitle2
|
|
|
|
Scripts
|
|
-------
|
|
|
|
emblentryname.script
|
|
emblaccession.script
|
|
embltitle.script
|
|
|
|
swissentryname.script
|
|
swissaccession.script
|
|
swisstitle.script
|
|
|
|
genbentrynamescript
|
|
genbaccession.script
|
|
genbtitle.script
|
|
|
|
pirentryname.script
|
|
piraccession.script
|
|
pirtitle.script
|
|
|
|
nrl3dentryname.script
|
|
nrl3dtitle.script
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|