Experiment files and their application during large-scale sequencing projects

The data for large scale sequencing projects are passed through several processing steps prior to assembly, and post-assembly processing generally requires knowledge of more than just the sequence of each reading. We address here the problem of providing data to individual programs and of combining all the tasks into a single process. The solution comprises two components: a file format (experiment file format) that stores information about readings, and a script (PREGAP) that controls the creation and use of experiment files by the processing programs. PREGAP can take a batch of data from a variety of sequencing instruments, gather information about each reading, and then scan the reading to select the 3' end of the good quality data, mark sequencing vector, other cloning vector sequences, and Alu segments. The results of all these operations are added to the experiment file for each reading, ready for processing by the assembly program. Experiment files also provide a mechanism for using alternative assembly engines with our package.