Working with huge files in Unix/Linux

The Unix or GNU/Linux command line has a variety of tools that are essential when working with gigantic files. Bioinformaticians often work with files that are many gigabytes in size. Recently, I was working with very large VCF and FASTA files and wanted to check their contents. Both files came gzipped so I needed to decompress them first.

~$ gzip -d file.vcf.gz
~$ gzip -d seq.fasta.gz

Now, what I had at the end of this was an 11GB VCF file and a 250MB fasta file - way too big to open up in any text editor. But I can use a variety of utilities available on the Linux command line to work with it.

To view the first few lines I could use head like so:

~$ head file.vcf

By default, this did not display enough lines to satisfy my requirements. I needed to view the first 100 lines so I used the -n paramter like so:

~$ head -n 100 file.vcf

I can also use head to grab the first few characters if I am reading from a FASTA file with a really long sequence on a single line. If I wanted to grab the first 1000 characters I can write:

~$ head -c 1000 seq.fasta

I can also use the same parameter to grab the last few characters or lines from the file using tail like so:

~$ tail -c 1000 seq.fasta

This is all good and well but what if I want to inspect only certain lines from a file - say from lines 10 - 20? I can use awk for that:

~$ awk 'NR==10,NR==20' file.vcf

I can also use awk to skip the first 10 lines like so:

~$ awk 'NR>10' file.vcf 

OK. What if I want to count how many characters are in the second line of a FASTA file where, in some cases, a whole sequence put onto one line? I can use a combination of awk and wc to count the letters:

~$ awk 'NR==2' seq.fasta | wc -c

If you want to count how many lines and how many characters there are in a file you can use wc like this passing it both the l and c flags:

~$ wc -lc seq.fasta

Now, if I want to navigate through the whole VCF file to see what it contained, I could not use a text editor. Instead, I would use 'less'. 'less' is better than 'more' if you are more familiar with that. It allows you to navigate to certain line numbers and you can use PgUp and PgDn and the up and down arrows to navigate through a file. You press Q to quit back to your console when you are done.

~$ less file.vcf

Less is OK for some things but it takes quite some time to read a really really big file. It may sometimes be easier to make a smaller sample file from the big file so you can comfortably inspect it. Use the redirection operator '>' to output say 1000 lines to a new file called smaller.vcf:

~$ head -n 1000 file.vcf > smaller.vcf

OK. There's something I didn't mention before which may save you a lot of time if you are in a rush or really short of disk space - you don't need spend all that time decompressing the VCF or FASTA files! You can decompress them on the fly on the command line and pipe a little of the decompressed output to one of those command line tools mentioned above. For example, you just want to grab 1000 lines from file.vcf.gz which is 11GB when decompressed, but you don't want to wait an hour for it to decompress and you only have 4GB of free disk space to spare. Try this:

~$ gzip -dc file.vcf.gz | head -n 1000 > just1000lines.vcf

The trick is that gzip can decompress things out to the console using the -c flag. So while it's decompressing, the output is piped to head, which continues to collect data until 1000 lines are read and then instead of printing all those lines out to the console, we redirect the output to insert into a file: just1000lines.vcf. This all happens very quickly and efficiently in memory and only 1000 lines are written to disk, saving you time and disk space.

Hopefully these command line tools - gzip, head, tail, awk, wc, less - and the redirection operator '>' will help you when you are working with large files.

Amusing Unix/Linux Commands

From the book 'The Unix-Haters Handbook'.

% rm meese-ethics
rm: meese-ethics nonexistent

% "How would you rate Dan Quayle's incompetence?
Unmatched ".

% ^How did the sex change^ operation go?
Modifier failed.

% If I had a ( for every $ the Congress spent, what would I have?
Too many ('s.

% make love
Make: Don't know how to make love. Stop.

% sleep with me
bad character

% got a light?
No match.

% man: why did you get a divorce?
man:: Too many arguments.

% ^What is saccharine?
Bad substitute.

% %blow
%blow: No such job.

These attempts at humor work with the Bourne shell:

$ PATH=pretending! /usr/ucb/which sense
no sense in pretending!


$ drink <bottle; opener
bottle: cannot open
opener: not found

$ mkdir matter; cat >matter
matter: cannot create

40 years of Unix

The computer world is notorious for its obsession with what is new - largely thanks to the relentless engine of Moore's Law that endlessly presents programmers with more powerful machines.

Given such permanent change, anything that survives for more than one generation of processors deserves a nod.

Think then what the Unix operating system deserves because in August 2009, it celebrates its 40th anniversary. And it has been in use every year of those four decades and today is getting more attention than ever before.

Work on Unix began at Bell Labs after AT&T, (which owned the lab), MIT and GE pulled the plug on an ambitious project to create an operating system called Multics.

The idea was to make better use of the resources of mainframe computers and have them serve many people at the same time.

"With Multics they tried to have a much more versatile and flexible operating system, and it failed miserably," said Dr Peter Salus, author of the definitive history of Unix's early years.

Time well spent

The cancellation meant that two of the researchers assigned to the project, Ken Thompson and Dennis Ritchie, had a lot of time on their hands. Frustrated by the size and complexity of Multics but not its aims of making computers more flexible and interactive, they decided to try and finish the work - albeit on a much smaller scale.

The commitment was helped by the fact that in August 1969, Ken Thompson's wife took their new baby to see relatives on the West Coast. She was due to be gone for a month and Thompson decided to use his time constructively - by writing the core of what became Unix.

He allocated one week each to the four core components of operating system, shell, editor and assembler. It was during that time and after as the growing team got the operating system running on a DEC computer known as a PDP-7 that Unix came into being.

By the early 1970s, five people were working on Unix. Thompson and Ritchie had been joined by Brian Kernighan, Doug McIlroy and Joe Ossanna.

The name was reportedly coined by Brian Kernighan - a lover of puns who wanted Unics to stand in contrast to its forebear Multics.

The team got Unix running well on the PDP7 and soon it had a long list of commands it could carry out. The syntax of many of those commands, such as chdir and cat, are still in use 40 years on. Along with it came the C programming language.

But, said Dr Salus, it wasn't just the programming that was important about Unix - the philosophy behind it was vital too.

"Unix was created to solve a few problems," said Dr Salus, "the most important of which was to have something that was much more compact than the operating systems that were current at that time which ran on the dinosaurs of the computer age."

Net benefits

Back in the early 1970s, computers were still huge and typically overseen by men in white coats who jealously guarded access to the machines. The idea of users directly interacting with the machine was downright revolutionary.

"It got us away from the total control that businesses like IBM and DEC had over us," said Dr Salus.

Word about Unix spread and people liked what they heard.

"Once it had jumped out of the lab and out of AT&T it caught fire among the academic community," Dr Salus told the BBC. What helped this grassroots movement was AT&T's willingness to give the software away for free.

That it ran on cheap hardware and was easy to move to different machines helped too.

"The fact that its code was adaptable to other types of machinery, in large and small versions meant that it could become an operating system that did more than just run on your proprietary machine," said Dr Salus.

In May 1975 it got another boost by becoming the chosen operating system for the internet. The decision to back it is laid out in the then-nascent Internet Engineering Task Force's document RFC 681, which notes that Unix "presents several interesting capabilities" for those looking to use it on the net.

It didn't stop there. Unix was adapted for use on any and every computer from mainframes to desktops. While it is true that it did languish in the 1980s and 90s as corporations scrapped over whose version was definitive, the rise of the web has given it new life.

The wars are over and the Unix specification is looked after by the Open Group - an industry body set up to police what is done in the operating system's name.

Now Unix, in a variety of guises, is everywhere. Most of the net runs on Unix-based servers and the Unix philosophy heavily influenced the open source software movements and the creation of the Linux desktop OS. Windows runs the communication stack created for Unix. Apple's OS X is broadly based on Unix and it is possible to dig into that software and find text remarkably similar to that first written by Dennis Ritchie in 1971.

"The really nice part is the flexibility and adaptability," said Dr Salus, explaining why it is so widespread and how its ethic fits with a world at home with the web.

"Unix is the best screwdriver ever built," said Dr Salus.

http://news.bbc.co.uk/1/hi/technology/8205976.stm