Pretty Microscope Pictures: Olympus Bioscapes Digital Photocompetition

1st Place 2010 Olympus BioScapes Digital Imaging Competition: Eyes of daddy longlegs. A frontal section of Phalangium opilio eyes. The lenses, retinasand optic nerves are visible. The image is a depth color-coded projection of a confocal image stack. (Igor Siwanowicz, Max Planck Institute for Neurobiology, Munich, Germany)

Click here to view more...



Unique value row count in SAS

You may have come across the need to count unique values in your dataset and put that as a row in your dataset/table - something like this:

id, name
1, Alan
2, Brad
2, Brad
3, David
3, David
3, David
4, George
5, Joe
6, Steven
7, Zed
7, Zed

The solution involves using the LAG() function. Lag is like a look-back function and for each row that is processed it looks back a row and fetches the value. So, if I were at observation (row) 2 and did LAG(name), it would return the value of "Alan". The retain function puts an initial temporary value in a variable so I can use it in my processing.

Anyway, here's the code:

data pupils;
    input name $;
    datalines;
Alan
Brad
Brad
David
David
David
George
Joe
Steven
Zed
Zed
;
run;


data store;
    set pupils;

    prevname = lag(name);

    format id BEST12.;
    retain id 0;
     if name^=prevname then id = id + 1;

    drop prevname;
run;

Biggest Genome ever? It's plant vs amoeba

A few days ago there was an article in Science magazine that they had discovered a plant (Paris japonica) with the largest genome ever found with 149 billion base pairs...

Except, I remembered there's an amoeba (Polychaos dubium - what a bloody weird name by the way!) with an even bigger genome of 670 billion base pairs.

So it got me wondering - how could Science magazine get it so wrong?! Well, this comment on the science mag site gave the reason why they would reject dubium:

"While this and some other Amoeba have been reported to have such very large genomes, some caveats regarding their reliability are perhaps in order.
The measurement for Amoeba dubia and other protozoa which have been reported to have very large genomes were made in the 1960s using a rough biochemical approach which is now considered to be an unreliable method for accurate genome size determinations. The method uses whole cells rather than isolated nuclei and thus will include not only DNA from the mitochondria but also any DNA in engulfed food organisms. Also some of the species are multinucleate.
The accuracy of the genome size estimates are also called into question given that a related species, Amoeba proteus, which was reported to have a genome size of 300 pg was more recently shown to be an order of magnitude smaller (34 - 43 pg DNA per cell). Like the situation for dinoglagellates (see below), the genomes of these Amoeba are clearly large but to know just how big requires their genome sizes to be estimated using modern best practice techniques available today. Only then will it be possible to know just how their genomes compare in size to those of Paris japonica." {Quoted from Ilia}

The reasons given in the quote can be ignored except for the clincher which is in bold. If we assume the same experimental conditions and level of accuracy, dubium will be found to have about 69pg DNA.

Game over, the plant wins.

Difficult SAS issue - restructuring datasets with proc transpose

I've been trying to solve this problem for a few hours and have tried many different things. In the end, it was two sets of a process that solved the problem with proc transpose. It's worth documenting so I'm blogging it.

The Problem:

I have an example dataset that contains observations like so:


(dataset: dayobs)
days_range,    value,    count
0-9,    300,    6
60-69,    250,    4
300-309,    76,    1

I wanted it to show all the ranges with value and count set to zero if it wasn't already set:

days_range,    value,    count

0-9,    300,    6

10-19,     0,     0

20-29,     0,     0

30-39,     0,     0

40-49,     0,     0

50-59,     0,     0

60-69,    250,    4

... etc ... 

290-299,    0,    0

300-309,    76,    1

That was easy, simply make a dataset that lists the day_range from 1-10 ... 390-400.

data ranges;
    input days_range $;
    format days_range $7.;
    datalines;
0-9
10-19
... etc ...
    ;
run;

and then merge both the datasets - dayobs and ranges - to create a new dataset called daysrange:

data daysrange;
    merge ranges dayobs;
    by days_range;
run;


OK. Now, I wanted to transpose it into a single row so I can insert that into a big dataset that logs the changes every day. I want to make it look like so:

v1_9, c1_9, v10_19, c10_19, v20_29, c20_29, ...etc... v60_69, c60_69, ...etc
300, 6, 0, 0, 0, 0, ...etc... 250, 4, ...etc

It might seem obvious what to do but I tried to be clever and did this:

data rowifieddaysrange;
    set daysrange;


    if days_range='0-9' then do;
        c1_9=count;
        v1_9=value;
    end;
    else if days_range='10-19' then do;
        c10_19=count;
        v10_19=value;
    end;
 etc...


keep c1_9 v1_9 c10_19 v10_19 etc...;
run;

and this is what the data looked a bit like in the end:

0, 0, 0, 0, 0, 300, 0, 0, 0, 0, 0, 0, 6
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
250, 0, 0, 0, 0, 0, 4, 0, 0, 0, 0, 0, 0
etc.

Basically, it had the correct columns, but they were spaced out across different rows. No matter what I tried - I look at all the different sas procs and their options - I was unable to collapse/compact it into one row... if anyone knows the solution to this problem I'd really like to see it please. Ta.

OK. So guess what the solution is... It turns out you need to use two sets of proc transpose and a couple of merges, to deal with each variable individually - then finally merging the two separate datasets together at the end! :


*transpose the daysrange dataset first by count;
proc transpose data=daysrange out=tdaysrange(drop=_:) prefix=c;
    id days_range;
    var count;
run;
*now transpose the table with all the categories;
proc transpose data=ranges out=tranges(drop=_:) prefix=c;
    id days_range;
run;
*now merge them;
data cdataset;
    merge tranges tdaysrange;
run;


*transpose the daysrange dataset next by value;
proc transpose data=daysrange out=vdaysrange(drop=_:) prefix=v;
    id days_range;
   var value;
run;
*now transpose the table with all the categories;
proc transpose data=ranges out=vranges(drop=_:) prefix=v;
    id days_range;
run;
*now merge them;
data vdataset;
    merge vranges vdaysrange;
run;


*now join the the value and count datasets into one dataset!
data rowifieddaysrange;
    merge vdataset cdataset;
run;

And finally it's all in one row that looks something like this:

v1_9, c1_9, v10_19, c10_19, v20_29, c20_29, ...etc... v60_69, c60_69, ...etc

300, 6, 0, 0, 0, 0, ...etc... 250, 4, ...etc

That's a lot of work for something that aught to be straightforward!

Got a new book: Pro HTML5 Programming

I got this book a few days ago and so far so good. I've read through chapter 1 and made my first HTML5 compliant page. The book is not written for absolute beginners - it assumes you've done HTML4/XHTML1, CSS2-3 and have a good grasp of Javascript - but I have these so it's all good. I'll report back once I've read the whole book and tried everything.

Amusing Unix/Linux Commands

From the book 'The Unix-Haters Handbook'.

% rm meese-ethics
rm: meese-ethics nonexistent

% "How would you rate Dan Quayle's incompetence?
Unmatched ".

% ^How did the sex change^ operation go?
Modifier failed.

% If I had a ( for every $ the Congress spent, what would I have?
Too many ('s.

% make love
Make: Don't know how to make love. Stop.

% sleep with me
bad character

% got a light?
No match.

% man: why did you get a divorce?
man:: Too many arguments.

% ^What is saccharine?
Bad substitute.

% %blow
%blow: No such job.

These attempts at humor work with the Bourne shell:

$ PATH=pretending! /usr/ucb/which sense
no sense in pretending!


$ drink <bottle; opener
bottle: cannot open
opener: not found

$ mkdir matter; cat >matter
matter: cannot create

Nil Points

I submitted a PHP class (Music Albums Year Analyzer) to PHPClasses.org about a month ago and they nominated it for the monthly innovation award contest. Guess how many people voted for my class...

Not a single person. lol.

I came joint last (12th). Seems no-one wants to analyse their music collection or they didn't like my coding style or something?

A few months ago I submitted another class, the Bounded Queue, and I came 9th. At least that's not last. I write classes I haven't seen been done in PHP before and that are somewhat useful. But most of the cool things have already been done so what's left is not that interesting usually.

Whales are great

I found this BBC resource called Great Whales a few days ago. It covers Blue, Fin, Right, Sei, Sperm, Bowhead, Bryde's, Humpback, Gray, and Minke whales. Check it:

10 years of the Human Genome

The completion of the draft human genome sequence was announced ten years ago. Nature 's survey of life scientists reveals that biology will never be the same again. Declan Butler reports.

Declan Butler

Download a PDF of this story.

"With this profound new knowledge, humankind is on the verge of gaining immense, new power to heal. It will revolutionize the diagnosis, prevention and treatment of most, if not all, human diseases." So declared then US President Bill Clinton in the East Room of the White House on 26 June 2000, at an event held to hail the completion of the first draft assemblies of the human genome sequence by two fierce rivals, the publicly funded international Human Genome Project and its private-sector competitor Celera Genomics of Rockville, Maryland (see Nature 405, 983–984; 2000).

Ten years on, the hoped-for revolution against human disease has not arrived — and Nature 's poll of more than 1,000 life scientists shows that most don't anticipate that it will for decades to come (go.nature.com/3Ayuwn). What the sequence has brought about, however, is a revolution in biology. It has transformed the professional lives of scientists, inspiring them to tackle new biological problems and throwing up some acute new challenges along the way.

Almost all biologists surveyed have been influenced in some way by the availability of the human genome sequence. A whopping 69% of those who responded to Nature 's poll say that the human genome projects inspired them either to become a scientist or to change the direction of their research. Some 90% say that their own research has benefited from the sequencing of human genomes — with 46% saying that it has done so "significantly". And almost one-third use the sequence "almost daily" in their research. "For young researchers like me it's hard to imagine how biologists managed without it," wrote one scientist.

“69% were inspired by the genome to become a scientist or change their research direction.”

The survey, which drew most participants through Nature 's print edition and website and was intended as a rough measure of opinion, also revealed how researchers are confronting the increasing availability of information about their own genomes. Some 15% of respondents say that they have taken a genetic test in a medical setting, and almost one in ten has used a direct-to-consumer genetic testing service. When asked what they would sequence if they could sequence anything, many respondents listed their own genomes, their children's or those of other members of their family (the list also included a few pet dogs and cats).
Some are clearly impatient for this opportunity: about 13% say that they have already sequenced and analysed part of their own DNA. One in five said they would have their entire genome sequenced if it cost US$1,000, and about 60% would do it for $100 or if the service were offered free. Others are far more circumspect about sequencing their genome — about 17% ticked the box saying "I wouldn't do it even if someone paid me".

Click for a larger version.

Nature 's poll also gauged where the sequence has had the greatest effect on the science itself. Although nearly 60% of those polled said they thought that basic biological science had benefited significantly from human genome sequences, only about 20% felt the same was true for clinical medicine. And our respondents acknowledged that interpreting the sequence is proving to be a far greater challenge than deciphering it. About one-third of respondents listed the field's lack of basic understanding of genome biology as one of the main obstacles to making use of sequence data today.

Sequence is just the start

Studies over the past decade have revealed that the complexity of the genome, and indeed almost every aspect of human biology, is far greater than was previously thought (see Nature 464, 664–667; 2010). It has been relatively straightforward, for example, to identify the 20,000 or so protein-coding genes, which make up around 1.5% of the genome. But knowing this, researchers note, does not necessarily explain what those genes do, given that many genes code for multiple forms of a protein, each of which could have a different role in a variety of biological processes. "The total sequence was needed, I think, to allow us to see that our one gene–one protein model of genetics was much too simplistic," wrote one respondent.

A decade of post-genomic biology has also focused new attention on the regions outside protein-coding genes, many of which are likely to have key functions, through regulating the expression of protein-coding genes and by making a slew of non-coding RNA molecules. "Now we understand," wrote another survey respondent, "that, without looking at the dynamics of a genome, determining its sequence is of limited use." Some big projects are under way to fill in the gaps, including the Encyclopedia of DNA Elements (ENCODE) and the Human Epigenome Project, an effort to understand the chemical modifications of the genome that are now thought to be a major means of controlling gene expression.

The biggest effects of the genome sequence, according to the poll, have been advances in the tools of the trade: sequencing technologies and computational biology. Technological innovation has sent the cost of sequencing tumbling, and the daily output of sequence has soared (see Nature 464, 670–671; 2010). "Deep sequencing technology is now becoming a staple of scientific research. Would this have occurred if it wasn't for the technological push required to finish the human genome?" read one response.

Data dreams, analysis nightmares

Cheaper and faster sequencing has brought its own problems, however, and our survey revealed how ill-equipped many researchers feel to handle the exponentially increasing amounts of sequence data. The top concern — named by almost half of respondents — was the lack of adequate software or algorithms to analyse genomic data, followed closely by a shortage of qualified bioinformaticians and to a lesser extent raw computing power. Other concerns include data storage, the quality of sequencing data and the accuracy of genome assembly. Commenting on the survey results, David Lipman, director of the US National Center for Biotechnology Information in Bethesda, Maryland, says that the worries about data handling and analysis were an issue even in the earliest discussions of the genome project. Perhaps, he suggests, "there's a sort of disappointment that despite having so much data, there is still so much we don't understand".

Click for a larger version.

Eric Green, director of the National Human Genome Research Institute (NHGRI) in Bethesda, says that the institute is well aware of the need for more bioinformatics experts, better software and a clearer understanding of how the differences between genomes influence human health. He says the institute is planning to publish in late 2010 its next strategic five-year plan for the genomics field. One possible solution to the computing challenge, which was discussed at an NHGRI workshop in late March, is cloud computing, in which laboratories buy computing power and storage in remote computing farms from companies such as Google, Amazon and Microsoft. The European Nucleotide Archive, launched on 10 May at the European Molecular Biology Laboratory's European Bioinformatics Institute in Cambridge, UK, will also offer labs free remote storage of their genome data and use of bioinformatics tools.

“13% have sequenced part of their own DNA.”

Given ten years' of hindsight and the current set of obstacles, it's no surprise that researchers now state somewhat modest expectations for what human genomics can deliver and by when. The rationale for sequencing and exploring the human genome — to revolutionize the finding of new drugs, diagnostics and vaccines, and to tailor treatments to the genetic make-up of individuals — is the same today. But almost half of respondents now say that the benefits of the human genome were oversold in the lead up to 2000. "While I do feel that the gains made by the human genome project are extraordinary and affect my research significantly, I still feel that it was overhyped to the general population," read one typical response. More than one-third of respondents now predict that it will take 10–20 years for personalized medicine, based on genetic information, to become commonplace, and more than 25% even longer than that. Some 5% don't expect it will happen in their lifetime. "Our understanding of the genome will not come in a single flash of insight. It will be an organized hierarchy of billions of smaller insights," says David Haussler, head of the Genome Bioinformatics Group at the University of California, Santa Cruz.

Green says that when the Human Genome Project was envisioned, scientific leaders of the day predicted that it would take 15 years to generate the first sequence, and a century for biologists to understand it. "I think they got that about right," he says. "While we still don't have all the answers — being a mere 10% of the way into the century with a human genome sequence in hand — we have learned extraordinary things about how the human genome works and how alterations in it confer risk for disease."
Haussler agrees. "All that happened in the first ten years is still just early rumblings of much more dramatic changes to come when we begin to truly understand the genome," he says.

http://www.nature.com/news/2010/100623/full/4651000a.html