Sunday, 6 February 2011

Drupal error - warning: Cannot modify header information

Upon uploading my site from the local to the remote server an error (warning) was received when attempting to run update.php. If I ignored the warning and continued using the site I would get a blank white page every now and again on the administration pages although it did carry out the requested operation. I was having none of it and it was something that needed to be solved.

The error message I got:

warning: Cannot modify header information - headers already sent by (output started at /home/{username}/public_html/drupal/sites/all/modules/{modulename}/{modulename}.module:1) in /home/{username}/public_html/drupal/includes/common.inc on line 148.

It took a while to figure out what the problem was. The first thing to say is that the line and file number it says is causing the problem usually is the file and line number causing the problem. So I opened up the custom module I'd written and here was what it had in the first two lines:

<?php
// $Id$

Nothing strange about that. And there was no white space preceding the opening of the php code tag.

To cut a very boring story short, the problem turned out to be that I'd saved the file as UTF-8 and before the opening tag the text-editor had stuck in an invisible UTF-8 marker character which was being sent to the browser before the headers and thus causing the error.

The solution is to change the text file to ASCII (or ANSI as some text-editors call it) or find another way to remove that preceding character(s?) and if any preceding characters show up in the ASCII file then delete them and save the file.

Tuesday, 18 January 2011

Buggy JavaScript Regex and parsing speeds in Firefox, Chrome, Opera and IE6

I was working on a little section of a small javascript application I'm writing when I came across this issue and I've decided it's worthy of documenting in the public domain.

The problem: This Javascript fragment of a regular expression replace statement requires exponential run-time to parse a string the longer it gets:

input.replace(/([^\s]+)*&radic;\{(.*?)\}/gi, '<sup>$1</sup>&radic;<span class="ol">$2</span>')

It replaces an example string - "&radic;{123}" - with this - "<sup></sup>&radic;<span class="ol">123</span>".

But with each additional character added inside the curly braces the time taken to replace it increases exponentially. I have 4 browsers installed on my system - Firefox 3.6.13, Google Chrome 5, Opera 11 and Internet Explorer 6. I haven't upgraded IE6 to the latest version because I use it to test how websites look for people using business/government computers whose technology department haven't upgraded the browsers in 10 years! (Shame on you!)

The results are surprising to say the least. The table and charts below show how long it took (in milliseconds) for each browser to parse a string with 1 to 20 characters inside the curly braces and output it to the screen.


 
Str Length Time FF Time CH Time OP Time IE6
1 5 0 0 0
2 5 1 0 0
3 5 0 0 0
4 6 1 0 0
5 8 1 0 0
6 12 1 0 0
7 19 1 0 0
8 33 3 0 0
9 60 5 0 0
10 115 8 0 0
11 224 17 0 0
12 448 35 0 15
13 892 71 0 0
14 1773 143 0 0
15 3544 286 0 15
16 7113 576 0 0
17 14167 1146 0 31
18 28356 2287 0 47
19 58691 4593 0 94
20 113196 9159 0 188




The results are very surprising. At 13 characters long, the browser suddenly starts becoming very slow and a pattern becomes apparent - that the time taken to execute the regex statement is taking exponentially longer.

IE6 only really starts doing this at 17+ characters but is, surprisingly, faster that both Firefox and Chrome! IE6 faster than Chrome! OMG! Oddly, IE6 spontaneously spikes to 15ms every now and again before reaching 17 characters whether 1 or 16 characters are entered.

Another thing to note is that Chrome is more than 10x faster at doing this than Firefox. The JagerMonkey needs to go on a JagerDiet.

What is even more surprising than that is that execution time in Opera is very close to 0ms and does not rise at all! In Opera this operation isn't exponential! I tested it with over 100 characters and performance in Opera is amazing going above no more than 1ms. (Much respect to the Opera Dev team and their Carakan engine!)

When I made this mistake I was really surprised why it took so long but quickly realised my mistake. I can't really explain what's going on under the hood of the regex engine (because I don't really know in that much detail) but I identified the bug in my regex and solved the problem, which is the single asterisk (*) highlighted:

input.replace(/([^\s]+)*&radic;\{(.*?)\}/gi, '<sup>$1</sup>&radic;<span class="ol">$2</span>')

and which should be replaced with a question mark to solve the problem.

input.replace(/([^\s]+)?&radic;\{(.*?)\}/gi, '<sup>$1</sup>&radic;<span class="ol">$2</span>')

What the regex code is doing, and don't blame me if I'm wrong - I'm no expert - is that when you use the * operator it checks to see if the pattern before it exists and it does this by trying to match the whole length of the string with the pattern [^\s] (which means any character but empty space) and then it reads the string one character at a time until it reaches the end and then it back tracks the length of the string to the beginning. It's a greedy operator which tries to find the longest possible matching string. After that, the rest of the regex is interpreted accordingly.

When you replace * with a ? it checks to see if there's anything before &radic; and if there isn't it simply moves on to carry out the rest of the regex. It only checks one character minimum before it moves on while using * checks the length of the string with each character it moves along. That's why using * becomes slow the longer the string gets.

I think the reason Opera is not affected by this coding bug is because their internal regex engine identifies the bug and realizes that it's stupid for the greedy operator to check the whole string when the next regex pattern after it matches instantly so internally it treats the * operator like a ? operator, thus reducing execution time. They have very clever people at Opera. Mozilla and Google should really think about incorporating this performance increasing technique into their own engines.

Saturday, 8 January 2011

Dolly the sheep lives

...on through her clones.


Dolly the sheep has been reborn. Four clones have been made by the scientist behind the original research.
The quads, which have been nicknamed ‘the Dollies’, are exact genetic copies of their predecessor, who was put down seven years ago.

Wednesday, 5 January 2011

SAS Macro functions and returning values and a bit about macro variable type casting

I wrote an article about this subject before but I was quite new to SAS and it was quite confusing so I'm writing it again better and clearer this time.

The SAS macro programming language is a strange language lacking data types we usually associate with strongly or even weakly typed programming languages. SAS macro variables are intrinsically treated as strings. The equivalent command for casting a type from one to another is an input() command. For example, if I have a string that contains the month and year (MMMYY), then I can convert it to a SAS date format like this:

%let mmmyy = JAN11;
%let datum = %sysfunc(input(&mmmyy., MONYY.)); 
*the second argument defines the format of the data (&mmmyy.) that you want to cast into SAS format. &datum. now holds the SAS date equivalent of 01/01/2011;

Datum now holds the SAS date format of the first of January 2011. SAS macro variable values like the integer date stored in datum are intrinsically stored as strings, so you can't just do anything with them. In order to treat them as an actual integer and do any mathematical operations on them you need to use eval() like so:

%let yesteryear = %eval(&datum. - 1); *holds the SAS date for 31/12/2010;

One problem often encountered with SAS macro variables is when working with dates. Often, people define a date like so:

%let bankholiday = "03JAN11"d;

This is acceptable, but if you try to do any mathematical operations using that variable outside of a data step it will throw an error message and literally show that you tried to carry out an operation on the string ' "03JAN11"d '. The best ways to convert such a date to the SAS date format is to use either intnx() or define the date using mdy():

%let bankholiday = %sysfunc(mdy(01, 03, 11));
or
%let bankholiday = %sysfunc((intnx(day, "03JAN11"d, 0)));

I've yet to figure out how to detect the type of the macro variable before I do anything with it, so I advise you to inform other programmers about the type of the variable your code works with before you let someone else use your code.

Now. When it comes to macro functions, SAS also proves to be strange when compared to other languages. What is a very natural and normal way of working in a procedural/Object Oriented language appears to be missing in SAS. I've yet to see an official documented example that shows a macro returning a value, but it is possible. I doubt I'm the first person to discover or use this method.

What is odd about macro functions that return values is that you should NEVER use single line comments inside the body of the macro otherwise it throws an error! Always use the multi-line comments like /* this comment */. And there is no "return" command preceding the variable to make it return but there is a %return statement and all this does is stop the execution of the rest of the code and jump out of the macro. The way to return a value is to just write the the name of the variable holding the value you wish to return, without a semicolon on the end.

Anyway. Here's an example bit of code that lets you ask if a date is a weekend or not. If it is a weekend it returns 1 else it returns 0.

/**
* %isWeekend(datum)
* Tells you if a date is a weekend or not by returning 1 (true) or 0 (false). If you leave it empty it will check todays date.
* Usage:
%let datum = %sysfunc(mdy(01,01,2011));
%put %isWeekend(&datum.); *shows 1 (true);
*
* @return    boolean    If the date you supplied is a weekend it will return 1, otherwise 0.
* @param    date    date    The date you want to check. If you leave this empty it will use todays date
* @date 20110105
* @author Ahmad Retha
**/

%macro isWeekend(date);
    %if date=  %then %do;
        %let date = %sysfunc(today());
    %end;

    %let wd = %sysfunc(weekday(&date.)) ; /*find the weekday of the date given. Sunday=1, Saturday=7*/

    %let iwe = 0; /*set the variable we wish to retune to 0 (false) initially*/
    %if &wd.=1 or &wd.=7 %then %do;
        %let iwe = 1;
    %end;

    /* return iwe (1 or 0) */
    &iwe.
%mend;


Allow me to explain what this code does. The first part of the code, the %if statement, checks to see if a date argument was supplied and if it isn't it uses today's date. This behaviour of setting an empty parameter to a default value makes our macro function more robust and easy to use - it is a recommended practice.

The next section assigns the weekday, Sunday, Monday, Tuesday through to Saturday to the variable &wd. as a number from 1 to 7, as there are 7 days in a week. In SAS, the week starts on Sunday, 1, and ends on Saturday, 7.

Next we create a macro variable, iwe, which holds the value we wish to to return. We want to return 1 (true) if the date is a weekend, or 0 (false) if it's not. Initially we set the value to 0 (false) as most days of the week return false.

The next part checks if the weekday is Sunday (1) or Saturday(7) and if the date's day is a weekend it sets iwe to 1 (true). If it's not a weekend then iwe already holds 0 (false).

Finally, we return the value held in iwe (either 1 or 0) by just writing the variable &iwe. by itself - note that you should NOT put a semicolon on the end otherwise it will throw an error. There is no "return" command - just put the variable name on a line of its own.

That's it. Easy right?

You can write many really useful macro functions that you will use often and stick them into a file and make a library of useful functions to %include and use in your projects. For example, for my job I wrote a macro called %isWeekend(date), like the one above (though as I'm writing this from home I couldn't just copy and paste and I wrote the above on the fly from memory), another macro called %getNext(day) which returns the date of the next weekday you give it, %getLast(day) which is similar but looks backwards, %isHoliday(date) and %getLastWorkday(date). I stuck those into a SAS file, a library, and now include it into my other scripts when I need the functionality. This approach promotes code re-use and makes coding quicker and easier and as there is less duplicate code it is easier to maintain. Naturally, all my code is highly documented and I recommend you comment up your code as well.

I hope this little tutorial proves useful.

- Ahmad Retha

Friday, 19 November 2010

Pretty Microscope Pictures: Olympus Bioscapes Digital Photocompetition

1st Place 2010 Olympus BioScapes Digital Imaging Competition: Eyes of daddy longlegs. A frontal section of Phalangium opilio eyes. The lenses, retinasand optic nerves are visible. The image is a depth color-coded projection of a confocal image stack. (Igor Siwanowicz, Max Planck Institute for Neurobiology, Munich, Germany)



Monday, 18 October 2010

Unique value row count in SAS

You may have come across the need to count unique values in your dataset and put that as a row in your dataset/table - something like this:

id, name
1, Alan
2, Brad
2, Brad
3, David
3, David
3, David
4, George
5, Joe
6, Steven
7, Zed
7, Zed

The solution involves using the LAG() function. Lag is like a look-back function and for each row that is processed it looks back a row and fetches the value. So, if I were at observation (row) 2 and did LAG(name), it would return the value of "Alan". The retain function puts an initial temporary value in a variable so I can use it in my processing.

Anyway, here's the code:

data pupils;
    input name $;
    datalines;
Alan
Brad
Brad

David
David
David
George
Joe
Steven
Zed
Zed
;
run;


data store;
    set pupils;

    prevname = lag(name);

    format id BEST12.;
    retain id 0;
     if name^=prevname then id = id + 1;

    drop prevname;
run;

Sunday, 10 October 2010

Biggest Genome ever? It's plant vs amoeba

A few days ago there was an article in Science magazine that they had discovered a plant (Paris japonica) with the largest genome ever found with 149 billion base pairs...

Except, I remembered there's an amoeba (Polychaos dubium - what a bloody weird name by the way!) with an even bigger genome of 670 billion base pairs.

So it got me wondering - how could Science magazine get it so wrong?! Well, this comment on the science mag site gave the reason why they would reject dubium:


"While this and some other Amoeba have been reported to have such very large genomes, some caveats regarding their reliability are perhaps in order.
The measurement for Amoeba dubia and other protozoa which have been reported to have very large genomes were made in the 1960s using a rough biochemical approach which is now considered to be an unreliable method for accurate genome size determinations. The method uses whole cells rather than isolated nuclei and thus will include not only DNA from the mitochondria but also any DNA in engulfed food organisms. Also some of the species are multinucleate.
The accuracy of the genome size estimates are also called into question given that a related species, Amoeba proteus, which was reported to have a genome size of 300 pg was more recently shown to be an order of magnitude smaller (34 - 43 pg DNA per cell). Like the situation for dinoglagellates (see below), the genomes of these Amoeba are clearly large but to know just how big requires their genome sizes to be estimated using modern best practice techniques available today. Only then will it be possible to know just how their genomes compare in size to those of Paris japonica." {Quoted from Ilia}
The reasons given in the quote can be ignored except for the clincher which is in bold. If we assume the same experimental conditions and level of accuracy, dubium will be found to have about 69pg DNA.

Game over, the plant wins.

Wednesday, 29 September 2010

Difficult SAS issue - restructuring datasets with proc transpose

I've been trying to solve this problem for a few hours and have tried many different things. In the end, it was two sets of a process that solved the problem with proc transpose. It's worth documenting so I'm blogging it.

The Problem:

I have an example dataset that contains observations like so:


(dataset: dayobs)
days_range,    value,    count
0-9,    300,    6
60-69,    250,    4
300-309,    76,    1

I wanted it to show all the ranges with value and count set to zero if it wasn't already set:

days_range,    value,    count
0-9,    300,    6
10-19,     0,     0
20-29,     0,     0
30-39,     0,     0
40-49,     0,     0
50-59,     0,     0
60-69,    250,    4
... etc ... 
290-299,    0,    0
300-309,    76,    1

That was easy, simply make a dataset that lists the day_range from 1-10 ... 390-400.

data ranges;
    input days_range $;
    format days_range $7.;
    datalines;
0-9
10-19
... etc ...
    ;
run;

and then merge both the datasets - dayobs and ranges - to create a new dataset called daysrange:

data daysrange;
    merge ranges dayobs;
    by days_range;
run;


OK. Now, I wanted to transpose it into a single row so I can insert that into a big dataset that logs the changes every day. I want to make it look like so:

v1_9, c1_9, v10_19, c10_19, v20_29, c20_29, ...etc... v60_69, c60_69, ...etc
300, 6, 0, 0, 0, 0, ...etc... 250, 4, ...etc

It might seem obvious what to do but I tried to be clever and did this:

data rowifieddaysrange;
    set daysrange;


    if days_range='0-9' then do;
        c1_9=count;
        v1_9=value;
    end;
    else if days_range='10-19' then do;
        c10_19=count;
        v10_19=value;
    end;
 etc...


keep c1_9 v1_9 c10_19 v10_19 etc...;
run;

and this is what the data looked a bit like in the end:

0, 0, 0, 0, 0, 300, 0, 0, 0, 0, 0, 0, 6
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
250, 0, 0, 0, 0, 0, 4, 0, 0, 0, 0, 0, 0
etc.

Basically, it had the correct columns, but they were spaced out across different rows. No matter what I tried - I look at all the different sas procs and their options - I was unable to collapse/compact it into one row... if anyone knows the solution to this problem I'd really like to see it please. Ta.

OK. So guess what the solution is... It turns out you need to use two sets of proc transpose and a couple of merges, to deal with each variable individually - then finally merging the two separate datasets together at the end! :


*transpose the daysrange dataset first by count;
proc transpose data=daysrange out=tdaysrange(drop=_:) prefix=c;
    id days_range;
    var count;
run;
*now transpose the table with all the categories;
proc transpose data=ranges out=tranges(drop=_:) prefix=c;

    id days_range;
run;
*now merge them;
data cdataset;
    merge tranges tdaysrange;
run;


*transpose the daysrange dataset next by value;
proc transpose data=daysrange out=vdaysrange(drop=_:) prefix=v;
    id days_range;
   var value;
run;
*now transpose the table with all the categories;
proc transpose data=ranges out=vranges(drop=_:) prefix=v;
    id days_range;
run;
*now merge them;
data vdataset;
    merge vranges vdaysrange;
run;


*now join the the value and count datasets into one dataset!
data rowifieddaysrange;
    merge vdataset cdataset;
run;

And finally it's all in one row that looks something like this:


v1_9, c1_9, v10_19, c10_19, v20_29, c20_29, ...etc... v60_69, c60_69, ...etc
300, 6, 0, 0, 0, 0, ...etc... 250, 4, ...etc

That's a lot of work for something that aught to be straightforward!

Monday, 13 September 2010

Got a new book: Pro HTML5 Programming

I got this book a few days ago and so far so good. I've read through chapter 1 and made my first HTML5 compliant page. The book is not written for absolute beginners - it assumes you've done HTML4/XHTML1, CSS2-3 and have a good grasp of Javascript - but I have these so it's all good. I'll report back once I've read the whole book and tried everything.