a = 2
a <- 2
2 -> a

I’m pretty sure all these things are equivalent. The only subtlety I’ve noticed so far is that to set default options in function definitions you have to use `=`.

This redundancy drives me absolutely nuts as an R beginner. From a human point of view code has two main jobs: to be written and to be read. Now for the first job, having three different ways to assign a value to a variable might be nice? I’m not sure how, but I guess it would be nice sometimes to write a function, then remember that you hadn’t assigned it a variable and do it at the end instead.

For the second job  though, reading R is awful. Now I’ve heard horror stories about perl, so maybe it’s just that I’m used to readable code and that, in the big scheme of things, R is nice and readable. But yesterday, ploughing through someone’s metaprogramming, tearing at my hair as to the obtuseness of everything, I was foiled for about an hour having not realised that the fsking arrow could go both ways.

I think it highlights a big flaw in R. Sometimes it’s like no “destructive” design decisions have been taken since the original work of Ihaka and Gentleman. I spend so much time messing about trying to convert a data.frame of factors into a list of “vectors” of strings or something. It seems that no-one has ever said “OK, we need a way to assign a value to a variable: let’s pick one”. Instead three have emerged. There are two different types of classes (which only seems to have any real effect on the help files). There are loads of data types that are in regular use, with somewhat bastardised names (i.e. a ‘vector’, ‘matrix’, ‘list’, ‘array’ are all different things – a vector doesn’t seem to be a subclass of matrix, for example).

Now, I’m sure that the pluralities in R could maybe be held up to be a good thing. They probably encode subtleties that I don’t understand, and that my bitching is just me exposing my naivety about the language. And I know that R is descended from LISP and S and that its user base either grew up with R or with one of these other languages. Therefore things that are natural to this background will seem alien to someone like me with a MATLAB and Python background. And there’s backwards compatibility to consider and so on.

So don’t get me wrong – I think R is an amazing piece of work, and the communities around R are incredibly devoted and responsive. It’s kind of like the “first” three episodes of Star Wars: there’s an amazing film in there somewhere, it just needs some editorial decisions to be made. Maybe a new “Hadley” language will be born whereby R is completely re-API’d into a single language that only uses data.frames…

It’s about finding geneome-wide homologies using bioconductor. It turns out that bioconductor has a package called biomaRt which allows you to query the Ensembl databases with ease. (Ensembl stores gene information for a bunch of different organisms).

I thought I’d write down my solution here, as a sort of extended answer to my question on biostar, in case anyone trips up on the question there and would like a more complete answer. You’ll need to read the question before any of this code makes sense!

library(biomaRt)
gen_hs2mm <- function(affyids){
    ensembl_hs <- useMart(
        "ensembl",
        dataset = "hsapiens_gene_ensembl"
    )
    hs2mm_filters <- c(
        "affy_hg_u133a",
        "with_mmusculus_homolog"
    )
    hs2mm_gene_atts <- c(
         "affy_hg_u133a",
        "ensembl_gene_id"
    )
    hs2mm_homo_atts <- c(
        "ensembl_gene_id",
        "mouse_ensembl_gene"
    )
    # the names in these lists are arbitrary
    hs2mm_value = list(
        affyid=affyids,
        with_homolog=TRUE
    )
    # get the human genes and mouse orthologues
    hs2mm_gene <- getBM(
        attributes = hs2mm_gene_atts,
        filters = hs2mm_filters,
        value = hs2mm_value,
        mart = ensembl_hs
    )
    hs2mm_homo <- getBM(
        attributes = hs2mm_homo_atts,
        filters = hs2mm_filters,
        value = hs2mm_value,
        mart = ensembl_hs
    )
    # merge the two lists!
    hs2mm <- merge(hs2mm_gene,hs2mm_homo)
}

gen_mm2hs <- function(affyids){
    ensembl_mm <- useMart("ensembl",
        dataset = "mmusculus_gene_ensembl")
        mm2hs_filters <- c(
        "affy_mogene_1_0_st_v1",
        "with_hsapiens_homolog"
    )
    mm2hs_gene_atts <- c(
        "affy_mogene_1_0_st_v1",
        "ensembl_gene_id"
    )
    mm2hs_homo_atts <- c(
        "ensembl_gene_id",
        "human_ensembl_gene"
    )
    # the names in these lists are arbitrary
    mm2hs_value = list(
        affyids=affyids,
        with_homolog=TRUE
    )
    # get the mouse genes and human orthologues
    mm2hs_gene <- getBM(
        attributes = mm2hs_gene_atts ,
        filters = mm2hs_filters,
        value = mm2hs_value,
        mart = ensembl_mm
    )
    mm2hs_homo <- getBM(
        attributes = mm2hs_homo_atts,
        filters = mm2hs_filters,
        value = mm2hs_value,
        mart = ensembl_mm
    )
    mm2hs <- merge(mm2hs_gene,mm2hs_homo)
}
source('load_data.r')
# here immgen and cd4T are different experession set objects 
# from Bioconductor.
# immgen is mouse data (from the Immunological Genome Project) 
# and cd4T is human data
# cd4T can be found on GEO using the accessionID GDS785 
# See ref[1]
immgen <- load_immgen()
cd4T <- load_GDS785()
hs2mm <- gen_hs2mm(rownames(exprs(cd4T)))
mm2hs <- gen_mm2hs(rownames(exprs(immgen)))
colnames(hs2mm)[1] <- 'human_ensembl_gene'
colnames(mm2hs)[1] <- 'mouse_ensembl_gene'
# the final thing is to merge the two tables to make a single 
# table containing all the probes that are homologous, along 
# with their respsective EnsemblIDs
homol <- merge(hs2mm,mm2hs)

[1] Lee MS, Hanspers K, Barker CS, Korn AP et al. Gene expression profiles during human CD4+ T cell differentiation. Int Immunol2004 Aug;16(8):1109-24. PMID: 15210650

Rprof()

Rprof("profile.out")

Rprof(NULL)

> summaryRprof("method1.out")

$by.self

self.time self.pct total.time total.pct

"strsplit"          4.08     94.0       4.10      94.5

"+"                 0.12      2.8       0.12       2.8

"is.character"      0.12      2.8       0.12       2.8

"as.logical"        0.02      0.5       0.02       0.5

$by.total

total.time total.pct self.time self.pct

"strsplit"           4.10      94.5      4.08     94.0

"+"                  0.12       2.8      0.12      2.8

"is.character"       0.12       2.8      0.12      2.8

"as.logical"         0.02       0.5      0.02      0.5

proftools

plotProfileCallGraph("profile.out")

profr

out <- profr(code_to_profile)

> head(out)

f level time start  end  leaf source

8      example     1 0.44  0.00 0.44 FALSE  utils

9  <Anonymous>     2 0.04  0.00 0.04 FALSE   <NA>

10     library     2 0.02  0.04 0.06 FALSE   base

11      source     2 0.38  0.06 0.44 FALSE   base

12  prepare_Rd     3 0.04  0.00 0.04 FALSE   <NA>

13        %in%     3 0.02  0.04 0.06 FALSE   base

> summary(out)

f          level             time          start

match        : 7   Min.   : 1.000   Min.   :0.02   Min.   :0.0000

%in%         : 6   1st Qu.: 5.000   1st Qu.:0.02   1st Qu.:0.1200

is.factor    : 6   Median : 7.000   Median :0.02   Median :0.1600

eval.with.vis: 6   Mean   : 8.113   Mean   :0.04   Mean   :0.1685

inherits     : 5   3rd Qu.:10.500   3rd Qu.:0.02   3rd Qu.:0.2200

<Anonymous>  : 3   Max.   :19.000   Max.   :0.44   Max.   :0.4200

(Other)      :82

end            leaf            source

Min.   :0.0200   Mode :logical   Length:115

1st Qu.:0.1400   FALSE:99        Class :character

Median :0.2000   TRUE :16        Mode  :character

Mean   :0.2085   NA's :0

3rd Qu.:0.2500

Max.   :0.4400

Here I should say that `code_to_profile` was the standard `example(glm)`. One can also use `plot()` or `ggplot()` to plot the ‘call tree’. I’m not quite sure what a ‘call tree’ is, and to be honest the (adimittedly pretty) graph that `ggplot(out)` produces doesn’t enlighten me a great deal. That’s OK though – all I really wanted was the time spent.

A quick point about Hadley’s code, that I think highlights something wrong with R in general (and right with Python) is that in `profr`, from a user perspective I don’t really have to learn anything new to make it work. I apply a function (handily called `profr`) to some code and I get back a `data.frame` on which I can use very standard tools in order to investigate. I don’t have to write to a file (or generate a temp file to write to or anything), I don’t have to learn to stop and start hidden things, or that to stop profiling I pass `NULL` to a function that started profiling. I don’t have to use a whole new tool to parse what is basically a table. And to plot I can just use `plot` rather than having to dissapear down the (admittedly fun) hole that is graphviz. It feels like every new thing I come across in R means learning a whole API and set of concepts and object structure. (Admittedly ggplot is the total opposite of this, but hey I guess that was the point)

I’m going to define the model again. Notation is important:

• : state transition probability
• : duration density
• : segment density
• : initial state distribution

Some things to note.

1. The state transition probability doesn’t really care how long the segment associated with q_i is. This means that the transition is invariant under changes to which kind of means the whole thing is still Markovian, though this enough for people to start using terms like ‘semi-Markovian’ which is kind of confusing.
2. The duration of the state at time is dependent on the state at time . This implies that:
3. The segment starts at time and lasts for time steps. This means that the last time index in the ‘current’ segment is .

What we’d like to do is write this out as a properly specified graphical model, but it turns out that this is pretty hard to do for the general case. As a first crack I tried introducing a counting variable . Unfortunately it doesn’t qutie make it, but was a useful learning experience. So, with a counting variable we can write the alternative model out as follows:

1. : counting variable transition distribution
2. : state transition probability conditioned on the counting variable at the previous time point
3. : segment density
4. : initial state distribution

The counting variable either decrements or resets, such that and . Here is simply the duration density, such that when the counter resets it starts off at a new duration.

Similarly, the state variable either stays the same or transitions depending on the counting variable. So and .

So we’ve swapped some awkardness in having two ideas of time’s passage – one for the underlying state and one for the observations – with some awkwardness of defining these conditional densities.

As it stands, though, we’re still stuck with a variable topology – the introduction of the counting variable hasn’t really changed the fact that the segment density changes dimension dependent on . If each observation was independent given the state then the counting variable would be all we need to get a nice regular graph.

The observations aren’t independent though! This is the big difference between Segment HMMs and what are known as explicit-duration HMMs. Hence to be able to draw a graph we must make some assumptions on the form of the output density. Murphy actually states this in one of his technical reports, but it seems to have taken a couple of months for it to actually sink in. Next step, then, is a graphical model for a th order explicit duration switching AR model! I think I’ll save that for the next post.

In [253]: 8**14
Out[253]: 4398046511104L

In [254]: 8**(numpy.int32(14))
Out[254]: 0

I would feel guilty about this if I had to specify the type of numbers I was using, but this is a dynamic programming language! How the hell am I supposed to know that raising an int by a numpy.int32 will force the result to also be an numpy.int32 (and therefore (silently) 0 because int32 can’t hold my number) whereas raising an int by an int will result in a long? Why didn’t the int32 become a long also? Or an int64 or something that could hold my number?

Gah!

delta[t][q] = max(
    [delta[tau][j] +
        pylab.log(
            pylab.array([
                output_dist(Q=q,L=(t-tau),Y=Y[tau+1:t]),
                duration_dist(Q=q,L=(t-tau)),
                transition_dist[q,j]]).prod())            
    for j in self.state_range])

Is this bad? The above is the max of a list. The list is made up using a list comprehension, where each element is the log of a product of a 1D array plus a bit. Each element of each array is a call to a function associated with my model. The trouble is, if I break it down into some for loops, then I start having to invent temporary names for my variables, which seems clunky.

Any opinions?

M

If you are young, do not read this book; it is not fit for you;
If you are old, throw it away; you have nothing to learn from it;
If you are unambitious, light the fire with it; you do not need its guidance.

But, if you are neither less than twenty-five years old, nor more than thirty;
And if you are ambitious withal, and your spirit hankers after academic politics;
Read, and may your soul (if you have a soul) find mercy!

Here, I just wanted to link to some coverage of the trial, which is illuminating. There’s Spectrical which is the sort of `official’ coverage, then there’s also TorrentFreak’s coverage which has an especially moving piece about one of the wife of one of the witnesses for the defence being sent a load of flowers. Then there’s also the mainstream coverage in the Guardian, in Cory Doctorow’s column Digital rights, digital wrongs. It all goes to show the lengths people go in order to restrict change. I wonder if I’ll be as motivated as the record companies when something I don’t want to change comes along? Hopefully, at least, I’ll be better informed than these dinosaurs…

There are three things we need to do to make beamer slides like this that don’t look like those endless maths-y beamer slides that everyone seems to like for some reason.

1. Get rid of all the junk on the top and the bottom! No-one can see it anyway!
2. Make the colours of the presentation very pretty.
3. Make the listings use a better font than courier, and also use pretty colours!

1: get rid of all the junk

This is an awesome command. It’s (maybe) the longest single command ever in the world, sort of like that village in Wales with the huge name:

/usetheme{default}
/beamertemplatenavigationsymbolsempty

So now, your beamer presentation is nice and clean and happy.

2: Make Pretty Colours

This is just like knowing the magic words. We define a colour, then we assign it to a magic word. So lets define three colours, a background, a foreground and a title colour:

/definecolor{fore}{RGB}{249,242,215}
/definecolor{back}{RGB}{51,51,51}
/definecolor{title}{RGB}{255,0,90}

These colours are awesome but obviously you can choose your own if you’re a philistine and disagree. I’ve chosen the names (like fore), but they can be whatever you like. You can use other colour specifications instead of RGB, though using RGB for a presentation makes sense to me because it’s probably never going to be read seriously on paper. Now we need to tell beamer to actually use them:

/setbeamercolor{titlelike}{fg=title}
/setbeamercolor{normal text}{fg=fore,bg=back}

there are millions of these magic words like ‘titlelike’ but no-one will tell you what they are.

3: Make the listings pretty!

To include code listing (like Python) we use the listings package because it’s brilliant. We need:

/usepackage{listings,bera}
/definecolor{keywords}{RGB}{255,0,90}
/definecolor{comments}{RGB}{60,179,113}
/lstset{language=Python,
keywordstyle=color{keywords},
commentstyle=color{comments}emph}

The packages are for the listsings (clearly) and bera is the font. It’s secretly Bitstream Vera but they had to change the name or something because the world is messed up! Anyway bera is a lovely font in my honest opinion and you should use this and not courier which is ugly and very last century.

The colours are for comments and keywords which you’ll see in a sec, be patient.

The funny lstset command tells the listings package some things. The first is that we’re using Python. There are other languages, I think, but why we wouldn’t be using Python is beyond me. Then we ask listings to make our keywords the colour we defined above (pink!) and our comments to be the comment colour and also emphasised!

Finally, we must make a sample page to demonstrate the lstlisting environment, and because there’s one more important thing that is NOT OBVIOUS! Here’s a sample page:

/begin{frame}[fragile]
/frametitle{A Page}
/begin{lstlisting}
def list_of_vectors(n,T):
 return [matlib.zeros((n,1)) for t in T]
/end{lstlisting}
 Here is an equation:
 /begin{equation}
 x_t = Ax_{t-1} + w_t
 /end{equation}
/end{frame}

The thing to note is the [fragile] command. Here’s what it does. It means that the weird error that happens if you don’t include it DOESN’T HAPPEN! Enjoy!