What to do when your Hessian matrix goes balmy !!!

So you ran some mixed models and got some balmy messages in return? Are these those messages?

“The Hessian (or G or D) Matrix is not positive definite. Convergence has stopped.”

OR

“The Model has not Converged. Parameter Estimates from the last iteration are displayed.”

Then this post is for you. First let’s try to understand right from the basics of matrix algebra itself. Before going into the Hessian matrix let’s take a detour into the murky world of mixed models and see what’s going on there and how come we get a thing called Hessian matrix !

A linear mixed model looks like this (from Wikipedia):

\boldsymbol{y} = X \boldsymbol{\beta} + Z \boldsymbol{u} + \boldsymbol{\epsilon}

where

  • \boldsymbol{y} is a known vector of observations, with mean E(\boldsymbol{y}) = X \boldsymbol{\beta};
  • \boldsymbol{\beta} is an unknown vector of fixed effects;
  • \boldsymbol{u} is an unknown vector of random effects, with mean E(\boldsymbol{u})=\boldsymbol{0} and variance-covariance matrix \operatorname{var}(\boldsymbol{u})=G;
  • \boldsymbol{\epsilon} is an unknown vector of random errors, with mean E(\boldsymbol{\epsilon})=\boldsymbol{0} and variance \operatorname{var}(\boldsymbol{\epsilon})=R;
  • X and Z are known design matrices relating the observations \boldsymbol{y} to \boldsymbol{\beta} and \boldsymbol{u}, respectively.

Let’s focus on the variance-covariance matrix G or some software refer to it as the D. It is the a matrix of the variances and covariances of random effects. The variances are the diagonal elements and the off-diagonal ones are covariances. So if you have a mixed model with two random effects say, a random intercept as well as the random slope, then we would have a 2 X 2 G matrix. The variances of the intercept and slope terms would be in the diagonal whereas the off-diagonal would contain the covariances.

Remember this G matrix is a one which contains variances so mathematically speaking, the matrix should be positive definite (for a matrix to be so, diagonal elements should be positive). As variances are always positive, hence this makes sense.

The Hessian matrix referred to in the warning messages you got is actually based on this G matrix which is used to calculate the standard errors of the covariance parameters. So, the algorithms which calculate them would be stuck and won’t be able to find an optimised solution if the given Hessian matrix calculated for the model doesn’t have positive diagonal elements.

So, the whatever results you may get out of the mixed model wouldn’t be correct or trustworthy. What that means is that the model which you specified couldn’t estimate parameters etc with your data. Some might choose to ignore this warning and move ahead, but my request is please don’t !!! This warning is indeed important, and NO the software doesn’t have a vendetta against you/your project.

 The next step is obviously to ask what can you do in this circumstance and what might be the solution. One method might be to check the scaling of your predictor variables in the model. If they are highly different then that can be a good reason why the software has trouble in variance calculation. So, just a change in scaling of the predictors can solve your problem here.

Another method is when some covariance estimates are 0 or have no estimates at all or don’t produce the standard errors at all (SPSS usually does this, and produces blank estimates). Now don’t go on ignoring this variable, as something is fishy with the model itself. For if the best estimate of your variance is zero, this means there is zero variance within your data for the effect under consideration. For example, you have introduced a random slope for that effect, but in actuality the slopes do not differ across the subjects of your study in that effect and possibly a random intercept component might well explain all the variation.

So just remember when something like this happens, the best possible solution for you to do is to respecify the random components in your model and that could be about removing a random effect. Sometimes you might feel or have been told that a given random effect has to be introduced because of the design of the study, you wouldn’t find any variation in the data. Another thing, is that you could specify perhaps a simpler covariance structure which contains lesser number of unique parameters to be estimated.

Let me give an example to highlight this situation:

A researcher wants to understand the behavioural responses of rats living in their cages in a lab building by doing standard behavioural tests. Since the cages are situated in different floors, in different corners in the lab building, the researcher wanted to see if before experimentation is there any change in their responses to simple behavioural tests. Now let’s suppose there are 1000 rats in each floor and there are 10 floors in the building. That makes it 10000 rats which would be a huge number to study all of them individually. So, we take samples of rats within each floor and the design indicates including a random intercept component for each floor, to account for the fact that rats in the same floor may be more similar to each other than would be the case in a simple random sample. So, if this is true, we would likely want to estimate the variance of behavioural responses among floors.

But we know that modern animal facility guidelines calls for rigorous protocols to be followed and because of that rats are kept in similar cages with as similar conditions as possible. Then we can easily see here that there wouldn’t be much variance in the behavioural responses among the floors. This leads to the scenario i put up before, i.e., variance for floors = 0 and the model would be unable to uniquely estimate any variation from floor to floor, above and beyond the residual variance from one sampled rat to another.

Finally, another option is to use a population averaged model instead of a linear mixed model. As population averaged models don’t have any random effects, but do contain the correlation of multiple responses by the sampled individuals.

For more, read these —

  1. West, B. T., Welch, K. B., & Galecki, A. T. (2007). Linear mixed models: A practical guide using statistical software. New York: Chapman & Hall/CRC
  2. Linear mixed models in R- http://www.r-bloggers.com/linear-mixed-models-in-r/
  3. Model Selection in Linear Mixed Models- http://arxiv.org/pdf/1306.2427v1.pdf
  4. Hessian matrix in statistics- http://www.slideshare.net/FerrisJumah/hessin

Why do we love? An empirical test…

Archetypal lovers Romeo and Juliet portrayed by Frank Dicksee

Yeah love is indeed a mysterious thing and has always captured our imaginations. One of the most famous tragic love stories was the Romeo and Juliet by William Shakespeare. Tragic in the sense that the main protagonists die at the altar of their own love. So, what makes love so special? Or indeed as a biologist i ask what is the need for love. Just look what goes into the love process. Endless dating games, elaborate preparations, endless flirtations, also many humiliations and finally if you are lucky the one acceptance.

But wouldn’t it be simpler to just think about procreation alone, i.e., reproduce for the sake of propagation?? Since, evolutionary struggles dictate that there exists differential reproduction and hence propagation of one’s own genes is the thing which ultimately matters. So, then why do we go for this protracted cycle?

To answer this question albeit in an indirect way authors – Malika Ihle, Bart Kempenaers and Wolfgang Forstmeier all from Department of Behavioral Ecology and Evolutionary Genetics, Max Planck Institute for Ornithology, Seewiesen, Germany conducted a remarkable experiment. The results of this experiment was published recently in PloS Biology – Fitness Benefits of Mate Choice for Compatibility in a Socially Monogamous Species.

As we know that to actually conduct a cost/benefit analysis of love is easier said than done and there would be innumerable ethical concerns regarding the bounds of experimentation with humans. This present study however, used a model animal in an elegant experiment which was designed to find the reproductive consequences of mate choice.

The Experiment

The model species used here was the –zebra finch (Taeniopygia guttata, a native bird of Australia).

Adult male at Dundee Wildlife Park, Murray Bridge, South Australia

Adult male at Dundee Wildlife Park, Murray Bridge, South Australia

They started off with a population of 160 birds that had recently been isolated from the wild, and then set them up on a sort of speed-dating session, with groups of 20 females to choose freely between 20 males (See figure 1 below). Once the birds had paired off, half of the couples (the “chosen” or C group) were allowed to live happily ever after. For the other half, however, the authors intervened like overbearing Indian parents, and split up the happy pair to forcibly pair them up with other broken-hearted individuals (the “non-chosen” or NC group). The bird couples of both C and NC groups were then left in aviaries to breed. The authors then measured the couple’s behaviour and the number and paternity of dead embryos, dead chicks and surviving offspring.

Experimental Design

Figure 1: Experimental Design

Results

Relative fitness estimates (mean ± SE) of males (n = 84) and females (n = 84) from chosen and non-chosen pairs

Figure 2: Relative fitness estimates (mean ± SE) of males (n = 84) and females (n = 84) from chosen (C) and non-chosen (NC)pairs

The first batch of results is elegantly shown in the figure above.The overall reproductive fitness (measured as the final number of surviving chicks) was 37% higher for individuals in chosen pairs than those in non-chosen pairs. But since reproductive fitness is the sum total of different effects which add up to the total number of offspring produced, it’s vital to look at those parameters and understand the mate choice in C group affected the fitness. To start off the authors noted that both the C and NC group laid similar number of eggs which suggests that their initial investment towards egg laying is not affected by the group they are in and also oblivious to the mismatched mate picked up by the authors. But the nests of NC group had almost three times as many unfertilized eggs as the chosen ones, and a greater number of eggs that were neglected (either buried or lost).

The authors in their earlier studies had known this fact that embryo deaths happened mainly due to genetic incompatibility between the parents, however the egg hatching related deaths happened due to behavioural incompatibility. So, the next step was to compare these two phenomena in the two C and NC groups. They found that though the embryo mortality was similar in both the groups, however mortality of the hatched chicks was comparatively
higher in the NC couples. This suggests that its the behavioral incompatibility
between the non-chosen (NC) parents, and not genetic incompatibility which might be the driving factor behind the observed reduction in overall fitness (Fig. 3, below).

Embryo (A) and offspring (B) mortality rates (parameter estimates [mean ± SE]) in chosen and non-chosen pairs.

Figure 3: Embryo (A) and offspring (B) mortality rates (parameter estimates [mean ± SE]) in chosen (C) and non-chosen(NC) pairs.

So, the next question which the authors asked was if it’s the behavioural incompatibility which leads to greater hatchling death then can it be observed during the elaborate courtship rituals which happened before pairing? What they found was that although the NC and C couples spent almost similar time in courtship rituals the NC group females were far less receptive to NC males and also tended to copulate lesser compared to C group. Harmonious behaviour during courtship in zebra finches have been studied in detail and is taken as sum total of these: friendliness, mutual following, synchronous activity etc. So, a couple showing these behaviours in a greater amount would be termed as the ones who show behavioural compatibility and in anthropomorphic terms ”are in love”. An analysis of this behaviour among the C and NC couples showed that on an average the NC couples showed far less such behaviour than the ones in C group.Apart from these results, when the chicks hatched what was seen that greater proportion of males in NC group showed infidelity than in C group and the majority deaths of chicks which happened in the critical period of first 48 hours was due to lesser paternal care in NC group than in C. 

Discussions

The authors in the end ascribe this difference in reproductive fitness to the behavioural incompatibility between the two groups. They also mention – ‘‘The mechanisms behind such behavioural compatibility, in terms of willingness or ability to cooperate with certain individuals and in terms of coordination between partners need further study, in particular in the context of offspring provisioning.”

In humans, some studies suggest that individuals are more satisfied, more committed, and less likely to engage in domestic violence, when involved in a love-based rather than an arranged marriage (2,3,4). The challenge there is also to find out whether stable and happy marriages result from motivation to cooperate (and to identify what stimulates such feelings, see 5-8), or from congruence in terms of partners’ intrinsic behavioural types [9].

References:

  1. Ihle M, Kempenaers B, Forstmeier W. Fitness Benefits of Mate Choice for Compatibility in a Socially Monogamous Species. PLoS Biol. 2015; 13(9): e1002248. doi:10.1371/journal.pbio.1002248
  2. Xu XH, Whyte MK (1990) Love matches and arranged marriages—A Chinese replication. Journal of Marriage and the Family 52: 709–722.
  3. Sahin NH, Timur S, Ergin AB, Taspinar A, Balkaya NA, Cubukcu S (2010) Childhood trauma, type of marriage and self-esteem as correlates of domestic violence in married women in Turkey. Journal of Family Violence 25: 661–668.
  4. Regan PC, Lakhanpal S, Anguiano C (2012) Relationship outcomes in Indian-American love-based and arranged marriages. Psychological Reports 110: 915–924. PMID: 22897093
  5. Asendorpf JB, Penke L, Back MD (2011) From dating to mating and relating: predictors of initial and long-term outcomes of speed-dating in a community sample. European Journal of Personality 25: 16– 30.
  6. Honekopp J (2006) Once more: Is beauty in the eye of the beholder? Relative contributions of private and shared taste to judgments of facial attractiveness. Journal of Experimental Psychology-Human Perception and Performance 32: 199–209. PMID: 16634665
  7. Meltzer AL, McNulty JK (2014) “Tell me I’m sexy . . . and otherwise valuable”: Body valuation and relationship satisfaction. Personal Relationships 21: 68–87. PMID: 24683309
  8. Todd PM, Penke L, Fasolo B, Lenton AP (2007) Different cognitive processes underlie human mate choices and mate preferences. Proceedings of the National Academy of Sciences of the United States of America 104: 15011–15016. PMID: 17827279
  9. Rammstedt B, Spinath FM, Richter D, Schupp J (2013) Partnership longevity and personality congruence in couples. Personality and Individual Differences 54: 832–835.