Working on a recent longitudinal study, the Ipsos SRI team found some results that reinforced the limitations of ‘traditional’ tracking research.
For those not intimate with ‘tracking’ research, here is a quick overview.
What is tracking research?
Tracking studies test the same measures at different points in time to look for changes. This is the kind of research you might use to assess the impact of, for example, a communications campaign. In this example, we would survey a sample or ‘cross-section’ of the population before a campaign launches, collecting information on the sorts of messages the campaign will communicate. This becomes our ‘benchmark’ – a baseline against which we can compare future results to look for differences.
How is a longitudinal study different from a cross-sectional one?
The approach described above is often called ‘cross-sectional’ because it relies on surveying a ‘cross-section’ of the population at different points in time. Another common approach is a longitudinal one – in a longitudinal study the same individuals are surveyed at different points in time. The power of a longitudinal design is that it allows us to look for changes in a particular individual’s attitudes or behaviors.
The study in question
The recent study that highlighted the limitations of traditional tracking was longitudinal. We were testing the impact of a behaviour change program about transport. Longitudinal analysis – tests that looked for changes in an individual’s behaviour at two points in time, showed no differences. However cross-sectional analysis – tests that treated each wave of interviewing as though they were separate samples found those who had been part of the behaviour change program had greater awareness of public transport routes compared to those who had not.
At face level, this result was confusing. Why was one analysis showing differences, when another supposedly more sensitive analysis, was showing none? So, we looked at the data more closely and found that the people who took part in the behaviour change program had greater awareness of public transport routes at the benchmark phase – before the behaviour change program had started! The program had not increased their awareness levels so longitudinal analysis did not show any differences.
If a researcher had looked at this situation from a purely cross-sectional perspective, they might have (falsely) concluded that being part of the behaviour change program had resulted in the increased awareness. In fact, it is more likely that being more aware of public transport options made people more likely to participate in the program.
What are the lessons?
The lesson for any user of tracking research was simple – just because a study finds a relationship between two variables, we can’t assume the relationship is causal. If people recall communications about sunscreen and are more likely to use sunscreen, we can’t assume that the one lead to the other. It is just as likely that people who are more concerned about sun safety are more likely to remember sun safety ads.