On the radio this week, talkback host Mike Hosking shared his reckons about AT’s annual research report into walking and cycling, which we covered yesterday. In particular, he simply couldn’t believe that 65% of Aucklanders think bike paths are good for the community.
He didn’t just doubt the results, he doubted the methodology – and described the research as an ‘opt-in’ online survey filled in by ‘keyboard warriors’ and ‘bored deadbeats with nothing to do.’ Similarly, the NZ Herald ran an editorial referring to AT ‘surveying self-selected samples online for the answers it wants.’
But just because something seems too good to be true doesn’t mean it’s false.
This study wasn’t an ‘opt-in’ poll that people found and shared online: it was a representative sample survey carried out in the industry-approved fashion by an independent research company. For avoidance of any doubt, the market research company TRA describes how the research was conducted:
The AT Active Modes study was conducted among a representative sample of 1,459 Aucklanders aged 15+ years old. To ensure a representative sample, this is managed in a few important ways:
1. Respondents invited to take part in the survey were proportional to the Auckland population as per 2013 census, reflecting Age, Gender and Auckland Regions (Rodney, North, Central, West, East, South, Franklin).
2. Respondents didn’t know what the survey was on or for before completing the survey, removing any participation bias.
3. We sample daily – meaning over the 5th – 20th April, we spread the number of interviews to minimise any impact of events or weather. This is the same time as the previous year – again to remove any seasonality bias.
4. We ‘post weight’ all respondents back to Auckland census data to ensure the final respondents who complete are representative of Aucklanders aged 15+ years old.
5. We boost sample for city fringe and regular cyclists for this survey – this is only used for specific analysis purposes. This boost in specific segments does not affect the overall data or results reported as they are weighted back to population and behaviour.
We look forward to swift corrections on this point by NewstalkZB and the NZ Herald. In the age of science denialism and accusations of ‘fake news’ aimed at damaging public trust of robust data, it would be unthinkable for two major news outlets to let such a enormous misapprehension stand.
How sampling works
For those who are a bit rusty about statistics, we asked a top statistician at the University of Auckland to explain in clear and simple terms how sample-based surveys work. Here’s what Professor Thomas Lumley told us:
Getting information from everyone — as the Census tries to do — is very expensive, so any realistic opinion polling has to rely on small numbers of people.
In order for a small sample to give a reliable estimate of public opinion, it has to be representative: different groups of people have to contribute to the sample in the same proportions that they contribute to the population. Professional polling organisations do this in two ways: they try to sample people at random, independent of their opinions, and they give extra weight to groups of people who end up being under-represented in the sample. When they do this, they can get a pretty good estimate of the likely accuracy of the poll.
Which is what TRA did. A balanced sample of around 1500 people out of a total population of 1.5m gives you a confidence level of 95% with a margin of error of around 2.5%. Interestingly, you could also survey a population of 15m using a sample roughly the same size. Because that’s how sampling works.
So what’s wrong with online polls?
By contrast, open-ended online polls are indeed open to all sorts of imbalances, which Hosking went on to demonstrate in unexpectedly entertaining fashion. As he explained this morning on the radio, he wanted to show that opt-in surveys are ‘open to corruption’ and thus ‘not news’. Which is not news to anyone who knows anything about statistics, and thus why no reputable research company would touch them with a bargepole. Here’s Prof. Lumley again:
If you don’t make any effort to select a representative sample or to reweight groups in the sample so that their views are representative, it is unlikely that the result will be close to actual public opinion, and it is not possible to say anything quantitative about how badly it could be wrong. Self-selected online polls with many thousands of responses can easily contain less information than representative samples of a few hundred.
Unselected online polls (‘bogus polls’ as I call them) also tend not to check their respondents carefully: for example, it can be easy for people who aren’t from Auckland to vote in a poll on Auckland transport policy.
If only we had an example of an unselected online poll at this point, to illustrate the discussion. Oh wait! We do: after expressing disbelief about the professional survey, Hosking started his own straw poll on Facebook (NB the post appears to have been geo-limited to New Zealand only, presumably to enhance its accuracy as a record of the local psyche).
About an hour after Hosking’s poll opened on Monday morning, he was saying on the radio: ‘We’ve got the same number of people opting in and we’ve got a completely different result. According to the Mike Hosking Breakfast Listeners, a fair and reasonable selection of New Zealanders from the far flung parts of this country, you don’t think that cycleways are good for the country.’
At this point, he’d reached out and connected with a thousand or so bike-dubious listeners, successfully illustrating the flaws of opt-in surveys – and why professional research outfits wouldn’t dream of using them. He added that if the people who’d texted and emailed him understood how to use Facebook, he’d have 50,000 votes against safe cycling infrastructure. (That’s quite a lot of texts and emails to count up first thing in the morning; he must have very good support staff).
But an even more interesting lesson was to come. By morning tea, the results were neck and neck – and by lunchtime, support for cycle lanes and bike paths had edged ahead with what would become unstoppable momentum. Across the day, support steadily inched upwards, one percentage point at a time.
Here it was at 2.30pm.
60/40 at 4.00pm
Nice.
By bedtime on Monday:
And as of dinnertime on Tuesday 19 June, Mike’s poll had achieved this pretty spectacular result… with another 11 hours still to go.
And that’s where it landed this morning, when the poll closed with 16,400 votes…
Does that mean even more people like bikes than Mike imagined?
It’s tempting to leap at that figure of 79% as the true representation of popular support for bike-friendly cities (or even split the difference between Mike’s poll and the professional survey, to arrive at 72%).
But, just like Mike – who is generally fond of citing political polls that use exactly the same industry standard approach applied by TRA – we prefer to trust the professionals, so we’ll settle for TRA’s 65%.
Still, it’s pretty striking that Mike’s straw poll (of the subset of New Zealanders who know how to use Facebook and can see his page) reached over 16,000 individuals, of whom 13,000+ say bike paths are great for the community.
How does this really quite enormous sample compare with TRA’s carefully balanced representative sample of 1459 Aucklanders? Is it a case of the more the merrier and the stronger the results? We asked Prof. Lumley again:
Good opinion-poll samples using traditional methods are in the range of 500 to 2000 people. Fewer than 500 doesn’t give you much precision. With more than 2000 the sample size stops being the limiting factor — you’ve already got as much accuracy as the method will allow. That’s why election polls are usually of 1000 people. Getting more people doesn’t make the result more accurate unless you can be sure those people are representative.
The Auckland Transport survey was conducted by market research company TRA and took a representative sample by age, gender, and location — everybody who was sampled was from Auckland, and people across all of Auckland were involved.
So, we’re comparing an ‘opt-in survey filled in by bored keyboard warriors… from the far flung parts of this country’ [Hosking’s Facebook poll] with the sort of professional research into Auckland opinions that Kiwi businesses will pay real money for.
Looks like we’re best to stick with 65% support for cycleways being good for the community as the authoritative data point for 2018, then.
Still, as wildly unscientific as it was, Mike’s straw poll backs up the professional TRA survey on one key point: despite a small and often vocal bikelash, there is a large ‘soft middle’ out there – a silent majority, a taciturn predominance, of regular folks who quite like bikes and certainly don’t hate them and were probably glad to be asked about it.
The truth is, ordinary New Zealanders – kids, parents, couples, millennials, older folk, professionals, shift workers, leisure lovers, locals – are rediscovering their love of bikes. They understand that everyday cycling is an essential element of liveable neighborhoods and efficient cities. They totally get that going by bike is one of the most efficient and healthy ways to get around Auckland. They’re keen to see safer streets that welcome all sorts of people on bikes. They’re also keen to stand up and be counted, when called upon.
And they’re not going anywhere. Because, as AT’s five years of research shows – because as well as being statistically robust, the survey is consistent with previous years, thus allowing for longitudinal comparisons – the trend is headed consistently in one direction: up. Auckland, we’re on a roll. As covered in the Herald on Monday, not only have attitudes to bike paths and the state of cycling done a 180 in the last five years, Aucklanders are actively embracing riding a bike as something we do here:
Over half a million Aucklanders (38 per cent) now identify as cyclists, meaning that they ride a bike at least occasionally. This is up from 20 per cent in 2014: that’s almost double in five years.
Throughout the city, the proportion of people riding a bike to connect to public transport has grown from 5 per cent in 2015 to 14 per cent today. Cycling to work has also grown, from 12 to 19 per cent.
The survey suggests e-bikes will bring further big changes. Only 3 per cent of respondents said they own an e-bike now, but 33 per cent said they were “thinking about” buying one in the next 12 months.
What next?
We look forward to the published retractions on this story, and an explainer for the public about how sample surveys work, as any responsible media outlet would wish to ensure that any future stories based on survey results aren’t instantly subject to public doubt. Not just bike stories, but political stories, stories about which radio stations are most listened to, etc.
We also look forward to the results of next year’s Active Modes survey – and to any alternative polls Mike may care to run to cross-check any research results he’s not sure about.
It’ll be especially interesting to hear what he and the Herald editorialists think about the walking half of the Active Modes research report. Pedestrians all over Auckland will naturally be curious about whether their fellow citizens support better footpaths and share their concerns about safe strolling in their neighbourhoods, for example. After all, it’s so important to tell the whole story.
PS This is one time it is actually safe to read the comments, if you’re one of the lucky ones who can actually see the page – there are some real gems in there!
