We all see these, and for some reason recently I’ve been seeing the NetRatings survey a lot when I go to the NZHerald site. Perhaps my cookies keep getting deleted, perhaps I’m using too many different browsers (3 on my mac, 1 at work). The data we in the industry get from these surveys is really valuable, so I’m going to attempt to constructively criticise the survey questions:
Thanks for asking. You should point out that the answers I give you will apply to and be available not just to the NZHerald owners, but for all sites I visit that subscribe to the NetRatings service. That’s pretty much everyone here in NZ (except for Yahoo!xtra).
1: looks ok
2: Actually I access the internet pretty much constantly, and from multiple devices. This makes me feel like the survey is stuck in 1999. Perhaps you could also include either “more than 3 hours a day, 1-2 hours a day” or even “from 2 or 3 places a day, from one place a day”.
3.4: I don’t really like giving out too many details – so an age range would be better thanks.
5: Does anyone actually chose “semi-skilled worker?” – perhaps some work is required on those descriptions. Also the list seems skewed in reverse order. The most common jobs are closer to the bottom that the top, and so I’d reverse the order to make things easier, and to help stop job inflation.
6: The NZ statistics NZ folk collate statistics of businesses by number of employees – but the breakdown stops at 100+ employees. However those employees at 100+ sized companies represent almost half the total number of employees. So I’d say NetRatings have this more right than the Stats.govt folk, though 10,000 seems a bit rich – perhaps that should be relabeled “Government and quasi-Government organizations”
7: The average income per household in New Zealand is pretty low – at $1,321 per week or $68,600. Meanwhile the 3rd Quintile weekly income for individuals is $576-759, or $30-40,000. The top quintile for individuals kicks in at just $53,800, which implies the top quintile for households is about $100,000. But I’m going with NetRatings here – while their lowest income category should be a bit lower (bottom quintile is $10,000 for the self employed), their higher brackets seem right, and they are targets for advertisers. Good to see the prefer not to say optiojn.
8:This question doesn’t acknowledge the existence of flats, where multiple people are responsible for significant amounts of shopping. Flats are not uncommon in NZ, so an option should be added.
9: This does adequately find out where you live in NZ, but does not distinguish between urban and rural. Doing so is tough without another question, but it may be an important one for advertisers.
10: Interesting how the Pacific Islanders are broken out by country, while the old bugbear ‘Asian’ is there for the non Chinese or Indians..
11: Where is Bebo? MySpace? YouTube? FindSomeone? This list is well out of date.
12: Clueless really. What is ‘downloaded a brochure or form’? Does that mean I filled out a form, viewed a pdf within my browser or looked at the marketing information for something? The question assumes old technology I think.
What is an ‘electronic newspaper’? I certainly don’t regard Stuff or NZHerald.com as electronic newspapers, but as online news sites. What will add to the confusion is the emergence of newspaper readers that let you see a pdf of the newsprint version. I am sure most people are confused by these options.
Why ‘joined a chat or discussion group’ instead of ‘participated in a chat or discussion group’. And what is a chat or discussion group anyway?
Where are myspace, bebo and the like? Where is banking? Where is skype? Where is ecommerce? where is international ecommerce? where is mapping? tourism? the list goes on..
13: A useless question for active users of the internet as you can tell by my answers.
14: fine, though some indication of frequency would be telling.
15: That’s a long list, but where are shares? where are subscriptions to online services such as the WSJ? One technical issue with this questions is that once you click on either of the options in a row then you cannot unclick that row. Maybe that is just Firefox on the mac but…
Lance Wiggs 
Hi Lance,
I am the Senior Analyst at Nielsen//NetRatings New Zealand and would like to thank you for your detailed comments, thoughts, and advice on our internet demographics survey.
Firstly some background that you and your readers may like to know. The survey was last updated in 2005, and all the questions were developed in close coordination with what is now known as the Interactive Advertising Bureau New Zealand (IAB NZ) and the New Zealand publishers delivering the survey at the time.
We are very aware that the online market is constantly on the move, and updating itself very quickly. However, we also have to balance this with the need to keep consistency in the survey so we can measure the important demographic trends that publishers and advertisers require over a reasonable period of time.
You will be pleased to know that we review the survey regularly and will be updating this during 2007, again in coordination with IAB NZ and publishers. We will also be taking take your very useful comments and suggestions into consideration as we revise the survey.
Please let me know if you would like to discuss this further. We certainly welcome any constructive feedback from the industry or internet users that would help to improve our demographics survey for respondents, and which that at the same time gathers even more useful data for publishers and advertisers in the New Zealand market.
Kind Regards
Tony Boyte
Senior Analyst
Nielsen//NetRatings
DDI: 09 970 4143
Mobile: 021 480 182
LikeLike
Hi Lance. I found your blog through Rowan’s and have been reading for a month or two; thanks for posting. Since this topic is dear to my heart I thought I’d emerge from lurking and throw my two cents in…
Although I agree with many of your points, some of the response options may actually be appropriate for the audience/purpose of the survey. For example, it is possible that the survey analysts have looked at the response distributions of some of the questions (e.g., income) and found that they do generate a good spread for online respondents to this survey. Similarly, the ethnicity question may have the options it does because they want to use the breakdown for comparative purposes (say, with census, which asks the question in a very similar way).
They have also followed best practice in asking the age question in the ‘year born’ format. I’ve recently done a piece of research on this and the yearborn format gives more detailed information than age categories without leading to a substantial increase in nonresponse or fibbing. The same is true of the format in a mail survey setting.
That said, there are some aspects of the methodology in addition to those you point out that are of particular concern…
First, the survey uses lots of drop down boxes. A nasty little feature of IE6 (on PC) allows the selection for these to be changed when a scroll mouse user attempts to scroll down the page after making an answer. Tonnes of people over the last few years have been using exactly this combination of hardware and software (in a survey I did in 2005 a very high proportion of people were using both). So, this survey will very probably have systematic error in its results for drop-down questions, with answers skewed toward the bottom of the response list.
Second, as a user of the information you will have no idea how representative it is of site visitors. On-site pop-up surveys have notoriously bad response rates (think click-through rates for banner ads) and there is no robust way of telling whether people have responded multiple times because access is not restricted. With no objective measure available to ensure a good mix of people decided to fill in the survey, you are left with data that may be way off the mark. IMO, this last point trumps any problems there may be with question wording or response options in the survey itself.
LikeLike