George Pinder
Research Manager
Getting a representative sample of ethnic minority adults via online panel research is a major problem for researchers and stakeholders. Skews and biases in sample composition are common—and have major implications for policies and campaigns affecting ethnic minorities.
The lack of online panel respondents from more deprived backgrounds means that the views, experiences and behaviours of ethnic minorities are misrepresented.
This has negative real-life impacts. Research data is influential in informing government policy, corporate decisions and strategies, communications and public health messaging. During the height of the COVID-19 pandemic, health data from the ONS demonstrated that ethnic minorities in low-skilled occupations and facing socio-economic deprivation were highly correlated with higher COVID-19 death rates. However, these dynamics are normally missed in market research data because the number of respondents from these groups is too low. With representative survey data about the views and experiences of deprived groups, the outcomes of policies and messaging can be improved.
The most important root cause for these problems is that ethnic minority adults from socio-economically deprived backgrounds are severely under-represented on online research panels. Without measures being taken to counteract this, ethnic minority samples invariably skew towards better-off and better-educated respondents—to a far greater degree than nationally representative samples of UK adults.
This skew is particularly pernicious because it persists despite controlling for other demographic variables. Even setting representative quotas by age, gender, and region fails to address socio-economic skews. A sample that can seem to be representative on the surface can turn out to be far from representative in reality.
Our analysis of published ethnic minority survey data shows that if no socio-economic controls are placed on the sample, the social grade profile will be heavily skewed towards those in more privileged groups:
This pattern can also be seen in terms of the political profile of ethnic minority samples. The below chart compares the 2019 ethnic-minority vote share derived from online panel research versus ethnic-minority vote share in the Ipsos MORI “How Britain Voted” research based on just under 30,000 telephone interviews.
Linked to the over-representation of respondents in more privileged socio-economic groups, there is also—without proper measures being taken—an over-representation of Conservative voters in ethnic minority online samples.
Solving these skews and biases is exceptionally difficult given how deep-seated some of the causes are. However, there are numerous immediate steps that can be implemented whenever your team or a research partner are conducting research with ethnic minorities:
When setting quotas and weighting, control for socio-economic and political variables, not just basic demographic variables (age, gender, region, ethnic group). Based on our analysis, we have found that setting quotas by education, and then additionally weighting by 2019 past vote is effective.
Each panel has its own biases and shortcomings. By using multiple panel sources, you can smooth these biases out and reduce the impact of any given panel skew on your sample.
To reach those respondents who typically don’t sign up for online market research, use suppliers that source respondents from other places as well (e.g. pop-up ads, banners) and couple this with rigorous data quality (see more on this here).
Work with your supplier to target ethnic minority respondents from deprived backgrounds early in fieldwork. Give them as much time and space to respond as possible, especially before any other quotas have already filled up. This way, you won’t be left chasing tough interlocked groups at the end (e.g. secondary school educated, 18-24-year-olds in Scotland).
Review the panels that you are working with and use ones that take this problem seriously. Ask them about their specific recruitment practices for ethnic minority respondents and how they reach respondents who wouldn’t otherwise sign up for their panels. It will be clear if they have thought about the problem.
Which?, the consumer watchdog, wanted to understand the experiences of adults from ethnic minority backgrounds in different areas of consumer life, and how this compared to the experiences of white people. In particular, they wanted to understand the scale and nature of discrimination faced by ethnic minority consumers.
Which? ran two surveys on the Focaldata platform. One among a representative sample of 1,000 ethnic minority adults in the UK, and one among a representative, “control” sample of 2,000 adults of all ethnicities. The survey covered:
In this context, it was critical that the ethnic minority sample properly represented the views of consumers from more deprived backgrounds and with fewer resources at their disposal. If they had not done this, the data would have painted a misleading picture of the experiences of ethnic minorities. If the data was skewed towards those from more advantaged socio-economic groups, the true nature of discrimination and its impacts would have been obscured.
It was also important that the Which? team was able to draw comparisons between different ethnic groups. For instance, the experiences of consumers from Black Caribbean backgrounds are very different to those from, say, Indian backgrounds, therefore the research had to be designed to allow for this nuanced analysis to be possible.
Focaldata accesses a huge network of panels in the UK, US and internationally. This gives us the largest possible accessible audience. After all, decentralised marketplaces tend to beat centralised bureaucracies. Critically our targeting technology enables us to find ethnic minorities and other niche groups far more quickly and more affordably. We use machine learning to predict which respondents are likely to be ethnic minorities, even if we don't actually store that information. This enables us to be 5x more effective in targeting niche audiences than traditional methods, which rely on sending a survey to the general population and then screening out certain groups.
To find out more about how our platform can help you deliver higher quality ethnic minority samples and other niche audiences, please get in touch: george@focaldata.com.
