The Gambling Commission has today (18 January 2020) commenced a consultation – that will close on 12 February 2021 – in which it shares its intentions with regard to changing the research methodology it uses to collect gambling participation and problem gambling prevalence statistics.

The Commission says that it believes that “this new approach will set the standard for authoritative research into gambling behaviour”.

It adds that “the specific focus of this consultation is to gather views on proposals to adopt a new methodology for our regular participation and prevalence research, to provide nationally representative data for the adult population of Great Britain”.

In terms of background information, the Commission states that it:

…. currently utilises a ‘combination approach’ for adult participation and prevalence research which has developed over time, by deriving these official statistics from several different surveys:

• As a section of separate Health Surveys for England, Scotland and Wales conducted approximately every 2 years (subject to availability), which are large scale face to face population surveys that provide our current ‘gold standard’ prevalence measurement
• A quarterly telephone survey which supplements the Health Surveys by providing a more regular measure of participation and problem gambling prevalence
• A quarterly online survey which supplements the telephone survey with more granular data about online gambling behaviour

Whilst the data collected is robust and authoritative, we have identified a number of limitations with the current arrangements which impact our ability to further develop our understanding of a fast moving and changing industry. These limitations are detailed in this consultation, but at a high level can be summarised as:

• Lack of control over our access to Health Surveys limits our ability to report representative data for the whole of Great Britain
• Different participation and prevalence questions on different surveys generate multiple figures
• Data from the different surveys is not directly comparable due to different methodologies being used
• The infrequency and long turnaround time of the Health Surveys from inception to reporting
• Traditional research methods (on which we rely) are in decline and under greater threat due to Covid-19 impacts

In terms of its intention, the Commission states that it proposes:

…. to replace the Health Surveys, telephone survey and potentially the online survey with a single, high quality methodology which will be more efficient, cost effective and timely, helping us to respond quickly to emerging consumer issues. We believe that a new approach will enable us to set the standard for authoritative research into gambling.

We have identified a number of criteria a new ‘gold standard’ approach needs to enable:

• The ability to accommodate core questions on gambling participation and prevalence on one survey (currently collected via the separate Health and telephone surveys)
• The ability to access a detailed set of demographics and to retain current questions on broader criteria such as other health conditions, which would be published as part of the results
• To address issues of currency by being able to alter questionnaire content, by amending questions, adding or deleting questions quickly
• Complete data that is representative of the adult GB population via a consistent approach in England, Scotland and Wales with a high quality sampling approach (preferably a random probability design)
• Delivery of a large, robust sample size, which can scaled up or down according to budget availability
• Control over the survey, such that we can ensure it provides an unbroken series of annual statistics (unlike the current Health Surveys which we are unable to access every year)
• A significantly faster turnaround than the Health Surveys from completion of data collection to reporting of the statistics
• The ability to conduct fieldwork regularly and therefore release updated statistics on a frequent basis
• Preferably, an approach which does not rely on face to face fieldwork and is therefore better able to withstand the threat posed by Covid-19 to this approach
• That the research should be conducted by a highly reputable provider which follows relevant research industry standards and enables continued compliance with official statistics production requirements

We also consider that changing the survey method could result in changes to the data and intend to undertake a pilot survey to assess the impact ahead of any permanent change.

In moving to a new approach, we are open to making use of existing general population surveys, and also to commissioning a new survey that would be built specifically for the Gambling Commission. It is important to emphasise that whatever option is chosen, ensuring objectivity and transparency in data collection and reporting would be of great importance to us. The Commission, and our lead Government Department, DCMS, are designated to produce official statistics and we are bound by the principles in the Code of Practice around Trustworthiness, Quality and Value. In addition to this, we would seek advice on methodology and questionnaire design from independent research experts and would publish full details of our survey design, response rates and quality assurance processes.

The consultation presents six alternative proposals, in relation to which the Commission states:

We recognise that our current set of surveys have a range of strengths and limitations. Taken as a whole, the current ‘combination’ approach of three surveys provides good breadth of coverage of key metrics and maintaining this mix for several years has allowed us to report data and trends in a consistent manner over time.

Actions could be taken to maintain and enhance the combination approach. For example, sample sizes could be increased to provide more robust data (if budget allowed) and questionnaire content could be reviewed and made more consistent. However, such changes would not address many of the limitations of the current approach. These are outlined in the following pages, together with our proposed actions.

We set out below, word for word, the six proposals:

Proposal I: Single survey for Great Britain

Lack of control over the inclusion of our questions on the Health Surveys limits our ability to report representative data for the whole of Great Britain

The issue: Due to the fact the current Health Survey approach relies on separate NHS/government-led surveys in England, Scotland and Wales, it has often not been possible for the Commission to secure space for our participation and prevalence questions on the surveys for all three nations in the same years. Consequently, it has so far only been possible to release a combined ‘Gambling behaviour in Great Britain’ report for 2015 and 2016. In 2020 we are therefore relying on data from four years ago (2016) for the most recent GB-representative Health Survey statistics on problem and at-risk gambling. This inability to publish complete, consistent data that represents the full geographical area that we are responsible for is one of the key issues that we are seeking to address through the methodology review.

Proposal: To replace our current usage of the separate Health Surveys for England and Scotland and equivalent survey in Wales with a new ‘gold standard’ population survey which covers the whole of Great Britain via a large and robust sample. We will consider both existing general population surveys that we can access, and new surveys, which would be designed for this purpose, as a means of meeting this objective.

Rationale: We will be able to report complete, consistent data that covers the whole of Great Britain via a new survey that reflects best practice and allows comparisons between the nations to be made with confidence. Consolidating control over the survey for England, Scotland and Wales will also strengthen our ability to change, add or remove questions in a consistent way. As a result, we will be better placed to respond to emerging policy issues, government interest, stakeholder concerns, changes in research funding or specific events. In an ideal world, if budget allowed, we would seek to generate a very large sample size which would not only provide robust national data but also provide more granular geographic data and facilitate comparisons between the widest possible range of demographic cohorts.

Proposal II: Consolidation of current surveys

Data from the different surveys is not directly comparable due to mode effects

The issue: For the Health Surveys alone, there are currently difficulties associated with combining data from three separate surveys for England, Scotland and Wales. The use of a different survey in Wales means that for Wales we do not have access to the broader health measures available via the HSE and SHeS and, from 2020, we also do not have access to DSM-IV problem gambling screen data. The problem of comparability is exacerbated when the quarterly telephone and online surveys are also considered. Data collected via the quarterly telephone and online surveys (while helpful in supplementing the Health Survey data with more timely statistics) is not directly comparable due to different survey methodologies.

Proposal: To reduce the number of surveys the Commission currently uses to produce official statistics on participation and prevalence to provide a single set of trusted metrics. As part of this, to absorb content from our existing surveys into the new ‘gold standard’ population survey.

Rationale: Consolidating our surveys will address the issue of multiple data points by using a single methodology (removing the issue of different mode effects) and the ability we will have to apply a single set of participation and prevalence questions.

Proposal III: Participation questions

Different participation questions on different surveys generate multiple figures

The issue: A further consideration which affects comparability is the current inconsistent application of participation questions. Currently, participation statistics are published from both the Health Surveys (based on past 12 month participation) and the telephone survey (based on the past 4 weeks). Further, the surveys incorporate different lists of gambling activities. The activity list used in the Health Surveys was originally developed for use in the 2012 survey. As such, it does not sufficiently reflect shifts towards online gambling in recent years, and it also provides limited granularity in terms of National Lottery games. Lack of available budget and a desire to retain comparability with previous data sets are the main reasons why the activity list has not been altered in subsequent years. There is a risk of confusion and misuse of statistics arising from multiple figures, leaving the Commission open to challenge and posing a threat to the credibility of our research. Some questions are duplicated (or questions on similar topics are asked in slightly different ways) – creating a need for the Commission to manage multiple data points (different ‘versions of the truth’) with associated risks of confusion or misuse of statistics

Proposal: Via a single preferred methodology, to gather more granular data on gambling participation and frequency. Also, to review and refresh the list of gambling activities included in the survey so that it better reflects the current diversity of gambling products and better facilitates analysis of problem gambling prevalence at a product level.

Rationale: The production of a single authoritative set of participation statistics will provide greater clarity to the use of this data and to policy debates. We believe it will increase user trust in the statistics who will have greater confidence that data is not contradictory and that the way we classify gambling participation accurately reflects the current product mix, both online and offline.

Proposal IV: Frequency and turnaround time

The infrequency and long turnaround time of the Health Surveys from inception to reporting

The issue: While we consider the Health Surveys to provide robust measurement of past 12 month gambling participation and problem gambling prevalence, the length of time between the surveys means that they cannot monitor shorter-term changes, and only measure changes approximately every two years (dependent on when we can secure space). Furthermore, the survey content cannot be adapted quickly to reflect new gambling products, and data from each survey is unavoidably out of date before the next survey is published. The slow turnaround of the surveys from inception to reporting is also an issue. Using 2016 fieldwork as an example, survey content was signed off in Autumn 2015, data collection ran from January to December 2016, and the combined GB report was published in September 2018 – approximately two years after the survey’s inception. This creates a major risk that emerging trends that may require action, will not be identified in a sufficiently timely manner. The quarterly telephone survey fills this gap to some extent albeit via a less robust methodology and problem gambling prevalence measure (the short-form PGSI screen).

Proposal: To explore surveys (including existing external surveys) which we would be able to access more frequently than the Health Surveys and which have a shorter turnaround time. To move towards at least annual publication of ‘gold standard’ participation and prevalence metrics.

Rationale: Running more regular ‘gold standard’ surveys and reducing the time lag from data collection to reporting will help meet best practice for official statistics and enable evidence-based discussion and action to take place based on the most up to date and high quality data.

Proposal V: Explore more ‘future proof’ methods

Traditional research methods, already in decline are now under greater threat due to Covid-19

The issue: It is important to refresh methods to keep pace with evolving best practice for population surveys. Further, reliance on face to face methods carries some risk given the impact of Covid-19 and potential future pandemics and government measures imposed on society

Proposal: To explore more ‘future proof’ methodologies for ongoing measurement which will be able to withstand threats posed to more ‘traditional’ research approaches. These methods include online, ‘push to web’ and mixed-mode surveys. Therefore an alternative approach could involve recruiting respondents via postal invitations (with addresses selected on a random probability basis), and conducting the survey either fully online, or online supplemented with other data collection methods such as postal returns or telephone interviews. Examples of other national population studies which have changed methodology in order to evolve and future proof the research include Sport England’s Active Lives Survey, the Community Life Survey commissioned by the Cabinet Office and Natural England’s People and Nature Survey. All three of these surveys have moved from more traditional telephone or face to face methodologies to an online or mixed methodology approach.

Rationale: Alternative methodologies exist which would be better able to withstand the threats posed by Covid-19 or future pandemics to interviewer-led in-home surveys. By continuing with the Health Surveys, there would be a risk that fieldwork may be adversely affected or may have to stop altogether, as has been the case in the HSE 2020. Coverage of the population may also increase under a mixed-mode approach compared with a single mode survey and bias should decrease in the combined estimates. A survey which has as wide a coverage of the population as possible should minimise bias.

Proposal VI: Survey pilot

Changing the survey method could result in changes to the data

The issue: The principal risk of a change to a new methodology is that the results will no longer be directly comparable with the existing surveys and their historical trend data. The impact of changing the methodology on trend data need to be understood. We also believe it is important to ensure that, if moving to a gambling-specific survey, it does not attract an over-representation of gamblers or problem gamblers. We therefore propose to pilot an alternative method to identify and understand the impact that this has on the data compared to our existing surveys. We will work closely with our stakeholders to manage any changes in the data and may consider applying weights to the data (if necessary) to take into account any discontinuity of the data series. Any changes to time series data will also be communicated via our website.

Proposal: To pilot questions using a potential new methodology in 2021 so that we can compare the results of the pilot with the telephone and online surveys that take place over a similar time frame, and with the most recent Health Survey data (2018). To take steps to ensure that the survey does not encourage an over-representation of gamblers, by taking care in the way the survey is branded and introduced to participants. To analyse and report on comparability of trend data. Subject to satisfactory pilot study data, to begin our new survey methodology in 2022.

Rationale: A pilot stage is necessary to be able to analyse and understand the impact of a change to the methodology on participation, prevalence and other important metrics (such as contextual data about physical and mental health co-morbidities) and to build sufficient confidence to support a permanent change.

The online consultation survey can be accessed here and downloaded below.

UPDATE: The deadline for responding to this consultation has been extended (by two weeks) until 26 February 2021