Research / Big and linked data

Capitalising on New Zealand’s health data

Using big and linked data to investigate non-communicable diseases

illustrated representation of the IDI simplified
Project Status: Completed Funding: $530,000 Timeframe: February 2016 – June 2019


What we investigated

Take | Issue

New Zealand has some of the richest ‘big and linked data’ in the world, available to researchers in anonymised form through the Statistics New Zealand Integrated Data Infrastructure (IDI). This data has enormous potential to answer research questions about the health of our population but to date it has not been fully utilised.

Whāinga | Aim

This project aimed to bring together big data experts from the Virtual Health Information Network (VHIN) with other researchers and health providers to answer research questions about non-communicable diseases.

A secondary aim was to build the capacity of the VHIN to support health researchers across the country using data held within the IDI.

Huarahi I Whāia | Approach

We interrogated linked health, social and census data sets, and helped to create new data linkages, to investigate a series of research questions:

      • What impact did the Canterbury earthquakes have on rates of cardiovascular disease?
      • What factors affect progression from prediabetes to diabetes in New Zealand?
      • What is the prevalence of cancer in New Zealand?

We also examined linkage in the IDI and the ways in which linkage error and bias may impact research that uses IDI data.


Outcomes and Impact

Putanga | Outcome

Our studies have demonstrated the value and potential of big and linked data to health research, policy and clinical practice in New Zealand:

      • We showed that in the first year after an earthquake there is a correlation between the level of earthquake damage to property in an area and the number of cardiovascular-related hospitalisations and deaths among residents. This will enable better planning for measures to address post-earthquake stress.
      • We identified factors associated with the progression of prediabetes to diabetes: being younger and male, and having a higher level of HbA1c (glycated haemoglobin, a measure of blood glucose control) or BMI (body mass index, a measure of fatness) are all associated with increased risk. We also made a novel finding that speaking Te Reo Māori is associated with reduced risk.
      • More people are now living beyond a cancer diagnosis so it is important to know the prevalence of cancer in New Zealand to estimate demand for ongoing support services. We found that 3.2% of New Zealand residents had been diagnosed with at least one cancer in the 18½ years up to 2013, and that prevalence is higher amongst those with greater access to healthcare.

New data has been brought into the IDI as a result of our studies, including Earthquake Commission residential building claims data, and primary health care data. Code and methodologies generated from these studies have been shared with other researchers.

The IDI is based on a huge number of data linkages. Our report on data linkage bias within the IDI informs researchers about data linkage processes in IDI and the potential impacts of this for their research. This is also available as a condensed accessible guide.

Te Ara Kei Mua | Next Steps

Further research is planned to examine community lab test data and investigate the use of integrated health data in understanding clinical pathways. We will also examine the extent of linkage error and bias in the IDI on measures of ethnic inequalities in cardiovascular disease, cancer and diabetes.

Nga Hua O Te Rangahau | Research Products

      • Tables and code tracking cardiovascular events after the Christchurch earthquake (available in IDI)
      • Tables identifying progression of prediabetes to diabetes in the study cohort after three years, by age, sex, ethnicity and socioeconomic position (see publication).
      • Tables identifying the 18½ year prevalence of cancer in New Zealand by sex, age, ethnicity and a variety of demographic factors (see publication).
      • Linkage error and linkage bias: a guide for IDI users
      • Training courses, guides and code sharing (available on VHIN website).
      • VHIN facebook page - a support network for researchers working with big data.

View the website for this project


Knowledge Exchange

Ētahi Atu Puka | Other Publications

Pāpāho | Media

Kōnae Whakaata | Video

Capitalising on New Zealand’s health data

Research Findings Briefs

The study identified the progression rates from prediabetes to diabetes in New Zealand, and key factors which protect against progression.

This study examined the impact of earthquake housing damage on cardiovascular disease (CVD)-related hospital admissions and deaths in the five years following the Canterbury earthquakes of 2010 and 2011.

Project Team

Related News Articles

Sign up to our Newsletter

"*" indicates required fields

Scroll to Top