Intro to public sector data

Want to know more about data but don’t know where to start? We’ve created a short introduction to help you understand what public sector data is and how it’s used in research.

What is public sector data?

Throughout a person’s lifetime, they generate lots of information which is collected by public sector bodies such as the NHS, local authorities, government bodies, schools, social care organisations, and many others.

For example, GP records might contain data about an individual’s medical history and diagnoses. Similarly, local authorities and government bodies may store information about a person’s employment status and housing information. Most public sector organisations will also hold general person-level data such as an individual’s name, date of birth and address.

This information is known as public sector data or administrative data.

This data allows public sector organisations to deliver their services, but it also provides a wealth of information which can be used for the wider public good.

The term ‘public sector data’ encompasses a very broad range of information about people, places and businesses, meaning it is a vital resource in understanding how a society operates and how it can be improved. When researchers are able to access public sector data, they can discover valuable insights which help develop systems and improve lives on a national scale.

How is public sector data kept safe?

With such a wide range of personal information collected, it’s very important that data is held safely and securely. Measures to keep data secure include physical controls such as keeping it in a locked, access-controlled environment and technical security measures, such as strong password protocols with multi-factor authentication, hardware encryption, firewalls and anti-virus software.

To learn more about the measures that keep data secure, read our explainer: What is the Five Safes framework?

Securing data also involves training staff to make sure they are using the correct processes and policies when they are processing data. The Information Commissioner’s Office (ICO) publishes guidance around the technical and organisational controls which organisations must implement to protect data, including those processing public sector data.

How is data made available for research?

When it is ready to be used for research, public sector data is usually stored in a Trusted Research Environment such as the National Safe Haven – a secure environment which houses Scottish data and can only be accessed by approved researchers.

To find out more about Trusted Research Environments, read our explainer: What are Trusted Research Environments?

There is legislation in place in the UK to protect people’s personal data, including personal public sector data, and these include the Data Protection Act 2018, UK GDPR and the Digital Economy Act 2017.

When data is shared with researchers or other public sector organisations, technical and organisational controls are implemented to ensure that it cannot be traced back to any individuals. Personal data can either be pseudonymised or anonymised to ensure that individuals are not identified or identifiable. To find out more about these methods of protecting personal data please read our explainer ‘What is data de-identification?’

How do researchers access public sector data?

The IG processes required to grant data access can vary depending on certain elements of the research project, for example the specific datasets required. Generally, projects will need to be reviewed by an approval panel involving those with responsibility for keeping the data safe. If the application is successful, the researcher will be able to access the data subject to certain conditions, such as only accessing it in a safe environment.

What are the barriers to accessing public sector data for research?

Often, public sector data is often locked away in lots of individual systems, across many different organisations. This makes it difficult and time-consuming for researchers to access and use data.

For example, if a researcher wanted to compare health data with homelessness data, they would need to apply to two organisations separately. This means more time is spent applying for data and waiting for access, which can delay research and lead to limited or outdated conclusions.

Researcher Access Service

This is where the Researcher Access Service comes in. Initially launched by Research Data Scotland in 2024, the Researcher Access Service streamlines and simplifies the data access process so that approved researchers can access data faster.

By making it quicker and simpler to access public sector data - without sacrificing security - Research Data Scotland aims to speed up research, making it easier for decision-makers to base their insights on up-to-date, good quality research built on good quality data.

Learn more about the Researcher Access Service

What is data linkage?

Data linkage is when an individual’s pseudonymised data can be connected across multiple datasets. This allows researchers to discover relationships between different areas of the public sector and understand how people interact with different services.

Before an individual’s data can be linked, their personal information is removed and replaced with a linkage key – a unique identifier which allows a person’s data to be connected across different datasets without containing information that could be used to identify them.

Find out more about data linkage in this short video, supported by funding from Health Data Research UK (HDR UK) and the Medical Research Council (MRC):

Click here to watch this video with British Sign Language (BSL) interpretation.

Open data

Lots of public sector data is sensitive and can be linked to individuals, meaning it needs to be kept secure. However, once it has been anonymised and aggregated to ensure it cannot be traced back to an individual, some public sector data can be safely shared publicly. This anonymised data is known as open data.

Open data is freely accessible to anybody and can help provide broad insights into certain areas. It includes information on topics such as business and economy, crime and justice, education, health, transport and others.

Platforms such as the Scottish Government’s open data platform provide portals for accessing open data from across Scotland.