The injustice of bad data

Every day, data is used to make decisions that impact our lives. Now, as artificial intelligence is being trained to automate decision making, data plays an increasingly important role. However, there is a problem with data. It holds a bias.

Dr. Ali Karimi, PhD, has taken a scholarly interest in understanding the impacts of data bias. He and his team received a Transdisciplinary Connector Grant in 2023 and in 2024 was appointed a SSHRC Tier II Canada Research Chair in Race, Social Justice, and Digital Media within the Faculty of Arts. 

"Having Ali as a Tier II Canada Research Chair is a tremendous asset to the Department of Communication, Media and Film," says department head, Dr. Samantha Thrift, PhD. "It recognizes the significance of his research and provides vital funding to advance this work, including the establishment of the Data Justice Research Centre."

As the director of the Centre, Karimi is advancing research with a mission to examine the impacts of data-driven governance and to highlight the ways digital inequality affects marginalized populations. The Centre’s projects are broadly focused on census data, digital archives and surveillance.

What data justice looks like

Karimi has firsthand knowledge of the impacts of biased data. Raised in Afghanistan, he was familiar with the persistent surveillance by aerial blimps that recorded civilian activity. 

As privacy is a luxury not afforded to the poor, surveillance data contributes to justice issues.

"If you are a marginalized person, if you are a poor person, you basically have no privacy" Karimi shares. "The state feels entitled to scrutinize every aspect of your life. Data justice looks at this issue and why our institutions scrutinize the powerless more than the powerful."

While surveillance collects a significant amount of data on the people of Afghanistan, there is no government census to officially record demographics. Due to what he calls “state opacity” portions of the population are simply not seen or considered, when it comes to distribution of economic and political resources. 

A government census is the most significant data set a country has. Despite its importance, a census is not immune to data bias. Questions that are not asked create gaps in the data, excluding marginalized groups.

"Sometimes it is difficult to identify these gaps," says Karimi. "The way we do it is to look at what questions are asked, and how data is collected and processed to identify the flaws and gaps through those processes."

The 2020 Australian census provided an example of the impact of asking the right questions. The decision to include the Hazara dialect, in addition to the Dari language, revealed an 87 per cent increase in the Hazara population on the 2021 census. This addition to the census gave significantly more visibility within the data.

Issues of bias, based on data gaps, are not exclusive to the census. Digital archives such as the federal government’s Library and Archives Canada are another source of data used in decision making. Data bias exists in national archives, as people ultimately decide what is preserved or not.

Transdisciplinary impact of data justice research

The Data Justice Research Centre is where researchers from the humanities work alongside colleagues from computer science and engineering to better understand how data bias correlates with justice issues.

Mansi Singhal, MSc'26, a former student of Karimi's who is currently pursuing a PhD in computer science, came to the Centre as a bioengineering student for her master's thesis. Singhal's research looked at the impact of Alzheimer’s research on patients within marginalized communities. She found gaps in the data, revealing a bias.

Singhal described how most of her Alzheimer's research is based on data sets that do not represent the diversity of the population they attempt to serve. She looked at 24 publicly available and widely used data sets from around the world. The majority of the data came from North America and Europe.

"We have around 80% white, 10% Black, 5% Asian and 5% other communities such as Hispanic and minority groups," Singhal says about the data demographics. "This is the underrepresentation and overrepresentation of certain groups. That is the data bias."

The results of data, skewed toward white populations in high-income countries, giving a limited understanding of the disease in the context of other populations. Singhal’s research found factors like genetics, diet, lifestyle and environment were not considered when working with Alzheimer's patients from marginalized groups or low-income countries.

AI and data issues

While the Digital Justice Research Centre's work has predominantly focused on the Global South, they have also been conducting research on the levels of AI use in Canadian federal and provincial decisions making.

Similar to the data bias issues Singhal discovered in the Alzheimer's data, putting too much faith in the data used by AI tools comes with risks. Health policy decisions, made through AI technology, will only be as good as the data the machines are trained with. Singhal maintains the impacts of having incomplete data will be felt most by marginalized communities.

"If we have data that is biased," says Singhal. "It's like garbage in, garbage out."

Karimi’s team at the Data Justice Research Centre is committed to finding more equitable and accountable approaches to data collection, use and governance in an increasingly digital world.

"Dr. Karimi’s research shows us that the ways information is collected, organized, and used are deeply connected to questions of power and inequality," says department head Thrift. "His work asks important questions about whose lives are made visible through data, whose experiences are overlooked and how information systems can reinforce existing forms of injustice.”