Artificial Intelligence (AI) can produce biased outputs. In part, this is because unrepresentative data is used to train, validate and test AI. To remedy skewed datasets and train fairer AI, advocates, policymakers and developers alike call for more comprehensive and systematic data production and processing about diverse people’s bodies and lives, including disabled people. Yet this response rests on a number of assumptions: that more disability data can correct AI bias, that disability data used to detect and correct AI bias can be effectively safeguarded, and that representative data is possible while adhering to human rights standards and norms. Drawing on disability data, I argue that we should be cautious about these assumptions when regulating AI.

Assumption 1: More Disability Data Can Correct AI Bias

As noted by other scholars, the term bias is undefined in doctrine. In reports and literature in this area, the word “bias” appears to be used in relation to at least three distinct effects related to AI: (1) inaccurate or discriminatory outputs that disadvantage historically marginalised individuals and groups, (2) AI outputs that represent individuals or groups as a negative stereotype or the perpetuation of stigma, and (3) the tendency for human operators to favour computer-generated recommendations. This lack of clarity about what is meant by AI bias is important because calls for more representative data primarily address only one form of bias: inaccurate or discriminatory outputs.

Despite Mehrabi and colleagues observing that “the field of algorithmic fairness is a relatively new area of research and work still needs to be done for its improvement”, the need for representative data to correct AI bias has long been accepted as a universal truism. This broad acceptance of the remedy has neglected to consider what representation actually entails for diverse identities and experiences. For instance, representative disability data presents a challenge not merely because disability is underrepresented in existing datasets, but because disability may resist representation through the kinds of categorisation and aggregation on which AI systems depend. Representation is not merely a matter of collecting existing information. It depends upon decisions about how people are categorised, measured and defined, making data production a social and political process rather than a purely technical one.

Disability is a difficult category to quantify. Under human rights law, disability is understood to arise from the interaction of a long-term impairment with environmental and attitudinal barriers that hinder participation in society (CRPD, Preamble (e), art. 1). For this reason, disability data is heterogeneous and contextual. Because of this complexity, health data is often used as a proxy to identify impairments, despite disability extending beyond impairment alone. But even then, the disability umbrella includes a diverse range of people. Subgroups are small, and variation remains high, particularly when you factor in intersectional identities and life circumstances. Disabled people may be highly identifiable within datasets because their experiences, impairments and life circumstances often place them at the statistical margins. Consequently, achieving representative disability data is difficult and, for some groups, may be practically impossible without comprehensive data collection, the shortcomings of which I discuss below. Guo and colleagues noted this diversity and the impact on AI training data as early as 2019:

“Fairness issues for PWD [people with disabilities] may be more difficult to remedy than fairness issues for other groups, particularly where people with particular classes of disability may represent a relatively small proportion of a population. Even if included in training and evaluation data, they may be overlooked as outliers by current AI techniques”.

Wald highlights that, of all the “protected characteristics” covered by the UK’s Equality Act, “disability is the most heterogeneous” (p. 2).

AI relies on statistical relationships derived from large datasets to generate predictions and outputs. Although expanding and diversifying training data is frequently presented as a means of mitigating bias, this assumes that sufficient representative data can be obtained for all groups. In practice, some populations are small, rare, or characterised by experiences that resist meaningful aggregation. These groups may remain persistently underrepresented in datasets, limiting the capacity of data-driven approaches to eliminate bias. As such, if representative data cannot remedy the problem it is intended to solve, claims of necessity become increasingly difficult to sustain.

Assumption 2: Disability Data Can Be Effectively Safeguarded

The Artificial Intelligence Act includes provisions intended to safeguard personal data used in AI development (art. 10,  art. 4a(1)(a-f)[proposed]). Implicit in these provisions is the assumption that such safeguards exist and are effective in practice. The focus of these efforts has primarily been on the safe production, processing and storage of data. For some time, data used in AI training was assumed to be effectively unretrievable once incorporated into a model. However, this assumption has been challenged by research demonstrating that, under certain circumstances, personal data can be extracted or inferred from AI systems. Of particular concern from the perspective of disability data is that rare or unique information may be at greater risk of being retrieved, inferred or exposed by AI. As discussed earlier, many disabled people are statistical outliers or otherwise unique within datasets, potentially making their data more vulnerable to exposure, inference or re-identification.

Beyond the risk of extraction from AI systems, the current negotiations on the Digital Omnibus proposal raise a more fundamental question: can some forms of disability data ever be rendered truly anonymous? The proposal includes amendments to the General Data Protection Regulation that seek to clarify the concepts of identifiability, personal data and pseudonymisation. These developments are particularly significant for disability data, given the potential hyper-visibility of some disabled people within datasets.  In addition, people with intersectional identities and life experiences are at risk of re-identification through data linkage even where direct identifiers have been removed. In some circumstances, a small number of linked characteristics, such as migration background, sexual orientation and disability or health status, may be sufficient to render individuals identifiable, raising questions about whether effective anonymisation can be achieved in practice.

Assumption 3: Representative Data Can Conform to Human Rights Standards

There are significant human rights concerns associated with incentivising the production and processing of greater quantities of data about marginalised groups, extending beyond privacy alone. One of these is coercive data practices that remove human autonomy, choice, and control. A routine example is the requirement to provide unnecessary personal information, i.e. information that is not relevant or important to the service being provided, to access an essential service when no alternatives exist. Ironically, maintaining human autonomy in interacting with AI was one of the concerns during the drafting of the Artificial Intelligence Act (Recital 27). Now, in pursuing a representative dataset, we are seeing the potential for human autonomy to be diminished earlier in what is becoming known as the AI supply chain. Changes in regulation are not only opening up existing personal data stores, but they may also lead to practices that produce more sensitive data. As highlighted by leading human rights monitoring organisations in Europe: “we are concerned that the proposed amendment could effectively legitimise a substantial increase in the scale and scope of the collection of sensitive data” (p.7).

The costs and risks associated with data disclosure are not evenly distributed, a point I explored in my research on disability data in refugee contexts. There are many legitimate reasons for marginalised groups, including disabled people, to withhold information about themselves and their identities, challenging the assumption that truly comprehensive or representative datasets can ever exist. Yet the regulatory framework in Europe is shifting to incentivise the production and processing of data as a resource for developing AI.

It has been noted for some time that power is increasingly concentrated with data controllers and digital technology providers and deployers rather than data subjects, with significant human rights implications, including for disabled people. Disabled people have a great deal at stake in AI. Consistent with the disability movement’s principle of “Nothing about us, without us”, the CRPD obligates states to closely consult and actively involve disabled people in matters affecting them (CRPD, art. 4(3)). Further, the CRPD recognises disabled people’s “individual autonomy including freedom to make one’s own choices” (art. 3(a)). When it comes to personal data, including sensitive data, this means that, at a governance level, disabled people should be closely consulted and actively involved in shaping regulation that determines how disability data are categorised and used. At the individual level, the CRPD’s emphasis on autonomy, participation, and self-determination underscores the importance of enabling people to self-identify and exercise meaningful agency over decisions regarding their data, including decisions about disclosure and use.

The European Union and its Member States are parties to the CRPD. This legally binding commitment requires them to translate their obligations into Union and Member State laws, policies, and practices (CRPD, art. 4(1)(a-e)). The Committee that monitors the implementation of the CRPD has mainly championed the production and processing of disability data under article 31 to address discrimination and ensure the implementation of rights. But changing digital technologies and power structures demand closer scrutiny of disability data practices by the CRPD Committee in future jurisprudence.

Conclusion

While representative data may appear neutral and directed towards a worthwhile objective, for disabled people and other groups who remain minorities or outliers within datasets, it poses significant human rights risks that extend beyond privacy. If human rights standards and norms are to be respected, individuals should retain meaningful choice and control over their data, including the right not to disclose sensitive aspects of their identity. Framing personal data as a resource in an urgent race to develop AI places increasing pressure on these standards while offering uncertain benefits. As the disability data example demonstrates, the assumptions underpinning the push for ever-greater data collection and processing deserve far greater scrutiny than they currently receive.

Pretending bias can be addressed by representative disability data creates no incentive to change practices. As a disability human rights scholar, I rely on engineers to propose solutions to biased systems. I note that those who raise concerns about representative disability data failing to address bias have sought alternative approaches, such as Wald, who suggested personalisation rather than classification. But the broader question I am posing is: why are we allowing infringement on disabled people’s fundamental rights when there is poor evidence about the effectiveness of disability data addressing bias?