We Put A Lot of Faith in Machines

Letter from the Executive Director, October 2024

We put a lot of faith in machines, from machines that move us around—cars and trains on the ground, airplanes in the sky—to our GPS systems to navigate us and thermostats to heat or cool our homes. And even though they present regular and obvious problems, such as when they crash, malfunction, get hacked, or simply stop working, we also rely on computers to mediate our interactions with the world. Somehow, we increasingly consider these technologies more reliable, even as we have less and less understanding of how they derive the results we see.  

Perhaps part of this foundation of trust is based on the computational and algorithmic design of the first computers: You take input A, you process it in this way with input B, and it yields output C. Who wouldn’t be troubled by a calculator if they typed “3” and “+” and “5” and got something other than “8” as a response? After decades of working with calculators and ever-increasingly complex computational tools (which is where “computer” is derived from), we’ve all become very trusting of computational results. Perhaps an implied rationale for this trust is that if we presume computers are fundamentally just doing basic math and simple algorithms very quickly, why wouldn’t we trust them?

Another part of this trust foundation is because of the heavy reliance on standards.  Computational platforms rely on a vast array of technology standards to function. From the codes to represent characters in machine language to markup languages for embedding meaning and display, and from their communication protocols to their device interfaces, computers are highly standardized devices. A 2010 paper identified at least 250 standards deployed in an average laptop, but the authors acknowledged that the actual number could be nearly double that, and it’s likely that it has only increased in the 14 years since the original study. A key benefit of standards is to ensure consistency and reliability of processes. As they have in so many other products, standards have certainly improved the perception (if not the fact) of reliability.  

This trust is being applied across so many applications, extending well beyond its foundations, and it is now being applied to significantly more complex scenarios. Years ago, when libraries purchased a discovery service, such as an abstract and indexing service, it would have been unthinkable to accept “just trust us” as a response to asking what was included. As Google became ubiquitous in the 2000s, people came to trust that “everything” was included in its search results, despite this being demonstrably false, particularly for non-English or non-northern-hemisphere content. While things have improved, it’s certainly not a solved problem. We’re far from having access to everything ever created or distributed in digital form. However, so long as one isn’t using someone else’s user profile, one can expect that the top results of one query will generally be the same from one search to the next. We’ve slowly moved to a place where we’re comfortable—at least implicitly—with “just trust us.”

The large language models that support the current generation of artificial intelligence–supported tools are more stochastic, which means they are far more probabilistic and random in their outputs. One shouldn’t expect that the same input would always yield the same output. But if the outputs aren’t the same, and we don’t know what the inputs are, and there isn’t any citation to sources, how reliable should we consider the outputs? And yet, with the hype surrounding large language models, it seems we are set to double down on the situation of misplaced trust because of a lack of transparency.

All these ideas were embedded within the discussions during the recent NISO Plus Global Online event, which NISO hosted last month. The session on Content Discovery in the Age of AI with a call for collaboration between publishers, libraries, discovery service providers, and policymakers to establish best practices for the ethical use of generative AI (GAI) in academic publishing. The NISO Open Discovery Initiative has launched a survey to gather insights from academic institutions and service providers on how GAI might impact content discovery. This survey marks a first step toward developing community-driven standards that will address key issues such as copyright integrity, data governance, privacy, and bias prevention in the evolving GAI ecosystem. Perhaps one of the concrete outcomes of this informative discussion will be potential new work from NISO, in the NISO Plus model where discussions lead to ideas, which lead to outputs and solutions. 

This was just one of the interesting topics during the virtual conference that intersected with trust and AI systems. There was another session focused on accessibility of content and the need for greater adherence to and adoption of accessibility functionality, with discussion of the use of GAI tools to support accessibility. While there’s some potential to increase capacity and accelerate application of accessibility features, there are obviously risks. For example, if any of you have used the automatic generator of image alt-text in some word processing tools, you may have noticed the vague, simplistic, and even inaccurate data generated by those tools. Among the ideas suggested during the session was guidance for the community on how to use AI-based tools to support accessibility. Beyond GAI, there was another thread of conversation about potential new work by NISO on describing quality alt-text  in the production of content and identifying who would be best engaged in that process. Far too often, even when it does exist, the alt-text field doesn’t contain valuable information and simply repeats the caption. Good alt-text should include relevant features of the image and why it is included, adding context to the image’s inclusion, not just describing its visual characteristics.

As we have done for each NISO Plus event, a report of the outcomes and potential next steps will be shared with the community. We have also announced the forthcoming NISO Plus in-person event in Baltimore, which will take place on February 10-12, 2025. We’ve released a call for session ideas for the conference.  We’re hoping to establish a cadence for larger worldwide virtual events in the fall so that they  interact meaningfully with the smaller in-person event, so ideas generated in one event will feed into and produce outcomes for discussion at the next event. We look forward to hearing your ideas about the next big idea that NISO should consider working on, be it on AI, discovery, accessibility, or something else about interoperability. We look forward to your session suggestions!

Sincerely,

Todd Carpenter