The Beetles sang “money can’t buy you love”. It’s not as catchy, but I also think it is true that money can’t buy you trust. In this article, we are going to take a look at how trust can be established in AI-enabled systems. To understand this, we must first look at the 'Build vs. Buy' decision because the source of your software often dictates how much flexibility you have in creating systems of trust that fit your business. Finally, we will explore how trust can be established in AI-enabled systems and some of the challenges in maintaining trust over time. Let's dive in!
Build vs buy
When deciding on the best way to solve a problem using software, there is always a key decision: should you build or buy? I.e. should you buy an existing “off-the-shelf” product and configure it or build a bespoke system that exactly meets your needs? It is a trade-off between initial investment and long-term gain. Off-the-shelf systems are often cheaper (at least in the short term) and quicker to implement. However, they can fail if the system is clunky or doesn’t cover all of your needs. In contrast, bespoke software often requires a much larger initial investment of time and money, but if done right, it can transform your business in the long term. At ClearSky, we earn our living by designing and implementing systems where the initial investment in the bespoke software that we build is far outweighed by the long term gains.
However, the decision is never as simple as buy vs build. It is not a binary choice. Even off-the-shelf products will almost always require work to integrate the system with the rest of a particular company’s IT infrastructure. Conversely, bespoke systems are seldom 100% custom code. If we were to build everything from the ground up, we would spend months building basic services such as user authentication, file storage systems, email sending systems, and more before we got onto solving the particular problems of our client’s business. Even when building bespoke software, we always use a range of pre-built components to handle generic functionality. This allows us to focus our efforts on creating the parts of the system that fit our customers’ particular needs.
Careful consideration is needed when deciding how to integrate off-the-shelf components into a bespoke software system - extra care is needed when the technology is novel and constantly evolving, like AI! A key way that we manage this risk when designing systems is to create an integration layer between the third-party library or service and then bespoke code. This integration layer can be large or small, depending on how closely the functionality of the third-party component meets the needs of the requirements for the system being built. There is always a balance to be struck between leaning on the dependency as much as possible (less development effort required) and writing integration code to “fill in the gaps” and get the functionality working exactly as we need for the application being built (more development effort required).
Shooting at a moving target
In order to effectively make progress, we need to avoid shooting at a moving target. Everyone who is even vaguely aware of AI knows that the technology is moving forward at warp speed. Seemingly every day a new capability, framework or tool is released. With this incredibly fast-paced innovation, it is hard to know where to invest engineering effort and where to wait for the technology to catch up.
Here’s an example. 2 short years ago, at the beginning of 2024, context windows were small (a few thousand characters), and so engineers developed solutions such as rag to manage context efficiently. Fast forward only 2 years to today, and we have models with a 1 million token+ context window readily available. This doesn’t eliminate all context engineering problems; for example, context rot is still an issue (the tendency for models to lose focus in very long documents). However, the exponential increase in context window size has made it significantly easier to integrate with LLMs when using large source data. The focus now is on cost management and speed of execution.
So where should we focus our effort? I think the answer is at the interface of humans and AI. There are whole classes of problems in this space that I don’t think can be engineered out by the LLM platforms. For example, arguably the biggest challenge facing businesses adopting AI is trust. AI implementers rightly advocate for “human in the loop”, but what does this mean in practice? We will be limited by our ability to adopt AI if humans have to cross-check and verify every line output by an LLM. Equally, we are far away from AI agents being given trust to ultimately be recognised by the law.
Trust is a very nuanced and human thing. Designing systems that foster trust between humans and AI requires a deep understanding and empathy with the users of the system and their goals. I don’t believe that there will be platform level solutions to these problems. To get the most out of the human AI relationship, we need to carefully design systems that are appropriate for the particular scenario and people involved.
Gaining trust over time
It is worth noting that trust is not a static thing. As the old saying goes, trust is earned. Therefore, when designing systems, we need to consider how trust will evolve over time. We need to design mechanisms that allow (human) users to progressively trust the AI components more and more over time. When the system initially goes live, the users will naturally be cautious; this is the same for any new system, AI-enabled or not. However, if designed well, over time the system will allow human users to give more trust to the AI components and take on more and more responsibility, freeing up the human users to do more and more of what humans are good at (what AI can take on vs what humans are good at is a conversation for another day!). Next, let’s look at a concrete example of how a system can be designed to support incremental trust.
In our Future Labs initiative at ClearSky, we have been developing the prototype to explore how AI can help improve the speed and accuracy of the loan underwriting process. In Polaris View an AI agent reviews the loan application (known as a proposal) along with supporting documentation and highlights key issues to the underwriter. The underwriter then reviews the issues returned by the AI agent and makes a decision to approve or reject the loan. As you can imagine, the underwriter is still ultimately responsible for the decision and so they have to review and check the AI’s output. Let’s take a close look at how trust is built here and the key distinction between false negatives and false positives.
If the AI agent does not find any issues for a proposal and recommends approval, the risk is a false negative, i.e. what if the AI agent has missed something? There is no easy way for the underwriter to check for a false negative. If there is no trust in the AI agent, then they will have to perform a full review of the proposal. Conversely, if the AI raises an issue and recommends rejection, the risk is a false positive. The issues raised by the AI agent in Polaris View always tie back to evidence in the proposal. This means that it is easy for the underwriter to check that the issue is correct and, if so reject the proposal. In summary, it is easy to see how Polaris View improves the time to reject, but much harder to see how it improves the time to approve. This is where incremental trust comes in.
Polaris View is just a prototype, but let’s imagine it is being rolled out to a client. Initially, there would be low/no trust in the output from the AI. If the AI flagged no issues against a proposal, then a full human review would be required to confirm that the proposal can be accepted. Over time, data could be collected about how many of the human reviews found issues that were missed by the AI, which would achieve two things:
- A feedback loop to inform improvements to the system to make the AI more accurate.
- An error rate metric i.e. how many times the AI missed issues vs the total number of proposals recommended for approval.
It is worth noting that humans are not infallible, and there will also be an error rate for the human underwriters that the businesses should be aware of. If a business is aware of this human error rate, then they could make a decision that the AI agent will be trusted once its error rate matches or surpasses that of the human underwriters. In order to ensure AI is adopted timeously, we shouldn’t be aiming for a perfect AI; we should aim for an AI that is demonstrably more reliable than the status quo. However, as mentioned before, trust is not a static thing. Even traditional systems need continuous auditing as they are developed, in case any functionality regresses. In fact, due to their very nature, AI-enabled systems will likely need a very careful approach for continuous audit due to the fact that they are non-deterministic.
Trust Regression
For those of you who haven’t heard the term “non-deterministic”, it basically means that if you ask an LLM the same question multiple times, you may not get the same answer. This is because there is inherent randomness in the behaviour of LLMs. “Traditional” (non-AI) software is different; it is often reasonable to expect that traditional systems are deterministic. What’s more, when making changes to traditional systems, we can often predict with a high level of certainty what the impact of the change will be. This is not the case for AI-enabled systems. A small change to a prompt could change the behaviour of the system in unpredictable ways, especially when multiple agents with multiple prompts are collaborating asynchronously.
So what does this mean for building trust? It is a tricky problem with no easy answers. One answer would be to completely reset the trust level when a change is made to the system and revert to complete human audits until the trust is rebuilt. This would be prohibitively laborious, as modern software solutions are continuously updated and improved. In traditional systems, a key strategy for reducing the possible blast impact of changes is to break them up into bite-sized testable chunks. This approach can be directly applied to AI-enabled systems. At a basic level, instead of a single prompt that completes a complex task, perhaps it can be broken up into multiple smaller prompts, each of which can be tested and verified independently.
Neither of these solutions is a silver bullet. The exact mechanisms for building and rebuilding trust are very human and often bespoke to the business context that the system is deployed into. The processes around trust management are also as important as the core functionality that the system provides. If a system is not trusted by the users or their businesses, then it won’t survive. As a consultancy building bespoke software for our clients, we at ClearSky are spending a lot of time thinking about how we build mechanisms to establish and maintain trust in our next generation of AI-enabled software.
Conclusion
Paying close attention to how systems are rolled out, adopted, and trusted is not a new idea, of course. It is “just” change management, which is a very well-documented process. However, people implementing software systems have a long and illustrious history of falling at this last hurdle of change management.
With AI-enabled systems, the change management problem isn’t really any different. What differs is that the reward/cost of getting it right/wrong is higher than ever. To ensure we get the highest reward from the next generation of AI-enabled software, we need to think early in the design process about how the new system will facilitate progressive adoption and build trust.
The exact mechanics of how trust is established are very particular to the workflows and nuances of each organisation. Trust is calibrated to each organistation’s appetite for risk and their specific human workflows. I’d say that it is unlikely that you will be able to find an off-the-shelf solution that fits these needs well enough.
If you’d like to learn more about our prototype Polaris View then please get in touch, we'd love to share the work with you.
Written by Matthew Jeorrett Techincal Delivery Manager