Trusting AI Series | Blog 4, Part 1
AI holds enormous potential, along with some very real risks, but the hype around it is getting out of control.
Follow the Trusting AI series: subscribe to The Scanner to get Part 2 when it is published.
I started writing this blog series a few months ago after reflecting on conversations about AI at the CDIA Connect conference in May of this year. Those conversations centered on the question "Can I trust AI?" I envisioned a four-blog series, with this blog being the final one.
In the months since starting this series, however, the concern in the mainstream press about AI-related risks has increased significantly. I will therefore split what would have been the final blog in the series into two parts, with this one focusing more conceptually on the process of creating real solutions (and separating that from the hype) and the next one getting into more specifics related to our areas of focus.
My goal for the overall series is to help professionals in our industry move forward with AI-enabled solutions that are truly helpful for their businesses and careers. I’m doing that by sharing my perspectives and hopefully connecting a few dots in ways that resonate with their own experiences.
My Bottom Line Up Front (BLUF): AI holds enormous potential, along with some very real risks, but the hype around it is getting out of control. Move forward while staying grounded in reality, and there are great opportunities to create value for your company and your career.
A useful AI solution starts with a specifically defined business task and measurable outcomes, including the related quality metrics. The model is only one part of it. Real value comes from breaking the work into the right pieces, combining AI with non-AI elements and controls, testing each step with realistic data, and keeping experienced professionals involved where judgment and nuance matter. In our testing, the most famous models are not necessarily the best fit for each of those pieces.
Now, let's shift to some perspectives on rolling out real solutions versus what we read and hear about in headlines.
For several months I have been feeling a growing disconnect between what I read about AI in the mainstream and financial press and what I see in our work. Most of the attention is on the "frontier" models (i.e., the models considered to have the most advanced overall AI capability), but most of the work we (and others) are doing does not need them.
At Bridgeforce Data Solutions, we carefully test our products across a wide variety of very specific tasks, in realistic use cases, using large volumes of ultra-realistic synthetic data. Exactly how we do that is part of our secret sauce, building upon our very deep and specialized expertise and data (I'll share a little more about it in my next blog). As part of this testing, we evaluate the performance of various models for specific sub-tasks.
What we see is that frontier models are not necessarily any better than far lesser-known models when evaluated in this careful way. No AI model can reliably perform the end-to-end tasks we test for on its own. Once you break the tasks into their appropriate pieces, develop the right non-AI elements of the solution, and evaluate performance at each step, the frontier models do not necessarily perform the best.
When I compare notes with others who are evaluating AI for use in specialized, highly regulated uses, they are seeing similar things.
I think an analogy would be helpful here (for anyone who read my last blog, you know that I like to use them).
In this case it is useful to think about AI models like engines. Think of the latest frontier models like Formula One racing engines. It's an interesting engineering challenge to make a better engine for Formula One, and you attract a lot of attention, but those are not the engines you would use for all applications.
Next, think of AI-enabled solutions as vehicles of all types, designed for their specific tasks. The engine is a critical component: if it does not work, then the rest of the vehicle is not particularly useful. But you can build very different vehicles, with very different quality, using the same engine.
Now think about how likely you would be to use a Formula One engine if you were tasked with creating each of the following:
That sheds light on one of the related disconnects. At any given moment, public attention focuses upon perhaps 2-4 models from the big names. The last time I looked at our model options in Amazon Bedrock (which I covered in my information security blog), there were over 100, pointing to how most real work is being done by the models not making headlines every day.
Abstracting back a level, the recent hype made me reflect on lessons from innovating throughout my career.
New technologies or ideas that seem exciting to the C-suite, boards of directors, and investors can quickly generate enormous hype. Turning that hype into real value, however, requires changing how companies operate and how they actually work in the belly of the beast.
That process requires getting into the messy details of real processes and real data, with people who understand those processes and data, and iterating through all the surprises until it is working well. The gap between the hype around a new technology and the work it takes to create real value is, in my view, the core reason for what is sometimes called the Gartner hype cycle.
A new technology attracts interest.
Early successes are generalized too broadly.
Implementation difficulties lead to delays and disappointment.
Teams work through details and create sustainable value.
Value becomes established and the curve flattens.
On the hype cycle, the period climbing toward the peak of inflated expectations is a time of great excitement about what some new technology could potentially do. The people selling the new tool (in this case the AI models) say that this time will be magically different: this innovation is so great that it will not be hard at all to make it work with the messy, real-world processes and data that you have.
There are generally a few examples of very specific use cases in which the technology really does do some amazing things, but those use cases end up being generalized more broadly than reality can support.
An example from earlier in my career comes to mind. We were marketing Yankees' co-branded credit cards by email to season ticket holders, and the response rates were fantastic. The next thing we knew, a big-name consulting firm was extrapolating those results across the entire bank, using assumptions that were completely unrealistic outside of that very specific audience. Those assumptions led to big budgets that returned very little.
At some point, reality starts to win. The new technology might be valuable, but it is not magic. The realization of how much work it would take to make the technology practically useful leads to delays, going over budget, and lots of overall disappointment.
Work continues, however, and the organizations that keep working through the details can create significant value. This is the slope of enlightenment, where sustainable value is created but takes longer than the early hype suggested.
For really powerful technologies like the Internet and AI, I think of a curve that is shaped a little differently. For many years, I have referred to this subset of the hype cycle concept as the Amazon curve. It starts with the same shape: quick excitement, followed by a bubble burst and disappointment (imagine Amazon stock from 1997 to 2001). The next step then starts the same way, with the long process of creating real value. The difference is that the eventual value creation ends up being enormous, even more than the initial hype, but it takes a long time to get there.
I believe this will be the shape for AI technologies. It feels like we are in a period of ever-increasing hype (which in this case is also bringing significant fear), but that the long term will generate enormous amounts of value.
Excitement creates high expectations.
Reality brings a reset and disappointment.
Progress takes sustained work over a long period.
In Matt’s concept, eventual value exceeds the initial hype.
I also think it is important to keep in mind some special characteristics about AI hype. Most of the attention on AI is focusing on the sorts of tasks that it can do very well. Some of those capabilities are shockingly impressive, and quite real. AI is rapidly and radically changing, for example, how software should be developed and tested.
The disconnect happens when imaginations run wild on what those impressive capabilities could mean for other use cases. Those capabilities are not magic. They are the result of math, using linear algebra and probabilities. I love math. I was a math nerd as a kid. It can help solve hard things, but it is also less helpful in some areas than others.
A model might be able to write computer code and do math impressively well, but that does not mean it could respond appropriately to a credit dispute without being part of a much more comprehensive solution.
Keep in mind that AI models depend on large amounts of good training data. A special subset of cases can be iterated in a simulation, effectively creating a near-infinite training data set. There is, therefore, a huge difference between AI performance for things that can be iterated infinitely versus those that can't, and the hype is about items with fast iteration loops.
For things that can be iterated infinitely, like playing chess or Go, AI can perform better than any human. For coding, it can learn from enormous amounts of training data and iteratively code and test, so it can perform impressively well too. For math, it can keep trying things and then check its work to see if it worked.
Much of life, however, does not have such fast feedback loops, and extrapolating AI performance into those areas would be a huge mistake. For many things, we do not know if we made the right decision for a long time, perhaps even years. Even when we do find out, the feedback loop is messy, with many variables involved.
This is a big part of why human experience and expertise remain, and will remain, enormously valuable. Human beings are capable of learning from a very small number of data points. How many times do you need to burn your hand on a stove before learning to be careful? We also have intelligence across many different categories, not just the math that powers AI.
Now, let’s start to tie all of this together, since I think the total picture is actually very encouraging for the sorts of thoughtful professionals who just might be reading this. Let’s use the diagram below to illustrate how this really works, and therefore what is next.
“Look at the possibilities!”
The messy middle
Understand the details. Plan the work. Measure results.
Procedures + controls + professional judgment
Where human thinking and nuance matter most
Excitement has two effects
Upside Secures resources and the push to overcome inertia.
Downside Hype can hide the effort, creating frustration and disconnects.
Detailed knowledge and nuance solve the problems here. This is where human interaction and judgment make the difference.
In this visual, leadership sees a shiny object (AI, the internet, etc.) and gets excited about the long-term possibilities. They can also underestimate the effort it takes to create it and make it useful from a business perspective.
This has both good and bad points. The downside is that this is where the hype grows, causing various frustrations and disconnects between leadership and the teams doing the work. The upside is that the excitement drives resource allocation and enough of a push to start overcoming the organizational inertia to do nothing or move slowly.
The "messy middle" is where all the details and processes must be understood, planned for, and measured to confidently and successfully implement new technology. The knowledge here usually cannot be modeled and gamed out like chess or Go. Instead, it embodies procedures and controls developed from years of effort that still require professional knowledge and judgment to operate well.
This messy middle is the area that many professionals know best, even better than their leadership. Domain experts here understand their processes better than AI does, and likely better than AI ever will. AI solutions are useless without the right controls and understanding wrapped around them.
If you want to see what this looks like in our own work, the interactive walkthroughs of our AI Research Assistant and AI Resolution Engine show how we built narrowly defined AI tasks with controls and human review around them.
I'm going to come back to a couple of things from my last blog, which I think need to be re-emphasized in this context:
"The key is this: Ask the most detailed questions about the things you know the most about. If you know the business outcomes that must occur and how to measure them, then be demanding and specific on those points. If the solution meets those metrics, then great. If not, then it is not good enough."
"The practical takeaway is that trust in AI needs to be narrowly and precisely defined to have meaningful answers, and for business and compliance professionals, the focus belongs on business outcomes. Which model is being used matters far less than how well the overall solution works, measured against existing quality metrics."
Those detailed questions are the ones that reflect how an organization differs from its competitors, and how work actually gets done. Yes, we all need the flexibility to change our processes and execute in different ways, but only if the new processes and tools address the nuances that the professionals involved know are important.
That holds whether you work at a bank, a credit union, a fintech, or a mortgage, auto, or student lender. The use cases differ, but the need to define the task, measure the outcome, and wrap the right controls around the model does not.
Wrapping up this blog, very real and valuable AI solutions can be built and deployed using the tools available to us today. Bridgeforce Data Solutions and many others are doing that and improving them every day.
We are living through a hype cycle that has many characteristics in common with prior ones, but with the volume turned up. This one really does have enormous potential. It also has many disconnects, which make the hype even bigger.
The key to success is to see this in the context of the decades of technological advancement before it. The hype around every shiny object throughout my career has promised that it will not require all of the detailed, messy work that prior efforts needed.
That hype eventually leads to disappointment. The real value is created by the diligent professionals who kept working through the messy details, whether internally or with vendors.
I hope this was helpful. Next time, I'll dive into some more specifics from our use cases, but I felt the need to address this a bit more broadly this time.
Please let me know any feedback that you have, and best wishes for your own innovation.
If your team is evaluating AI for credit reporting, furnishing investigations, or dispute resolution, we can talk through the business outcome you need and the controls and measurement required to support it.
Share your current priorities. We’ll follow up to discuss relevant DQS capabilities and next steps.
No. In our testing, frontier models are not necessarily any better than far lesser-known models. Once tasks are broken into their appropriate pieces and performance is evaluated at each step, the frontier models do not necessarily perform the best.
No AI model can reliably perform the end-to-end tasks we test for on its own. A model might write code and do math impressively well, but that does not mean it could respond appropriately to a credit dispute without being part of a much more comprehensive solution.
It is Matt's term for how really powerful technologies like the Internet and AI play out. It starts like the hype cycle, with quick excitement followed by a bubble burst and disappointment. The eventual value then ends up being even more than the initial hype, but it takes a long time to get there.
AI does best where tasks can be iterated quickly, like chess, Go, coding, and math. Much of life does not have such fast feedback loops. For many decisions, we do not know if we made the right call for a long time, perhaps even years.
Ask the most detailed questions about the things you know the most about. If you know the business outcomes that must occur and how to measure them, be demanding and specific on those points.
Connect With Us
Connect with company leadership and follow the official page for perspectives on credit reporting, disputes, data quality, DQS, and industry developments.
