AI will expose your data quality issues

We wrapped up another successful Open24 + MOps-Apalooza 2024 conference in the first week of November. Talking to all the Ops professionals at this event is a great way to take the pulse on what everyone’s struggles and priorities are. Though AI was a hot topic, I was surprised by how slowly it is gaining traction within the Ops space. There seem to be three major roadblocks slowing down AI adoption:

• InfoSec mandate – For most medium to large enterprises, the information security team has blocked the use of commercial AI services due to security, compliance, and IP lawsuit concerns. These organizations are building internal AI models, but they are slow coming.
• Usability and feedback loop – Current GenAI is better at open-loop tasks such as summarizing or writing drafts. It is more difficult to get AI to produce precise and repeatable results. To help you understand the challenge, here is a good article on why AI can’t spell the word “strawberry”. Prompt engineering often takes as much effort as traditional methods to get precise and repeatable output. The feedback loop for how to improve AI output is also different from traditional software configurations, and the impact of the feedback is not immediate and precise.
• AI doesn’t exist in a vacuum – No one is ripping out existing solutions wholesale and replacing them with AI versions. Rather, the need is to inject AI into existing processes and technology stack for improvement, which introduces integration, change management, and data quality complexities just like every other technology introduction.

Data quality is obviously one of the biggest prerequisites for AI success. AI is possibly the most data-driven technology humans have ever invented, so the classic garbage-in, garbage-out challenge applies to AI in spades. As such, AI is likely the one technology that will expose your data quality issues the most, which may help to raise the awareness and urgency of dealing with your GTM data quality issues.

Prompt quality = data quality

A prompt is the instruction you give to an AI model to execute the task you want it to perform. Just as you would expect from giving instructions to a junior employee, AI’s ability to successfully complete the task heavily depends on the quality of the instruction; thus, the emergence of the new profession, “prompt engineer.” For most RevOps use cases, constructing AI prompts will involve embedding various GTM data into the prompt, such as account name, website, contact name, LinkedIn profile, and email address. If your GTM data is of poor quality, it would be hard for AI to perform the task successfully. For example, if you’re asking an AI agent to perform web crawl and research on a prospect, but the person name and LinkedIn URL you provide in the prompt do not match and the company name and job title are from two jobs ago, you’re hampering your AI agent right out of the gate. And unlike a human employee, AI has yet to master common sense and ask good clarifying questions.

Trust but verify your hallucinating AI

If you have read about or used AI at all, you are familiar with AI’s tendency to hallucinate. On a side note, if you’re open to reading a hilarious yet extremely informative academic paper on this topic, ChatGPT is bullshit discusses why “hallucination” is the wrong word and why “bullshit” more accurately describes this AI shortcoming. After reading this paper, you can enlighten your friends at the next cocktail party that “bullshit” is actually a technical term. Humor aside, hallucination is a major technical challenge we have to account for if we want to use AI to scale operations. For the second time in this newsletter, comparing AI to a junior employee turns out to be a very useful framework to understand what it takes to get the most out of AI. Any manager knows you have to trust but verify, especially with junior employees, and this is equally applicable to AI. To verify AI’s work, you will need high-quality data. If you don’t verify your AI’s output, your AI’s hallucination can pollute your data with fiction and create more data quality issues than it can remediate, at record speed.

Until AI can explain itself, you will have to

All the cutting-edge AIs we see today are black box technologies. AI cannot yet explain how it comes to a certain conclusion; as much as we would like to believe it, AI doesn’t actually exercise logic and reasoning. Instead, it’s pattern matching and trying to generate humanlike responses. The paper cited above also explains that what people believe to be AI explanations are actually not logical explanations, but simulated responses of what a human explanation would sound like—facts be damned. So, if you’re going to use AI and AI can’t show its work, then when challenged, you will have to explain AI’s output on behalf of AI. If your data quality is good–and you therefore have confidence in AI’s output—you at least have a fighting chance to explain why and how using the data you prompted AI with. If you have no confidence in your data’s quality, then you shouldn’t have any confidence in the AI’s output, and certainly shouldn’t try to explain or defend AI’s output.

Personalized content is irrelevant without precision targeting

One of generative AI’s most powerful use cases is the creation of personalized content to power tailored buyer’s journeys. However, personalization only works if it is delivered with precision to the target persona and buying group. For example, you can create highly personalized content targeted at a CIO, but can end up using that content on a security engineer prospect because you don’t have his job title or cannot properly segment his title to a job function and a job level that can guide the targeting algorithm. All that investment in AI personalization is at best a wasted effort; at worst, it backfires and creates a terrible buyer experience. Personalized content is only relevant if you can target precisely, and good-quality data is essential to the process. Without the ability to target precisely, AI personalization is not so much a precision GTM weapon, but a weapon of mass destruction.

AI is an executive’s wormhole into bad data

Unfortunately, in many companies it’s hard to make executives care about data quality and invest in it. One of the reasons is that most executives consume data in the form of reports and dashboards. They don’t ever see the raw data and the sheer amount of human effort that goes into cleaning and preparing the data for these reports. A data analyst might have to spend 20 hours collecting, stitching, and scrubbing data every month just to generate one report, but none of that pain, effort, and cost are visible when an executive looks at a report. One of the hypes of AI is how easily it could put answers at the executive’s fingertips. The vision being sold by AI vendors and visionaries is that every executive will have a co-pilot on their desktop and smartphone home screen, where they can type in their questions, such as “What is the difference in ACV and NRR between our ICP and non-ICP customers?”, and AI will serve up the answers instantaneously. For those of you data people who have managed to stop ROTFLYAO, AI is promising your executives a wormhole directly to your company’s raw data. Out goes the human abstraction layer. Maybe this is a good thing, because AI may finally give the executives an unadulterated view of how bad most companies’ data quality is. Maybe this will finally make the executives care about and invest in their data quality and infrastructure.

Curious about how AI is transforming RevOps? Check out the sessions from Open24 and MOps-Apalooza 2024.

Recommended Resources