Over time, and as tools have shifted, some of these issues have been resolved or changed, but many remain. I've seen countless individuals work diligently to build something that looks great and works fast, but leads to shallow insights.
The most common struggle I'm seeing and hearing today revolves around the expectation versus the reality for AI's ability to provide insights. I've seen both internal team members and customers struggle with the expectation that AI access would be an easy click of a button that would single-prompt guide them to the next correct answer. One customer recently expected they could simply point Claude at the Alli MCP (the model connection to PMG’s proprietary operating system) to access their brand’s media data tables and have it explain the "whys" behind performance changes across media channels.
Companies are investing heavily in AI tools, and the cost is rising and harder to manage. Yet many organizations are not seeing the expected benefits, either in revenue generation or efficiency gains. If you polled your organization, would they say AI is having a profound impact on efficiency today?
The Gap Between AI Access and Useful Insight
Today, there is a gap between expectations and the reality for AI-generated insights when analyzing business outcomes. The central issue is a misunderstanding of the context and data structures needed to produce quality insights.
I do want to address the dangers of trusting LLMs to freely handle the math and calculations side of the analysis, but that’s just the barrier to entry when using AI to generate insights. The models are improving. Coding agents are getting better at building math calculations, but they still need guidance and validation. The bigger point is that even after you fix those challenges with skills and coding rigor, the insights will only be as good as the context you give the agents.
More Data Is Not More Context
More access doesn’t close the gap, whether it’s the open web or every table in the warehouse. Context windows keep getting larger. However, context windows can still be a limiting factor and degrade results, even as they grow. Context Engineering is the discipline that closes this gap. I see it as combining the right amount and content of context with instructions on how to best use it. Each new release and higher-thinking model has more capability to source some of this information itself, but then it becomes a question of curation costs.
Anthropic, which builds many of these models, makes the strongest counterargument: “smarter models require less prescriptive engineering, allowing agents to operate with more autonomy.” Yet the same perspective concludes that “context windows of all sizes will be subject to context pollution and information relevance concerns.” I have no doubt that Anthropic is correct on sourcing context, but the human-developed curation decisions and instructions on how to utilize it, and how to weight various portions of it, are still going to be critical, especially when getting into nuances of specific businesses.
“Insights will only ever be as good as the context you provide the agents writing them.”
Adobe research found that when an answer shares a keyword with a question, models hold near-perfect accuracy at long context lengths. However, when the answer requires an inference with no shared keywords, accuracy drops to about 26%. That’s the line between reporting “what” changed and explaining “why” something changed.
This becomes more nuanced as you move down the marketing spend funnel into specific channels, publishers, tactics, and creatives. Each requires its own set of contextual considerations. As we move up the organization, the detailed context from the lower levels is synthesized and carried forward. The details can be thinned and shed as higher-level agents run their synthesis, and additional macro context is pulled in.
I have only seen the components of what I would consider a complete implementation of an Insights Engine that provides recommendations. For one customer, we've built a weekly insights generator as a set of skills that pulls in external context. For another, a day-over-day search performance dashboard with insights down to the query and keyword level, along with monitoring of the change log and automated prompting of the team to provide change context, which we call the Context Logger. We also use it for social campaign insights. Lastly, we’ve built a Trends Recommendation Engine that reviews external trending topics and keywords that could affect performance and suggests actions.
All of these in isolation are pieces of the puzzle, but I haven't seen any implementation yet that pulls all of these (and more) together into an Insights Engine that can provide recommended actions across the entire marketing funnel.
Leaders should therefore treat an Insights Engine as a decision system, rather than another AI feature. It requires clear requirements, deliberate architecture, and thoughtful choices about when people should guide, review, or override the output.
Build Toward Decisions, Not Outputs
Start by defining the output. A good insights outcome provides the "why" changes happened and the recommended actions, both alongside the most obvious "what" changed and by what delta. The customer who wanted the "whys" behind their media performance changes was missing both the context curation and that decision definition.
Phase 1: Start by sourcing and organizing relevant content based on human decision-making. That may include AI-gen erated aggregations, but relevance must be determined by a human. The curation of context is still critical and should be done at the channel level first.
Phase 2: Build independent sub-agents for each channel's insights generation that can review performance data and relevant context to provide meaningful insights.
Phase 3: Aggregate the insights across channels, with additional macro-level business context, including longer trended data views, to get budget-level decision-making insights.
Keep Humans at the Critical Decision Points
Human review should be intermittent and intentional rather than a permanent layer in every workflow. For the foreseeable future, people will need to make the most consequential decisions about which context to include, how the system should use it, and how the resulting narrative should guide action.
Guardrails and validation are going to be unique depending on the situation and the business, but in general that means human review gates, reevaluation of context libraries, metric QA against non-AI-generated outputs, and, when projecting an outcome based on an insight, a confidence threshold.
At PMG, we use Alli to curate and organize data, both structured and unstructured, to store and provide context for insight generation. We’ve seen pockets of success within channels and practices, but we are still working toward a holistic approach that can provide confident insights and recommendations in an automated output.
The Compounding Cost of Weak Context
Currently, the cost of getting this wrong is frustration and token costs (which can be substantial), but as AI becomes more and more integrated into workflows, if the contextual foundation is not properly put into place, the cost could become exponential in the form of bad optimizations and negative performance. The ones who get it right faster will more quickly reach the fundamental promise of AI efficiency, and they will have the ability to focus on broader issues like strategy and token economics.
This has implications far beyond marketing: the human skill of providing context, guardrails, checkpoints and specific goals to AI in order to reap the benefits. I've known since that first Custom GPT that building a truly insightful recommendation engine is going to take discipline, work and lots of context. As our founder and CEO, George Popstefanov often states: “Data is everywhere, intelligence is not.”
