There is one prediction that seems to come and go in every end-of-year business magazine or blog. This common prediction is more frequent than even calls for “The Year of Mobile.” The claim we “gurus” make and miss time and time again is that companies in the year ahead will mine deep data pools to develop brilliant, personalized recommendations on the products and services that you and I will want next. But decades since people started filling up storage facilities with data, we still see few marketers putting data to use. Amazon continues to be common case study, cited for its “You Might Like” feature that makes recommendations based on past purchases. But we all know how even Amazon’s algorithm fails to learn how to remove something you had wrapped as a gift, Banana Republic continues to send me offers for women’s apparel, and Jet Blue can’t even nail a basic user experience on its ticket-selling website. This is the kind of question that keeps me up at night in my quest to shift the world to Marketing with Meaning; but I think I have an answer.

Over the holidays I read an interesting article in the MIT Technology Review about how a handful of companies are actually putting their consumer knowledge to work to do some pretty interesting and successful micro-marketing. The concept, called predictive analytics, actually uses a combination of an individual’s specific personal information and purchase history, as well as traits from the entire customer database, to predict which types of products and services people might be interested in next. For example, the Cabela’s brand, which sells outdoor equipment through catalogs, online, and physical stores:

built a model that ranks customers from those with the best buying history to those with the worst. Next, it adds more than 15 predictive variables, including a customer’s preferred product categories and zip code. Each customer is then assigned a score from 1 to 100—the higher the ’star rating,’ the greater the revenue projection. The score determines whether and when Cabela’s mails that customer each of its catalogues.  The goal is ‘to determine how much each customer is going to spend with us over the next 12 months,’ says Corey Bergstrom, director of market research and analysis for the $2.6 billion company. Some high-value customers get special perks, such as more sophisticated telephone support. ‘If you deserve the white-glove type treatment,’ Bergstrom adds, “we need to know who you are so that we can go the extra mile for you.’”

Cabela’s has quadrupled the rate of response to its catalogs since implementing the system. That’s an almost unbelievable improvement on an age-old business model. The article notes that “predictive analytics software is a fairly small part of a $1.4 billion global market for business intelligence software”—and goes on to suggest that companies need more of a top-down commitment from the CEO to customer service support.

But let’s consider this question: Why can’t more companies make this commitment to smarter, personalized marketing through tools such as predictive analytics? Why are we getting the same catalogs, the same email specials, and the same credit card mail offers when so much of our data lies there for the interpreting—and sales are there for the taking? I believe there are four organizational behavior hurdles that stop this no-brainer from happening at the companies we work with and buy from every day:

Known Costs, Unknown Upside

We humans are often pretty bad at examining and judging risk and reward. For example, we have a bias against taking risks when we know the costs but are unsure of the rewards. I’ve been in too many meetings where companies are unable to achieve the revenue upside of a marketing investment because there simply isn’t available budget. No matter how high sales could go with a predictive analytics tool, for example, the $750,000 price tag for implementation is hard to swallow.

There is one piece of business where advanced data mining and personalized predictive analytics is hot, though: credit card fraud prevention. According to a recent article in Wired, in the U.S. alone, retailers and the big credit card issuers lose more than $3.4 billion a year due to lost and stolen cards. That is a very real number sitting on the income statements of these companies, thus providing them with a guaranteed bang for their buck by investing in analytics platforms. No wonder they are using tools to, say, predict that a card used to buy gasoline and then a piece of jewelry is likely a theft and flagged.

Too bad such technology isn’t being used nearly as often by these companies to make smart sales recommendations. The problem is that these lost sales opportunities never hit the books.

No “Burning Platform”

We humans and organizations also shy away from risks when things seem to be going well in our current situations. And such is the scenario of most businesses that could or should be using tools such as predictive analytics. Big, successful companies—and even not-so-successful companies that are still hanging on—are safe from the ravages of radical change, so they tend to keep on keepin’ on. It’s easier to do a study on something new and kill it in a committee, and then return to our everyday habits and do what we’ve always done. Sure, sales might even be flat or down a bit, but most jobs seem secure and we can always blame it on the economy, which we hope will pick up soon.

Individuals and companies take more risks when their jobs lives and livelihoods are actually on the line. Whether it’s Sears jumping into personalization and community tools that outstrip anything Walmart and Target are doing, or a start-up such as Zappos embracing killer customer service in order to stand out in a crowded marketplace and create a company from nothing, organizations that are standing on “a burning platform” recognize the need for something new much faster and more often than the leaders. They know their jobs are at risk so they take big risks that lead to big rewards. This is the heart of the concept of The Innovator’s Dilemma, and it is incredibly hard to overcome.

The Law of Big Numbers

Another classic marketing trap happens when a business is so large that improvement goes unnoticed and unappreciated. The big, national brands that most of us use every day rely on millions of purchase decisions and scale, scale, scale. The incentive is to spend your time and money on sweeping activities that impact many people at once. Personalized offers and actions fly in the face of mass marketers’ business models.

Take, for example, a major retail grocery chain that has a sophisticated data analytics tool that brands can use to gain insights about how people buy their products and shift marketing efforts accordingly. If I am the brand manager at a soup company, I can do basket analysis of what people buy, and how often and at what price points they buy my brand versus others, and then do some personalized offers through direct mail and checkout systems that the retailer makes available. Let’s say I spend several hours of my time testing various offers and hit a remarkable result—say, doubling sales at this one chain. The problem is that this retailer represents only about 1% of my annual sales. So after adjusting for time lost on other efforts and share that was simply stolen from another retailer, that doubling of sales in one chain has negligible impact on my business. It’s simply a waste of time to even think about. That’s why you see the same grocery store ads in the newspaper rather than a personalized list of specials in your email inbox.

Many Marketers Prefer Art to Science

A few years ago I was in a new business pitch with a major, global financial services company and brought forward an idea for a killer predictive analytics tool that would still be remarkable. I reasoned that because the company had access to individual credit card purchases, we could make smart recommendations about what people might be interested in now—and next. For example, we could see when someone was starting to pick up a new hobby or interest such as skiing or wine, and offer products, services, and experiences that could help him take that growing interest to the next level. Or when someone landed in a new city we could make proactive recommendations on restaurants. As I excitedly presented the idea to a CMO and his team I saw their eyes glaze over. They simply didn’t get it. And I think the main reason was that this company and all of its marketing staff was used to creating ad campaigns, not smart buying-recommendation tools. It simply didn’t compute.

I believe many people in marketing and advertising got into the jobs in large part because they lean toward art over science. We want to create cultural icons and memorable ads. We are attracted to creativity and imagination. And most of the big-company CMOs in charge today got to where they are because they learned how to judge whether or not a 30-second advertisement will break through the clutter during prime-time television. Many marketers find the concept of writing algorithms, analyzing personal buying habits, and testing hypotheses to be completely foreign.

…But Hope Springs Eternal

To paraphrase my favorite sci-fi author, William Gibson, the future of marketing is here; it just needs a user manual. The purpose of this blog, my book, and a big part of my everyday work at digital ad agency Bridge Worldwide, is to help guide marketers into the new world around us and give them new habits and tools. The first step for anyone in marketing is to recognize the need for change, and the reality that what made us successful in the past will not necessarily be the skills and methods needed going forward.