My worst and most intriguing experience in recent months happened one Friday in March when my wife, two teenage daughters, and I were trapped in the Apple store in downtown Palo Alto.  Our salesperson there spent a full four hours trying to get something done for us with AT&T.  I thought it would be pretty easy but turned out to be absurdly complicated --  we were buying one new iPhone and replacing another that had been stolen from my daughter.  The Apple guy ultimately succeeded despite dozens of obstacles put up by AT&T's people, system, and rules (which were interpreted differently by just about every employee he and we dealt with, by the way).  Our Apple guy succeeded only through his raw persistence and because, as he explained, he had learned that such a high percentage of the AT&T people are so incompetent, that sometimes the best thing to do is to just hang-up and start from scratch (in hopes the next one will be competent).

I believe that, in the process of making this happen, at least 10 different phone calls were made to At&T, some by him and some by us.  During this time, we talked to virtually every employee and manager in the place, and each assured us that our salesperson was among their best people.  The problem, they explained, was that AT&T can be impossible and time-consuming to deal with -- and their system meshes very poorly with Apple's in many ways.

An added problem is that the AT&T people are apparently on a  flawed incentive system. So rather than actually trying to what was best for us as customers or relationships with Apple, there was constant upselling. Our salesperson also reported that, on multiple occasions, AT&T employees resisted doing what was needed to get our phones working because it meant they would get no incentive pay (I never quite understood this, but I heard him say things to AT&T employees things like "I know this will mean you don't get your incentive, but this is how what we have to do it to serve the customer.")   I was amazed to find that AT&T apparently does not have a dedicated hot line that enables Apple salespeople and "Geniuses" to connect directly to AT&T people who are especially trained to deal with Apple stores given that Apple sells so many AT&T accounts -- but apparently that isn't the case.

I would love to have a film of our experience in the Apple store to show to AT&T executives.  We were there so long that virtually every employee in the place at one time or another came up to us and told us there favorite story about how much AT&T sucked and how lucky we were to have the most skilled and persistent person in the place helping us. Also, quite a few customers overheard the stories or asked us what was going on, and jumped into the conversation with their own bad experiences.  This was a busy Friday night at the store closest to Steve Jobs' house, and in fact, it was the one where he made a surprise appearance the day the iPad was released.  Perhaps AT&T ought to spend less money advertising and brag less about how wonderful they are and devote more attention to fixing their defective system and improving their training.  I especially believe that they don't quite fathom how much damage their incentive system does because it focuses their people away from helping customers and toward getting as much money as possible out of them. Perhaps they should read Steve Kerr's classic on the damage done by lousy incentive systems:  "On The Folly of Rewarding A, While Hoping for B."

In any event, for better worse, I suspect that the effect of all this is that tens of thousands of customers a day get to experience Apple's competence and AT&T's incompetence side-by-side in a public arena.  This contrast not only affects the particular employees and customers involved in a given transaction, it often spreads to many others in the setting -- especially when it is a long ugly one like ours.

If you are an AT&T executive, you don't need a fancy survey, you don't need a marketing consultant, just walk into a few Apple stores and ask employees and customers what they think of your company and why. And stand around awhile and watch the dynamics surrounding the especially bad customer experiences.  Apple stores create experiences that teach customers and key opinion leaders to despise your company and see it as greedy, incompetent, and out of touch. 

As always, I assume I am biased and my experiences are not representative. Am I being unfair to AT&T? Have others had good experiences with them, especially in Apple stores?  Note that I had good experiences when I simply bought my iPhone, but whenever anything at all complicated has happened, it has been awful.

P.S. Check out the reactions to an earlier version of this post on my personal blog. I was especially struck by the comments from former Apple Store employees, like the one who wrote that "In my experience about 1 in 10 customers will face hurdles taking several hours due to ATT."  And his reports about the incentive system were equally troubling: "The incentives for ATT reps is absolutely correct, and they'll even activate unsold phones that Apple stores are trying to activate which gives them credit for the contract, but then stops Apple's systems from selling the phone. Once an iPhone is activated, it cannot be sold, it must be activated post-purchase."