Background

This is the story of Bob, who is a senior analyst working in the Customer Strategy group of a US based retailer focusing on the online side of the business. Net Promoter Score was adopted as the corporate standard for measuring Customer Satisfaction. The corporate goal was to use the Net Promoter score as a visibility and accountability metric across all departments within the company.

photos/bob-buried-in-data.jpg

Bob reports up to the Chief Customer Officer and is tasked with managing the company’s Net Promoter Score program. A homegrown surveying system integrated into the CRM system is used to push out the survey and collect responses from customers. Five days after an order is delivered, each customer gets a personalized survey response email. Responses from each customer can be linked back to the CRM data. The responses to this survey is the basis of the Net Promoter Score calculation in the company.

Before using Insight Magnet for the analysis, Bob would spend most of his day and evenings in Excel. Over time he had developed a lot of macros that would import the data, tabulate the results and create customized spreadsheets that needed to be sent out to various departments. Finance, Sales, Merchandising, Marketing, Logistics, Customer Support and IT all had their unique needs and wants. Bob and his team just didn't have the bandwidth to go beyond the top 5 for each week.

Challenges

Timeliness of Analysis

The collection of the data, organizing it in the appropriate format and processing was a very time consuming process. The responses had to be downloaded from the CRM system, scrubbed, and loaded into Excel. Then an additional extract from the CRM system was needed to make the associations between the responses and the customer and order details. Based on the reports more data had to be imported from other systems.

Bob had a weekly meeting with the Chief Customer Officer and he would focus on having the slides ready for that meeting. Very often he would walk away from that meeting with a lot of follow-up questions that would take up considerable amount of time to respond. The best they could do within the constraints was to have a monthly NPS score for all departments and weekly report for the corporate level.

Analyzing the verbatim responses

Compared to analyzing the numerical responses, analyzing the text responses was challenging and onerous. Bob had tried using all sorts of Excel add-ins and macros but the results were mostly superfluous and not relevant to the business. He and his coworkers would typically decide to read a few responses and summarize the themes for the management reporting. He was aware that lot of useful feedback, suggestions, comments were not being utilized. Some business partners would ask for the verbatim responses occasionally -- usually in cases when their scores showed a dip. Bob knew that the root cause behind the variations lay hidden deep within the verbatims, but he did not have the tools to distill it out of the large amount of data that was collected every day.

Marketing wanted to start giving users a choice of language to answer the survey. That would mean that the verbatims would be in Spanish and French in addition to English. Bob and his team simply didn’t have the language expertise, available headcount or other resources to deal with non-English responses.

Why? Why? Why? -- the root cause conundrum

The weekly meeting resulted in some unhappy managers, typically those whose departments had a drop in the scores. And then the Why? questions would begin. Bob had studied the Total Quality theory used at Toyota and would see that play out every week. Unfortunately, there were no answers to many of the Why? questions.

Using his analysis, Bob could narrow down the variations with the hope of pinpointing the area of impact. The management discussion would focus on the known attributes and it was always inside out thinking. Stakeholders at the table would speak about their department, their responsibilities and their part in the process. But what mattered to the customer was usually based on his or her overall experience. Without the root cause it was left to the individual managers to come up with an action plan.

Further there was no way to track the causality between the action and the effect. So the scores may improve or decrease. But could that be related and attributed to the action taken? That was not always an easy question to answer.

Responding to ad-hoc requests

Bob was adept at using Excel and would work long hours to respond to ad-hoc requests. It was however, the coordination with other teams that was out of his control. If the scores were lower for a newly introduced product line, they would want to know how does that compare with other new products from the past 9 months. Bob would have to run around trying to get the data in order to attempt answering the question. The CRM system was run and managed by IT. So any extract that was not designed and provisioned by IT, meant that there was either a delay or custom coding work for Bob to get that data.

Generating an plan of action

Managers had begun to demand a starter plan of action. Their logic was simple -- if we are going to be measured by this metric, you should be able to tell us what can we do to improve our scores. That involved finding the segments or micro-segments of dissatisfied customers and having a first guess as to why they were not completely satisfied.

Our customers should be more satisfied today than they were yesterday. And they should be more satisfied tomorrow than they are today.

The company did try hiring an Industry expert and they had benchmarks for other comparable companies. It was gratifying to learn that their scores were in the top quartile as compared to the other companies but that still didn't help them follow through on the corporate mantra - Our customers are more satisfied today than they were yesterday. And they will be more satisfied tomorrow than they are today.

Solution

Bob reached out to Insight Magnet and discussed his situation. Having experimented with a few solutions before, he was skeptical and he clearly stated that “I do not want another Excel in the cloud”. To get started he uploaded the first five months of data he had from the Customer Satisfaction survey. The data also contained Siilent attributes for the orders. He was interested in analyzing the results along Payment Methods, Order Types, Promotions used, and an attribute tracking the value of the order as Low, Medium and High.

ss/nps_bob/nps-trend-asof-may31.png

Being an analyst at heart, Bob wanted to discover the trends and patterns in the data. He started off by looking at the time trend in the monthly NPS score. The first thing he noticed (in fact he already knew this from his Excel based analysis) was that the scores had dipped in April and May. It was the Why? question that Bob wanted to pursue further.

Without assuming detailed knowledge of the business changes, he wanted to evaluate what can the data and Insight Magnet analytics do to help him iteratively isolate and identify the root cause. He began by looking at the Siilent attributes of the orders. What he was trying to look for is the convergence and divergence of individual trends as compared to the overall trend. If the individual trends follow the overall trend, then it is unlikely that the root cause causally relates to the Silent Attribute in question.

ss/nps_bob/nps-trend-with-order-type.png

He began with the Order Type. Order type indicated the type of the order (as it was tracked in the CRM system). He did not see the prominent dip seen in April and May for the Order Type analysis. Bob compared the trends for the various values of Order Type and did not see a the strong deviation that he could observe in the aggregate results.

ss/nps_bob/nps-trend-with-card.png

Moving on, Bob next looked at Payment Method. Here the form of payment used by the customer was tracked. Available choices for this attribute were were Credit Card, Debit Card, Gift Card and Other. Of the four lines, there was a marked deviation as seen by the downward dip for customers using Debit Cards for their purchase. He continued and noticed a similar dip for Customers who had used a promotion with their order.

Need to dive deeper.

ss/nps_bob/time-segment-card.png

Based on the trends, Bob had noticed that the dip was for the past 60 days. He then zoomed in on the past three months and looked at the time segment analysis. For the time period, Debit Card was the second most frequent method of payment but ranked last in the Net Promoter Score. It also had the highest absolute number of Detractors.

ss/nps_bob/time-segment-promo.png

Close of half of the respondents had used a promotion but their satisfaction was noticeably lower than the other segments. So now there is a relevant segment of users that is trending lower than the overall and in turn is dragging the overall scores down. Having identified the “Who?” with a few clicks, the next step was to look for the “Why?”.

Insight Magnet provides a variety of options to browse the open-ended text feedback. You can certainly review the verbatims as entered by respondents but most users prefer to look at the trends that emerge by looking at the relevant subset of the respondents. So, Bob started with the keyword clouds for the same time period. Here, the top trending keywords are shown in a cloud view. Very basic text analytics is performed at this stage. Stop words are filtered and case is harmonized. When looking at the clouds contrasted with the segments based on the satisfaction scores, the fog begins to clear.

ss/nps_bob/keyword-bands.png

Looking at the detractors, and not surprisingly, Debit Card ranks high in the word cloud. This gave Bob the additional validation of some issue with Debit Card linked to detractors as he was able to combine customer entered input with the CRM captured data elements. In addition to the debit card the word clouds flag the prominence of immediate charges and shipping.

To confirm the localized correlation of Debit Card with detractors, Bob looked for Debit Card in promoters and passives. He noticed that Debit Card does not figure in the top keywords for either the passives or promoters.

ss/nps_bob/ngram-debit-card.png

Establishing more context is easy using the n-grams. While keywords are analyzed in isolation, now we look at the proximity of two or more words and show you the n-grams -- which are patterns of words as they appear in the verbatims. There seems to be something to do with the “free shipping promotion” and “debit card usage”.

ss/nps_bob/detractor-word-tree.png

Proceeding further, Bob looked at the word tree. This view shows a much dynamic visualization of the captured feedback. An analyst can refine the search in real time and look for patterns and trends in the feedback. Bob, who was not a linguist or did not have any format training in text-analytics or natural language processing was able to navigate and research only the verbatims needed for the root cause analysis. It was clear now that the Debit Card was getting charged when the order was placed. A customer will see her bank account immediately debited with such transaction and in most cases the industry practice is to charge the card when the order ships, not when the order is placed. Charge on a debit card is much more immediately noticed as compared to a charge on a credit card. There was also a correlation between the use of Debit Card and a promotion of some kind that Bob was not completely familiar with.

ss/nps_bob/keyword-band-delay.png

As Bob navigated through the word tree, shipment delays also caught his eye. Delayed order is never a contributing factor to customer delight, but he wanted to correlate this in the context of this specific time period.

ss/nps_bob/promoter-word-tree.png

Was this an across the board phenomenon, Bob wanted to know. He looked at the same word tree for promoters. And no, it was glowing with praises for excellent customer service that this company had worked very hard to put in place.

Armed with the “Who” and “Why”, Bob now had a very directed and well scoped mission to answer the “How”? He called marketing and inquired about promotions in the past 60 days that predicated the use of a Debit Card. He also reached out to the StoreFront Development Team and asked if there were any “back-end” changes in payment processing in the last 60 days. And he got his answers within a few hours.

Yes, there was a promotion that encouraged customers to use their Debit Card and get 5% off their purchases made with a Debit Card. And, yes the merchant bank payment processor had changed in the last two months. IT investigated and realized that while the credit card were pre-authorized before the order was processed, debit cards used on the order were actually fully authorized. This meant the funds were debited from the customer’s bank account the moment they saw the order confirmation screen.

To confirm the hypothesis and connect the dots, Bob asked for an extract from the CRM system that included the order delivery status. Getting that extract took more than a week and he did not have it for his weekly management meeting. Regardless, he was able to correlate the really low scores (1-3) to a congruence of three causes -

  • Customers had participated in the promotion for Debit Card Usage (promo code on the order).
  • Their debit cards were charged at order confirmation -- not on shipment (verified by IT)
  • They received their orders more than 4 days late (CRM extract on delivery status).

Having more than one of these in the customer experience provided a multiplier effect in the wrong direction. But the good news was, Bob knew what needed to be fixed.

After Bob’s presentation, the stakeholders at the meeting walked away with very specific action items that needed to be addressed in a timely manner. And fix they did. When Bob looked at the summary results three months later, here is what he found -

ss/nps_bob/nps-trend-asof-aug31.png

The dip in the NPS scores was short lived. A timely corrective action based on the root cause had brought the scores up. The Debit Card promotion was still running and the CFO was getting better DSO and lower transaction fees without sacrificing the satisfaction metric that was driving the business.

Takeaways

This is a classic case of a skeptical user trying out Insight Magnet without any customization or configuration. This story highlights the “out-of-the-box” experience. This workflow is particularly suited for an quantitative minded analysts who likes to be in the driver’s seat. There are several advanced features in Insight Magnet which simplify the process further by bubbling up items to review for you. Think of this as “stick-shift” vs “automatic shift” -- not good or bad, but a matter of choice and preference. Rest assured we have both trims available in Insight Magner.

Fast. Really Fast!

The onerous task of data loading was now reduced to a single file upload. After loading the file manually a few times, loading can also be automated. That leaves the analyst with a lot of time to focus on the analysis and research.

All the data

Insight Magnet can analyze large amounts of numerical, text, structured and unstructured data in real time. You are not forced to limit the data analyzed because of technical limitations or time constraints.

Zero Training

Bob is not a statistician, linguist or a database expert specializing in advanced queries. He understands the business and Insight Magnet provides him the capability of asking business driven questions that he can research purely based on the data.

Top down or bottom up

The dashboards are now ready within hours of loading the data into the system. You can start looking at top down if you wish. If researching a specific issue, you can navigate the system from the most atomic data element upwards.

Personalized Access

Gone are the custom Excel files that Bob used to email. Now all the stakeholders have a personalized login into Insight Magnet and they show up for meetings with their iPad in hand.

Easy in, Easy Out

It is very easy to get data into Insight Magnet and the same is true to get data out. Insights discovered can just as easily exported to other systems to follow through on the business decisions.