Pop Stars Without Clothing (not really)

Today marks a milestone in our countdown of the Top Ten Stupid Six Sigma Tricks. A subjective and unimportant milestone to be sure, but a milestone nonetheless—we have now reached the halfway point. Forget the New Year, this is cause for celebration.

To commemorate this trivial event, let us turn our laser focus onto more technical mistakes. So far we have been talking about systemic problems. Let’s focus now on some specific tools and how they are misapplied, leading to lost money and increased exasperation. I call this one Stupid Six Sigma Trick #5: Pop stars without clothing.

Well, O.K., not really. But would you have gotten this far if I told you that it was really Stupid Six Sigma Trick #5: Bad stats, part 1?

No! No! Wait! Don’t leave now. I promise it won’t get too dry.

For the fearless readers who are left, I’d like to talk about some specific technical errors made by Six Sigma’s technical resources—Black Belts and Master Black Belts.

One of the benefits of Six Sigma is that very difficult problems in business can be solved through the application of the scientific method (a.k.a. DMAIC). To get this benefit, we take a small cadre of good people and subject them to intensive training in advanced problem solving. The nature of difficult problems in business is that they are not generally of the “turn this to the right and the problem starts, turn it to the left and the problem stops” type. If they were, they’d have already been solved. In business, lots of things are happening at once and whether a factor is a cause of the problem tends to be assessed on probabilities rather than certainties, so our problem-solvers need to understand statistics.

Now I am going to tell you something that a lot of Six Sigma people don’t want you to hear: Doing statistics properly, especially in a business environment, requires knowledge. Sometimes a lot of knowledge. The nature of numbers is that you can do about any fool thing you can think of to those numbers, but the nature of inferential statistics is that you can get very wrong answers if you don’t know what you are doing.

Some in the Six Sigma world call for cookbook statistics—“Just follow steps one through eight, and you too can set up and analyze an experiment like an expert!” I wish it were so easy. Blindly entering data into a software program and executing a macro will not result in a warning window popping up, but it can lose your company millions of dollars.

How about a fun activity to prove the point? If you are a Black Belt or Master Black Belt, you could encounter the following situations. If you are neither, feel free to spread the love and forward this to your favorite Black Belt. Just don’t ask how they did, because—unlike my readers—they will lie. Grab a sheet of paper or open up a blank Notepad session, and record your answers. No peeking!

1. You are in the Measure phase, and, having just shown the sufficiency of your measurement system, you begin to track the time to answer customer complaints. Luckily, we gather a lot of data quickly, and you are soon ready to determine if this process is in control. However, you accidentally hit the histogram button and see on the graph that the data appear mooshed over to the left (near zero) with a long tail heading off to the right. Interestingly, before you can shut your eyes you notice that the normal curve superimposed on the histogram has helpfully placed natural process limits (±3 standard deviations) at –20 to +40 minutes. Do you:

a. Curse that stupid histogram button, click on the individuals chart button nearby, and use that to determine if the process is running with statistical control?
b. Curse your luck and test for normality to determine if you can even use the standard control limits that appear on an individuals chart to determine if the process is running with statistical control?
c. Praise Chebychev and his theorem and proceed to use the individuals chart to determine if the process is running with statistical control?
d. Do something else (record what you do)?

2. You are analyzing survey results with data that has been coded as follows: 1 = Disagree Strongly, 2 = Disagree, 3 = Neutral, 4 = Agree, 5 = Agree Strongly. Since it is a strategic objective to improve customer service, you are interested in determining if recent changes in your customer service activities have changed the answers on this year’s survey from last year’s to “Acme Inc. has the best customer service in the business.” Do you:

a. Use either the F-test (if normally distributed) or Levene’s Test (if not) to determine if there has been a change in the response variability, then use either the equal or unequal variance two-sample t-test to see if the mean response has changed?
b. Call your headhunter?
c. Do something else (record what you would do)?
d. Great question! I wish I knew!

3. After each customer complaint, your customers fill out a satisfaction survey. You are interested to find out if the time to respond to a particular customer complaint (as measured in hours between complaint received and complaint resolved, assume ratio level data) is related to the answer that person gives on the survey to, “I am going to use Acme Inc. for my future needs” with the data coded as: 0 = Yes, 1 = No.

a. Ah-ha! You can’t trick me! That is a correlation and I would use that r thing—what’s it called, the Pearson Product Moment Correlation Coefficient or something. And then I would, uhmm, see if that number is pretty big, you know, close to 1.0 or –1.0.
b. Oh wait! I remember now: I would calculate the r and then test it for significance using that formula that changes r and n into a t score.
c. Do something else (record what you would do)
d. Customers? My job would be a lot easier with fewer of those dweebs anyway.

4. Your company has three different locations, all similar in purpose, and using the principle that more vendors mean more competition, you have many suppliers for a critical process input. You wonder if having all these different vendors actually does affect your output, so you design an experiment to test this. You randomly select ten vendors from all the possible vendors. Further, you know that the processes are slightly different across those three locations, so you also want to know if the output is affected differently for the different locations. You set up an experiment, and you know that the model is:

a. A one-way fixed-effect ANOVA—you have one factor (vendor) at ten levels
b. A two-way fixed-effect ANOVA—you have two factors (vendor at ten levels and location at three levels)
c. Something else (record what kind of model you have)
d. A nova? Isn’t that what happens to old stars?

O.K., time’s up. You can find the answers and discussions here.

Now here is the point I am trying to prove. I made up general—and realistic—questions that could easily come up in a Six Sigma project, manufacturing or not. I am also willing to bet that fewer than 5 percent of trained BBs and MBBs got the right answers (well, except for number 3, where loyal readers might remember a hint from last month’s column).

Keep in mind that it is not the Black Belt or Master Black Belt’s fault if they do not get the right answer, O.K.? The fault lies with people out there saying that Black Belts don’t need to know statistics, and so they train them in how to move data from eye to finger (for data entry) without ever passing through the brain. If we keep training our Black Belts to put the data in the software and push a button, this is what we will have: a bunch of people who sometimes accidentally get it right and sometimes, in all ignorance, get it horribly wrong. These are not esoteric, academic differences. By choosing the wrong answers to the above questions, you could cost your company millions of dollars in lost opportunities or false conclusions.

So, bad stats cost money. Frequently, Black Belts don’t even know what (or that) they are doing wrong. Even worse, when an expensively trained Black Belt looks at the data and knows that something is screwy but does not have the training to know what, either they go ahead with the analysis they know, losing integrity and unknowingly making bad decisions, or maybe they go to a Master Black Belt, who probably doesn’t have the answer either and so faces the same conundrum. Is either of them going to stand up and say, “Oh my goodness! We were robbed in our training! I need to learn more!” Well, believe it or not, some do. But many do not, especially to their boss in charge of procuring training.

The effect of SSST #5

This is not about statistical purity and how we need more statisticians. I was trained as an engineer, and I value tools for their utility. These statistical tools are very powerful and very economical and can solve problems you thought were impossible to solve. They really can.

But look, the purpose of research done under the aegis of Six Sigma is to learn something that makes or saves money. If that research is flawed, though, wrong decisions will be made and money will be lost. Doing good research, perhaps especially in business, requires that you know a fair amount about the tools you are using. Would you want someone who had only studied 1973 motorcycle disc brakes to rebuild your new BMW’s anti-lock brakes? They might get it right, but there might be something they don’t even know they don’t know, and that would be dangerous.

I’m not calling for Black Belts to be made from statisticians. I think there are a number of benefits that result from training your process experts in statistical techniques. Heck, I think everyone should be trained to think statistically in order to understand the concepts of variation, even if they are not trained in statistics. However, there is just no substitute for competency, and to achieve competency you need to know more than how to enter data into an analytical program. The interesting—and difficult—thing about statistics is that the correct analytical choice comes from your knowledge of what is going on, not from the data.

If Black Belts do not achieve this competency, they will get wrong answers, and DMAIC will be seen as something that works only sometimes. To achieve this competency, and avoid SSST #5, they need to be trained competently, and my experience is that this is not happening for a large number of Black Belts.

So after all, perhaps it’s not the pop stars, but the emperor who has no clothes.

But I could be wrong.