What about Ms. Forbes' proposed use of "science" and "numbers" to provide evidence? These words sound good, but let's examine her plan under a strong light. We can do this only in general way because we would have to pay a significant amount to see the actual survey questions, but that's okay because the basic research plan is the most important part.

The proposed study seems to be a "before-and-after" study, a type which I discussed in a post several weeks ago. It will involve comparisons between participants' answers to survey questions before they take the online course, after they have finished it, and probably at several points between the beginning and the end of the course. If there are differences between the earlier and later responses, can the changes be attributed to the effects of the classes? No, of course not. A large number of uncontrolled, and unknown variables are confounded with the effect of the course. For example, the passage of time after the major winter holidays (if the survey began now) may lead to a calmer, more satisfactory family atmosphere. As Spring comes on, the possibility of spending more time outdoors may take some pressure off cooped-up families. Similar causes of change could be found for other times of year, too, and for all sorts of altered circumstances. The point is that a "before-and-after" study, even if it shows clear improvement for participants, cannot provide evidence that the treatment, intervention, course, or whatever was the actual cause of that change.

Let's look at some other issues in this deceptively appealing research design. How about the fact that parents are electing to start a class at a given time? Why would they do that? The most common reason is that they are feeling that things are going quite badly at home. People don't choose to do a parenting class at the same time that they are patting themselves on the back for good parenting. Matters are likely to be at a low ebb when people decide to seek help. And what usually happens when matters are at their worst? Unless we're talking about a fatal illness, the chances are that things will spontaneously improve a bit. Family problems don't stay the same in their severity all the time, but they get a bit better and a bit worse for a variety of reasons. (The term "regression to the mean" is used to describe the tendency to go downhill a bit following a really good period, and to improve after a really miserable time.) A big problem with "before-and-after" is that there's no way to tell how much (if any) improvement was a result of an intervention like parenting classes, and how much would have occurred because of regression to the mean.

There's another important problem in Ms. Forbes' design. Remember how this worked? People signed up and paid for the course. They were also invited to-- but didn't have to - participate in the research component, which involved answering survey questions electronically and anonymously. So, which people are likely to sign up in the first place? Would these be parents with open minds or even some wariness about the course, or are they likely to be people who expect to benefit and anticipate an excellent experience? And, which of those registrants are most likely to volunteer to do the surveys? It's hard to imagine that someone would do this without high positive expectations for the course, and even a wish to provide help and support for Ms. Forbes and her institute. People with very negative attitudes would not have signed up for the course to begin with, so they will not be in the survey group; people who are indifferent probably can't be bothered to do a survey. We are left with a group of survey respondents most of whom are enthusiastic, and whose enthusiasm can make their assessments of the course's effects very positive.

It takes a lot more than a before-and-after study to prove this pudding or similar interventions. However, to be quite fair, if there is more to her study than there appears to be, if I have missed a point, I hope Ms.Forbes will comment and correct me.