I spend a lot of time thinking about content quality and how to measure it.
This won't come as a surprise to anyone who's worked with me in the past. Much of my consulting business involves quantifying-- how many pieces of content created, of what types, with which characteristics, over what time frame, collecting how many pageviews, generating how many leads, etc.
But those are simple metrics of performance and production. There's not much challenge in them.
Measuring content quality, however, is something else entirely.
That's hard. That's complicated, challenging and -- particularly to some journalists -- controversial.
But I like the the difficult and disputatious. Which probably explains why I run in circle-eights and giggle whenever a client asks "how do we know if our content is good?"
New gig
A client asked me a version of that question a few weeks ago. And so, for the next few months I'll be helping a major B2B publisher measure quality.
How? The way I always do it:
I'll work with editors across the company to get consensus on a list of content characteristics that indicate quality and can be measured objectively. (For example, one measurement involves scoring a piece of content against a quality scale, i.e., an enterprise story is worth more than a story based on a press release; a story based on a press release is worth more than an edited press release, an edited press release is worth more than an unedited press release, etc.)
I don't attempt to measure quality indicators that are subjective. For example, I admire a well-crafted sentence as much as the next guy, but I won't count the number of elegant phrases per 1,000 words, because I can't get consensus on what makes for an elegant phrase.
More, better or both?
There can be quality scores for individual writers, brands, departments, etc.
The truly interesting work begins when you analyze those scores along with other metrics.
Suppose, for example, a B2B brand has a below-average quality score, as well as poor Web metrics (low pageviews, uniques, etc.) and below-average output (content/per author/per week) figures. The most common reaction is that the editor should be fired.
But what if the brand served an industry that was contracting and where company research indicated that potential revenue growth is nonexistent.
In such a case does it make sense to hire and train someone new? Or should you stick with the sub-par performer and make plans to shutter the brand in the near future?
Or imagine a brand that makes great money, scores well-above-average on quality and where output per editor is nearly double the company norm. Should you cut the training budget and give those folks raises? Turn the editor-in-chief into the division's content director?
How about a brand with extraordinarily high quality scores, painfully low output numbers and pageviews that are half the size of its competitors? Sounds like senior-writer syndrome, to me. You've got a great journalist producing a small number of wonderful pieces per month. Does the brand have a monthly print product? If so, there may not be a problem. But does the cost/expense ratio suggest a Web-only future for the brand? Then you have a huge problem.
Culture clash
In my experience, measuring the quality of content inevitably leads to a discussion about the quantity of content.
The assumption is that there is an inverse relationship between quality and quantity. Journalists tend to argue that they can do more or they can do better, but they can't do both.
But my experience is that this is untrue.
First, as a general rule, I've found that the worst writers almost always also produce less content than their peers. (There are, however, some talented writers with low productivity, just as there are awful writers who produce an awful lot.)
Second, I've found that even the most talented and prolific people in any content operation often have work habits that hurt quality and lower productivity.
The relationship between quality and quantity was at the center of a recent blog post by the always insightful John Bethune.
Citing a recent ad from ReadWriteWeb seeking an editor "“to produce 5 solid web tech news articles a day, 5 days a week,” John suggests that to traditional, print-based journalists this "new ethos of digital productivity is not just foreign, it’s al-Qaeda foreign. They are publishing terrorists, threatening the placid print way of life."
John goes on to say that although he sympathizes with his fellow print veterans, he wonders if "it just our old print ways, our preconceptions and work habits, that make digital workloads look so extreme? We say that quality will invariably suffer with increased output. But does it?"
Before you answer that question for yourself, I strongly recommend that you read John's entire post.
And pay particular attention to the comments, where my friend Robin Sherman raises his concerns about the quality/quantity relationship, where John responds to those concerns, and where I said:
"As an industry, we’ve sunk into a seemingly endless series of arguments in which we compare apples to oranges. We argue over the quality levels that can be achieved with Twitter versus what is possible in long-form narrative. That’s as ridiculous as comparing the works of Shakespeare to a newspaper headline. The truth is that there are great plays and there are great hedes. There are great stories and there are great tweets. There are epics and sonnets and there is also haiku. There are wonderful stories written on the fly. And there are magnificent works that consume a lifetime."
