Beware of Averages – Especially with Benchmarks
I ran across a situation recently that made me scratch my head. A very smart group of managers were discussing their newly discovered “benchmark” for a particular classification of job. While I do get excited about gaining insight through the use of matrix, I was disappointed in the overall effort or structure they used to develop their “benchmark.”
Here was their system or process for developing a broad benchmark for a job position. Seems they took a number of people who had been selected as “top performers” and then ran some assessments on this group, averaged the data across the group and decided the data from the average was good for a usable benchmark.
Well, this plan had several major flaws and will probably get them in trouble legally with the EEOC and other watchdog agencies. Here is a list of the flaws I found in their system…
- No Performance Standards – when I asked what performance standards they used to determine who were included in the “high performer group,” I got this deer in the headlights look. This was my first clue of a lack of due diligence in their process. It became obvious to me they had used a very subjective selection process – probably a good buddy technique – in other words, they selected people they liked rather than performance based selection without the personal bias.
- No Control Group – There was no effort to – at a minimum – select a cross section of low to mid level performers as a control group to compare the data with from the “high performer group.” This is a minimum effort to have any chance of passing a review by the Federal and State agencies.
- No Same Job Specification – A couple of questions uncovered a lack of clear job position selection. Here again, there was no discussion about identifying truly similar jobs for their study. In other words, not all job positions were exactly the same – in fact, some were very different and took highly specialized talent to achieve any results. This meant the benchmark was done with people who actually have unique positions within the job classification (which was quite broad) thus impacting the “benchmark” averages.
- Lack of Clear Job or Position Analysis – Related to the above, yet, different since here I’m talking about a clear discussion and bias free analysis of the Job or Position itself – without the personalities of people creating bias. One of the fastest ways to create a REAL benchmark is to “let the Job Talk” and thereby identify the prioritized attributes necessary for the job to be highly successful. This is the key for future hiring and selection as well as serious “On Boarding” processes to improve both performance and retention.
- Review of the Variances in Data – Once again our group failed this step as they never looked at the detail of the arrays of data, the diversity of the variables. If they had looked at this information, the information would have shown large gaps between the participants of the study with very few “clusters” or tight knit data indicating a truly unique pattern. Without this tight pattern, the variables are too large and the hiring and selection process takes a hit regarding finding the potential high performer.
- Lack of a Clear System for Comparison – This team of managers did not have a clear system for comparing the “Job benchmark” to candidates and understanding the uniqueness of each candidate in relation to the position. A system is simply a process designed to uncover potential high performers during the selection process. The system provides a step by step process for every candidate to follow and the system is legal and will pass any examination by the labor agencies.
- Lack of Targeted Interview Questions – Here are your tie breaker and verification system. Performance based behavioral interview questions which uncover the details and depth of knowledge held by a candidate. Without this structure, interviews take on a “wild west” form and can get the company and the interviewers in trouble with the labor agencies. A true system will supply a set of targeted questions to be used by the interviewers to verify the capabilities of every candidate.
There you have the list of fatal flaws of the manager group I talked with about Benchmarking their top performers. While they probably did get some valuable information and experience in the process, it had major holes in the validity of their benchmark. One gross error was the huge gaps between the so-called high performers thus making the average meaningless in my opinion. There was a lack of definitive data separating the high performers from the low to mid level performers. And, there was no data to show what a low or mid level performer even looked like regarding the data.
In summary, take the time to set up a real “benchmark system”, one that focuses upon the job or position and what it will take for the job to be successful. This exercise alone has identified many unique attributes necessary for job success. Use the system with a complete set of assessments that uncover the how, why and will be necessary for job success and then use a similar assessment tool to use in matching to the job.
Latest posts by Voss Graham (see all)
Tags: Beware of Averages, fatal flaws to watch out for in benchmarking, hiring and selection, job benchmarking, job or position analysis, performance standards, Targeted Interview questions, using a control group, Voss Graham