appropriately, it will not lead to the desired outcome. Managers need to understand how to gather and use performance information and be empowered to communicate performance information to employees in a way that will produce the appropriate outcomes as well as support organizational goals.
BANKS Industries, a client of yours, is attempting to train managers to use performance information effectively. It has implemented a system where both the immediate supervisor and the department head are evaluating employees using a graphic rating scale. The employees being evaluated are all call center representatives for a help desk that handles problems with its personal computing devices product line. You have had an opportunity to review its job descriptions and performance evaluation document.
A group of twenty call center representatives has already been evaluated. The evaluation uses a graphic rating scale that results in an average score per employee. The graphic rating scale is weighted as follows:
•Needs Improvement = 1
•Meets Expectations = 2
•Exceeds Expectations = 3
A score of 2–2.49 is considered acceptable. Anything over 2.5 exceeds expectations. Scores under 2 indicate a need for improvement in at least one area. BANKS Industries has decided that for call center representatives, the evaluations will be conducted by both the call center manager and the district manager. Both managers are present and walk the floor on a regular basis, and both are reasonably familiar with the employees. The call center manager reports to the district manager.
After looking at the ratings from this round of evaluations, management is concerned that there may be some inter-rater reliability issues. It looks like the two managers do not always agree on ratings for these employees. Here is a comparison of the average scores:
Employee Name Call Center Manager District Manager
Alfrey, Lisa 2 2.2
Armstead, Catherine 1.8 2.3
Bartlett, Kelly 2.2 2.7
Cannon, Violet 2.7 2.5
Cantley, Candice 1.2 1.3
Cost, Tonja 1.8 2.5
Gomez, Andrea 2.6 2.3
Hahn, Frida 2.6 2.8
Hasselhoff, David 3 3
Holman, Patricia 1.9 2.3
Karter, Kim 1.9 2.3
Peraza, Teresa 1.6 2
Reynolds, Burt 2 1.8
Royals, Dillon 2.3 2.6
Schull, Leonard 1.4 1.8
Scriber, Jacqueline 3 2.4
Smith, Bert 2.3 1.9
Stenberg, Madison 2.4 3
Wellington, Dawn 2.8 2.4
Wydra, Cindy 2.8 2.5
The client is not sure whether this is an effective approach to evaluating performance. You have been contacted to consult with this client and assist in identifying issues and training managers to better use the appraisal instrument.
•Define and discuss a graphic rating scale as an appraisal instrument.
•Conduct a statistical analysis of the two sets of data using a t-test in Microsoft Excel to determine whether the differences in ratings are statistically significant (include your computations).
•Explain the results of your t-test, including a table/graphic representation and the implications of your results.
•Assess the appraisal instrument and make at least two recommendations for improvement.
•Identify training topics for management to ensure better inter-rater reliability in the future. Include a justification.