Shifts in Rating Bias due to Scale Saturation

Rating scales are used widely in online surveys and cardinal peer evaluations. There has been work on modelling bias and reliability of workers under the assumptions that these parameters remain static and unaffected by the observed distribution of the underlying tasks. Anecdotal experiences in grading often points to the contrary and consequently in this work-in-progress we seek to construct experiments that clearly demonstrate bias changes. Better understanding of this phenomenon can lead to improved models and algorithms. We specifically observed that the order of tasks can lead to scale saturation which may lead to bias shifts.