Published on November 24th, 2014
“No one has tried this kind of work before in such a way — could I, as the first attempt, be successful?”
For three months, research associate Arielle Cardona attempted to code for incivility within online news comments in new ways –using paid Workers in Amazon’s Mechanical Turk, undergraduate students in the Mechanical Turk sandbox, and by machine training the content analysis program WordStat. Using a previously coded data set as the comparison “Exemplar” coding, she compared how well these groups and program were at accurately detecting incivility.
Although we never reached perfect agreement, tweaking and editing our instructions and question format consistently made us better. This document serves as a progress report, with some thoughts on the next possible steps in research.