I just sent this out to a common mailing list, so pardon me if you got this same message in an email, but Cynthia Zeller and I have put together a database to aid in determining consensus on Analysis. I call it the Analysis Consensus Evaluator (ACE). Ok, maybe it's a poor acronym...
The idea is simple: Rank the Minutiae, Clarity, Quality and overall Value/No Value determination (based on sample cases). Then have the database run a common inter-rater reliability statistical analysis on the data that shows how strongly people agree on what is of value, which is reportable by all the factors that are ranked. By using rankings, you avoid hard coding elements like minutiae counts which we all know are contextual.
http://www.fs2d2.com/ACE.zip
Lastly, I’ve coded a way to get the data back to me. If you use the database, I’d appreciate it if you sent me a copy of your data. I’d be interesting in seeing the results.
Enjoy!
Boyd