Systems and methods for the determining annotator performance in the distributed annotation of source data
- Creators
- Welinder, Peter
-
Perona, Pietro
Abstract
Systems and methods for determining annotator performance in the distributed annotation of source data in accordance embodiments of the invention are disclosed. In one embodiment of the invention, a method for clustering annotators includes obtaining a set of source data, determining a training data set representative of the set of source data, obtaining sets of annotations from a set of annotators for a portion of the training data set, for each annotator determining annotator recall metadata based on the set of annotations provided by the annotator for the training data set and determining annotator precision metadata based on the set of annotations provided by the annotator for the training data set, and grouping the annotators into annotator groups based on the annotator recall metadata and the annotator precision metadata.
Additional Information
Application filed: May 27, 2016. Application published: September 22, 2016. Patent granted: February 20, 2018. STATEMENT OF FEDERAL FUNDING: This invention was made with government support under Grant No. IIS0413312 awarded by the National Science Foundation and under Grant No. N00014-06-1-0734 and Grant No. N00014-10-1-0933 awarded by the Office of Naval Research. The government has certain rights in the invention. CROSS-REFERENCE TO RELATED APPLICATIONS: The current application is a continuation of U.S. patent application Ser. No. 13/921,493, filed Jun. 19, 2013 and issued as U.S. Pat. No. 9,355,360 on May 31, 2016, which claims priority to U.S. Provisional Patent Application No. 61/663,138, titled "Method for Combining Human and Machine Computation for Classification and Regression Tasks" to Welinder et al. and filed Jun. 22, 2012, the disclosures of which are hereby incorporated by reference in their entirety.Attached Files
Published - US9898701B2.pdf
Files
| Name | Size | Download all |
|---|---|---|
|
md5:ca16110f5dff035e7c8605bd3277b1d5
|
52.5 kB | Preview Download |
Additional details
- Eprint ID
- 87079
- Resolver ID
- CaltechAUTHORS:20180613-144511351
- NSF
- IIS0413312
- Office of Naval Research (ONR)
- N00014-06-1-0734
- Office of Naval Research (ONR)
- N00014-10-1-0933
- Created
-
2018-07-05Created from EPrint's datestamp field
- Updated
-
2019-10-03Created from EPrint's last_modified field