Copied
Docs

Contact Us

If you still have questions or prefer to get help directly from an agent, please submit a request.
We’ll get back to you as soon as possible.

Please fill out the contact form below and we will reply as soon as possible.

EMPLOYEE LOGIN
  • Home
  • Getting Started
  • Annotate
  • Tasks
  • API
  • Recipes
  • Tutorials
  • Integrations

Automated Feedback

Updated at February 25th, 2025

Generating Feedback Automatically

It is possible to generate quality feedback for a set of annotations by inferring the mistakes committed from the edits made to the annotations. Instead of flagging feedback manually, which is time consuming, users can focus on fixing the annotations and the quality feedback will get generated automatically and passed to the original annotator. 

Workflow Setup

Within the production annotation workflow, any submission made on a step of type Review or QA will generate automated feedback. The feedback will be generated by comparing the submissions for the previous steps in the current round. While QA steps are only allowed to be the last step in the workflow, Review steps can be configured to be at any place within the workflow.

Feedback

The following feedback can be generated from fixes made on annotations:

Name Detail
Missing Shape A shape was added in the final submission (it was missing from the original submission).
Extra Shape A shape was removed from the final submission.
Inaccurate Shape  A shape had its position modified in the final submission. See the thresholds section below for details on what edits count as errors. 
Incorrect Label A shape had its primary class modified in the final submission. We consider nested outputs with names `Class` or `objectClass` to be primary classes for the purpose of automated feedback. 
Inaccurate Attribute An attribute which is not the primary class was changed on a shape in the final submission. For these fixes, we will store in the Comment of the feedback what specific attribute changed, the original value, and the final value. See the reporting section below for more details.
Missing Metadata A non-annotation output was added in the final submission (it was missing from the original submission).
Extra Metadata A non-annotation output was removed from the final submission.
Incorrect Metadata The value of a non-annotation output was modified in the final submission.

Automated Comments

We also generate extra information within the Comment field for the Incorrect Attribute and Incorrect Metadata errors. We store the original and final value of the answers as text in the following format:

 `ORIGINAL_VALUE` -> `FINAL_VALUE`

If the value was not filled in either submission, we display the value as `<empty>`.

Inaccurate Shape Thresholds

Not all position edits should count as errors. We allow configuration of thresholds to control what edits count to trigger quality feedback. These can be configured within Output settings.

By default, we use the following thresholds

Sensor Type Shape Type Tolerance Units
3D Cuboid 0.05 Customer's units (usually meters)
3D Polyline 0.95 IoU (intersection over union)
2D Point 3.0 Pixels
2D Polygon 0.95 IoU (intersection over union)
2D Rectangle 3.0 Pixels
2D Slider Rectangle 3.0 Pixels
2D Line 0.95 IoU (intersection over union)
2D Arrow 0.95 IoU (intersection over union)
2D Cuboid 0.05 Pixels
2D Keypoint 3.0 Pixels

The default thresholds can be changed in the individual output settings, at the bottom of the page. 

As of September 2024, polylines, 2D cuboids, lines, and arrows, are not properly supported and data generated may be inaccurate.

Attribution

When a submission occurs during a QA or Review step, feedback is generated based on the outputs in the current Review step and at least one previous step. Here’s how the feedback attribution works:

Identify the Last Submission with the Output: Feedback is linked to the step where the output was last annotated before the Review step.

Example 1

  • OutputA is annotated in Step A and Step B.
  • OutputA is fixed in Step C (Review step).
  • Feedback for OutputA is attributed to Step B (the last step where OutputA was annotated before the Review step).
 
 

Example 2

  • OutputB is annotated in Step A.
  • OutputB is not present in Step B.
  • OutputB is fixed in Step C (Review step).
  • Feedback for OutputB is attributed to Step A (the only previous step where OutputB was annotated).
 
 



 

 

mechanical input robotic response

Was this article helpful?

Yes
No
Give feedback about this article
Generating Feedback Automatically Workflow Setup Feedback Automated Comments Inaccurate Shape Thresholds Attribution

The first B Corp-certified AI company

  • Security
  • Terms
  • Privacy
  • Quality & Information

Copyright © 2023 Samasource Impact Sourcing, Inc. All rights reserved.


Knowledge Base Software powered by Helpjuice

Expand