Automated Feedback
Updated at September 4th, 2024
Generating Feedback Automatically
It is possible to generate quality feedback for a set of annotations by inferring the mistakes committed from the edits made to the annotations. Instead of flagging feedback manually, which is time consuming, users can focus on fixing the annotations and the quality feedback will get generated automatically and passed to the original annotator.
Workflow Setup
Within the production annotation workflow, any submission made on a step of type Review or QA will generate automated feedback. The feedback will be generated by comparing the submissions for the previous steps in the current round. While QA steps are only allowed to be the last step in the workflow, Review steps can be configured to be at any place within the workflow.
Feedback
The following feedback can be generated from fixes made on annotations:
| Name | Detail |
| Missing Shape | A shape was added in the final submission (it was missing from the original submission). |
| Extra Shape | A shape was removed from the final submission. |
| Inaccurate Shape | A shape had its position modified in the final submission. See the thresholds section below for details on what edits count as errors. |
| Incorrect Label | A shape had its position modified in the final submission. See the thresholds section below for details on what edits count as errors. |
| Inaccurate Attribute | An attribute which is not the primary class was changed on a shape in the final submission. For these fixes, we will store in the Comment of the feedback what specific attribute changed, the original value, and the final value. See the reporting section below for more details. |
| Missing Metadata | A non-annotation output was added in the final submission (it was missing from the original submission). |
| Extra Metadata | A non-annotation output was removed from the final submission. |
| Incorrect Metadata | The value of a non-annotation output was modified in the final submission. |
Automated Comments
We also generate extra information within the Comment field for the Incorrect Attribute and Incorrect Metadata errors. We store the original and final value of the answers as text in the following format:
`ORIGINAL_VALUE` -> `FINAL_VALUE`If the value was not filled in either submission, we display the value as `<empty>`.
Inaccurate Shape Thresholds
Not all position edits should count as errors. We allow configuration of thresholds to control what edits count to trigger quality feedback. These can be configured within Output settings.
By default, we use the following thresholds
| Sensor Type | Shape Type | Tolerance | Units |
| 3D | Cuboid | 0.05 | Customer's units (usually meters) |
| 3D | Polyline | 0.95 | IoU (intersection over union) |
| 2D | Point | 3.0 | Pixels |
| 2D | Polygon | 0.95 | IoU (intersection over union) |
| 2D | Rectangle | 3.0 | Pixels |
| 2D | Slider Rectangle | 3.0 | Pixels |
| 2D | Line | 0.95 | IoU (intersection over union) |
| 2D | Arrow | 0.95 | IoU (intersection over union) |
| 2D | Cuboid | 0.05 | Pixels |
| 2D | Keypoint | 3.0 | Pixels |
The default thresholds can be changed in the individual output settings, at the bottom of the page.
As of September 2024, polylines, 2D cuboids, lines, and arrows, are not properly supported and data generated may be inaccurate.
Attribution
When a submission occurs during a QA or Review step, feedback is generated based on the outputs in the current Review step and at least one previous step. Here’s how the feedback attribution works:
Identify the Last Submission with the Output: Feedback is linked to the step where the output was last annotated before the Review step.
Example 1
- OutputA is annotated in Step A and Step B.
- OutputA is fixed in Step C (Review step).
- Feedback for OutputA is attributed to Step B (the last step where OutputA was annotated before the Review step).
Example 2
- OutputB is annotated in Step A.
- OutputB is not present in Step B.
- OutputB is fixed in Step C (Review step).
- Feedback for OutputB is attributed to Step A (the only previous step where OutputB was annotated).