Training
Updated at September 27th, 2024
To train an annotation team on a new workflow, you can use gold tasks. Gold tasks are examples with known correct answers that serve as a benchmark. By comparing the team's responses to these gold tasks with the known correct answers, you can evaluate their performance. This comparison helps in calculating a score that reflects how well the team understands and executes the new workflow.
Certification Projects
To use training tools, you must first create a project of type Certification. In this type of project, you'll find an additional tab called Training Templates.
💡 Please note the following restrictions for Certification projects:
- Single Workflow Step: Certification projects can only have one step in their workflow.
- No Manual Uploads: Tasks are generated from templates, so manual uploads are not allowed.
- No Client Batch ID or Primary Key: You cannot set an input as a Client Batch ID or Primary Key.
- Task Rejection Disabled: It is not possible to reject tasks in these projects.
Training Templates
Training templates are pre-defined models from which training tasks are created. If you want each member of an entire training cohort to work on the same 10 tasks, you only need to create 10 training templates, and tasks will get created automatically for all trainees.
Creating Templates
To create a template, follow these steps:
Step 1: Click on the Create Training Template button
To create a template, click on the Create Training Template button at the top right of the template table
Step 2: Specify the following fields
| Field | Description |
| Title | Use a descriptive name to differentiate it from others, for example “busy-intersection-with-pedestrians”. |
| Task ID | Templates are created from existing tasks. You must specify the task ID of the task to use, and its assets and answers will get copied into the template. |
| Start / End Frame | If the original task you chose is a sequence (video or 3D), you can cut it into a smaller template by specifying the frame range to use. This is a good way to ensure that the TPT on training tasks is not too high, and that trainees can work on many diversified tasks across the training period instead of only a few very long sequence tasks. |
| Type | Templates can be of type Practice or Evaluation. For practice templates, trainees will have access to all the training tools while annotating. For evaluation templates, trainees will not have access to any hints or feedback. Their score will get computed, but they won't have had access to any tools in order to evaluate how ready they are for production. |
| Pass mark | This is the score that trainees should get before submitting their tasks. This is not enforced, meaning that trainees will still be able to submit without reaching the score, but the pass mark will be shown visually in the workspace. |
| Target TPT | This is the target time the trainees should complete the task in. |
Template Status
Templates can be either Enabled or Disabled:
- Enabled Templates: When a template is enabled, users can work on tasks from that template. Any user who clicks on "Start Work" on the production page will receive one task from each active template in the project.
- Disabled Templates: When a template is disabled, it will not be included in the trainees' task queue. Disabling a template also deletes all associated training tasks that are either New or In Progress. However, it does not affect tasks that have already been submitted.
💡By default, newly created templates are Disabled.
Archived Templates: Archived templates are hidden from the template table by default. However, you can search for them by filtering for the Archived status in the Status column.
Editing Templates
To edit a template, follow these steps:
- Ensure the template is in the Disabled state.
- Click on the row in the table corresponding to the template you want to edit.
- You can modify the template's title and type.
- Additionally, you can edit the gold answers in the workspace.
Cloning Templates
To clone a template, follow these steps:
- Click on the template you want to clone to open the edit modal.
- Click on the Clone button at the top of the edit modal and confirm you want to duplicate it.
- Optionally, edit the name and type of the cloned template.
Working on Training Tasks
When a user on a Certification project starts work on the production page, a task for each enabled template will get created for them. They can stop work and pick back up from where they left off. Once they submit their task for all enabled templates, they will see the normal message signaling that their queue is empty and get redirected to the Production page. If they click on Start Work again, they will be able to work on the same enabled templates again.
Training Workflow
On practice tasks, trainees have access to training tools. The tools will only become available once they click on “Compute Score” for the first time. This will save their current answers as their first attempt. This first attempt serves to give visibility on their performance without any training tools.
The TPT for the first attempt submission is the time from task load to the time to first submission. The TPT for the final submission is the time from task load to the time to the final submission, so it will also include the first attempt time.
Once they click on Compute Score for the first time, they will see their current score and gain access to the training tools. The score is computed by comparing their answers to the gold answers. There are two training tools that they can use:
Feedbacks
After computing their score, users can see detailed feedback on the errors they made. Errors are generated by comparing the user's answers to the gold answers of the training template. The feedback will be shown as shape and field level feedback on the workspace.
Clicking on Update Score after making edits to the answers will show a fresh set of feedback which applies to the latest answers.
Feedback is also shown for non-annotation outputs. The feedback in this case is shown on the output itself.
Hints
To help trainees get to the correct answers, they can use hints to temporarily show the gold answers.
For non-annotation outputs, the value of the answer is shown on the output.
📘 Note
- Trainees must submit their first attempt before gaining access to these tools, preventing them from cheating. We also track the number of times trainees use the tools.
- For evaluation tasks, trainees do not have access to any training tools. They complete tasks as they would in a production environment, and their scores are computed in the background.
Thresholds
For certain projects, it may be acceptable for bounding boxes to be off by 5 pixels, versus only 2 pixels on another project. It is therefore possible to configure custom thresholds to flag inaccurate shapes to match the workflow requirements. This can be configured in the output settings of the certification project. These can be set at the level of individual shape types. If annotated shapes fall under the configured thresholds, they will not be flagged as inaccurate shapes.
Thresholds for polygons, lines, and arrows are measured using Intersection over Union (IoU). The number specified should be between 0 and 1. The thresholds for all other annotation types are measured in with point-wise distance, and so the number should be above a non-negative number.
Changing thresholds will affect future feedback data generated only. So modifying them at a certain time will only affect tasks which are submitted by trainees after the change was made.
Reporting
The Standard Certification Report (link) contains the relevant data to ascertain whether trainees are learning properly and ready for production. At a glance, you can see the quality of the trainees' first submission (before they get access to training tools) and final submission (the best score they could get using the training tools). The first submission score is an indicator of how well prepared the trainees are for production, where they will not have access to hints and live feedback. The final submission score allows you to monitor whether they are using the tools properly.