1. In the Projects Table, find and select your project by clicking on its row.
2. Once inside your project, click on the Sampling tab.
3. From here, you can sample delivered tasks and review their quality.

Follow these steps to create a sample for reviewing tasks:

1. Click "Create Sample": Located at the top right of the page.
2. Set Filters : In the popup window, apply filters to select tasks for review (e.g., tasks delivered in the past five days).
3. Click "Next" : Proceed to the next step.
4. Choose Sampling Method: Select whether to review a percentage or a fixed number of tasks from those returned by the filters.
   • The total count of tasks matching the filters is displayed.
   • Some tasks may be excluded if they are already part of an in-progress sample.
5. (Optional) Create Multiple Samples Choose to:
   • Create one sample per last submitter.
   • Generate a fixed number of samples and distribute tasks evenly.
6. Click "Create Sample" : Finalize and generate the sample.

💡 The new sample will now appear highlighted at the top of the sample list. The row contains information about the creation date, sample creator, sample ID, sample size, review progress, average quality score, filters used during creation, and number of pieces of feedback given.

💡 Samples created on tasks in the Completed or Approved states are considered Internal samples, while those created on Delivered or Acknowledged states are considered External. Learn more about filtering for specific samples in the next section.

To find a sample previously created, you can toggle the available filters by clicking on the Filter icon next to the Create sample icon. Enter the appropriate filters to find the sample you are looking for.

1. Navigate to the next task by using the navigation in the navigation bar at the top of the workspace.

2. Navigate to a specific task by opening the dropdown menu. The progress bar shows how many tasks have been reviewed and which ones have feedback. Once opened, the dropdown contains all the tasks in the sample and shows which ones have been reviewed and which ones have feedback.

 3. When finished reviewing a task, click on the MARK AS REVIEWED button to finish your review. This will take you to the next unreviewed task in the sample.

💡 Note that if a sample has already been finalized, then you cannot edit feedback for a task.

1. To provide feedback on a specific shape, select the shape from the list and click on the Flag icon.

2. After clicking Flag icon, the feedback panel will open and allow you to select the error committed by the annotation team.The error panel allows you to select the Error type, the start, and end frames of the error, and add any comments for the error. This panel can also be opened by: Selecting a shape and right-clicking on the shape in the canvas to open the context menu, then clicking “Add feedback”. Selecting a frame range in the timeline, right-clicking to open the context menu, and clicking “Add feedback”. Note: Feedback on missing shapes can be added by right-clicking on the canvas to flag that a shape is missing.

3. To provide feedback at the output level, click the flag icon next to the output and select an error tag.

4. Add a comment at the bottom of the workspace for more details on the overall output. This does not flow to our reporting and is only for guidance. To flag errors, you must use the feedback tools outlined in points 2 and 3 above.

In sampling, you can view the original response, click Compare, and see the modifications made.

💡 Notes

• In this mode, you can only compare the response to the original version.
• Additionally, you can highlight specific parts of the response and provide feedback.
• The error types are currently fixed to be Factuality, Instruction Following, and Language

The sampling portal allows users to review tasks completed by the annotation team and compare them against the original response. This process helps verify whether the delivered work meets the required standards and offers insights into potential mistakes the model might be making.

Review the Delivered Task

1. To provide feedback on a specific task, highlight the text and click "Add Feedback". 
2. This will open the feedback panel, where you can select the error made by the annotation team and include a comment.
    

3. Once the feedback is saved, a red flag will appear next to the response, indicating that feedback has been added. Clicking on the flag will open the feedback for review.

4. The response from the annotation team can then be reviewed and compared to the original version in the workspace.

When you are finished with a review of a sample, you should finalize your review. This will ensure that no one (including yourself) can edit your feedback, and will also send the quality data to Sama's data warehouse for review. Once you have marked all tasks as reviewed in a sample and are on the final task, the icon in the top right will change to FINALIZE SAMPLE. Click this icon to finalize the sample.

You can also finalize a sample on the sample list pagae. Click on the three-dots to the right of the sample. Click on Finalize. On the confirmation screen, click Finalize.

After finalization, a green badge will appear next to the kebab menu on the specific sample. This will let Sama's team that you are finished your review so that they can look at the quality feedback given.

⚠️ Be careful! This cannot be undone. You cannot modify the quality feedback you gave on specific tasks after finalizing a sample and will not be able to delete the sample after finalization.