Annotation tools

For workspace outputs, a variety of different tools are available to annotators in the workspace depending on what type of output they're working on. For Image Annotation workspace outputs, the available tools will also depend on whether the image is in vector or raster format.

In some cases, in the project settings, you can customize what tools to enable or disable in the workspace.

Here's a breakdown of what tools appear and when:

Workspace outputLayer typeAvailable toolsCustomizable in project settings?
Image AnnotationVectorPoint
Line
Arrow
Rectangle
Slider rectangle
Polygon
Cuboid (2D)
Keypoint
Yes
Image AnnotationRasterPaintbrush
Fill
Superpixel
Yes
VideoVectorPoint
Line
Arrow
Rectangle
Slider rectangle
Polygon
Cuboid (2D)
Keypoint
Yes
Point CloudVectorCuboid (3D)No
Sensor FusionVectorCuboid (3D)*

*The Sensor Fusion output is constructed using a main Point Cloud workspace output and one or more optional reference Video/Image Annotation workspace outputs. Since the annotation takes place on the main Point Cloud workspace, the only tool available for now is the cuboid.
No


Vector tools

Examples are shown below of what each type of vector tool looks like in the workspace, as well as how the results of that tool are structured in JSON export files.



Point

16001600
"shapes": [
  {
    "tags": {
      ...
    },
    "type": "point",
    "index": 1,
    "key_locations": [
      {
        "tags": {
          ...
        },
        "points": [
          [
            262,
            57
          ]
        ],
        "visibility": 1,
        "frame_number": 0
      }
    ],
    ...
  }


Line

16001600
"shapes": [
  {
    "tags": {
      ...
    },
    "type": "line",
    "index": 2,
    "key_locations": [
      {
        "tags": {
          ...
        },
        "points": [
          [
            491,
            113
          ],
          [
            547,
            56
          ],
          [
            650,
            63
          ],
          [
            682,
            28
          ]
        ],
        "visibility": 1,
        "frame_number": 0
      }
    ],
    ...
  }


Arrow

16001600
"shapes": [
  {
    "tags": {
      ...
    },
    "type": "arrow",
    "index": 3,
    "key_locations": [
      {
        "tags": {
          ...
        },
        "points": [
          [
            867,
            104
          ],
          [
            950,
            50
          ],
          [
            1029,
            51
          ],
          [
            1065,
            82
          ]
        ],
        "visibility": 1,
        "frame_number": 0
      }
    ],
    ...
  }


Rectangle

16001600
"shapes": [
  {
    "tags": {
      ...
    },
    "type": "rectangle",
    "index": 4,
    "key_locations": [
      {
        "tags": {
          ...
        },
        "points": [
          [
            180,
            256
          ],
          [
            322,
            256
          ],
          [
            180,
            384
          ],
          [
            322,
            384
          ]
        ],
        "visibility": 1,
        "frame_number": 0
      }
    ],
    ...
  }


Slider rectangle

16001600
"shapes": [
  {
    "tags": {
      ...
    },
    "type": "slider_rectangle",
    "index": 5,
    "key_locations": [
      {
        "tags": {
          ...
        },
        "points": [
          [
            554,
            253
          ],
          [
            700,
            253
          ],
          [
            554,
            384
          ],
          [
            700,
            384
          ],
          [
            641,
            253
          ]
        ],
        "visibility": 1,
        "frame_number": 0
      }
    ],
    ...
  }


Polygon

15181518
"shapes": [
  {
    "tags": {
      ...
    },
    "type": "polygon",
    "index": 6,
    "key_locations": [
      {
        "tags": {
          ...
        },
        "points": [
          [
            911,
            264
          ],
          [
            1030,
            232
          ],
          [
            1122,
            325
          ],
          [
            979,
            396
          ],
          [
            870,
            348
          ],
          [
            923,
            328
          ]
        ],
        "visibility": 1,
        "frame_number": 0
      }
    ],
    ...
  }


Cuboid

The Cuboid tool is different depending on whether it is being used in 2D (Image Annotation, Video) or 3D (Point Cloud, Sensor Fusion).



Cuboid (2D)

16001600

2D cuboids are described in the JSON output file as two sets of points: points and keypoints (denoted by P and KP, respectively, in the figure above).

For the keypoints:

  • KP1 corresponds to the horizon
  • KP2 and KP3 define which of the six sides of the cube is facing forward (shown by a yellow face in the workspace)
  • KP4 denotes the depth of the cube
  • KP5 denotes the height of the cube

For the points:
The points are computed from the keypoints above.

  • P1, P2, P3, and P4 correspond to the corners of the front face of the cube
  • P5, P6, P7, and P8 correspond to the corners of the back face of the cube
"shapes": [
  {
    "tags": {
      ...
    },
    "type": "cuboid",
    "index": 7,
    "key_locations": [
      {
        "tags": {
          ...
        },
        "points": [
          [
            411,
            632
          ],
          [
            411,
            539
          ],
          [
            312,
            629
          ],
          [
            312.0,
            537.5942857142857
          ],
          [
            449,
            602
          ],
          [
            449.0,
            524.9428571428572
          ],
          [
            366.1883084984444,
            599.9343376453251
          ],
          [
            366.1883084984444,
            523.9749467823809
          ]
        ],
        "visibility": 1,
        "frame_number": 0,
        "key_points": [
          [
            193,
            457
          ],
          [
            312,
            629
          ],
          [
            411,
            632
          ],
          [
            449,
            602
          ],
          [
            411,
            539
          ]
        ]
      }
    ],
    ...
  }


Cuboid (3D)

The (x,y,z) vertices of each 3D cuboid are always exported in the following pattern: one full face and then the other full face. In this diagram, the side bounded by 3-1-2-0 is one face of the cuboid and the side bounded by 7-5-6-4 is the other face of the cuboid. These faces may be “front” and “back,” “left” and “right,” or “top” and “bottom,” depending on the cuboid’s rotation in the data.

The cuboid vertex export order can be used to define a face of interest. The face of interest is defined by the first four vertices of the cuboid listed in the export file. A face of interest, or side of the cuboid, can be used to designate the front of a car or the direction of movement of an object bounded by a cuboid. Therefore, there is always a default face of interest.

16001600

As an illustrative example, consider the Point Cloud workspace below, where a single vehicle has been delimited by a cuboid.

The JSON for this object would be as follows:

"Point Cloud Scene": [
  {
    "shapes": [
      {
        "tags": {
          "Parent": "None",
          "Answers": "TYPE_VEHICLE",
          "Occlusion": ""
        },
        "type": "cuboid",
        "index": 3,
        "key_locations": [
          {
            "tags": {
              "Occlusion": "Visible"
            },
            "points": [
              [
                -2.64196,
                3.93875,
                -1.82775
              ],
              [
                -2.64196,
                3.93875,
                -0.40086
              ],
              [
                -2.64196,
                5.85652,
                -1.82775
              ],
              [
                -2.64196,
                5.85652,
                -0.40086
              ],
              [
                1.83147,
                3.93875,
                -1.82775
              ],
              [
                1.83147,
                3.93875,
                -0.40086
              ],
              [
                1.83147,
                5.85652,
                -1.82775
              ],
              [
                1.83147,
                5.85652,
                -0.40086
              ]
            ],
            "visibility": 1,
            "frame_number": 0,
            "position_center": [
              -0.40525,
              4.89764,
              -1.1143
            ],
            "direction": {
              "roll": 0.0,
              "pitch": 0.0,
              "yaw": 3.141592653589793
            },
            "dimensions": {
              "length": 4.47343,
              "width": 1.91777,
              "height": 1.42689
            }
          }
        ],
        ...
 }
  • The first 4 points represent the face of interest.
  • The position_center (x,y,z) is the center point of the cuboid.
  • Dimensions are the length(along x-axis), width(along y-axis) and height(along z-axis) of the cuboid.
  • Direction is the roll, pitch and yaw of the cuboid; applied in that respective order and in radians. Using direction, it is easy to determine any faces of interest (such as the front of a cuboid that may represent the front of a vehicle). For specific faces of interest that depend on annotation instructions, use points as above.


Keypoint

16001600

In the JSON export file, the keypoint options (classes and connections) are summarized in the nested JSON object _extras, as shown below.

{
  "tasks": [
    {
      "id": "6124f0ceda72d104a1e0b23b",
      "project_id": 8613,
      "priority": 0,
      "created_at": "2021-08-24T13:17:34.241Z",
      "state": "completed",
      "data": {
        ...
      },
      "answers": {
        ...
      },
      "batch_id": 70718,
      "task_url": "https://api.sama.com/projects/8613/tasks/6124f0ceda72d104a1e0b23b",
      "_extras": {
        "keypoint_options": {
          "Video": {
            "classes": {
              "Keypoint Class": {
                "points": [
                  "Keypoint 1",
                  "Keypoint 2",
                  "Keypoint 3",
                  "Keypoint 4",
                  "Keypoint 5",
                  "Keypoint 6"
                ],
                "connections": [
                  {
                    "from": "Keypoint 1",
                    "to": "Keypoint 2"
                  },
                  {
                    "from": "Keypoint 1",
                    "to": "Keypoint 4"
                  },
                  {
                    "from": "Keypoint 1",
                    "to": "Keypoint 3"
                  },
                  {
                    "from": "Keypoint 4",
                    "to": "Keypoint 5"
                  },
                  {
                    "from": "Keypoint 4",
                    "to": "Keypoint 6"
                  }
                ]
              }
            }
          }
        }
      }
    }
  ]
}


Raster tools

There are three different raster tools, which each have different options in the secondary toolbar in the workspace:

  • Paintbrush (a)
  • Fill (b)
  • Superpixel (c)

The tools are used to assign a category to each pixel of the image. All pixels are assigned a "no label" category to start with, and the annotation process consists of assigning categories to the pixels of the different objects in the image.

The output of this type of annotation takes the form of a greyscale image, a raster mask, where the value of a pixel represents the category to which it has been assigned.

10381038

The location of the raster mask image is listed in the output JSON file under the mask_url key:

{
  "tasks": [
    {
      "id": "6125acb53928ed00e6fc55fa",
      "project_id": 8616,
      "priority": 0,
      "created_at": "2021-08-25T02:38:38.847Z",
      "state": "completed",
      "data": {
        "Image Raster Original file name": "dog.png",
        "Image Raster Size": "1414221",
        "Image Raster Height": "1152",
        "Image Raster Width": "1729",
        "Image Raster Original url": "https://assets.samasource.org/8616/dog.png?Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly9hc3NldHMuc2FtYXNvdXJjZS5vcmcvMjMvamVyb21lLXRlc3QvODYxNi9kb2cucG5nIiwiQ29uZGl0aW9uIjp7IkRhdGVMZXNzVGhhbiI6eyJBV1M6RXBvY2hUaW1lIjoxNjI5ODY4NTAyfX19XX0_&Signature=IE2FAi7aWTedNtV7OPBNCVQsPnUGON1EcGJDX5LR0Py3HVHtrzo6QwUI1dk9qiEeZLReKRc0tE-sBB0JG-r2gVFhGUVy~V0qiLSfvKYc8KQ3gdNTjr0ZzxaevtuUHnJQvggcugFoZ490NLPEe99ggD0ACtaXDNdLUPqt~oLXpN~iDREovNe-TmFI~CGtuU4Dq6R6ktBJkdZ8~XijRtBLyYYly7nb01hCT~eE8Tf5oKRpaq30E1SFkjHbacRw4IBy4mrMkpjMVPtbc0Z034CLh1zufIK~btVu9zxNwFSWY3PbZNsKFSemZIu0mkZGk08j8xikqGlllnCWh4HWMxs-ag__&Key-Pair-Id=APKAI3VFKBKGZNMXJ2MQ",
        "image": "https://assets.samasource.org/8616/dog.png?Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly9hc3NldHMuc2FtYXNvdXJjZS5vcmcvMjMvamVyb21lLXRlc3QvODYxNi9kb2cucG5nIiwiQ29uZGl0aW9uIjp7IkRhdGVMZXNzVGhhbiI6eyJBV1M6RXBvY2hUaW1lIjoxNjI5ODY4NTAyfX19XX0_&Signature=IE2FAi7aWTedNtV7OPBNCVQsPnUGON1EcGJDX5LR0Py3HVHtrzo6QwUI1dk9qiEeZLReKRc0tE-sBB0JG-r2gVFhGUVy~V0qiLSfvKYc8KQ3gdNTjr0ZzxaevtuUHnJQvggcugFoZ490NLPEe99ggD0ACtaXDNdLUPqt~oLXpN~iDREovNe-TmFI~CGtuU4Dq6R6ktBJkdZ8~XijRtBLyYYly7nb01hCT~eE8Tf5oKRpaq30E1SFkjHbacRw4IBy4mrMkpjMVPtbc0Z034CLh1zufIK~btVu9zxNwFSWY3PbZNsKFSemZIu0mkZGk08j8xikqGlllnCWh4HWMxs-ag__&Key-Pair-Id=APKAI3VFKBKGZNMXJ2MQ",
        "name": "task1"
      },
      "answers": {
        "Image Raster": {
          "layers": {
            "raster_coding": {
              "mask_url": "https://samahub3.s3.amazonaws.com/clients/23/external/projects/8616/tasks/6125acb53928ed00e6fc55fa/outputs/Image%20Raster/d1bdde4b-3d01-47d3-8536-4952eb871969.png"
            }
          }
        }
      },
      "submitter_ids_by_step": {
        "Step A": 15131
      },
      "batch_id": 70772,
      "task_url": "https://api.sama.com/projects/8616/tasks/6125acb53928ed00e6fc55fa"
    }
  ]
}

If you have been provided with the right credentials, you can download the mask using the AWS CLI.

aws s3api get-object --bucket samahub3 --key "clients/23/external/projects/8616/tasks/6125acb53928ed00e6fc55fa/outputs/Image Raster/d1bdde4b-3d01-47d3-8536-4952eb871969.png" raster-mask.png