Working with Annotations


Overview

Annotations in Frame.io are represented as a serialized JSON blob of drawing objects, complete with colors, coordinates, and sizes. In the blob, all quotation marks apart from the outer bounding pair are escaped with forward slashes (e.g. \"key\":\"value\").

There are three basic shapes -- arrow, rect, and line -- as well as free-hand drawing, which is outside the scope of this guide, but follows the same basic principles of serialized coordinate data.

The coordinate system

All shapes are placed on a cartesian coordinate system with an upper-left origin, and a scale of 0 to 1 on each axis.

coordinate_plane.png

Basic shape data

All shapes are placed on the above plane with a combination of coordinates and style data. Each shape has some common attributes, and then each has a slightly different way of addressing the coordinate system.

All attributes for distance/width/height use the same 0 to 1 scale as the main coordinate grid itself, but are generally marked from the shape's point of origin.

In other words, if a video is being displayed in a 640x480 viewport, x=1 will equal an absolute 640px in that context. This enables annotations to scale seamlessly while maintaining their relative position as viewing screen, size, and context shift.

Note: the origin of a rect is always the upper-left corner of the shape, regardless of drawing origin.

Common attributes

AttributeDescription
toolarrow, rect, or line. States which shape is being made.
colorSix-digit hex color for the shape.
sizePixel size for the shape's line. For arrows, this refers only to the start of the line, as end width is determined programmatically.

Arrow

AttributeDescription
xx-coordinate of the arrow's origin.
yy-coordinate of the arrow's origin.
wDifference in arrow's y-axis position from start to end, relative to the arrow's origin.
hDifference in arrow's x-axis position from start to end, relative to the arrow's origin.

Rect

AttributeDescription
xx-coordinate of the upper-left corner of the rectangle.
yy-coordinate of the upper-left corner of the rectangle.
wWidth of the rectangle, relative to upper-left rectangle origin.
hHeight of the rectangle, relative to upper-left rectangle origin.
ixThe drawing x-origin of the rectangle. (Not required).
iyThe drawing y-origin of the rectangle. (Not required).
radiusRadius of the rectangle's corner roundedness. Must be positive. Values above 150 are visually indistinguishable.

Line

AttributeDescription
x1x-coordinate of the line's origin.
y1y-coordinate of the line's origin.
x2x-coordinate of the line's terminus. Absolute value on the image plane.
y2y-coordinate of the line's terminus. Absolute value on the image plane.

Simple example: the coordinate plane

plane-example.png

Serialized
JSON
{"annotation": "[{\"tool\":\"line\",\"color\":\"#E74A3C\",\"size\":3,\"x1\":0.0028530670470756064,\"y1\":0.012680297987002695,\"x2\":0.9964336661911555,\"y2\":0.9903312727849104},{\"tool\":\"line\",\"color\":\"#34A3DB\",\"size\":3,\"x1\":0.004992867332382311,\"y1\":0.98779521318751,\"x2\":0.9942938659058488,\"y2\":0.012680297987002695}]}

Breakdown

blue-plane.png

JSON
{
   "tool":"line",
   "color":"#34A3DB",
   "size":3,
   "x1":0.004992867332382311,
   "y1":0.98779521318751,
   "x2":0.9942938659058488,
   "y2":0.012680297987002695
}

plane-example-red.png

JSON
{
   "tool":"line",
   "color":"#E74A3C",
   "size":3,
   "x1":0.0028530670470756064,
   "y1":0.012680297987002695,
   "x2":0.9964336661911555,
   "y2":0.9903312727849104
}

Complex example: lots of boxes

lots-of-boxes.png

Serialized
JSON
{"annotation": "[{\"tool\":\"rect\",\"color\":\"#34A3DB\",\"size\":3,\"x\":0.008426966292134831,\"y\":0.02663337494798169,\"w\":0.0702247191011236,\"h\":0.10986267166042447,\"ix\":0.008426966292134831,\"iy\":0.02663337494798169,\"radius\":8},{\"tool\":\"rect\",\"color\":\"#1ABCA1\",\"size\":3,\"x\":0.8932584269662921,\"y\":0.0449438202247191,\"w\":0.08895131086142327,\"h\":0.35122763212650854,\"ix\":0.8932584269662921,\"iy\":0.0449438202247191,\"radius\":8},{\"tool\":\"rect\",\"color\":\"#E67422\",\"size\":3,\"x\":0.023408239700374533,\"y\":0.7873491468997087,\"w\":0.1254681647940075,\"h\":0.2047440699126093,\"ix\":0.023408239700374533,\"iy\":0.7873491468997087,\"radius\":8},{\"tool\":\"rect\",\"color\":\"#E74A3C\",\"size\":3,\"x\":0.6919475655430711,\"y\":0.5409904286308781,\"w\":0.29868913857677903,\"h\":0.4427798585101955,\"ix\":0.6919475655430711,\"iy\":0.5409904286308781,\"radius\":8},{\"tool\":\"arrow\",\"color\":\"#E74A3C\",\"size\":3,\"x\":0.25374531835205993,\"y\":0.23137744486059092,\"w\":0.24250936329588013,\"h\":0}]}"

Breakdown

With all that in mind, it's helpful to lay out each element in the above image to develop a clearer picture of the annotation coordinate system. Note that the size and radius for each of the rectangles in this example are taking the Frame.io UI defaults.

Blue rect

blue-box.png

JSON
{
   "tool":"rect",
   "color":"#34A3DB",
   "size":3,
   "x":0.008426966292134831,
   "y":0.02663337494798169,
   "w":0.0702247191011236,
   "h":0.10986267166042447,
   "ix":0.008426966292134831,
   "iy":0.02663337494798169,
   "radius":8
}

Green rect

box-green.png

JSON
{
   "tool":"rect",
   "color":"#1ABCA1",
   "size":3,
   "x":0.8932584269662921,
   "y":0.0449438202247191,
   "w":0.08895131086142327,
   "h":0.35122763212650854,
   "ix":0.8932584269662921,
   "iy":0.0449438202247191,
   "radius":8
}

Red rect

red-box.png

JSON
{
   "tool":"rect",
   "color":"#E74A3C",
   "size":3,
   "x":0.6919475655430711,
   "y":0.5409904286308781,
   "w":0.29868913857677903,
   "h":0.4427798585101955,
   "ix":0.6919475655430711,
   "iy":0.5409904286308781,
   "radius":8
}

Orange rect

orange-box.png

JSON
{
   "tool":"rect",
   "color":"#E67422",
   "size":3,
   "x":0.023408239700374533,
   "y":0.7873491468997087,
   "w":0.1254681647940075,
   "h":0.2047440699126093,
   "ix":0.023408239700374533,
   "iy":0.7873491468997087,
   "radius":8
}

Red arrow

red-arrow.png

JSON
{
   "tool":"arrow",
   "color":"#E74A3C",
   "size":3,
   "x":0.25374531835205993,
   "y":0.23137744486059092,
   "w":0.24250936329588013,
   "h":0
}