Working with Annotations


Overview

Annotations in Frame.io are represented as a serialized JSON blob of drawing objects, complete with colors, coordinates, and sizes. In the blob, all quotation marks apart from the outer bounding pair are escaped with forward slashes (e.g. \"key\":\"value\").

There are three basic shapes -- arrow, rect, and line -- as well as free-hand drawing, which is outside the scope of this guide, but follows the same basic principles of serialized coordinate data.

The coordinate plane

All shapes are placed on a coordinate plane with an upper-left origin, and a scale of 0 to 1 on each axis.

coordinate_plane.png

Basic shape data

All shapes are placed on the above plane with a combination of coordinates and style data. Each shape has some common attributes, and then each has a slightly different way of addressing the coordinate plane.

All attributes for distance/width/height use the same 0 to 1 scale as the main coordinate plane itself, but are generally marked from the shape's point of origin.

In other words, if a video is being displayed in a 640x480 viewport, x=1 will equal an absolute 640px in that context. This enables annotations to scale seamlessly while maintaining their relative position as viewing screen, size, and context shift.

Note: the origin of a rect is always the upper-left corner of the shape, regardless of drawing origin.

Common attributes

AttributeDescription
toolarrow, rect, or line. States which shape is being made.
colorSix-digit hex color for the shape.
sizePixel size for the shape's line. For arrows, this refers only to the start of the line, as end width is determined programmatically.

Arrow

AttributeDescription
xx-coordinate of the arrow's origin.
yy-coordinate of the arrow's origin.
wDifference in arrow's y-axis position from start to end, relative to the arrow's origin.
hDifference in arrow's x-axis position from start to end, relative to the arrow's origin.

Rect

AttributeDescription
xx-coordinate of the upper-left corner of the rectangle.
yy-coordinate of the upper-left corner of the rectangle.
wWidth of the rectangle, relative to upper-left rectangle origin.
hHeight of the rectangle, relative to upper-left rectangle origin.
ixThe drawing x-origin of the rectangle. (Not required).
iyThe drawing y-origin of the rectangle. (Not required).
radiusRadius of the rectangle's corner roundedness. Must be positive. Values above 150 are visually indistinguishable.

Line

AttributeDescription
x1x-coordinate of the line's origin.
y1y-coordinate of the line's origin.
x2x-coordinate of the line's terminus. Absolute value on the image plane.
y2y-coordinate of the line's terminus. Absolute value on the image plane.

Simple example: the coordinate plane

plane-example.png

JSON
{"annotation": "[{\"tool\":\"line\",\"color\":\"#E74A3C\",\"size\":3,\"x1\":0.0028530670470756064,\"y1\":0.012680297987002695,\"x2\":0.9964336661911555,\"y2\":0.9903312727849104},{\"tool\":\"line\",\"color\":\"#34A3DB\",\"size\":3,\"x1\":0.004992867332382311,\"y1\":0.98779521318751,\"x2\":0.9942938659058488,\"y2\":0.012680297987002695}]}
{
    "annotation": [
        {
            "tool": "line",
            "color": "#E74A3C",
            "size": 3,
            "x1": 0.0028530670470756064,
            "y1": 0.012680297987002695,
            "x2": 0.9964336661911555,
            "y2": 0.9903312727849104
        },
        {
            "tool": "line",
            "color": "#34A3DB",
            "size": 3,
            "x1": 0.004992867332382311,
            "y1": 0.98779521318751,
            "x2": 0.9942938659058488,
            "y2": 0.012680297987002695
        }
    ]
}

Breakdown

blue-plane.png

SerializedDeserialized
"tool":"line", "color":"#34A3DB", "size":3, "x1":0.004992867332382311, "y1":0.98779521318751, "x2":0.9942938659058488, "y2":0.012680297987002695json { "tool":"line", "color":"#34A3DB", "size":3, "x1":0.004992867332382311, "y1":0.98779521318751, "x2":0.9942938659058488, "y2":0.012680297987002695 }

plane-example-red.png

SerializedDeserialized
"tool":"line", "color":"#E74A3C", "size":3, "x1":0.0028530670470756064, "y1":0.012680297987002695, "x2":0.9964336661911555, "y2":0.9903312727849104json { "tool":"line", "color":"#E74A3C", "size":3, "x1":0.0028530670470756064, "y1":0.012680297987002695, "x2":0.9964336661911555, "y2":0.9903312727849104 }

Complex example: lots of boxes

lots-of-boxes.png

{"annotation": "[{\"tool\":\"rect\",\"color\":\"#34A3DB\",\"size\":3,\"x\":0.008426966292134831,\"y\":0.02663337494798169,\"w\":0.0702247191011236,\"h\":0.10986267166042447,\"ix\":0.008426966292134831,\"iy\":0.02663337494798169,\"radius\":8},{\"tool\":\"rect\",\"color\":\"#1ABCA1\",\"size\":3,\"x\":0.8932584269662921,\"y\":0.0449438202247191,\"w\":0.08895131086142327,\"h\":0.35122763212650854,\"ix\":0.8932584269662921,\"iy\":0.0449438202247191,\"radius\":8},{\"tool\":\"rect\",\"color\":\"#E67422\",\"size\":3,\"x\":0.023408239700374533,\"y\":0.7873491468997087,\"w\":0.1254681647940075,\"h\":0.2047440699126093,\"ix\":0.023408239700374533,\"iy\":0.7873491468997087,\"radius\":8},{\"tool\":\"rect\",\"color\":\"#E74A3C\",\"size\":3,\"x\":0.6919475655430711,\"y\":0.5409904286308781,\"w\":0.29868913857677903,\"h\":0.4427798585101955,\"ix\":0.6919475655430711,\"iy\":0.5409904286308781,\"radius\":8},{\"tool\":\"arrow\",\"color\":\"#E74A3C\",\"size\":3,\"x\":0.25374531835205993,\"y\":0.23137744486059092,\"w\":0.24250936329588013,\"h\":0}]}"
JSON
{
    "annotation": [
        {
            "tool": "rect",
            "color": "#34A3DB",
            "size": 3,
            "x": 0.008426966292134831,
            "y": 0.02663337494798169,
            "w": 0.0702247191011236,
            "h": 0.10986267166042447,
            "ix": 0.008426966292134831,
            "iy": 0.02663337494798169,
            "radius": 8
        },
        {
            "tool": "rect",
            "color": "#1ABCA1",
            "size": 3,
            "x": 0.8932584269662921,
            "y": 0.0449438202247191,
            "w": 0.08895131086142327,
            "h": 0.35122763212650854,
            "ix": 0.8932584269662921,
            "iy": 0.0449438202247191,
            "radius": 8
        },
        {
            "tool": "rect",
            "color": "#E67422",
            "size": 3,
            "x": 0.023408239700374533,
            "y": 0.7873491468997087,
            "w": 0.1254681647940075,
            "h": 0.2047440699126093,
            "ix": 0.023408239700374533,
            "iy": 0.7873491468997087,
            "radius": 8
        },
        {
            "tool": "rect",
            "color": "#E74A3C",
            "size": 3,
            "x": 0.6919475655430711,
            "y": 0.5409904286308781,
            "w": 0.29868913857677903,
            "h": 0.4427798585101955,
            "ix": 0.6919475655430711,
            "iy": 0.5409904286308781,
            "radius": 8
        },
        {
            "tool": "arrow",
            "color": "#E74A3C",
            "size": 3,
            "x": 0.25374531835205993,
            "y": 0.23137744486059092,
            "w": 0.24250936329588013,
            "h": 0
        }
    ]
}

Breakdown

With all that in mind, it's helpful to lay out each element in the above image to develop a clearer picture of the annotation coordinate system. Note that the size and radius for each of the rectangles in this example are taking the Frame.io UI defaults.

Blue rect

blue-box.png

SerializedDeserialized
"tool":"rect", "color":"#34A3DB", "size":3, "x":0.008426966292134831, "y":0.02663337494798169, "w":0.0702247191011236, "h":0.10986267166042447, "ix":0.008426966292134831, "iy":0.02663337494798169, "radius":8json { "tool":"rect", "color":"#34A3DB", "size":3, "x":0.008426966292134831, "y":0.02663337494798169, "w":0.0702247191011236, "h":0.10986267166042447, "ix":0.008426966292134831, "iy":0.02663337494798169, "radius":8 }

Green rect

box-green.png

SerializedDeserialized
"tool":"rect", "color":"#1ABCA1", "size":3, "x":0.8932584269662921, "y":0.0449438202247191, "w":0.08895131086142327, "h":0.35122763212650854, "ix":0.8932584269662921, "iy":0.0449438202247191, "radius":8json { "tool":"rect", "color":"#1ABCA1", "size":3, "x":0.8932584269662921, "y":0.0449438202247191, "w":0.08895131086142327, "h":0.35122763212650854, "ix":0.8932584269662921, "iy":0.0449438202247191, "radius":8 }

Red rect

red-box.png

SerializedDeserialized
"tool":"rect", "color":"#E74A3C", "size":3, "x":0.6919475655430711, "y":0.5409904286308781, "w":0.29868913857677903, "h":0.4427798585101955, "ix":0.6919475655430711, "iy":0.5409904286308781, "radius":8json { "tool":"rect", "color":"#E74A3C", "size":3, "x":0.6919475655430711, "y":0.5409904286308781, "w":0.29868913857677903, "h":0.4427798585101955, "ix":0.6919475655430711, "iy":0.5409904286308781, "radius":8 }

Orange rect

orange-box.png

SerializedDeserialized
"tool":"rect", "color":"#E67422", "size":3, "x":0.023408239700374533, "y":0.7873491468997087, "w":0.1254681647940075, "h":0.2047440699126093, "ix":0.023408239700374533, "iy":0.7873491468997087, "radius":8json { "tool":"rect", "color":"#E67422", "size":3, "x":0.023408239700374533, "y":0.7873491468997087, "w":0.1254681647940075, "h":0.2047440699126093, "ix":0.023408239700374533, "iy":0.7873491468997087, "radius":8 }

Red arrow

red-arrow.png

SerializedDeserialized
"tool":"arrow", "color":"#E74A3C", "size":3, "x":0.25374531835205993, "y":0.23137744486059092, "w":0.24250936329588013, "h":0json { "tool":"arrow", "color":"#E74A3C", "size":3, "x":0.25374531835205993, "y":0.23137744486059092, "w":0.24250936329588013, "h":0 }