Inputs and Outputs

Introduction

Capsules are defined by the data they take as input and the information they give as output. Applications use this information to connect capsules to each other and schedule their execution. These inputs and outputs are defined by NodeDescription objects and realized by DetectionNode objects.

class DetectionNode(*, name: str, coords: List[List[Union[int, float]]], attributes: Dict[str, str] = None, children: List[DetectionNode] = None, encoding: Optional[numpy.ndarray] = None, track_id: Optional[uuid.UUID] = None, extra_data: Dict[str, object] = None)

Capsules use DetectionNode objects to communicate results to other capsules and the application itself. A DetectionNode contains information on a detection in the current frame. Capsules that detect objects in a frame create new DetectionNodes. Capsules that discover attributes about detections add data to existing DetectionNodes.

Parameters
  • name – The detection class name. This describes what the detection is. A detection of a person would have a name=”person”.

  • coords – A list of coordinates defining the detection as a polygon in-frame. Comes in the format [[x,y], [x,y]...].

  • attributes – A key-value store where the key is the type of attribute being described and the value is the attribute’s value. For instance, a capsule that detects gender might add a “gender” key to this dict, with a value of either “masculine” or “feminine”.

  • children – Child DetectionNodes that are a “part” of the parent, for instance, a head DetectionNode might be a child of a person DetectionNode

  • encoding – An array of float values that represent an encoding of the detection. This can be used to recognize specific instances of a class. For instance, given a picture of person’s face, the encoding of that face and the encodings of future faces can be compared to find that person in the future.

  • track_id – If this object is tracked, this is the unique identifier for this detection node that ties it to other detection nodes in future and past frames within the same stream.

  • extra_data – A dict of miscellaneous data. This data is provided directly to clients without modification, so it’s a good way to pass extra information from a capsule to other applications.

class NodeDescription(*, size: vcap.node_description.NodeDescription.Size, detections: List[str] = None, attributes: Dict[str, List[str]] = None, encoded: bool = False, tracked: bool = False, extra_data: List[str] = None)

Capsules use NodeDescriptions to describe the kinds of DetectionNodes they take in as input and produce as output.

A capsule may take a DetectionNode as input and produce zero or more DetectionNodes as output. Capsules define what information inputted DetectionNodes must have and what information outputted detection nodes will have using NodeDescriptions.

For example, a capsule that encodes people and face detections would use NodeDescriptions to define its inputs and outputs like so:

>>> input_type = NodeDescription(
...     detections=["person", "face"])
>>> output_type = NodeDescription(
...     detections=["person", "face"],
...     encoded=True)

A capsule that uses a car’s encoding to classify the color of a car would look like this.

>>> input_type = NodeDescription(
...     detections=["car"],
...     encoded=True)
>>> output_type = NodeDescription(
...     detections=["car"],
...     attributes={"color": ["blue", "yellow", "green"]},
...     encoded=True)

A capsule that detects dogs and takes no existing input would look like this.

>>> input_type = NodeDescription(size=NodeDescription.Size.NONE)
>>> output_type = NodeDescription(
>>>                 size=NodeDescription.Size.ALL,
>>>                 detections=["dog"])
Parameters
  • size – The number of DetectionNodes that this capsule either takes as input or provides as output

  • detections – A list of acceptable detection class names. A node that meets this description must have a class name that is present in this list

  • attributes – A dict whose key is the classification type and whose value is a list of possible attributes. A node that meets this description must have a classification for each classification type.

  • encoded – If true, the DetectionNode must be encoded to meet this description

  • tracked – If true, the DetectionNode is being tracked

  • extra_data – A list of keys in a NodeDescription’s extra_data. A DetectionNode that meets this description must have extra data for each name listed here.

Examples

detections

A capsule that can encode cars or trucks would use a NodeDescription like this as its input_type:

NodeDescription(detections=["car", "truck"])

A capsule that can detect people and dogs would use a NodeDescription like this as its output_type:

NodeDescription(detections=["person", "dog"])

attributes

A capsule that operates on detections that have been classified for gender use a NodeDescription like this as its input_type:

NodeDescription(
    attributes={
        "gender": ["male", "female"],
        "color": ["red", "blue", "green"]
    })

A capsule that can classify people’s gender as either male or female would have the following NodeDescription as its output_type:

NodeDescription(
    detections=["person"],
    attributes={
        "gender": ["male", "female"]
    })

encoded

A capsule that operates on detections of cars that have been encoded use a NodeDescription like this as its input_type:

NodeDescription(
    detections=["car"],
    encoded=True)

A capsule that encodes people would use a NodeDescription like this as its output_type:

NodeDescription(
    detections=["person"],
    encoded=True)

tracked

A capsule that operates on person detections that have been tracked would use a NodeDescription like this as its input_type.

NodeDescription(
    detections=["person"],
    tracked=True)

A capsule that tracks people would use a NodeDescription like this as its output_type:

NodeDescription(
    detections=["person"],
    tracked=True)

extra_data

A capsule that operates on people detections with a “process_extra_fast” extra_data field would use a NodeDescription like this as its input_type:

NodeDescription(
    detections=["person"],
    extra_data=["process_extra_fast"])

A capsule that adds an “is_special” extra_data field to its person-detected output would use a NodeDescription like this as its output_type:

NodeDescription(
    detections=["person"],
    extra_data=["is_special"])