How to write a GStreamer filter in C

A guide to writing a bounding box drawing GStreamer element for Computer Vision and AI applications

It’s a common task to need to draw bounding boxes while writing video processing and artificial intelligence applications. In this tutorial you will learn how to write a GStreamer filter that draws bounding boxes at the co-ordinates you specify. You can then insert the element in your pipeline, and emit signals on it to add and remove bounding boxes as as when your algorithm detects or recognizes objects. We’ll take a test video and draw two bounding boxes on it.

Let’s get started. The steps to writing the element follow the same rough outline as writing a source element which we learnt in the previous tutorial.

How to write a GStreamer source element in C

This element does what cairooverlay and cariotextoverlay would do together if you were to use them to draw a labelled bounding box. However, it does so without the need to know how write Cairo code. Some of the code for this element has been borrowed from gstcairooverlay.c in gst-plugins-bad.

Here are the steps

1. Generate the Filter element

2. Remove functions we are not using

3. Add signals

4. Set the capabilities of the element to BGRx, BGRA, RGB16

5. Add the code to init, set_caps, dispose, transform_ip

6. Compile and Test

7. Write Test application

1. Generate the Filter element

We’ll use gst-element-maker from gst-plugins-bad just like we did with the source element. Download gst-plugins-bad-1.20.4.tar.xz from https://gstreamer.freedesktop.org/src/gst-plugins-bad/. The base class to use is basetransform.

Also, clone my writings repo for referring to the filter source code.

./gst-element-maker basic_overlay basetransform
# clone writings repository
git clone https://www.github.com/mndar/writings

2. Remove functions we are not using

Keep set_property, get_property, set_caps, dispose and transform_ip functions in gstbasicoverlay.c and remove all other overrides. We will be modifying set_caps, dispose and transform_ip. These lines can be found in class_init after element metadata is set with gst_element_class_set_static_metadata(…). The last one, transform_ip is the most important function. This is the one that will be doing the part of drawing bounding boxes on the video frame.

Here are the relevant sections of the code.

static void
gst_basic_overlay_class_init (GstBasicOverlayClass * klass)
{
    ...
    ...
    gobject_class->dispose = gst_basic_overlay_dispose;
    base_transform_class->set_caps = GST_DEBUG_FUNCPTR (gst_basic_overlay_set_caps);
    base_transform_class->transform_ip = GST_DEBUG_FUNCPTR (gst_basic_overlay_transform_ip);
    ...
    ...
}
 

3. Add Signals

We need two signals, drawbox and removebox. drawbox will accept parameters box name, box label, x co-ordinate, y co-ordinate, width of bounding box, height of bounding box. removebox will accept just the box name as a parameter. Stay in class_init and add the g_signal_new lines to it as shown below.

Rest of the code relevant to the step is also shown in the code block below.

enum 
{
  SIGNAL_ADD_BOX,
  SIGNAL_REMOVE_BOX,
  N_SIGNALS
};

static void
gst_basic_overlay_class_init (GstBasicOverlayClass * klass)
{
  gst_basic_overlay_signals[SIGNAL_ADD_BOX] =
      g_signal_new ("addbox",
      G_TYPE_FROM_CLASS (klass),
      G_SIGNAL_RUN_FIRST, 0, NULL, NULL, NULL,
      G_TYPE_NONE, 6, G_TYPE_STRING, G_TYPE_STRING, G_TYPE_UINT, G_TYPE_UINT, G_TYPE_UINT, G_TYPE_UINT);  
  gst_basic_overlay_signals[SIGNAL_REMOVE_BOX] =
      g_signal_new ("removebox",
      G_TYPE_FROM_CLASS (klass),
      G_SIGNAL_RUN_FIRST, 0, NULL, NULL, NULL,
      G_TYPE_NONE, 1, G_TYPE_STRING);  
}

4. Set the capabilities of the element to BGRx, BGRA, RGB16

Since we are using Cairo, we need to set the caps to BGRx, BGRA, RGB. This is done with the following code for both sink and src pads. To feed this video frame to the element, we will be using videoconvert before the basicoverlay element.

We will also need a videoconvert after basicoverlay for autovideosink to be able to display the frame.

static GstStaticPadTemplate gst_basic_overlay_src_template =
GST_STATIC_PAD_TEMPLATE ("src",
    GST_PAD_SRC,
    GST_PAD_ALWAYS,
    GST_STATIC_CAPS (GST_VIDEO_CAPS_MAKE("{ BGRx, BGRA, RGB16 }"))
    );

static GstStaticPadTemplate gst_basic_overlay_sink_template =
GST_STATIC_PAD_TEMPLATE ("sink",
    GST_PAD_SINK,
    GST_PAD_ALWAYS,
    GST_STATIC_CAPS (GST_VIDEO_CAPS_MAKE("{ BGRx, BGRA, RGB16 }"))
    );

5. Add the code to init, set_caps, dispose, transform_ip

We’ve reached the stage where we have to add our custom code. Let’s describe how the element will function.

The list of bounding boxes will be held in a list, each element being of type BoxInfo. In set_caps, we get information about the video. In init, we set the signals addbox and removebox to have callbacks in the element itself. addbox adds a BoxInfo to the list and removebox removes it. A bounding box is identified by a box name. In dispose, we are just clearing memory allocated to the list of boxes.

The function transform_ip is where we are drawing the bounding boxes. _ip stands for ‘in place’ i.e we are not copying the video frame to modify it. The code block below shows the relevant code

Note: If you are trying to understand the cairo code in the third code block, do read a primer to Cairo co-ordinates and scaling first. A good resource is this one. In this code, we are scaling such the all co-ordinates lie between 0 and 1.0

//gstbasicoverlay.h

typedef struct {
  gchar *boxname, *boxlabel;
  guint x, y, width, height;
}BoxInfo;

struct _GstBasicOverlay
{
  GstBaseTransform base_basicoverlay;

  GstVideoInfo info;
  GList *boxlist;
  cairo_t *cr;

  guint video_width, video_height;
};

//gstbasicoverlay.c init function
static void
gst_basic_overlay_init (GstBasicOverlay *basicoverlay)
{
  basicoverlay->boxlist = NULL;
  g_signal_connect (G_OBJECT (basicoverlay), "addbox", G_CALLBACK(gst_basic_overlay_add_box), NULL);
  g_signal_connect (G_OBJECT (basicoverlay), "removebox", G_CALLBACK(gst_basic_overlay_remove_box), NULL);
}

//gstbasicoverlay.c
static gboolean
gst_basic_overlay_set_caps (GstBaseTransform * trans, GstCaps * in_caps,
    GstCaps * out_caps)
{
  GstBasicOverlay *basicoverlay = GST_BASIC_OVERLAY (trans);

  if (!gst_video_info_from_caps (&basicoverlay->info, in_caps))
    return FALSE;

  return TRUE;
}


static void 
gst_basic_overlay_add_box (GstBasicOverlay *basicoverlay, gchar *boxname, gchar *boxlabel, guint x, guint y, 
                           guint width, guint height, gpointer data) {
  g_print("adding box %s\n", boxname);
  BoxInfo *box = g_new0 (BoxInfo, 1);
  box->boxname = g_strdup (boxname);
  box->boxlabel = g_strdup (boxlabel);
  box->x = x;
  box->y = y;
  box->width = width;
  box->height = height;

  basicoverlay->boxlist = g_list_append (basicoverlay->boxlist, box);
}

void
free_boxinfo (gpointer data) {
  BoxInfo *box = (BoxInfo *) data;
  g_free (box->boxname);
  g_free (box->boxlabel);
}

gint
compare_names (gconstpointer a, gconstpointer b) {
  BoxInfo *box_iter;
  gchar *boxname;

  boxname = (gchar *) b;
  box_iter = (BoxInfo *) a;

  g_print ("Comparing %s %s\n", boxname, box_iter->boxname);
  
  return g_strcmp0 (boxname, box_iter->boxname);
}

static void 
gst_basic_overlay_remove_box (GstBasicOverlay *basicoverlay, gchar *boxname) {
  g_print("Removing box %s\n", boxname);
  
  GList *box_item;

  box_item = g_list_find_custom (basicoverlay->boxlist, boxname, compare_names);
  if (box_item) {
    free_boxinfo (box_item->data);
    basicoverlay->boxlist = g_list_remove (basicoverlay->boxlist, box_item->data);
    g_print ("Box List Length: %d\n", g_list_length (basicoverlay->boxlist));
  }
  else {
    g_print ("Box Item %s Not Found\n", boxname);
  }
}

void
draw_box (gpointer data, gpointer user_data) {
  BoxInfo *box = (BoxInfo *) data;
  GstBasicOverlay *basicoverlay = GST_BASIC_OVERLAY (user_data);
  cairo_t *cr = basicoverlay->cr;
  double x, y, width, height;

  x = (double) box->x / basicoverlay->video_width;
  y = (double) box->y / basicoverlay->video_height;
  width = (double) box->width / basicoverlay->video_width;
  height = (double) box->height / basicoverlay->video_height;

  g_print ("Drawing Box %s %lf %lf %lf %lf\n", box->boxname, x, y, width, height);

  cairo_set_source_rgb (cr, 0, 0, 0);

  //draw bounding box 
  cairo_rectangle (cr, x, y, width, height);
  cairo_set_line_width (cr, 0.005);
  cairo_stroke (cr);

  //show Text below bounding box
  cairo_set_source_rgb (cr, 0, 0, 0);
  cairo_select_font_face (cr, "arial",
			    CAIRO_FONT_SLANT_NORMAL,
			    CAIRO_FONT_WEIGHT_NORMAL);
  cairo_set_font_size (cr, 0.03);
  cairo_move_to (cr, x, y + height + 0.04);
  cairo_show_text (cr, box->boxlabel);

}

static GstFlowReturn
gst_basic_overlay_transform_ip (GstBaseTransform * trans, GstBuffer * buf)
{ ...
  GstVideoFrame frame;
  cairo_surface_t *surface;

  ...
  if (!gst_video_frame_map (&frame, &basicoverlay->info, buf, GST_MAP_READWRITE)) {
    return GST_FLOW_ERROR;
  }
  
  basicoverlay->video_width = GST_VIDEO_FRAME_WIDTH (&frame);
  basicoverlay->video_height = GST_VIDEO_FRAME_HEIGHT (&frame);

  //code borrowed from gstcairooverlay.c
  surface =
        cairo_image_surface_create_for_data (GST_VIDEO_FRAME_PLANE_DATA (&frame,
            0), CAIRO_FORMAT_RGB24, GST_VIDEO_FRAME_WIDTH (&frame),
        GST_VIDEO_FRAME_HEIGHT (&frame), GST_VIDEO_FRAME_PLANE_STRIDE (&frame, 0));
  if (G_UNLIKELY (!surface))
    return GST_FLOW_ERROR;

  basicoverlay->cr = cairo_create (surface);
  if (G_UNLIKELY (!basicoverlay->cr)) {
    cairo_surface_destroy (surface);
    return GST_FLOW_ERROR;
  }
  
  cairo_scale (basicoverlay->cr, GST_VIDEO_FRAME_WIDTH (&frame), GST_VIDEO_FRAME_HEIGHT (&frame));

  //iterate over bounding boxes
  g_list_foreach (basicoverlay->boxlist, draw_box, basicoverlay);

  cairo_destroy (basicoverlay->cr);
  cairo_surface_destroy (surface);

  gst_video_frame_unmap (&frame);

  return GST_FLOW_OK;
}

void
gst_basic_overlay_dispose (GObject * object)
{ 
  ...
  /* clean up as possible.  may be called multiple times */
  g_list_free_full (basicoverlay->boxlist, free_boxinfo);
  basicoverlay->boxlist = NULL;
  
  ...
}

6. Compile and Test

The compile command for this element contains cairo-gobject with pkg-config as we are using the Cairo library to draw the bounding boxes.

gcc -Wall -Werror -fPIC $CPPFLAGS `pkg-config --cflags gstreamer-1.0 gstreamer-base-1.0 gstreamer-video-1.0 cairo-gobject` -c -o gstbasicoverlay.o gstbasicoverlay.c
gcc -shared -o gstbasicoverlay.so gstbasicoverlay.o `pkg-config --libs gstreamer-1.0 gstreamer-base-1.0 gstreamer-video-1.0 cairo-gobject`

The output of gst-inspect-1.0 should show the element details. You should be able to see the two signals we are going to emit to draw and remove boxes.

$ GST_PLUGIN_PATH=. gst-inspect-1.0 basicoverlay|cat
Factory Details:
  Rank                     none (0)
  Long-name                Basic Overlay
  Klass                    Generic
  Description              Basic Overlay
  Author                   Mandar Joshi <emailmandar@gmail.com>

Plugin Details:
  Name                     basicoverlay
  Description              Basic Overlay
  Filename                 ./gstbasicoverlay.so
  Version                  0.0.1
  License                  LGPL
  Source module            Basic Overlay
  Binary package           Basic Overlay
  Origin URL               http://mndar.github.io

GObject
 +----GInitiallyUnowned
       +----GstObject
             +----GstElement
                   +----GstBaseTransform
                         +----GstBasicOverlay

Pad Templates:
  SINK template: 'sink'
    Availability: Always
    Capabilities:
      video/x-raw
                 format: { (string)BGRx, (string)BGRA, (string)RGB16 }
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]
  
  SRC template: 'src'
    Availability: Always
    Capabilities:
      video/x-raw
                 format: { (string)BGRx, (string)BGRA, (string)RGB16 }
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]

Element has no clocking capabilities.
Element has no URI handling capabilities.

Pads:
  SINK: 'sink'
    Pad Template: 'sink'
  SRC: 'src'
    Pad Template: 'src'

Element Properties:
  name                : The name of the object
                        flags: readable, writable, 0x2000
                        String. Default: "basicoverlay0"
  parent              : The parent of the object
                        flags: readable, writable, 0x2000
                        Object of type "GstObject"
  qos                 : Handle Quality-of-Service events
                        flags: readable, writable
                        Boolean. Default: false

Element Signals:
  "addbox" :  void user_function (GstElement* object,
                                  gchararray arg0,
                                  gchararray arg1,
                                  guint arg2,
                                  guint arg3,
                                  guint arg4,
                                  guint arg5,
                                  gpointer user_data);
  "removebox" :  void user_function (GstElement* object,
                                     gchararray arg0,
                                     gpointer user_data);

7. Write Test application

To test the element we will be writing a simple application that emits the signals on the element to draw remove box. The following code starts the pipeline and with a 2 second delay adds 2 bounding boxes and with a 6 second delay removes one of the bounding boxes.

#include <stdio.h>
#include <gst/gst.h>

gboolean timeout_add_box (gpointer data) {
    g_print ("Adding Box\n");
    GstElement *pipeline = (GstElement *) data;
    GstElement *element;
    element = gst_bin_get_by_name (GST_BIN (pipeline), "boverlay");
    g_signal_emit_by_name (element, "addbox", "firstbox", "First Box", 100, 10, 300, 300);
    g_signal_emit_by_name (element, "addbox", "secondbox", "Second Box", 1024, 400, 300, 400);

    return FALSE;
}

gboolean timeout_remove_box (gpointer data) {
    g_print ("Removing Box\n");
    GstElement *pipeline = (GstElement *) data;
    GstElement *element;
    element = gst_bin_get_by_name (GST_BIN (pipeline), "boverlay");
    g_signal_emit_by_name (element, "removebox", "firstbox");
    return FALSE;
}

int main(int argc, char *argv[]) {
    gchar *pipeline_string = "videotestsrc pattern=5 ! video/x-raw,width=1920,height=1080 ! videoconvert ! basicoverlay name=boverlay ! videoconvert ! autovideosink";
    GstElement *pipeline;
    GMainLoop *loop;
    GstElement *element;

    gst_init (&argc, &argv);
    loop = g_main_loop_new (NULL, FALSE);

    pipeline = gst_parse_launch (pipeline_string, NULL);
    if (pipeline == NULL) {
        printf("Could Not Create Pipeline");
        exit(1);
    }

    
    gst_element_set_state (pipeline, GST_STATE_PLAYING);

    g_timeout_add_seconds (2, timeout_add_box, pipeline);
    g_timeout_add_seconds (6, timeout_remove_box, pipeline);
    g_main_loop_run (loop);
    return 0;
}

To compile the application, run the following command

gcc -o testelement testelement.c `pkg-config --cflags --libs gstreamer-1.0`

Run the application with the following command and you will get the output shown the the screenshot at the top

GST_PLUGIN_PATH=. ./testelement

Conclusion

That’s it! You have a filter element that draws bounding boxes on a video. Writing a GStreamer filter in C is possible as you saw in this tutorial and is not too difficult if you know how to operate on images in C or C++. We’ll see how to write a GStreamer filter in Python in the next tutorial.