How to write a GStreamer Source Element in C

A 10-Step guide culminating with creating a flip book video with your code

How does it sound to be able to write your own GStreamer Element? It’s not a daunting task. It just takes you to know which functions do what in which GStreamer classes. In the last tutorial we learned the basics of how to use a GStreamer pipeline in C and Python. In this tutorial we will write a source element. The only assumption made your are familiar with writing C code, especially GStreamer code.

Getting Started With GStreamer: A beginner's guide

I will outline the steps in the 10 Step process to write a GStreamer Source Element starting with gst-element-maker from gst-plugins-bad. The element we will write will insert jpeg/jpg files from a directory into the pipeline. We will demonstrate a flip book (each picture will appear on the screen for 1 second) with jpeg/jpg files and in the end create a video with the pictures.

The 10-Step Process:

1. Get the sources

2. Generate the source element from the template

3. Delete unused functions

4. Search and Replace to change from GstBaseSrc to GstPushSrc

5. Correct include file

6. Set the Capabilities

7. Describe the functioning of the element

8. Add a Property to take input from the user, configure our element

9. Write the code for the init, start, fill, dispose function

10. Compile and Test

Let’s begin!

Step 1: Get the sources

Let’s start by downloading all the necessary sources. We need gst-element-maker which is in the gst-plugins-bad tarball. I downloaded version 1.20.4 here https://gstreamer.freedesktop.org/src/gst-plugins-bad/gst-plugins-bad-1.20.4.tar.xz . Next, clone my github repo writings, it contains the source code for the source element in the source_element directory

#Download and unpack gst-plugins-bad
wget https://gstreamer.freedesktop.org/src/gst-plugins-bad/gst-plugins-bad-1.20.4.tar.xz
tar Jxvf gst-plugins-bad-1.20.4.tar.xz

#CLone the repository containing the source element
git clone https://www.github.com/mndar/writings

Step 2: Generate the source element from the template

GStreamer elements are written by sub-classing a base class and over riding functions in them. To write a source element, you can either use GstBaseSrc or GstPushSrc. GStPushSrc is a actually a subclass of GstBaseSrc. This is the one we will use. However, there is no template for GstPushSrc. So, let’s start with GstBaseSrc and then edit manually.

cd gst-plugins-bad/tools
./gst-element-maker my_jpeg_streamer basesrc

This will generate the source code for the element and compile it. It will also show you the output of gst-inspect-1.0. The source code files generated will be gstmyjpegstreamer.c and gstmyjpegstremaer.h. The compile command for the element is

gcc -Wall -Werror -fPIC $CPPFLAGS `pkg-config --cflags gstreamer-1.0 gstreamer-base-1.0` -c -o gstmyjpegstreamer.o gstmyjpegstreamer.c
gcc -shared -o gstmyjpegstreamer.so gstmyjpegstreamer.o `pkg-config --libs gstreamer-1.0 gstreamer-base-1.0`

Step 3: Delete unused functions

The way to write an element in GStreamer is to over ride functions in the base class. So, open up gstmyjpegstreamer.c and keep only the over rides we are going to use. The function to edit is gst_my_jpeg_streamer_class_init

static void
gst_my_jpeg_streamer_class_init (GstMyJpegStreamerClass * klass)
{
  ...
  ...
  GstBaseSrcClass *base_src_class = GST_BASE_SRC_CLASS (klass);
  ...
  ...
  gobject_class->set_property = gst_my_jpeg_streamer_set_property;
  gobject_class->get_property = gst_my_jpeg_streamer_get_property;

  base_src_class->start = GST_DEBUG_FUNCPTR (gst_my_jpeg_streamer_start);
  push_src_class->fill = GST_DEBUG_FUNCPTR (gst_my_jpeg_streamer_fill);
  gobject_class->dispose = GST_DEBUG_FUNCPTR (gst_my_jpeg_streamer_dispose);
  ...
  ...
}

Step 4: Search and Replace to change from GstBaseSrc to GstPushSrc

Since we have generated the code with gst-element-maker using the basesrc class, short for GstBaseSrc, we have the do a case-sensitive search and replace of the following terms

Repace GstBaseSrc with GstPushSrc in the implementation file gstmyjpegstreamer.c
Replace base_src with push_src in the implementation file gstmyjpegstreamer.c
Replace base with push in the header file gstmyjpegstreamer.h

Step 5: Correct the include file

Change gstbasesrc.h to gstpushsrc.h since we are sub-classing GstPushSrc and not GstBaseSrc.

...
#include <gst/base/gstpushsrc.h>
...

Step 6: Set the Capabilities

Since this is a source element, it has just one pad. We set it’s capabilities to image/jpeg as we will be inserting jpeg files into the pipeline.

/* pad templates */

static GstStaticPadTemplate gst_my_jpeg_streamer_src_template =
GST_STATIC_PAD_TEMPLATE ("src",
    GST_PAD_SRC,
    GST_PAD_ALWAYS,
    GST_STATIC_CAPS ("image/jpeg")
    );

Step 7: Describe the functioning of the element

The element will read a property from the user, the directory, scan the directory for all jpeg/jpg files, make a list and send these files one at a time over the output pad of the source element.

To do so, we will add a property called directory, initialize variables in the init function, call functions that need to be executed only at the begininig in start, call functions that fill the GstBuffer in fill and call all functions that clean up memory in dispose. Here is a snippet of what we will use

  base_src_class->start = GST_DEBUG_FUNCPTR (gst_my_jpeg_streamer_start);
  push_src_class->fill = GST_DEBUG_FUNCPTR (gst_my_jpeg_streamer_fill);
  gobject_class->dispose = GST_DEBUG_FUNCPTR (gst_my_jpeg_streamer_dispose);

Step 8: Add a Property to take input from the user, configure our element

We use a property called directory to get input from the user. The images from this directory will be used by our element. Make sure all the images of the same resolution. As we are making a flip book, we need all images of the same size.

We add the property to our element in class_init and implement set and get in set_property and get_property respectively.

enum
{
  PROP_0,
  PROP_DIRECTORY
};
...
...
static void
gst_my_jpeg_streamer_class_init (GstMyJpegStreamerClass * klass)
{
  ...
  ...
  g_object_class_install_property
      (gobject_class, PROP_DIRECTORY,
      g_param_spec_string ("directory", "Directory",
          "Value of the Directory containing JPEG files",
          "myjpegstreamer", G_PARAM_READWRITE | G_PARAM_STATIC_STRINGS));
}

...
...
void
gst_my_jpeg_streamer_set_property (GObject * object, guint property_id,
    const GValue * value, GParamSpec * pspec)
{
  ...
  ...
    case PROP_DIRECTORY:
      const gchar *directory;
      directory = g_value_get_string (value);
      myjpegstreamer->directory = g_strdup (directory);
      g_print ("Setting JPEG Directory to %s\n", myjpegstreamer->directory);
      break;
  ...
  ...
}

void
gst_my_jpeg_streamer_get_property (GObject * object, guint property_id,
    GValue * value, GParamSpec * pspec)
{
  ...
  switch (property_id) {
    case PROP_DIRECTORY:
      g_value_set_string (value, myjpegstreamer->directory);
      break;
  ...
  ...
}

Step 9: Write the code for the init, start, fill, dispose function

The init function initializes all the variables we will be using. start is called once when the element starts, dispose is called when the element is destroyed and fill is called everything the GstBuffer needs to be filled.

The snippets of code are given below. In start, we scan the directory for all jpeg/jpg files. In fill we read part of the file and fill GstBuffer to its size.

The important part to remember is how to fill the GstBuffer. All you to do is call gst_buffer_map containing GstMapInfo. GstMapInfo.data points to the data inside the buffer. We have to fill the entire buffer. To get the size of the buffer, use gst_buffer_get_size. Once you are done, unmap the buffer using gst_buffer_unmap. Incase you are not able to fill the GstBuffer, use gst_buffer_resize to resize it to a smaller size.

//all operations on GstBuffer
...
gst_buffer_map (buf, &info, GST_MAP_WRITE);
buf_size = gst_buffer_get_size (buf);
...
...
count = fread (info.data, 1, buf_size, src->curr_file);
...
...
gst_buffer_resize (buf, 0, count);
...
...
gst_buffer_unmap (buf, &info);

// In gstmyjpegstreamer.h
#include <stdio.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
...
...
struct _GstMyJpegStreamer
{
  GstPushSrc push_myjpegstreamer;

  gchar *directory;
  GList *jpeg_files;
  FILE *curr_file;
  guint file_index;
  guint num_files;
};

// In gstmyjpegstremaer.c
static void
gst_my_jpeg_streamer_class_init (GstMyJpegStreamerClass * klass)
{
  ...
  ...
  base_src_class->start = GST_DEBUG_FUNCPTR (gst_my_jpeg_streamer_start);
  push_src_class->fill = GST_DEBUG_FUNCPTR (gst_my_jpeg_streamer_fill);
  gobject_class->dispose = GST_DEBUG_FUNCPTR (gst_my_jpeg_streamer_dispose);
  ...
  ...
}

static void
gst_my_jpeg_streamer_init (GstMyJpegStreamer *myjpegstreamer)
{
  myjpegstreamer->directory = NULL;
  myjpegstreamer->jpeg_files = NULL;
  myjpegstreamer->curr_file = NULL;
  myjpegstreamer->file_index = 0;
}

static gboolean
gst_my_jpeg_streamer_start (GstBaseSrc * src)
{
  GstMyJpegStreamer *myjpegstreamer = GST_MY_JPEG_STREAMER (src);

  GST_DEBUG_OBJECT (myjpegstreamer, "start");

  GDir *dir;
  GError *error = NULL;
  const gchar *file;

  dir = g_dir_open(myjpegstreamer->directory, 0, &error);
  while ((file = g_dir_read_name(dir))) {
      if (g_str_has_suffix (file, ".jpeg") || g_str_has_suffix (file, "jpg")) {
        g_print("%s\n", file);
        myjpegstreamer->jpeg_files = 
          g_list_append (myjpegstreamer->jpeg_files, g_strdup (file));
      }
  }

  g_dir_close (dir);

  myjpegstreamer->num_files = g_list_length (myjpegstreamer->jpeg_files);

  return TRUE;
}

gboolean
stream_list_file (GstMyJpegStreamer *src, gchar *directory, gchar *filename, GstBuffer *buf) {
  g_print("Copying File %s To Buffer\n", filename);
  
  long count;
  gsize buf_size;
  gchar *file_path;
  
  GstMapInfo info;

  if (!src->curr_file) {
    file_path = g_strdup_printf ("%s/%s", directory, filename);
    src->curr_file = fopen (file_path, "rb");
    g_free (file_path);
  }

  gst_buffer_map (buf, &info, GST_MAP_WRITE);
  buf_size = gst_buffer_get_size (buf);
  
  count = fread (info.data, 1, buf_size, src->curr_file);
  
  g_print ("File Index: %d Count: %ld, Buf Size: %ld\n", src->file_index, count, buf_size);

  if (count < buf_size) {
    fclose (src->curr_file);
    src->curr_file = NULL;
    gst_buffer_resize (buf, 0, count);
    gst_buffer_unmap (buf, &info);
    return FALSE;
  }

  gst_buffer_unmap (buf, &info);
  return TRUE;
}

static GstFlowReturn
gst_my_jpeg_streamer_fill (GstPushSrc * src, GstBuffer * buf)
{
  GstMyJpegStreamer *myjpegstreamer = GST_MY_JPEG_STREAMER (src);
  GList *file;
  gchar *filename;
  gboolean ret;

  GST_DEBUG_OBJECT (myjpegstreamer, "fill");
  
  if (myjpegstreamer->num_files == 0)
    return GST_FLOW_EOS;

  if (myjpegstreamer->file_index >  myjpegstreamer->num_files - 1) {
    return GST_FLOW_EOS;
  }

  file = g_list_nth (myjpegstreamer->jpeg_files, myjpegstreamer->file_index);
  filename = (gchar *) file->data;
  g_print ("Streaming File %s\n", filename);

  ret = stream_list_file (myjpegstreamer, myjpegstreamer->directory, filename, buf);
  if (!ret)
    myjpegstreamer->file_index++;
  
  return GST_FLOW_OK;
}

void free_list_string (gpointer data) {
  g_print("Freeing List Data\n");
  g_free (data);
}

static void
gst_my_jpeg_streamer_dispose (GObject *gobject) {
  GstPushSrc *src = GST_PUSH_SRC (gobject);
  GstMyJpegStreamer *myjpegstreamer = GST_MY_JPEG_STREAMER (src);

  g_free (myjpegstreamer->directory);
  g_list_free_full (myjpegstreamer->jpeg_files, free_list_string);
  G_OBJECT_CLASS (gst_my_jpeg_streamer_parent_class)->dispose (gobject);
}

Step 10. Compile and Test

To compile our code, run the following commands.

gcc -Wall -Werror -fPIC $CPPFLAGS `pkg-config --cflags gstreamer-1.0 gstreamer-base-1.0` -c -o gstmyjpegstreamer.o gstmyjpegstreamer.c
gcc -shared -o gstmyjpegstreamer.so gstmyjpegstreamer.o `pkg-config --libs gstreamer-1.0 gstreamer-base-1.0`
# Check the element with
GST_PLUGIN_PATH=. gst-inspect-1.0 myjpegstreamer

The output of the gst-inspect-1.0 command should be

$ GST_PLUGIN_PATH=. gst-inspect-1.0 myjpegstreamer|cat
Factory Details:
  Rank                     none (0)
  Long-name                FIXME Long name
  Klass                    Generic
  Description              FIXME Description
  Author                   FIXME <fixme@example.com>

Plugin Details:
  Name                     myjpegstreamer
  Description              FIXME plugin description
  Filename                 ./gstmyjpegstreamer.so
  Version                  0.0.FIXME
  License                  LGPL
  Source module            FIXME_package
  Binary package           FIXME_package_name
  Origin URL               http://FIXME.org/

GObject
 +----GInitiallyUnowned
       +----GstObject
             +----GstElement
                   +----GstBaseSrc
                         +----GstPushSrc
                               +----GstMyJpegStreamer

Pad Templates:
  SRC template: 'src'
    Availability: Always
    Capabilities:
      image/jpeg

Element has no clocking capabilities.
Element has no URI handling capabilities.

Pads:
  SRC: 'src'
    Pad Template: 'src'

Element Properties:
  blocksize           : Size in bytes to read per buffer (-1 = default)
                        flags: readable, writable
                        Unsigned Integer. Range: 0 - 4294967295 Default: 4096 
  directory           : Value of the Directory containing JPEG files
                        flags: readable, writable
                        String. Default: null
  do-timestamp        : Apply current stream time to buffers
                        flags: readable, writable
                        Boolean. Default: false
  name                : The name of the object
                        flags: readable, writable, 0x2000
                        String. Default: "myjpegstreamer0"
  num-buffers         : Number of buffers to output before sending EOS (-1 = unlimited)
                        flags: readable, writable
                        Integer. Range: -1 - 2147483647 Default: -1 
  parent              : The parent of the object
                        flags: readable, writable, 0x2000
                        Object of type "GstObject"
  typefind            : Run typefind before negotiating (deprecated, non-functional)
                        flags: readable, writable, deprecated
                        Boolean. Default: false

Let’s generate few test images of size 1280x720 which we will use in the flip book

# Generate test images of size 1280x720
mkdir generated
cd generated
gst-launch-1.0 videotestsrc pattern=0 num-buffers=1 ! video/x-raw,width=1280,height=720 ! jpegenc ! filesink location=1.jpg
gst-launch-1.0 videotestsrc pattern=1 num-buffers=1 ! video/x-raw,width=1280,height=720 ! jpegenc ! filesink location=2.jpg
gst-launch-1.0 videotestsrc pattern=2 num-buffers=1 ! video/x-raw,width=1280,height=720 ! jpegenc ! filesink location=3.jpg
gst-launch-1.0 videotestsrc pattern=3 num-buffers=1 ! video/x-raw,width=1280,height=720 ! jpegenc ! filesink location=4.jpg
gst-launch-1.0 videotestsrc pattern=4 num-buffers=1 ! video/x-raw,width=1280,height=720 ! jpegenc ! filesink location=5.jpg
gst-launch-1.0 videotestsrc pattern=5 num-buffers=1 ! video/x-raw,width=1280,height=720 ! jpegenc ! filesink location=6.jpg
gst-launch-1.0 videotestsrc pattern=6 num-buffers=1 ! video/x-raw,width=1280,height=720 ! jpegenc ! filesink location=7.jpg
gst-launch-1.0 videotestsrc pattern=7 num-buffers=1 ! video/x-raw,width=1280,height=720 ! jpegenc ! filesink location=8.jpg
gst-launch-1.0 videotestsrc pattern=8 num-buffers=1 ! video/x-raw,width=1280,height=720 ! jpegenc ! filesink location=9.jpg
gst-launch-1.0 videotestsrc pattern=9 num-buffers=1 ! video/x-raw,width=1280,height=720 ! jpegenc ! filesink location=10.jpg

To view the flip book, use the following pipeline

GST_PLUGIN_PATH=. gst-launch-1.0 myjpegstreamer directory=`pwd`/generated/ ! jpegdec ! videoconvert ! video/x-raw,width=1280,height=720,framerate=1/1  ! videorate ! autovideosink

To generate a H264 encoded Matroska video file with these images use the following pipeline

GST_PLUGIN_PATH=. gst-launch-1.0 myjpegstreamer directory=`pwd`/generated/ ! jpegdec ! videoconvert ! video/x-raw,width=1280,height=720,framerate=1/1  ! videorate ! openh264enc ! h264parse ! matroskamux ! filesink location=flipbook.mkv

And there you go, you wrote a GStreamer source element. To summarize what we did here, we use the GStremer base class GstPushSrc, added a property to take input from the user and set the function pointers to point to our functions that do what we want to do to insert data into the GStremaer pipeline.

Hope you enjoyed creating the flip book. See you next time!