gstreamer – Video pipelines for phyCAM & Vision Kits

With our Ezpire building block for GStreamer You receive pre-configured video pipelines for all supported phyCAM modules in the Vision BSPs. Demo scripts for live streaming, JPG/RAW saving, and more are included. phytec-vision-image Integrated and ready to use immediately. Use it as a basis for your own image processing or video applications.

What is GStreamer?

GStreamer is an open-source multimedia framework for Linux that is ideally suited for embedded video applications. It allows the creation of flexible pipelines for capturing, processing, encoding, and outputting video and audio data.

In the PHYTEC context, GStreamer uses the V4L2 (Video4Linux2) interface for controlling phyCAM cameras. phytec-vision-images Includes pre-configured pipelines for all supported phyCAM modules (VM-x16, VM-x17, VM-x20, VM-x24, etc.) with detailed, easy-to-read scripts.

GStreamer out-of-the-box

We take the complexity of camera integration off your hands. GStreamer is already fully integrated into our system. phytec-vision-images (Vision BSPs) are integrated and optimized for all our video kits. You receive a collection of ready-to-run demo scripts covering the most important video use cases:

  • Live video streaming: Output the camera signal directly and smoothly to a connected display (e.g. via HDMI).

  • Image and video capture: Seamlessly save images as compressed JPGs or in lossless RAW format at full sensor resolution.

  • Sensor-specific optimization: We provide customized configurations for our various phyCAM modules. The scripts automatically detect the connected camera type (color/monochrome) and optimally adjust the parameters.

  • Hardware acceleration (ISP):  To use an existing Image Signal Processor (ISP), configuration files are required. Selected board-level cameras from our phyCAM series are already pre-calibrated for use with selected processor boards. This allows you to use basic ISP functions directly. These include, among others: demosaic, Denoise/Sharpen Filter, AEC (Auto Exposure Control) AWB (Auto White Balance) BLC (Black Level Correction) Defect Pixel Cluster Correction and WDR3 (Wide Dynamic Range).

  • Multi-camera setups: Out-of-the-box support for the synchronous operation of two cameras (dual-camera), ideal for complex machine vision scenarios. We offer corresponding support for certain processors that have the capability of "virtual channel" technology. This allows multiple cameras to be connected to a single MIPI CSI-2 channel.

Simple configuration scripts allow for presetting in the camera and the processor's camera interface. The user then has access to the image/stream configured according to their wishes as a V4L2 device.

Technical Deep Dive

Our Vision images utilize V4L2 subdevices in combination with media-ctl for dynamic camera configuration (routing and format negotiation). GStreamer then retrieves the image data directly from the video device via V4L2. Each of our phyCAM modules (VM-x16, VM-x17, VM-x20, VM-x24, ...) is assigned to a video device. The included shell scripts in the /gstreamer-examples/ directory of the root partition abstract the complexity of the call.

  • Live streaming (Wayland/frame buffer): When `cam-fbdev_1280x720.shi` is called, the configuration script is executed first. Then, certain camera properties are set using `v4l2-ctrl`. Finally, a GStreamer pipeline is set up that outputs the V4L2 video stream to the display via a suitable sink (e.g., `waylandsink` or direct framebuffer), optimized for minimal CPU load.

  • Raw and JPG capture: Scripts like cam-save_jpg_full_res.sh call pipelines that capture the image data, compress it using jpegenc, and save it via filesink. With cam-save_raw_full_res.sh, the raw sensor data is dumped to the file system without any processing.

A look under the hood reveals how GStreamer optimally utilizes the SoC hardware. For example, an efficient H.264 encoding pipeline for a VM-016 module (AR0144) looks like this:
 
bash
gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-bayer,format=grbg,width=1280,height=800 ! bayer2rgbneon ! queue ! vpuenc_h264 ! rtspclientsink

Here, the Bayer raw data is converted using the highly optimized plugin bayer2rgbneon (utilizing the ARM NEON instruction set extension). The converted image is then fed into the dedicated hardware video encoder of the NXP SoC (vpuenc_h264), thus freeing up CPU resources for the actual application.

The V4L2 nodes are set up modularly. Cameras are loaded during boot via U-Boot overlays in the bootenv.txt file (e.g., overlays=imx8mp-isi-csi1.dtbo imx8mp-vm016-csi1.dtbo).
If an ISP is available, we offer extended pipelines in the directories marked "isp". Here, the V4L2 stream is not accessed directly, but routed through the hardware ISP of the NXP chip. GStreamer then accesses /dev/video[X], where finished YUV/RGB images are already present, where, for example, auto-exposure, auto-white balance, and debayering have already been applied latency-free in hardware.

Recommended Hardware

To evaluate our System on Modules and phyCAM modules in practice, we offer tailored solutions. Embedded Vision Development Kits These bundles contain everything you need to get started: System on Module (SoM), Baseboard, camera, lens and a pre-installed Vision BSP with all GStreamer examples.

For demanding video applications, we particularly recommend:

Our benchmark for embedded vision – with dual ISP and dedicated AI unit (NPU) for high-performance multi-camera setups.

Highly efficient and cost-effective. The UI runs via the software/DRM backend without the need for a powerful 3D GPU.

You can find more Vision setups for other SoCs and form factors on our website. Imaging Kit Overview Page.

The PHYTEC phyCAM portfolio
The GStreamer integration unfolds its full potential through our Extensive range of industrial-grade camera modules (phyCAM series). Whether rolling shutter or global shutter, monochrome or color, 1 megapixel to 4k resolutions – our hardware and software drivers are perfectly matched.

Which setup best suits your requirements? Camera modules, for example, can be easily exchanged via the standardized phyCAM interface. 
Talk to us – We will gladly help you choose the right hardware.

Getting Started

Get started with GStreamer and your phyCAM in just a few steps.

All GStreamer demo scripts and the necessary camera drivers are already included in our special phytec-vision-image Integrated. Depending on the hardware you are using, the initial setup process is as follows:

If you are using a PHYTEC Imaging Kit:
If you have purchased one of our vision kits (e.g., the phyBOARD-Pollux Imaging Kit), you don't need to do anything else. The phytec-vision-image is already pre-installed on these kits. You can boot up and get started right away.

If you are using a standard kit or a single board computer (SBC):
If you are using a regular development kit or a standard board, you can easily flash the Vision image yourself. To do this, go to the PHYTEC website and select the following: Product page of your module (zBiMX 8M Plus) and navigate to the area DownloadsThere you will find the latest phytec-vision-image for download and flashing onto an SD card.

Where can I find the GStreamer scripts and instructions?
Once the Vision image is running on your hardware, use our Getting Started Guide, to start the GStreamer pipelines.

  • Find documentation: Go to the product page of your module and navigate to Downloads > Image Processing > Quick Start Guides with cameras. Download the "Getting Started Guide phyCAM" there.

  • Start GStreamer demos: In a separate section of these documents you will find a detailed overview and instructions for all the GStreamer demo scripts we have prepared (such as live streaming, image capture and hardware-accelerated encoding).