Decoding video with a mainline kernel on i.MX6

The i.MX6 processors contain a Video Processing Unit (VPU) that allows video decoding and encoding to be done in hardware.

Thanks to the excellent work from Philipp Zabel at Penguntronix the VPU driver is supported in the mainline kernel at drivers/media/platform/coda/.

VPU Firmware

The VPU hardware needs a firmware to operate and this firmware is available via NXP BSP.

Firmware are usually installed in the root file system at the lib/firmware directory.

The coda driver in mainline used to be selected as built-in, but this causes problems because the coda driver could be probed prior to the rootfs getting mounted. In this case the VPU firmware is not found and the coda driver does not probe successfully.

In order to fix this problem I changed the coda driver to be built as module in this commit

At the time of this writing the most recent stable kernel is 4.13.5 and the above commit will land in the 4.14 kernel.

Adjusting the CMA size

In order to prevent memory allocation errors when displaying full HD videos at full HD resolutions it is necessary to adjust the contiguous memory allocation size properly.

The default CMA area in mainline is 16MB, which is not enough for the full HD playback case.

In this case passing cma=128M in the kernel command line is needed.

Patch for adding coda support in Buildroot

Here is a patch I did for adding VPU support for the imx6-sabresd_qt5_defconfig target:

Add support for it. Find this patch here.

From a0004f5547d4ae54b5bb5a68dfbcaecce5468d0c Mon Sep 17 00:00:00 2001
From: Fabio Estevam <fabio.estevam@nxp.com>
Date: Mon, 9 Oct 2017 14:45:53 -0300
Subject: [PATCH] configs/imx6-sabresd: Add VPU decoding support

Mainline kernel is able to decode video via the coda driver.

Signed-off-by: Fabio Estevam <fabio.estevam@nxp.com>
---
 board/freescale/imx6-sabresd/linux_qt5.fragment |  1 +
 configs/imx6-sabresd_qt5_defconfig              | 20 ++++++++++++++++++++
 2 files changed, 21 insertions(+)

diff --git a/board/freescale/imx6-sabresd/linux_qt5.fragment
b/board/freescale/imx6-sabresd/linux_qt5.fragment
index ffa5f63..d4f26b1 100644
--- a/board/freescale/imx6-sabresd/linux_qt5.fragment
+++ b/board/freescale/imx6-sabresd/linux_qt5.fragment
@@ -2,3 +2,4 @@
 # ondemand governor with a mainline kernel.
 # CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND is not set
 CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE=y
+CONFIG_VIDEO_CODA=m
diff --git a/configs/imx6-sabresd_qt5_defconfig
b/configs/imx6-sabresd_qt5_defconfig
index 832e326..7199cb0 100644
--- a/configs/imx6-sabresd_qt5_defconfig
+++ b/configs/imx6-sabresd_qt5_defconfig
@@ -75,3 +75,23 @@ BR2_PACKAGE_QT5BASE_GIF=y
 BR2_PACKAGE_QT5BASE_JPEG=y
 BR2_PACKAGE_QT5CINEX=y
 BR2_PACKAGE_QT5CINEX_HD=y
+
+# NXP firmware package
+BR2_PACKAGE_FREESCALE_IMX=y
+BR2_PACKAGE_FREESCALE_IMX_PLATFORM_IMX6Q=y
+BR2_PACKAGE_FIRMWARE_IMX=y
+
+# libdrm tests
+BR2_PACKAGE_LIBDRM_INSTALL_TESTS=y
+
+# gstreamer-1.0
+BR2_PACKAGE_GSTREAMER1=y
+BR2_PACKAGE_GST1_PLUGINS_GOOD=y
+BR2_PACKAGE_GST1_PLUGINS_GOOD_PLUGIN_V4L2=y
+BR2_PACKAGE_GST1_PLUGINS_GOOD_PLUGIN_V4L2_PROBE=y
+BR2_PACKAGE_GST1_PLUGINS_BAD=y
+BR2_PACKAGE_GST1_PLUGINS_BAD_PLUGIN_VIDEOPARSERS=y
+BR2_PACKAGE_GST1_PLUGINS_BAD_PLUGIN_KMS=y
+
+# For automatic firmware loading
+BR2_ROOTFS_DEVICE_CREATION_DYNAMIC_MDEV=y
--
2.7.4

This patch basically does the following:

• Select the coda driver as module in the defconfig,
• Install the VPU firmware
• Install some basic Gstreamer plugins
• Install BR2_PACKAGE_LIBDRM_INSTALL_TESTS package, so that modetest application can be used.

After applying this patch, generate a Buildroot image for the imx6-sabresd board:

$ make imx6-sabresd_qt5_defconfig
$ make

Flash the SD card:

$ sudo dd if=output/images/sdcard.img of=/dev/mmcblk0; sync

Download a video sample file and copy it to the root directory of the SD card:

$ wget http://linode.boundarydevices.com/videos/trailer_1080p_h264_mp3.avi
$ sudo cp trailer_1080p_h264_mp3.avi /media/<user>/<rootfs-patch>/root/; sync

Insert the SD card in the mx6sabresd and boot it.

Running video decoding with Gstreamer

As the mx6sabresd has an LVDS and HDMI outputs, it is necessary to know what is the connector that corresponds to the HDMI output.

This information can be found by running:

# modetest

This commands shows lots of information. These are the relevant lines that gives the HDMI connector id number. In our case it shows up as 37:

Connectors:
id      encoder status          name            size (mm)       modes   encoders
37      36      connected       HDMI-A-1        510x280         33

Now we are ready to play a video file.

In the mx6sabresd prompt run the following pipeline:

# gst-launch-1.0 filesrc location=/root/trailer_1080p_h264_mp3.avi ! \
avidemux ! h264parse ! v4l2video1dec capture-io-mode=dmabuf ! \
kmssink connector-id=37 name=imx-drm sync=0

Then the video will be displayed in the HDMI output.

The VPU interrupts can be checked to confirm that the VPU is actually being used:

# cat /proc/interrupts | grep vpu
 29:       1085          0          0          0       GPC  12 Level  2040000.vpu

Running the Gstreamer pipeline example above with a & symbol, allows us to run a top command to observer the CPU utilization while the video is running: it only consumes 1% of CPU as all the decoding work is done by the hardware VPU block. Awesome! :-)

At the time of this writing the kmssink is not able to display a video into a display, whem their resolutions do not match. For example, playing a full HD video on the 1024x768 LVDS display is currently not supported via IPU scaler. It is possible to use the standard scaling plugins from Gstreamer though.