Vmeta: Difference between revisions

From OLPC
Jump to navigation Jump to search
No edit summary
 
(45 intermediate revisions by 2 users not shown)
Line 1: Line 1:
vMeta is a hardware-accelerated multimedia encoding and decoding engine included in the XO-1.75 and XO-4.
vMeta is a hardware-accelerated video encoding and decoding engine included in the XO-1.75 and XO-4.


== Installation and test ==
== Installation ==


To obtain the packages referred to below, contact OLPC support. You will be given a URL of a directory which contains packages. In the instructions below, this is represented as $FIXME.
Tested on 12.1.0 on XO-1.75.


These instructions are for OLPC OS [[14.1.0]]:
# Install gstreamer-plugins-marvell and libvmeta-marvell
# Download http://techslides.com/demos/sample-videos/small.mp4 and play it in Totem in GNOME


For XO-1.75, the base package is ''marvell-ipp-mmp2'' and the gstreamer-0.10 plugin packages are ''gstreamer-plugins-marvell-mmp2'' and ''gstreamer-plugins-vmetaxv''.
For a beefier test video, try http://archive.org/download/hellmans_mayonnaise_commercial/hellmans_mayonnaise_commercial.mpeg


For XO-4, the base package is ''marvell-ipp-mmp3'' and the gstreamer-0.10 plugin packages are ''gstreamer-plugins-marvell-mmp3'' and ''gstreamer-plugins-vmetaxv''
Note that these packages do include software decoders too (not sure why) so just seeing the video play back doesn't mean that it is being done by the vmeta hardware engine. The best way to be confident that hardware accel is being used is by looking at totem standard output, which should be limited to:

These packages may be added to your build, or added after install.

See [[OS_Builder/Add_Restricted|How to add restricted software to build]].

=== Add after Install ===

For XO-1.75:
cd /etc/yum.repos.d
sudo wget -q -O restricted.repo $FIXME/restricted.repo
sudo yum install -y marvell-ipp-mmp2 gstreamer-plugins-marvell-mmp2 gstreamer-plugins-vmetaxv

For XO-4:
cd /etc/yum.repos.d
sudo wget -q -O restricted.repo $FIXME/restricted.repo
sudo yum install -y gstreamer-plugins-marvell-mmp3 gstreamer-plugins-vmetaxv

=== gstreamer-1.0 ===

Install gstreamer1-plugins-vmeta (both laptop models) and gstreamer1-plugins-bad-free (for commonly required container support).

== Testing ==

Test videos:
* http://techslides.com/demos/sample-videos/small.mp4 (403 forbidden, as of 2013-10-01)
* http://download.openbricks.org/sample/H264/SampleClip.m4v
* http://archive.org/download/hellmans_mayonnaise_commercial/hellmans_mayonnaise_commercial.mpeg (might be a [http://dev.laptop.org/ticket/12741 bad sample]?)

You can play these in Totem under GNOME.

In some cases software decoders are available, so seeing the video play back doesn't mean that it is being done by the vmeta hardware engine. The best way to be confident that hardware accel is being used is by looking at totem standard output, which should be limited to:


<pre>
<pre>
vmetadec istance(0x....) found
vmetadec istance(0x....) found
At finalizing, Gstvmetadec instance(0x....) DecErroOccured is 0x0
At finalizing, Gstvmetadec instance(0x....) DecErrorOccured is 0x0
Gstvmetadec instance(0x....) is finalized.
Gstvmetadec instance(0x....) is finalized.
</pre>
</pre>


Also look in <tt>dmesg</tt>, check that there are not any vmeta-related complaints there.
Also look in <tt>dmesg</tt>, check that there are not any vmeta-related complaints there. You can also watch /proc/interrupts contents, the vmeta interrupts should increase while playing back the sample videos.

For a more direct gstreamer-0.10 test:
gst-launch filesrc location=hellmans_mayonnaise_commercial.mpeg ! decodebin ! vmetaxvimagesink

For the same thing under gstreamer-1.0:
gst-launch-1.0 filesrc location=hellmans_mayonnaise_commercial.mpeg ! decodebin ! vmetaxvsink


== Troubleshooting ==
== Troubleshooting ==
Line 26: Line 62:
Check that regular audio playback is working (e.g. with <tt>speaker-test</tt>).
Check that regular audio playback is working (e.g. with <tt>speaker-test</tt>).


=== Video plays back with no audio ===
== gstreamer-plugins-marvell ==

Check that you have the appropriate audio codecs installed. Note that while the gstreamer-0.10 packages do include some software audio codecs, the gstreamer1 plugins do not include any audio codecs, you have to go and install them from other sources.

== Software component breakdown ==

=== uio_vmeta ===

This is an in-kernel UIO driver that allows the vmeta hardware engine to be driven from userspace (registers and interrupts). Some changes to the UIO core are needed as well.

=== libphycontmem ===

This library provides a userspace API to allocate and deal with a chunk of physically contiguous DMA-capable main memory. The buffer needs to be quite large, at least 32mb, 40mb in order to view higher quality videos.

Marvell code drops include backends for pmem and bmm (Buffer Management Module, probably a special kernel driver that provides such a buffer). An ion backend is also available from OLPC. This remains a sticky area.

* pmem was really designed to deal with fixed memory areas (e.g. graphics memory), but vmeta is something that we see as optional and on-demand; we don't want to hold this allocation even while vmeta is not being used, and we don't even know that vmeta ''will'' be used.
** When OLPC shipped this for 13.1.0 and previous, we unconditionally set up a DMA-coherent allocation in early boot.
** For 13.2.0 and newer, OLPC ships this solution for XO-1.75, but hacked to allocate the buffer only upon first use of pmem.
* ion is a little better, when combined with the DMA backend (which naturally uses CMA), now the allocation is done on-demand. However, ion provides no API to pass physical addresses to userspace, which is a requirement here.
** OLPC ships this for XO-4, plus a hack (a new ioctl) to provide physical addresses to userspace.
* OLPC has not explored the BMM approach (to the best of my knowledge); this does seem to have been abandoned in newer Marvell code drops though.
* An ideal approach would be for UIO to expose on-demand contiguous DMA allocations; while that has been discussed/proposed a few times, no implementation has gone upstream.

OLPC's version originates from the qseven alpha4 code drop. It has been modified to support ion, in addition to pmem, and to detect which one to use at runtime (we use ion on XO-4 and pmem on XO-1.75). pmem and ion helper code is built directly into the library.

Git repo: http://dev.laptop.org/git/users/dsd/libphycontmem

=== libvmeta ===

This library provides an API for interacting with the low-level vmeta hardware: management of DMA buffers, power, clocks, etc. It uses libphycontmem for memory management and interacts with the uio_vmeta kernel interface for the rest.

OLPC's current version originates from the qseven alpha4 code drop. OLPC's changes are limited to build system tweaks for packaging.

Git repo: http://dev.laptop.org/git/users/dsd/libvmeta

=== marvell-ipp, libvmetahal and libcodecvmetadec ===

: ''Continuing the tradition that IPP is lumped into the vmeta category...:''

The IPP libraries are a set of proprietary '''software''' encoders/decoders. That makes them unrelated to vmeta, which is a hardware-based engine. There is also an overlap, the IPP codecs support some video formats that vmeta can play without software help. Shipping software codecs alongside a hardware-based decoding/encoding engine sounds odd but there is some basis for it:

* The vmeta specifications place some restrictions on video parameters for the supported formats. Maybe there are some files that vmeta cannot play, even though the base format is (generally) supported.
* vMeta only accelerates video, the audio stream must be decoded in software.
* These decoders are optimized for the hardware; we have versions that use iwMMXt2 and other versions that use NEON.
* While there is some overlap, the IPP codecs do support some video formats that are not supported by the hardware (e.g. wmv).

Further confusing the IPP software vs hardware situation, marvell-ipp (software codec package) ships the libraries required for hardware-accelerated vmeta playback.
libcodecvmeta{dec,enc} link against libvmetahal. These are closed source libraries provided by Marvell.


OLPC provides marvell-ipp-mmp3 (for XO-4) and marvell-ipp-mmp2 (for XO-1.75). These packages include both the IPP codec libraries, and the vmeta "codec" library, libvmetahal, and the supporting libmiscgen.
This RPM package is built from two source tarballs: gst-plugins-marvell and gst-plugins-vmetaxv.


In marvell-ipp-mmp3, most libraries are hardfp. The files here come from the Marvell qseven alpha4 code drop. There are two incompatibilities with XO-1.75:
=== gst-plugins-marvell ===
* Some IPP libraries are compiled with NEON instructions, XO-1.75/MMP2 does not support this instruction set
* There is some incompatibility between one of the new vmeta libraries (libmiscgen, libvmetahal, and/or libcodecvmetadec) that causes a bus error in libvmeta when used on XO-1.75.


In marvell-ipp-mmp2, most libraries are softfp. It is puzzling why they seem to work on our hardfp system. These files originate from OLPC's previous libvmeta-marvell-005 package. Beyond that, the source of these files is somewhat unknown. Possibly from a mixture of Android code drops and incomplete tarballs posted on symplicity.
This codebase seems to originate from Marvell. It provides several gstreamer plugins that do encoding/decoding using libvmeta (and friends) - apparently making existing gstreamer apps use hardware acceleration (via vmeta) for encoding and decoding purposes.


Git repo: <tt>/home/dsd/private_git/marvell-ipp</tt> branch (mmp2/mmp3 branches)
==== gstreamer-plugins-marvell-0.10-1 ====


=== gstreamer-plugins-marvell ===
No source was published by the developer. Details unknown.


This is a set of gstreamer plugins that provide playback via vmeta or with the IPP codecs. libgstvmetadec links against libcodecvmetadec, and the IPP-based elements link against the IPP libraries.
==== gstreamer-plugins-marvell-0.10-2 ====


The IPP-based gstreamer plugins are set with a higher priority than default, so that they get preferred (even for existing applications). And the vmeta plugins have an even higher priority for the same reason. This means that existing gstreamer apps automatically use hardware-accelerated or IPP-optimized playback.
The [http://download.solid-run.com/pub/solidrun/cubox/packages/cubox-packages-source/cubox-packages.tar.gz Cubox package bundle] (see [http://www.solid-run.com/mw/index.php/Downloads their wiki]) includes gst-plugins-marvell0.10_0.2.0-0ubuntu1~ppa14.tar.gz. This source tarball has full git history from the developers at Marvell, with the latest change committed in March 2011.


OLPC ships two versions:
Our RPM was based on this, with [http://dev.laptop.org/~dsd/20130408/gst-plugins-marvell-mmp2.patch this code change] applied by OLPC. There is no associated info/changelog for this patch; presumably it is the minimum needed to make it work on MMP2.
# <b>gstreamer-plugins-marvell-mmp3</b> for XO-4. It requires the libraries and headers from marvell-ipp-mmp3, making it unsuitable for XO-1.75.
# <b>gstreamer-plugins-marvell-mmp2</b> for XO-1.75. It requires the libraries and headers from marvell-ipp-mmp2.


Git repo: http://dev.laptop.org/git/users/dsd/gst-plugins-marvell (mmp2/mmp3 branches)
==== gstreamer-plugins-marvell-0.10-3 ====


=== gstreamer-plugins-vmetaxv ===
There were no changes to the gst-plugins-marvell source used to build this version.


This package adds a fork of the xvimagesink plugin. This plugin uses a special hack to detect which buffers have come from the vmeta pipeline. When detected, physical addresses of such buffers can be sent straight to the graphics driver, avoiding a memcpy.
==== gstreamer-plugins-marvell-0.10-4 ====


When incoming buffers have not originated from vmeta, the normal Xv display code is used (including a memcpy of the frame data).
This version switches to an alternative gst-plugins-marvell codebase found in the private Android code-drops from Marvell (specifically the toughpad-a1 drop). No git history is included. Looking at the files and changes through each code drop, it would appear that this codebase shares an origin with the codebase found in the cubox download, but later diverged, while still merging in some of the changes from the cubox-acquired git tree.


Git repo: http://dev.laptop.org/git/users/dsd/gst-plugins-vmetaxv
For example, the Android codebase includes a debian/changelog file, with the latest entry from 2009. The cubox codebase comes with the same changelog but with entries going all the way to 2011. However, some of the things <em>described</em> in the post-2009 cubox codebase changelog <u>are</u> implemented in the Android codebase. This suggests a fork of the codebase and some confused history.


=== gstreamer1-plugins-vmeta ===
Despite the confusion, it does seem clear that this Android codebase is more recent than the Cubox gst-plugins-marvell0.10_0.2.0-0ubuntu1~ppa14.tar.gz (with last commit date March 2011). The gingerbread-beta3 code drop was received in October 2011, and this same codebase was updated in the toughpad-a1 code drop received in November 2012. This new version has some small fixes and other changes.


Carlos Rafael Giani has implemented a new vmeta decoder (including the equivalent vmetaxv replacement) for gstreamer-1.0. The decoder is written from scratch and has several advantages over the previous implementation. See https://github.com/dv1/gst-vmeta for more info.
On top of this, OLPC applies some changes (no changelog/info available):
* g711dec and g711enc compilation enabled in configure.ac
* Marvell code compiles mpeg2dec with the FORBID_WMMX_DEINTERLACE flag; we drop that flag.
* Add missing include to fix compile of gstwmvdec
* Enable compilation of libgstvmetaenc (looks like the Marvell codebase only compiles the decoders)
* Update to the latest GMutex API to fix compile warnings/failures


One important difference here is that these plugins do not include support for MPEG container formats and similar, whereas the 0.10 gstreamer plugins did. So you probably want to install gstreamer1-plugins-bad-free from Fedora for the container support.
=== vmetaxv ===


Another change is that there are no gstreamer plugins included here for access to the IPP audio libraries. So, while the gstreamer-0.10 setup does include some additional audio support, for gstreamer-1.0 you will need to find the required audio codecs (e.g. AAC decoder) from elsewhere.
This source package is a fork of one of the official gstreamer packages. It doesn't seem to come from Marvell, suggesting that it originates from OLPC.


== Previous packages ==
It looks like the motivation here is that xvimagesink always does a copy of the data it is going to display. vmetaxvimagesink is a modified xvimagesink that, in the case where it detects that the buffer comes from vmeta, it avoids the memcpy. The physical address is sent to the Xv driver and it is handled directly.


See [[/Historical packages]]
== libvmeta ==

Latest revision as of 03:51, 4 February 2015

vMeta is a hardware-accelerated video encoding and decoding engine included in the XO-1.75 and XO-4.

Installation

To obtain the packages referred to below, contact OLPC support. You will be given a URL of a directory which contains packages. In the instructions below, this is represented as $FIXME.

These instructions are for OLPC OS 14.1.0:

For XO-1.75, the base package is marvell-ipp-mmp2 and the gstreamer-0.10 plugin packages are gstreamer-plugins-marvell-mmp2 and gstreamer-plugins-vmetaxv.

For XO-4, the base package is marvell-ipp-mmp3 and the gstreamer-0.10 plugin packages are gstreamer-plugins-marvell-mmp3 and gstreamer-plugins-vmetaxv

These packages may be added to your build, or added after install.

See How to add restricted software to build.

Add after Install

For XO-1.75:

cd /etc/yum.repos.d
sudo wget -q -O restricted.repo $FIXME/restricted.repo
sudo yum install -y marvell-ipp-mmp2 gstreamer-plugins-marvell-mmp2 gstreamer-plugins-vmetaxv

For XO-4:

cd /etc/yum.repos.d
sudo wget -q -O restricted.repo $FIXME/restricted.repo
sudo yum install -y gstreamer-plugins-marvell-mmp3 gstreamer-plugins-vmetaxv

gstreamer-1.0

Install gstreamer1-plugins-vmeta (both laptop models) and gstreamer1-plugins-bad-free (for commonly required container support).

Testing

Test videos:

You can play these in Totem under GNOME.

In some cases software decoders are available, so seeing the video play back doesn't mean that it is being done by the vmeta hardware engine. The best way to be confident that hardware accel is being used is by looking at totem standard output, which should be limited to:

vmetadec istance(0x....) found
At finalizing, Gstvmetadec instance(0x....) DecErrorOccured is 0x0
Gstvmetadec instance(0x....) is finalized.

Also look in dmesg, check that there are not any vmeta-related complaints there. You can also watch /proc/interrupts contents, the vmeta interrupts should increase while playing back the sample videos.

For a more direct gstreamer-0.10 test:

 gst-launch filesrc location=hellmans_mayonnaise_commercial.mpeg ! decodebin ! vmetaxvimagesink

For the same thing under gstreamer-1.0:

 gst-launch-1.0 filesrc location=hellmans_mayonnaise_commercial.mpeg ! decodebin ! vmetaxvsink

Troubleshooting

Hangs after displaying first frame

Check that regular audio playback is working (e.g. with speaker-test).

Video plays back with no audio

Check that you have the appropriate audio codecs installed. Note that while the gstreamer-0.10 packages do include some software audio codecs, the gstreamer1 plugins do not include any audio codecs, you have to go and install them from other sources.

Software component breakdown

uio_vmeta

This is an in-kernel UIO driver that allows the vmeta hardware engine to be driven from userspace (registers and interrupts). Some changes to the UIO core are needed as well.

libphycontmem

This library provides a userspace API to allocate and deal with a chunk of physically contiguous DMA-capable main memory. The buffer needs to be quite large, at least 32mb, 40mb in order to view higher quality videos.

Marvell code drops include backends for pmem and bmm (Buffer Management Module, probably a special kernel driver that provides such a buffer). An ion backend is also available from OLPC. This remains a sticky area.

  • pmem was really designed to deal with fixed memory areas (e.g. graphics memory), but vmeta is something that we see as optional and on-demand; we don't want to hold this allocation even while vmeta is not being used, and we don't even know that vmeta will be used.
    • When OLPC shipped this for 13.1.0 and previous, we unconditionally set up a DMA-coherent allocation in early boot.
    • For 13.2.0 and newer, OLPC ships this solution for XO-1.75, but hacked to allocate the buffer only upon first use of pmem.
  • ion is a little better, when combined with the DMA backend (which naturally uses CMA), now the allocation is done on-demand. However, ion provides no API to pass physical addresses to userspace, which is a requirement here.
    • OLPC ships this for XO-4, plus a hack (a new ioctl) to provide physical addresses to userspace.
  • OLPC has not explored the BMM approach (to the best of my knowledge); this does seem to have been abandoned in newer Marvell code drops though.
  • An ideal approach would be for UIO to expose on-demand contiguous DMA allocations; while that has been discussed/proposed a few times, no implementation has gone upstream.

OLPC's version originates from the qseven alpha4 code drop. It has been modified to support ion, in addition to pmem, and to detect which one to use at runtime (we use ion on XO-4 and pmem on XO-1.75). pmem and ion helper code is built directly into the library.

Git repo: http://dev.laptop.org/git/users/dsd/libphycontmem

libvmeta

This library provides an API for interacting with the low-level vmeta hardware: management of DMA buffers, power, clocks, etc. It uses libphycontmem for memory management and interacts with the uio_vmeta kernel interface for the rest.

OLPC's current version originates from the qseven alpha4 code drop. OLPC's changes are limited to build system tweaks for packaging.

Git repo: http://dev.laptop.org/git/users/dsd/libvmeta

marvell-ipp, libvmetahal and libcodecvmetadec

Continuing the tradition that IPP is lumped into the vmeta category...:

The IPP libraries are a set of proprietary software encoders/decoders. That makes them unrelated to vmeta, which is a hardware-based engine. There is also an overlap, the IPP codecs support some video formats that vmeta can play without software help. Shipping software codecs alongside a hardware-based decoding/encoding engine sounds odd but there is some basis for it:

  • The vmeta specifications place some restrictions on video parameters for the supported formats. Maybe there are some files that vmeta cannot play, even though the base format is (generally) supported.
  • vMeta only accelerates video, the audio stream must be decoded in software.
  • These decoders are optimized for the hardware; we have versions that use iwMMXt2 and other versions that use NEON.
  • While there is some overlap, the IPP codecs do support some video formats that are not supported by the hardware (e.g. wmv).

Further confusing the IPP software vs hardware situation, marvell-ipp (software codec package) ships the libraries required for hardware-accelerated vmeta playback. libcodecvmeta{dec,enc} link against libvmetahal. These are closed source libraries provided by Marvell.

OLPC provides marvell-ipp-mmp3 (for XO-4) and marvell-ipp-mmp2 (for XO-1.75). These packages include both the IPP codec libraries, and the vmeta "codec" library, libvmetahal, and the supporting libmiscgen.

In marvell-ipp-mmp3, most libraries are hardfp. The files here come from the Marvell qseven alpha4 code drop. There are two incompatibilities with XO-1.75:

  • Some IPP libraries are compiled with NEON instructions, XO-1.75/MMP2 does not support this instruction set
  • There is some incompatibility between one of the new vmeta libraries (libmiscgen, libvmetahal, and/or libcodecvmetadec) that causes a bus error in libvmeta when used on XO-1.75.

In marvell-ipp-mmp2, most libraries are softfp. It is puzzling why they seem to work on our hardfp system. These files originate from OLPC's previous libvmeta-marvell-005 package. Beyond that, the source of these files is somewhat unknown. Possibly from a mixture of Android code drops and incomplete tarballs posted on symplicity.

Git repo: /home/dsd/private_git/marvell-ipp branch (mmp2/mmp3 branches)

gstreamer-plugins-marvell

This is a set of gstreamer plugins that provide playback via vmeta or with the IPP codecs. libgstvmetadec links against libcodecvmetadec, and the IPP-based elements link against the IPP libraries.

The IPP-based gstreamer plugins are set with a higher priority than default, so that they get preferred (even for existing applications). And the vmeta plugins have an even higher priority for the same reason. This means that existing gstreamer apps automatically use hardware-accelerated or IPP-optimized playback.

OLPC ships two versions:

  1. gstreamer-plugins-marvell-mmp3 for XO-4. It requires the libraries and headers from marvell-ipp-mmp3, making it unsuitable for XO-1.75.
  2. gstreamer-plugins-marvell-mmp2 for XO-1.75. It requires the libraries and headers from marvell-ipp-mmp2.

Git repo: http://dev.laptop.org/git/users/dsd/gst-plugins-marvell (mmp2/mmp3 branches)

gstreamer-plugins-vmetaxv

This package adds a fork of the xvimagesink plugin. This plugin uses a special hack to detect which buffers have come from the vmeta pipeline. When detected, physical addresses of such buffers can be sent straight to the graphics driver, avoiding a memcpy.

When incoming buffers have not originated from vmeta, the normal Xv display code is used (including a memcpy of the frame data).

Git repo: http://dev.laptop.org/git/users/dsd/gst-plugins-vmetaxv

gstreamer1-plugins-vmeta

Carlos Rafael Giani has implemented a new vmeta decoder (including the equivalent vmetaxv replacement) for gstreamer-1.0. The decoder is written from scratch and has several advantages over the previous implementation. See https://github.com/dv1/gst-vmeta for more info.

One important difference here is that these plugins do not include support for MPEG container formats and similar, whereas the 0.10 gstreamer plugins did. So you probably want to install gstreamer1-plugins-bad-free from Fedora for the container support.

Another change is that there are no gstreamer plugins included here for access to the IPP audio libraries. So, while the gstreamer-0.10 setup does include some additional audio support, for gstreamer-1.0 you will need to find the required audio codecs (e.g. AAC decoder) from elsewhere.

Previous packages

See /Historical packages