GStreamer: Difference between revisions

From OLPC
Jump to navigation Jump to search
(→‎RPM Fusion: update gstreamer console code, using generic for getting latest version (applies for all fc10 and up). Old code commented out.)
No edit summary
 
(18 intermediate revisions by the same user not shown)
Line 2: Line 2:


== Introduction ==
== Introduction ==
GStreamer is a library that supports multimedia, ranging from simple audio/video playback of [[Ogg]]/Vorbis files to complex audio mixing and video processing.
GStreamer is a library that supports multimedia, ranging from playback to complex mixing and processing.


* The [[Totem plugin]] uses GStreamer to play [[multimedia]] URLs in the [[Browse]] activity.
* the [[Totem plugin]] uses GStreamer to play [[multimedia]] URLs in the [[Browse]] activity,
* The [[Record]] activity uses GStreamer to deal with streams from the camera and microphone.
* the [[Record]] activity uses GStreamer to deal with streams from the camera and microphone, and;
* the Jukebox activity uses GStreamer to play video or audio objects in the journal.


== Adding codecs ==
== Adding codecs ==
As configured for OLPC the GStreamer framework supports a variety of multimedia [[File formats]].


<div style="background:#99cc55; color:white; padding:4px;">This section describes third-party software that does not meet the [[OLPC_on_open_source_software|OLPC standards for software freedom]]. The use of such software is unsupported; OLPC does not endorse or encourage its use.</div>
Expert uses can install software packages that add support for other file formats.
However,
'''NOTE:''' Release [[8.2.0]] uses an old backported version of [[GStreamer]], so most web guides to installing additional codec support ''will not work''.


<div style="background:#cc9955; color:white; padding:4px;">rpmfusion repositories no longer exist for Fedora 18, and irregardless require later cryptographic standards than are supported by the OLPC XO.</div>
=== MP3 ===


Some codecs can be [[OS_Builder/Add_Restricted|added during build]]. Adding during build is the method supported by OLPC, because it works with our operating system {{Code|[[olpc-update|olpc-update]]}} feature.
To enable MP3 playback in Sugar or GNOME, an extra codec has to be installed.


Some codecs are ''not'' included in OLPC OS, because of restrictions on redistribution. But they can be added after install. Such changes are lost on {{Code|[[olpc-update|olpc-update]]}}.
==== RPM Fusion ====


To add after install, on the XO laptop, start Terminal and paste this [[Compound_commands|compound command]]:
<div style="background:#99cc55; color:white; padding:4px;">This section describes third-party software that does not meet the [[OLPC_on_open_source_software|OLPC standards for software freedom]]. The use of such software is unsupported; OLPC does not endorse or encourage its use.</div>


sudo yum localinstall -y \
On OLPC XO-1.5 and XO-1 (tested for releases [[Release_notes/10.1.2|10.1.2]] and [[Release_notes/12.1.0|12.1.0]]):<ref>[http://rpmfusion.org/Configuration]</ref>
http://download1.rpmfusion.org/free/fedora/rpmfusion-free-release-$(rpm -E %fedora).noarch.rpm \
http://download1.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-$(rpm -E %fedora).noarch.rpm && \
sudo yum install -y \
gstreamer1-{plugins-{ugly,bad-free{-extras,world},base-tools,good-extras,bad-free},libav} \
gstreamer-{plugins-{ugly,bad},ffmpeg} && echo Complete!


When the prompt reappears, with "Complete!" above the prompt, it is ready to use. You may need to stop and restart any activity using GStreamer, once.
su
yum localinstall --nogpgcheck http://download1.rpmfusion.org/free/fedora/rpmfusion-free-release-$(rpm -E %fedora).noarch.rpm http://download1.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-$(rpm -E %fedora).noarch.rpm
yum install -y gstreamer-plugins-ugly gstreamer-plugins-bad gstreamer-ffmpeg
<!--
On OLPC OS [[Release_notes/10.1.2|10.1.2]] and [[Release_notes/10.1.3|10.1.3]], on XO-1.5 and XO-1:


These instructions have been tested for releases 10.1.2, 12.1.0, 13.2.0, 13.2.1, 13.2.2, 13.2.6 and 14.1.0.
su
rpm -Uvh http://rpm.livna.org/livna-release.rpm http://download1.rpmfusion.org/free/fedora/releases/11/Everything/i386/os/rpmfusion-free-release-11-1.noarch.rpm http://download1.rpmfusion.org/nonfree/fedora/releases/11/Everything/i386/os/rpmfusion-nonfree-release-11-1.noarch.rpm
yum install -y gstreamer-plugins-ugly

However, these Fedora FAQ instructions are not enough, since Fedora 11 used by OLPC OS 10.x has been marked end-of-life.
-->
When it says "Complete!" it is ready to use.


For more information, see:
For more information, see:
*[http://www.fedorafaq.org/#yumconf http://www.fedorafaq.org/#yumconf]
*http://www.fedorafaq.org/#yumconf
*[http://www.fedorafaq.org/#mp3 http://www.fedorafaq.org/#mp3]
*http://www.fedorafaq.org/#mp3
*[http://rpmfusion.org/FAQ http://rpmfusion.org/FAQ]
*http://rpmfusion.org/FAQ
*http://rpmfusion.org/Configuration (the basis for these instructions)

==== Fluendo ====

<div style="background:#99cc55; color:white; padding:4px;">This section describes third-party software that does not meet the [[OLPC_on_open_source_software|OLPC standards for software freedom]]. The use of such software is unsupported; OLPC does not endorse or encourage its use.</div>

See [[Fluendo mp3 decoder]] instructions.

== Using gst at the command line ==
The rest of this page is oriented towards developers using GStreamer.

In addition, expert users can run GStreamer commands similar to those below in a [[Terminal Activity]] to do things like
* play audio files in the background
* stream XO video to other servers

Home page: http://gstreamer.freedesktop.org/

{{anchor|Getting started}}

{{developers}}

==Getting started programming==

[http://www.jonobacon.org/?p=750 Getting started with GStreamer with Python]

== Recipes ==
=== Camera ===
* Have to script v4l2 via gstreamer to capture a single frame from the camera. See sugar/shell/intro/glive.py for sample code
* Try this from the command line: <tt>gst-launch-0.10 v4l2src ! ffmpegcolorspace ! pngenc ! filesink location=foo.png</tt>
* See also: [[Programming the camera]]
* You can simulate this hardware via a file source in gstreamer

=== Camera-as-video-camera (v4l2) ===
* Is a regular v4l2 device available via gstreamer (gst module)
** If you do not have hardware, you can simulate the camera interactively with an regular v4l2 source (many cheap web-cams provide this type of source)
** For test-driven development, you can use any gstreamer source (such as a regular file), hooking up your code to use a file source instead of a v4l2src
* Try this from a terminal in the developer console: <tt>gst-launch-0.10 v4l2src ! ximagesink</tt> (The image will appear <i>behind </i>the developer console window so you'll need to move the window aside. In addition, it will have strange colors, since you are displaying the YCrCb colorspace as though it is RGB data. Use <tt>gst-launch-0.10 v4l2src ! ffmpegcolorspace ! ximagesink</tt> to run slowly and get the right colors.)
* See also: [[Programming the camera]]
* You can simulate this hardware via a file source in gstreamer

==GStreamer 101==

===Examples===
v4l2src can always be replaced by videotestsrc, and alsasrc by audiotestsrc.

Beginning of a normal video pipeline:
v4l2src ! queue ! videorate ! video/x-raw-yuv,framerate=15/1 ! videoscale ! video/x-raw-yuv,width=160,height=120 ! ...
Beginning of a normal audio pipeline:
alsasrc ! audio/x-raw-int,rate=8000,channels=1,depth=8 ! ...
Video encoding:
... ! ffmpegcolorspace ! theoraenc ! ...
Audio encoding:
... ! audioconvert ! vorbisenc ! ...
Video output:
... ! ffmpegcolorspace ! videoscale ! ximagesink
Audio output:
... ! audioconvert ! alsasink ''sync=false''
Stdout:
... ! fdsink ''fd=1''

Encode video+audio as ogg:
v4l2src ! queue ! ffmpegcolorspace ! theoraenc ! queue ! \
oggmux name=mux alsasrc ! queue ! audioconvert ! vorbisenc ! queue ! mux. mux. ! queue ! ...

A long version of videotestsrc ! ximagesink:
videotestsrc ! theoraenc ! oggmux ! oggdemux ! theoradec ! ffmpegcolorspace ! videoscale ! ximagesink
A long version of videotestsrc ! ximagesink &amp;; audiotestsrc ! alsasink:
videotestsrc ! ffmpegcolorspace ! theoraenc ! queue ! \
oggmux name=mux audiotestsrc ! audioconvert ! vorbisenc ! queue ! mux. mux. ! queue ! \
oggdemux name=demux ! queue ! theoradec ! ffmpegcolorspace ! \
ximagesink demux. ! queue ! vorbisdec ! audioconvert ! alsasink
:Hangs on current fc6. [[User:MitchellNCharity|MitchellNCharity]] 17:43, 12 June 2007 (EDT)

Live video streaming to an icecast server:
v4l2src ! ffmpegcolorspace ! videoscale ! video/x-raw-yuv,width=320,height=240 ! theoraenc quality=16 ! oggmux ! shout2send ip=192.168.1.100 port=8000 password=hackme mount=olpc.ogg

Live streaming to an icecast server:
v4l2src ! queue ! ffmpegcolorspace ! theoraenc quality=16 ! queue ! oggmux name=mux alsasrc ! audio/x-raw-int,rate=8000,channels=1,depth=8 ! queue ! audioconvert ! vorbisenc ! queue ! mux. mux. ! \
queue ! shout2send ip=... port=... password=... mount=/''whatever''.ogg

Time lapse photography (1 frame every 30 seconds):
gst-launch v4l2src ! videorate ! video/x-raw-yuv,width=640,height=480,framerate=\(fraction\)1/30 ! ffmpegcolorspace ! jpegenc ! multipartmux ! filesink location=lapse1.mjpeg
press ctrl-c to stop

Playback:
gst-launch filesrc location=lapse1.mjpeg ! multipartdemux ! jpegdec ! ffmpegcolorspace ! autovideosink

===Notes===
====Colorspace====
The video coming raw out of v4l2src is color formatted YUYV. That is, there are 4 bytes per 2 pixels. The first 8 bit Y represents the luma (brightness) of the first pixel, the second 8 bit Y is the brightness of the second, the 8 bit U is the blue chrominance of both pixels, and the 8 bit V is the red chrominance of both.
====video size, framerate, and theoraenc quality====

====audio encoding====
glive.py used wav rather than vorbis. Why?

===Elements===

Adapters:
*[http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer-plugins/html/gstreamer-plugins-tee.html tee]
*[http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer-plugins/html/gstreamer-plugins-queue.html queue]
*[http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-base-plugins/html/gst-plugins-base-plugins-videoscale.html videoscale] Convert video size.
*[http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-base-plugins/html/gst-plugins-base-plugins-videorate.html videorate] Convert video rate.
*[http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-base-plugins/html/gst-plugins-base-plugins-ffmpegcolorspace.html ffmpegcolorspace] Convert video colorspace.
*[http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-base-plugins/html/gst-plugins-base-plugins-audioconvert.html audioconvert] Convert audio format.

Video characteristics:
*[http://gstreamer.freedesktop.org/data/doc/gstreamer/head/pwg/html/section-types-definitions.html#table-video-types video/x-raw-yuv]
*[http://gstreamer.freedesktop.org/data/doc/gstreamer/head/pwg/html/section-types-definitions.html#table-video-types framerate=15/1]
*[http://gstreamer.freedesktop.org/data/doc/gstreamer/head/pwg/html/section-types-definitions.html#table-video-types width=160,height=120]
Audio characteristics:
*[http://gstreamer.freedesktop.org/data/doc/gstreamer/head/pwg/html/section-types-definitions.html#table-audio-types audio/x-raw-int,rate=8000,channels=1,depth=8]

File I/O:
*[http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer-plugins/html/gstreamer-plugins-fdsrc.html fdsrc]
*[http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer-plugins/html/gstreamer-plugins-fdsink.html fdsink]
*[http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer-plugins/html/gstreamer-plugins-filesrc.html filesrc]
*[http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer-plugins/html/gstreamer-plugins-filesink.html filesink]
*[http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-base-plugins/html/gst-plugins-base-plugins-gnomevfssrc.html gnomevfssrc]
*[http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-base-plugins/html/gst-plugins-base-plugins-gnomevfssink.html gnomevfssink]

Video sources:
*[http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-base-plugins/html/gst-plugins-base-plugins-videotestsrc.html videotestsrc]
*[http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-good-plugins/html/gst-plugins-good-plugins-v4l2src.html v4l2src]

Audio sources:
*[http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-base-plugins/html/gst-plugins-base-plugins-audiotestsrc.html audiotestsrc]
*[http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-base-plugins/html/gst-plugins-base-plugins-alsasrc.html alsasrc]

Coding video:
*[http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-base-plugins/html/gst-plugins-base-plugins-theoraenc.html theoraenc]
*[http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-base-plugins/html/gst-plugins-base-plugins-theoradec.html theoradec]

Coding audio:
*[http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-base-plugins/html/gst-plugins-base-plugins-vorbisenc.html vorbisenc]
*[http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-base-plugins/html/gst-plugins-base-plugins-vorbisdec.html vorbisdec]
*speexenc

Wrapping:
*{{broken link|[http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-base-plugins/html/gst-plugins-base-plugins-oggmux.html oggmux]}} ''([[User:MitchellNCharity|MitchellNCharity]] 11:44, 13 June 2007 (EDT))''
*[http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-base-plugins/html/gst-plugins-base-plugins-oggdemux.html oggdemux]

Outputing video:
*[http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-base-plugins/html/gst-plugins-base-plugins-ximagesink.html ximagesink] ''or'' [http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-base-plugins/html/gst-plugins-base-plugins-xvimagesink.html xvimagesink]

Outputting audio:
*[http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-base-plugins/html/gst-plugins-base-plugins-alsasink.html alsasink]

Not sure about:
*[http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-base-plugins/html/gst-plugins-base-plugins-vorbisparse.html vorbisparse]

Sending ogg to an icecast streaming server:
... ! shout2send ip=... port=... password=... mount=/whatever.ogg
gst-launch ... ! fdsink '''|''' oggfwd ''host port password mount''

:Regretabbly, it appears fc6, and thus olpc, does not include shout2send in gst-plugins-good-plugins. Nor does it have rpms for [http://www.v2v.cc/~j/ffmpeg2theora/oggfwd/ oggfwd], or other possible alternatives. You can grab a random binary of oggfwd from it's site, or compile shout2send from source. :( [[User:MitchellNCharity|MitchellNCharity]] 17:43, 12 June 2007 (EDT)

:Correction, shout2send is on the olpc. oggfwd is not. --[[User:Damonkohler|Damonkohler]] 23:14, 13 September 2007 (EDT)

===Doing live streaming video from an xo===
Method1

xo sending video:

gst-launch v4l2src ! video/x-raw-yuv,width=320,height=240,framerate=\(fraction\)5/1 ! ffmpegcolorspace ! jpegenc ! multipartmux ! tcpserversink host=192.168.2.1 port=5000

xo recieving video:

gst-launch tcpclientsrc host=192.168.2.2 port=5001 ! multipartdemux ! jpegdec ! autovideosink

Method2

xo sending video:

gst-launch v4l2src ! video/x-raw-yuv,width=320,height=240,framerate=\(fraction\)2/1 ! ffmpegcolorspace ! smokeenc keyframe=8 qmax=40 ! udpsink host=192.168.1.1 port=5000

xo receiving video:

gst-launch udpsrc port=5000 ! smokedec ! autovideosink


Test cases for Browse activity:
Method3 Video+Audio
*http://www.quirksmode.org/html5/tests/video.html


== Adding codecs using xo-custom or other scripting ==
Sending XO:
gst-launch v4l2src ! queue ! video/x-raw-yuv,width=320,height=240,framerate=\(fraction\)4/1 ! videorate ! videoscale ! ffmpegcolorspace ! queue ! smokeenc ! queue ! udpsink host=192.168.2.129 port=5000 alsasrc ! queue ! audio/x-raw-int,rate=8000,channels=1,depth=8 ! audioconvert ! speexenc ! queue ! tcpserversink host=192.168.2.129 port=5001


On a laptop of the same architecture as the target, download the required packages:
Receiving XO:
mkdir gs
gst-launch-0.10 udpsrc port=5000 ! queue ! smokedec ! queue ! autovideosink tcpclientsrc host=192.168.2.129 port=5001 ! queue ! speexdec ! queue ! alsasink sync=false
cd gs
sudo yum install -y yum-utils
sudo yum localinstall -y --nogpgcheck \
http://download1.rpmfusion.org/free/fedora/rpmfusion-free-release-$(rpm -E %fedora).noarch.rpm \
http://download1.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-$(rpm -E %fedora).noarch.rpm
yumdownloader --resolve gstreamer{1,}-plugins-ugly gstreamer-plugins-bad gstreamer-ffmpeg


Copy the packages. On the target laptops, install the packages using RPM.
===Section TODO===
*Fill in blank sections.
*Include snapshot recipes.
*Discuss python interfacing.


Last tested 2014-01-21 on [[XO-1.75]] with [[13.2.0]].
== See also ==
* [[Programming the camera]]
* [[Video]]
* [[Sound]]
* [[Display]]
* [[Compiling_GStreamer_On_The_XO]] for cutting-edge GStreamer development
* http://gstreamer.freedesktop.org/
* http://gstreamer.freedesktop.org/documentation/
* [http://www.jonobacon.org/?p=750 Getting started with GStreamer with Python]
* [[Software components]] ; [[Sugar Architecture/API#Third_Party_Packages]]


== Other resources ==
== Developers ==
*http://www.mediamods.com/public-svn/camera-activity/Record.activity/glive.py
*https://coderanger.net/svn/projects/olpc/games/olpcgames/camera.py ''(said to not really be working (trouble knowing when gs is done) [[User:MitchellNCharity|MitchellNCharity]] 18:24, 12 June 2007 (EDT))''
*''Where is the camera code in sugar?'' [[User:MitchellNCharity|MitchellNCharity]] 18:16, 12 June 2007 (EDT)
*[http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer-plugins/html/index.html gstreamer-plugins]
*[http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer-plugins/html/gstreamer-plugins-plugin-coreelements.html gstreamer-plugins-plugin-coreelements]
*[http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-base-plugins/html/index.html gst-plugins-base-plugins]
*[http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-good-plugins/html/index.html gst-plugins-good-plugins]
*[http://www.google.com/search?hl=en&q=site%3Agstreamer.freedesktop.org&btnG=Search google search "site:gstreamer.freedesktop.org"]
*http://xiph.org/ Ogg/Theora/Vorbis


See [[GStreamer/Developers]].
[[Category:Camera]]
[[Category:Audio]]
[[Category:Developers]]
[[Category:Software development]]

Latest revision as of 02:13, 15 February 2021

GStreamer is our multimedia framework library.

Introduction

GStreamer is a library that supports multimedia, ranging from playback to complex mixing and processing.

  • the Totem plugin uses GStreamer to play multimedia URLs in the Browse activity,
  • the Record activity uses GStreamer to deal with streams from the camera and microphone, and;
  • the Jukebox activity uses GStreamer to play video or audio objects in the journal.

Adding codecs

This section describes third-party software that does not meet the OLPC standards for software freedom. The use of such software is unsupported; OLPC does not endorse or encourage its use.
rpmfusion repositories no longer exist for Fedora 18, and irregardless require later cryptographic standards than are supported by the OLPC XO.

Some codecs can be added during build. Adding during build is the method supported by OLPC, because it works with our operating system olpc-update feature.

Some codecs are not included in OLPC OS, because of restrictions on redistribution. But they can be added after install. Such changes are lost on olpc-update.

To add after install, on the XO laptop, start Terminal and paste this compound command:

sudo yum localinstall -y \
http://download1.rpmfusion.org/free/fedora/rpmfusion-free-release-$(rpm -E %fedora).noarch.rpm \
http://download1.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-$(rpm -E %fedora).noarch.rpm && \
sudo yum install -y \
  gstreamer1-{plugins-{ugly,bad-free{-extras,world},base-tools,good-extras,bad-free},libav} \
  gstreamer-{plugins-{ugly,bad},ffmpeg} && echo Complete!

When the prompt reappears, with "Complete!" above the prompt, it is ready to use. You may need to stop and restart any activity using GStreamer, once.

These instructions have been tested for releases 10.1.2, 12.1.0, 13.2.0, 13.2.1, 13.2.2, 13.2.6 and 14.1.0.

For more information, see:

Test cases for Browse activity:

Adding codecs using xo-custom or other scripting

On a laptop of the same architecture as the target, download the required packages:

mkdir gs
cd gs
sudo yum install -y yum-utils
sudo yum localinstall -y --nogpgcheck \
    http://download1.rpmfusion.org/free/fedora/rpmfusion-free-release-$(rpm -E %fedora).noarch.rpm \
    http://download1.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-$(rpm -E %fedora).noarch.rpm
yumdownloader --resolve gstreamer{1,}-plugins-ugly gstreamer-plugins-bad gstreamer-ffmpeg

Copy the packages. On the target laptops, install the packages using RPM.

Last tested 2014-01-21 on XO-1.75 with 13.2.0.

Developers

See GStreamer/Developers.