Talk:Emulating the XO

From OLPC
Revision as of 00:59, 4 December 2007 by MitchellNCharity (talk | contribs) (→‎Status of Emulation Summary: +comments)
Jump to: navigation, search

VMWare TODO

  • clean up the VMWare page to describe how to quickly install VMWare (and nothing extraneous)
  • add a section describing how to download and install an activity bundle
  • summarize this in a two-paragraph overview describing how long the whole process should take, and what scripts to download and run, to emulate and test a single Activity.
    • test how fast people can make the script by tweaking it


  • Explicitly note the compatibility and accuracy of emulation of : audio in/out, video in, screen resolution, mouse in (how to emulate the touchpad? possible at any level?), network
  • reference explicit build #s for VMWare.


Article TODO

The top of the article should:

  • be more specific about ease of installation. Eg, if kqemu is easily installed on Ubuntu (unlike fc6), then it might be easier than the LiveCd.
  • mention hardware support. Once it becomes clear whether the LiveCd actually supports camera, it should be mentioned.

Does TamTam sound work under qemu in the release we recommend? (It hasn't lately in latest-release). Should test, and mention if it doesn't. MitchellNCharity 13:14, 5 June 2007 (EDT)


Xen Player

Is anybody doing such an emulation? Nitpicker 21:59, 5 October 2006 (EDT)

Moved from abandoned Talk:OS_images_for_emulation. MitchellNCharity 03:04, 24 May 2007 (EDT)

Temporary summary of current state

currently: quickstart is a "choice of livecd/sound/no-library/read-only or qemu/non-devel.img/no-sound/library/read-write". and devel is "mix of qemu/livecd.iso/sound/ro and qemu/-devel.img/no-sound/rw". MitchellNCharity 13:06, 24 May 2007 (EDT)

Modifying the LiveCD .iso - adding the olpc Library

The LiveCd is intended for development, and thus doesn't include the Library sample content. One wants the library when demo'ing sugar. Here is how to add it.

The next release (June?) of the LiveCd will have google as the default browser page (rather than an error message). Making adding the Library less important. A nice, simple fix. MitchellNCharity 13:03, 5 June 2007 (EDT)

The LiveCD is currently created manually every month or two.

Getting /home/olpc/Libary

Alternatives:

  • Snarf a copy from an xo disk image (the ext3.img variety).
mkdir tmp_img
mount -o loop,offset=33174 olpc-redhat-stream-development-ext3.img tmp_img

The Libary is in tmp_img/home/olpc/Library. The offset=NNNNN may need to be adjusted. See Emulating the XO/Help and tips.

  • As of 2006-06-01, there is a snapshot:
wget http://dev.laptop.org/pub/content/lib-4-6.tar.gz

Get squashfs

Get squashfs. On fc6, 

yum install squashfs-tools

Add the library

wget http://olpc.download.redhat.com/olpc/streams/sdk/latest/livecd/olpc-redhat-stream-sdk-livecd.iso

As root 

mkdir tmp_iso
mount -o loop -t iso9660 olpc-redhat-stream-sdk-livecd.iso tmp_iso
/usr/sbin/unsquashfs -dest tmp_squashfs tmp_iso/squashfs.img  # about 4GB

file tmp_squashfs/os.img
## tmp_squashfs/os.img: x86 boot sector; partition 1: ID=0x83, starthead 1, startsector 62, 8388290 sectors, extended partition table (last)\011
## 62 * 512 = 31744
mkdir tmp_os
mount -o loop,offset=31744 -t ext3 tmp_squashfs/os.img tmp_os

mkdir tmp_os/home/olpc/Library
(cd dev.laptop.org/pub/content/library/; tar cf - .)|(cd tmp_os/home/olpc/Library; tar xf -)
chown -R 500:500 tmp_os/home/olpc/Library

umount tmp_os
/sbin/mksquashfs tmp_squashfs squashfs.img.new

mkdir tmp_iso2
(cd tmp_iso; tar cf - .)|(cd tmp_iso2; tar xpf -)
umount tmp_iso
mkisofs -v -r -T -J -V 'Modified LiveCd mumble' -l -R -c boot.catalog -b squashfs.img -o new.iso tmp_iso2  ##XXX - wrong - doesn't work

Notes:

Future possibilities

It would be nice to have:

  • LiveCd with Library. To be an "intro" livecd for non-developers to try sugar.
    • Perhaps create a mutant LiveCd. Take LiveCd, unsquash its squashfs, add library, resquash, and mkisofs. Downsides: LiveCd isn't updated very frequently (but since an update is coming rsn, perhaps we don't care just now). See Talk:Emulating_the_XO.
  • a small(?) qemu image with full fedora host detection/configuration (like Big FC6 Build). To be an "intro" qemu image for non-developers. A "virtual XO".
    • Perhaps derive from LiveCd? The bottleneck is getting a clean sugar-jhbuild. LiveCd is the only production source of this human intensive process, other than the latest-stable-builds, which are xo-specific. It's not clear that one could get from any xo disk image to a hardware-independent build. Perhaps one might find out what versions of things went into a latest-stable-build? Perhaps sugar-jhbuild works not-infrequently for some particular environment, eg fc7 x86? Perhaps take someone's fc6+sugar image (Big FC6 Build lacks etoys), attempt updating it, and strip it down?
    • Re derive from LiveCd, one might be able to unsquash the fs, customize for this role, make a qemu image with it, and add boot stuff?
    • The LiveCd image, once unsquashed, is not small (4GB). Is this Gnome and such? Does the xo have some non-readonly compression which we can shrink that with? Is it worth a performance hit? MitchellNCharity 23:38, 1 June 2007 (EDT)
    • If sound starts fully working on xo disk images, given that qemu doesn't support usb cameras, is this option needed? Isn't the non-devel image equivalent? MitchellNCharity 23:38, 1 June 2007 (EDT)

Emulation overhaul

moved from User_talk:Mcfletch

Hi. It's great to have emulation being updated. Here are a couple of observations which drove changes vis the n-1 version of the emulation pages:

  • There is a lot of overlap between different emulation platforms.
    • In "front", image selection and prep. Perhaps someday the build images will consistently just work in emulation. So the documentation for grabbing an image will simply be a link. But it's never been true so far, and I don't see any sign of that changing. So the selection of which image to use seems likely to remain something which requires a "weather report" of the status of recent builds. We obviously don't want to replicate that info. Two approaches might be to send people through a single "grab an image" page, or to transluced such a page by platform-specific recipes,
    • In "back", setting up a development environment and dealing with bugs. Getting from emulation is "running", to emulation is running correctly and usably. Display size, sound, camera, shell access, file access. The tuples of <emulation platform, operating system> share content on both axes. The current big Emulating the XO/Help_and_tips collects information which was previously scattered among emulation platform specific pages. It's inelegant. I'm unclear on what the right solution is here.
  • Emulation platforms get used for different things. XO devel images; 32-bit fc/ubuntu to run sugar-jhbuild; xs school server images. Live CDs. So separating out platform installation and general usage from their assorted uses seems worthwhile. But installation especially can be os dependent. So we need to deal with <{qemu,vmware,vbox} on {windows,mac,ubuntu,fc,other-linux}> and some extras like Parallels and Virtual PC.
  • I kept the OS images for emulation name as a redirect into Emulating the XO, and the access point used elsewhere on the wiki. The core team folk were continuing to use it, and it provides them an easy way for them to clip off Emulating the XO and pursue a different approach. Even though the title is really unfortunate ("which OS?!?").
  • One possible role for the article is to provide a "if you want to do X, and you are running OS Y, here's what we recommend". It seems unlikely that any one solution will become the one true way to emulate the XO. I've never really bought the "develop on Live CD" story, but even so, sugar-jhbuild is too big and fragile to be a general solution, different OSes have different prefered emulation platforms, qemu likely remains in as something non-proprietary, we will hopefully have bootable images (LiveCD and or USB stick), etc. Emulating the XO was envisioned as the place which helps visitors figure out their options, and organizes resources they collectively need. A counter argument is it would be really nice to have something, anything, which "just works", and can serve as the focus of effort, even if it is suboptimal for some users/uses. I'm not sure if trying for a sugar-jhbuild/some-emulator duopoly actually addresses a bottleneck. For the last few months at least, the bottleneck has mostly been emulation isn't a priority for the project (doesn't appear on roadmap; isn't kept working in the run-up to releases; isn't tested/tinderboxed; even simple tickets tend to be unaddressed for weeks). But perhaps having a single official and working emulation story might help change that.

MitchellNCharity 13:12, 28 August 2007 (EDT)

Old 'Comparison of alternatives' section

I copied out the recommendations and updated them. Here is the old version, now removed. MitchellNCharity 23:26, 19 September 2007 (EDT)

Comparison of alternatives

An OLPC laptop is custom hardware, running a stripped-down Red Hat linux, running Sugar. But what if you don't have a real olpc laptop? There are a several options, which can each be used in a couple of ways.

recommendations

platform purpose recommendation
Windows development emulated xo disk image; or emulated Ubuntu with sugar-jhbuild; (or develoepr CD or LiveCD)?. (need your reports)
Mac development emulated xo disk image; or emulated Ubuntu with sugar-jhbuild; (or developer CD or LiveCD)?. (need your reports)
Mac with Parallels development install Ubuntu 32-bit, and Sugar on Ubuntu Linux. See /Mac.
Ubuntu 32-bit development Sugar with sugar-jhbuild and Sugar on Ubuntu Linux
32-bit linux development Sugar with sugar-jhbuild
64-bit linux development xo disk images under qemu; or LiveCd once it gets updated (currently it's April)
I don't really believe the development option of Windows/Mac developer/live CD. I've not yet heard of it actually being used. They are months out of date (April). But Mcfletch said (in May) it's an option, so I added it. MitchellNCharity 09:19, 23 June 2007 (EDT)

Mcfletch It's probably no longer an option, Red Hat seems to have stopped releasing them. Even the developer's CDs are getting too old to be usable.

background

option updated sound? camera? library? etoys? R/W? development software? net? comments
Installing Sugar continuous yes yes? yes? yes yes your own yes On some platforms, it takes time and space (hours and ~3GB), but "just works". On others, it is quite difficult to install.
Installing Sugar, on a virtual 32-bit Ubuntu, running in an emulator (qemu, parallels, or vmware)
LiveCd April yes(boot) yes?(qemu) yes?(boot) no(qemu) no yes no toolchain, Gnome yes? Alternative Quick Start. Can both be booted from, and used in qemu. Problems: A.
XO disk images:
LATEST-STABLE-BUILD http://olpc.download.redhat.com/olpc/streams/development/LATEST-STABLE-BUILD/
...development-ext3.img ~monthly yes/B no yes yes yes no after config Quick Start for getting a look at Sugar.
...development-devel_ext3.img ~monthly yes/B no no yes yes a little after config
LATEST http://olpc.download.redhat.com/olpc/streams/development/LATEST/
...development-ext3.img ~daily yes/B no yes yes yes no after config
...development-devel_ext3.img ~daily yes/B no no yes yes a little after config Good for new developers (requires some command-line comfort).
OTHER
FC6+Sugar April yes? ? ? no yes Full Fedora Softare Dev. yes 6GB (April version: no etoys)

Key:

library: A library of sample content is included (english version). Ie, pretty text to web browse without having to get network.
R/W (writable): With an .img, you can save things between sessions. With a .iso, your environment is the same each time you start.
development software: are development tools included?
Net (network): does the network "just work" or "manual" steps are required?

Problems:

A: LiveCd (April) on fc6 x86_64, kernel panics under kqemu. A squashfs problem.
B: Tam Tam is silent (though EToys makes noise). Cause unknown.

Notes:

  • re "does sugar-jhbuild support camera/mic?": "yes, if the camera uses v4l2 and 640x480, I'd suspect so." Can someone confirm this? (#olpc, now) MitchellNCharity 00:16, 2 June 2007 (EDT)

Doables:

  • Remove LiveCd's "?" above.
  • Find out what's going on with Tam Tam sound. It would be nice for the xo disk images to all be sound-yes.
  • Add a microphone column? Reconsolidate sound/camera/mic into a H/W column?

QEMU or VirtualBox or VMWare?

Some observations:

  • Qemu is apparently the only officially "supported" emulator at present.
  • And as yet has a much bigger track record supporting development.
  • VirtualBox/VMWare supporting TamTam sound is a definite (if hopefully transient :) advantage.
    Do these actually support TamTam sound? Reports are ambiguous. MitchellNCharity 11:28, 9 October 2007 (EDT)
  • VirtualBox (and VMWare?) may be able to support cameras, which qemu is unlikely to any time soon. But I don't know of this being tested.
    VMWare support for USB 2.0 is said to be iffy. Does that mean no camera? What about VirtualBox? MitchellNCharity 11:28, 9 October 2007 (EDT)
  • It may be easier for people to install VirtualBox/VMWare than Qemu on some platforms. Eg Windows. And perhaps Fedora. But not Ubuntu.
  • Other possible VirtualBox/VMWare advantages:
    • Easier host/xo data interchange, yes?
    • 1200x900 support may be possible without Xephry. But I don't know of this being tested.
      VMWare apparently only supports 1280x1024. VirtualBox? MitchellNCharity 11:28, 9 October 2007 (EDT)
  • Display side: QEMU's default is 1024x768. VMWare supports 1280x1024. VirtualBox was once said to support arbitrary resolution? MitchellNCharity 14:03, 9 October 2007 (EDT)
  • Running a Xephyr-like remote X is said to be very difficult on Mac and Windows. Details? MitchellNCharity 14:03, 9 October 2007 (EDT)
  • Given the open source emphasis of the project, VMWare should perhaps need some clear advantage over VirtualBox to be recommended over it.
  • VirtualBox and VMWare apparently support USB sticks. But VMWare support for USB 2.0 is apparently iffy? What about VirtualBox? MitchellNCharity 11:28, 9 October 2007 (EDT)

Emulation Speed

Does anyone have a quick-and-dirty solution for limiting cpu cycles available to the emulation, outside or inside the guest OS? Running dmesg | grep -i bogo in build 613 on VirtualBox gives me something around 4800 BogoMIPs; real XO cpus hit around 740 BogoMIPs or so, I believe.

Suggest you explain at http://forums.virtualbox.org/ that you want to throttle your olpc guest os for purposes of realism. It could be in the undocumented API. 207.229.150.96 19:45, 11 October 2007 (EDT)
On linux, you might create a cpu eating process, and nice the emulator below it? Eg, http://kerneltrap.org/node/11773 . While ok for steady state, I believe the kernel optimizes for ui response, so you still might see the emulator "bursting" a faster-than-realistic response. MitchellNCharity 23:42, 11 October 2007 (EDT)

VirtualBox and 1200x900

Moved from User talk:MitchellNCharity MitchellNCharity 19:22, 11 October 2007 (EDT)

VirtualBox on Windows XP does not change to 1200x900 when all olpc's xconf files specify that screen resolution. The screen settles on 1024x768, getting this from vesafb according to the boot scroll. I have no idea how to alter that. Altering the conf files isn't impossible. Keying Alt-= gets the developer console, selecting the Terminal tab gets the command line. The commands cd /, cd etc, cd X11, su, ls (shows 4 confs: dcon-xorg.conf geode-xorg.conf qemu-xorg.conf xorg.conf), vi qemu-xorg.conf, i, Modes "1200x900" (in the Display SubSection of Screen), Esc, wq! changes the file; but the OS still boots 1024x768. It would have been easier for a vi novice to do this if help files for vi had been included in olpc developer build 557. Maybe you know someone who knows what files to change. 207.229.150.96 14:30, 11 October 2007 (EDT)

Re vi, that's a nice idea. Re who knows what files to change, I'm afraid I haven't looked at it. Anyone? MitchellNCharity 19:22, 11 October 2007 (EDT)
Trying res depends on what virtualbox tells olpc, I found Advanced Topic, Custom Vesa Resolutions in the virtualbox help file. Requesting VBoxManage setextradata "olpc" "CustomVideoMode1" "1200x900x16" got the virtualbox CLI's attention, but seemed to change nothing - looks like I ought to follow my own advice to User:Sj above and post in a virtualbox forum. 207.229.150.96 20:36, 11 October 2007 (EDT)
Strike three - time for another batter. For a 1200x900 screen VBox can install its video driver, which exists on the VBoxGuestAdditions.iso located in the host's "innoteck VirtualBox" directory, into the fedora core 6 based olpc guest. Normally, this would be done via the VM's virtualized cdrom drive. Either olpc needs to be adjusted to recognize a cdrom, or olpc must network to "VBoxLinuxAdditions.run" on a cd or in a dir in order to try the install. The additions seem to be perfectly willing to install on the host, rather than the guest. It would be sensible to image one's computer's system drive in order to be able to restore the drive image in case one's computer got wiped out due to making this mistake - trying to install the additions on the host instead of the guest. A cautious approach would be to run " VBoxLinuxAdditions.run help " rather than " VBoxLinuxAdditions.run all ". If you can install the driver(s) on olpc, then you are likely to get the 1200x900 screen. It's even possible that res can be changed on the fly, from the host, with " VBoxManage controlvm setvideomodehint " which doesn't work for all guest systems. So. Root's fc6 command looks like "sh /pathofimage/VBoxLinuxAdditions.run" plus help or all. VirtualBox can present the VBoxGuestAdditions.iso to olpc as a cd sitting in a virtual cdrom drive. Bash in olpc won't mount it. I don't know how to network olpc to VBoxLinuxAdditions.run. But, I saw somewhere here that changing the olpc kernel, which I guess the Additions do, crums up any development build. Oh, well - I'm out. - 207.229.150.96 16:31, 12 October 2007 (EDT)
Thanks for pitching in. :P MitchellNCharity 15:08, 14 October 2007 (EDT)
I got 1200x900 resolution on Vista using the CustomVideoMode1 trick above. In order for this to work you also need to set vga=0x360 on boot (or modify /boot/grub/grub.conf to use vga=0x360). I'm not sure if it is required to add the Modes "1200x900" to X11, but I did anyway. In summary:
  1. On host: VBoxManage setextradata "olpc" "CustomVideoMode1" "1200x900x16"
  2. On olpc: modify /boot/grub/grub.conf to use vga=0x360
  3. On olpc: modify /etc/X11/xorg.conf to include Modes "1200x900" in the Display subsection of Screen
The next reboot should come up in 1200x900. At least it did for me... 69.140.167.77 02:01, 18 November 2007 (EST)
I got 1200x900 without modifying xorg.conf. After applying CustomVideoMode1 on the host (Windows XP), I changed just the first listing in olpc's grub.conf from vga=0x317 to vga=0x360. Here's how to make the change: Key Alt-= for the developer console, select the Terminal tab to get the command line. The commands cd /, cd boot, cd grub, su, vi grub.conf, i, (change 317 to 360), Esc, :, wq! (does it).

Before: 

default=0
timeout=10
splashimage=(hd0,0)/boot/grub/splash.xpm.gz
hiddenmenu
title OLPC for qemu target (Full size)
        root (hd0,0)
        kernel /boot/vmlinuz ro vga=0x317 video=safb:ypan selinux=0 console=tty50.115200 console=tty0 emu
        initrd /boot/olpcrd.img

After: 

default=0
timeout=10
splashimage=(hd0,0)/boot/grub/splash.xpm.gz
hiddenmenu
title OLPC for qemu target (Full size)
        root (hd0,0)
        kernel /boot/vmlinuz ro vga=0x360 video=safb:ypan selinux=0 console=tty50.115200 console=tty0 emu
        initrd /boot/olpcrd.img
So that's all there is to it. Thank you, 69.140.167.77, for finding that. - 12.202.127.204 08:22, 23 November 2007 (EST)

Removed "Simulating the XO's 1200x900 display" section

This text was removed from the article:

The emulators listed have varying success at emulating the high-res display of the XOs. VMware comes close, but does not get the resolution quite right; particularly when emulating what the browser looks like (the browser does its own image scaling). Be sure to calibrate any image work you are doing with someone who has an actual XO before trusting the emulator to guide your further work.

To get 1200x900, use a remote X display. See /Help and tips#Run a Xephyr remote X display and How to set up for development on linux emulation#getting 1200x900. MitchellNCharity 11:06, 9 October 2007 (EDT)

How can "VMware comes close"? Either it's doing 1200x900, in which case it's better than close. Or it's doing 1280x1024 or 1024x780, which every one of the emulators can do, and so is not VMware specific. Though it would be nice the grub options were updated to make this easier (#4952). MitchellNCharity 20:45, 3 December 2007 (EST)

Status of Emulation Summary

Working out here what the current status of development and testing environments is:

  • Over the last few months, churn on sugar-jhbuild has dropped substantially. It is now possible to build sugar-jhbuild natively on at least Fedora 7, Ubuntu Gutsy and Gentoo amd64 with only a minimum of work and a few hours. This doesn't mean it's not a fragile approach, but the horrors of the past seem to have faded and most C/C++/sysadmin-friendly Linux developers seem to be able to complete the build readily.
  • Ubuntu Gutsy and Fedora 7 *should* be able to use packages to install Sugar natively. Testing is needed to ensure that it's usable for development once installed, but this is where we want to go with all platforms, with only core devs using sugar-jhbuild.
  • Three different LiveCD projects are underway. One is a Developer's Desktop LiveCD, the other two are more of a laptop emulation (that is, they run essentially the same code as an XO).
  • Qemu + KQemu have been shown to deliver reasonable performance for emulating the XO, and other than csound not working, they seem basically functional. We should probably move to recommending them as the standard way to work with the images, given that they are already integrated into the build process and ready to go.
  • VirtualBox and VMWare both seem to work perfectly well, and they are far more popular than Qemu, so more familiar to many possible devs, but they suffer from my not having enough time (or bandwidth) to do the conversions, testing and the like. Their advantages (mostly ease of use and hardware support) seem to have largely evaporated over the past few months. Windows and Mac platforms should be considered before we abandon them, as should the possibility of just enabling the conversions in the build system so that they are automatically and immediately available.
  • Image downloads have been scattered due to the exigencies of getting the Mass Production set out the door. We need to collect links to the images somewhere so people can find e.g. a Ship.2 image for emulation/testing.
    Suggest Emulating_the_XO#Build_recommendations. MitchellNCharity 23:59, 3 December 2007 (EST)
  • We might consider having a simple launcher script on Linux that can download a given stream-and-build, unpack it, and run qemu on it with appropriate options.
    There is a prototype on in http://dev.laptop.org/~mncharity/xo-qemu-wrapper/ . MitchellNCharity 23:59, 3 December 2007 (EST)
  • We likely should break out the descriptions of the various pages so that we see e.g. "Emulating an XO" on one page, with another page having "Creating a Development Environment", with Emulation linked in there as a way to test on a development environment and/or as an alternate approach.
    I'm unclear on the proposal, but note Getting_started_programming. MitchellNCharity 23:59, 3 December 2007 (EST)
  • We should consolidate the various pages describing Qemu installation and configuration on Linux such that there is a single page describing "How to install Qemu on Linux", another on "How to Configure Qemu on Linux".
    How does Emulating_the_XO/Quick_Start/Linux differ from the first? For the second, the right thing is perhaps to make available a one-liner shell script so people don't have to care. MitchellNCharity 23:59, 3 December 2007 (EST)
  • The Developer's Desktop-style LiveCDs should include Qemu as a testing environment as a link on the desktop. That is, the hypothetical script to download and install the emulator image should be a link on the desktop/menu/whatever so that devs can work on the sugar-jhbuild or package-based environment on the desktop and test on the emulated image when ready for the "next level". That should always work on a COW overlay so that the developers can roll back to a pristine state as needed.
    Neat idea. One thing the xo-qemu-wrapper script does not do at the moment is provide COW's. That would be a nice enhancement. MitchellNCharity 23:59, 3 December 2007 (EST)

Thoughts? --Mcfletch 23:23, 3 December 2007 (EST)

Retrieved from "http://wiki.laptop.org/mediawiki/index.php?title=Talk:Emulating_the_XO&oldid=82238"