Wiki Slice

Goal

The Wiki Slice project aims to create snapshots of wiki sites for use as offline reading material.

Code

The code will be hosted on dev.laptop soon. For the moment, you can pull from Julius Lucks' public git repo for the project. Do

git clone http://slyjbl.hopto.org/~olpc/git/wikipedia_olpc_scripts.git

or if you already have a git repo

git fetch http://slyjbl.hopto.org/~olpc/git/wikipedia_olpc_scripts.git master:my_new_branch

Files

See 'Implementation' below for an explanation of these files

old_perl_scripts/ # the old perl implementation
op.pl #mncharity's new perl implementation
wiki_slice.py # stub python implementation
wiki_page.py
wiki_slice_config.y
sandbox/ # some python play-around code

Implementation

Right now there is an existing perl implementation that I don't know too much about. You can have a look inside the old_perl_scripts directory. There is a new perl proof-of-concept/implementation in perl called op.pl by mncharity, and a new outline implementation in python. Both of these are outlined below.

Workflow

Taking wikipedia as an example of a wiki that we would want to slice, the general workflow is as follows:

1. Get a list of pages to be included in the slice. This can be done via a configuration file, or can be 'harvested' from a web-page.
2. For each of the pages:
   1. grab the wikitext of the page using the action=raw cgi parameter passed to the wiki's index.php script
   2. remove unwanted templates. These are specified in a configuration file.
   3. get the wiki text from expanding the templates (wia the Special:ExpandTemplates page)
   4. for wiki-links that refer to pages also included in the slice, turn these into local anchor tag links
   5. for wiki-links that refer to pages not included in the slice - de-link them (possibly highlight them)
   6. generate html. There are several methods to do this:
      • create a sandbox entry and generate the page
      • create a page preview (does not always work)
      • use the preview generated by Special:ExpandTemplates
   7. save the html
   8. find <img> tags and:
      1. download the image
      2. make the url refer to the local image
   9. wrap the html inside olpc's html wrapper
   10. save the html