Literacy Project/Data Processing Notes

Some notes on processing .zip files received from the field:

Prerequisites

Prerequisites: an account on hydro, membership in the literacy group, write access to /home/ethiopia:

$ ssh hydro.laptop.org
cscott@hydro:/home/ethiopia$ groups
cscott literacy
cscott@hydro:/home/ethiopia$ cd /home/ethiopia/
cscott@hydro:/home/ethiopia$ touch do-i-have-write-access
cscott@hydro:/home/ethiopia$ rm do-i-have-write-access 

Archiving, unpacking, and checking the data

wolonchete/ and wonchi/ are rsynced to worldliteracy.media.mit.edu

cscott@hydro:/home/ethiopia$ ls wolonchete/ wonchi/
wolonchete/:
2012-03-30  results.txt            wolonchete_2012-05-27  wolonchete_2012-06-24
2012-04-06  wolonchete_2012-05-01  wolonchete_2012-06-03
2012-04-13  wolonchete_2012-05-08  wolonchete_2012-06-10
2012-04-20  wolonchete_2012-05-17  wolonchete_2012-06-17

wonchi/:
2012-02-14  2012-03-30  results.txt        wonchi_2012-05-23  wonchi_2012-06-21
2012-02-20  2012-04-06  wonchi_2012-04-26  wonchi_2012-05-31
2012-02-28  2012-04-13  wonchi_2012-05-03  wonchi_2012-06-07
2012-03-08  2012-04-20  wonchi_2012-05-10  wonchi_2012-06-14
cscott@hydro:/home/ethiopia$ 

Note that some of these directories have prefixed. The unprefixed versions are guesstimated dates -- we originally recorded the date the USB key arrived at OLPC. We fixed that and started recording the collection dates. The prefixed directories have the actual collection dates. The unprefixed directories are named with our best guess at the collection date.

The archive/ directory contains zip files copied from the data collection keys. There should be two files per usb key (a wolonchete .zip and a wonchi .zip). BE SURE to verify that contents of .zip file match the name, so you don't inadvertently overwrite something.

cscott@hydro:/home/ethiopia$ ls archive
wolonchete_2012-06-03.zip       wonchi_2012-05-31.zip
wolonchete_2012-06-10.zip       wonchi_2012-06-07.zip
wolonchete_2012-06-17.orig.zip  wonchi_2012-06-14.zip
wolonchete_2012-06-17.zip       wonchi_2012-06-21.zip
wolonchete_2012-06-24.zip

These are the two most recent files received from Ethiopia:

cscott@hydro:/home/ethiopia$ ls ~rsmith/w*.zip
/home/rsmith/wolonchete_2012-07-01.zip  /home/rsmith/wonchi_2012-06-28.zip
cscott@hydro:/home/ethiopia$ unzip -v /home/rsmith/wolonchete_2012-07-01.zip | head
Archive:  /home/rsmith/wolonchete_2012-07-01.zip
 Length   Method    Size  Cmpr    Date    Time   CRC-32   Name
--------  ------  ------- ---- ---------- ----- --------  ----
       0  Stored        0   0% 2012-07-04 09:37 00000000  wolonchete_2012-07-01/
       0  Stored        0   0% 2012-07-04 09:27 00000000  wolonchete_2012-07-01/01/
       0  Stored        0   0% 1979-12-31 16:00 00000000  wolonchete_2012-07-01/01/edu.mit.media.funf.bgcollector/
    8200  Defl:N     2543  69% 2012-06-24 09:06 5209af9a  wolonchete_2012-07-01/01/edu.mit.media.funf.bgcollector/00000180_0388920440601417_7c429084-7a15-48f1-a458-957681c105fd_1340465667_mainPipeline.db
    6152  Defl:N     1415  77% 2012-06-24 09:06 5d96a57f  wolonchete_2012-07-01/01/edu.mit.media.funf.bgcollector/00000181_0388920440601417_7c429084-7a15-48f1-a458-957681c105fd_1340552241_mainPipeline.db
    6152  Defl:N      811  87% 2012-06-24 09:06 541732f9  wolonchete_2012-07-01/01/edu.mit.media.funf.bgcollector/00000182_0388920440601417_7c429084-7a15-48f1-a458-957681c105fd_1340552241_mainPipeline.db
  114696  Defl:N    16867  85% 2012-06-24 09:37 a86306dd  wolonchete_2012-07-01/01/edu.mit.media.funf.bgcollector/00000183_0388920440601417_7c429084-7a15-48f1-a458-957681c105fd_1340554005_mainPipeline.db
cscott@hydro:/home/ethiopia$ 

Note that the directory name does match the .zip file name. (We should check the wonchi file, too.) Good. This has recently (late July) become a slight problem, as the files for wolonchete are now being spelled wolenchite. We've adopted wolonchete as the normalized spelling, so rename anything that needs it (after unzipping).

Now let's unzip the files:

cscott@hydro:/home/ethiopia$ cd
cscott@hydro:~$ mkdir temp ; cd temp
cscott@hydro:~/temp$ unzip /home/rsmith/wonchi_2012-06-28.zip 
cscott@hydro:~/temp$ unzip /home/rsmith/wolonchete_2012-07-01.zip 

Check for the following:

• Spelling of wonchi/wolonchete
• Tablet directories should be two digits: "01" not "1"

We're going to check for duplicates:

cscott@hydro:~/temp$ cp ~rsmith/bin/rdfind ~/bin
cscott@hydro:~/temp$ rdfind wolonchete_2012-07-01/ wonchi_2012-06-28/
Now scanning "wolonchete_2012-07-01", found 13727 files.
Now scanning "wonchi_2012-06-28", found 7630 files.
Now have 21357 files in total.
Removed 0 files due to nonunique device and inode.
Now removing files with zero size from list...removed 36 files
Total size is 3198679889 bytes or 3 Gib
Now sorting on size:removed 6240 files due to unique sizes from list.15081 files left.
Now eliminating candidates based on first bytes:removed 3424 files from list.11657 files left.
Now eliminating candidates based on last bytes:removed 9294 files from list.2363 files left.
Now eliminating candidates based on md5 checksum:removed 2363 files from list.0 files left.
It seems like you have 0 files that are not unique
Totally, 0 b can be reduced.
Now making results file results.txt
cscott@hydro:~/temp$ 

Ok, this gives us confidence that there aren't gross errors in the data, like having the wonchi and wolonchete zips the same, etc. We should repeat the rdfind across the entire dataset (later) as well, to ensure that we don't have stale data here.

OK, this looks good. Let's move them into the ethiopia directory:

cscott@hydro:~/temp$ mv /home/rsmith/wonchi_2012-06-28.zip /home/rsmith/wolonchete_2012-07-01.zip /home/ethiopia/archive/
cscott@hydro:~/temp$ mv wolonchete_2012-07-01/ /home/ethiopia/wolonchete/
cscott@hydro:~/temp$ mv wonchi_2012-06-28/ /home/ethiopia/wonchi/

Normalize the permissions:

cscott@hydro:~/temp$ cd /home/ethiopia/archive/
cscott@hydro:/home/ethiopia/archive$ chmod 644 *
cscott@hydro:/home/ethiopia/archive$ cd ..
cscott@hydro:/home/ethiopia$ chmod -R a+X,o-w wolonchete/wolonchete_2012-07-01/ wonchi/wonchi_2012-06-28/
cscott@hydro:/home/ethiopia$ chmod -R g+sw wolonchete/wolonchete_2012-07-01/ wonchi/wonchi_2012-06-28/
cscott@hydro:/home/ethiopia$ chown -R :literacy wolonchete/wolonchete_2012-07-01/ wonchi/wonchi_2012-06-28/

Now we're going to run rdfind across all of the data, in order to catch more errors. As of early September, 2012, this step took about 45 minutes to inspect over 4 million files.

cscott@hydro:/home/ethiopia$ rdfind wolonchete/ wonchi/

The output will be in a results.txt file.

Pushing the data into git

Make sure you have a copy of the git repo somewhere handy:

cscott@hydro:/home/ethiopia$ cd
cscott@hydro:~$ git clone ssh://dev.laptop.org/home/literacy/git/data ethiopia-data
 or
cscott@hydro:~$ git clone ~rsmith/data/.git ethiopia-data
cscott@hydro:~$ cd ethiopia-data/
cscott@hydro:~/ethiopia-data$ git remote add dev ssh://dev.laptop.org/home/literacy/git/data

Make sure your repo is up-to-date. The scripts may have changed since the last import you did:

cscott@hydro:~$ cd ~/ethiopia-data/
cscott@hydro:~/ethiopia-data$ git pull origin

Add the ethiopia-data/scripts directory to your path (or double-check that it's there and before ~/bin, if you've done this before):

cscott@hydro:~/ethiopia-data$ export PATH=$HOME/ethiopia-data/scripts:$PATH

You might want to be inside screen or tmux for the following, since it takes a long time:

cscott@hydro:~/ethiopia-data$ cd /home/ethiopia/
cscott@hydro:/home/ethiopia$ process_by_week.sh wolonchete/wolonchete_2012-07-01
cscott@hydro:/home/ethiopia$ process_by_week.sh wonchi/wonchi_2012-06-28/
[...]
Mon Jul  9 17:46:31 EDT 2012: Finished processing wonchi/wonchi_2012-06-28/
cscott@hydro:/home/ethiopia$

You might see a few error messages which look like:

Unable to parse file: 00000034_0388920540e09217_d4c0820a-2ddc-4c19-8adf-872c835dd6b7_1340339595_mainPipeline.db

These are harmless; usually caused by too-small/empty .db files (check this).

Now that we've run this command, we have moved the original db files to encrypted_db and created merged.db and *.csv files:

cscott@hydro:/home/ethiopia$ ls wolonchete/wolonchete_2012-07-01/20/edu.mit.media.funf.bgcollector/encrypted_db/
[...]
00000689_0380614241409357_ffce5f83-9400-4699-9d6c-300ccfd58e85_1341117255_mainPipeline.db.orig
cscott@hydro:/home/ethiopia$ ls wolonchete/wolonchete_2012-07-01/20/edu.mit.media.funf.bgcollector/merged_20.db 
wolonchete/wolonchete_2012-07-01/20/edu.mit.media.funf.bgcollector/merged_20.db
cscott@hydro:/home/ethiopia$ ls wolonchete/wolonchete_2012-07-01/20/edu.mit.media.funf.bgcollector/csv_20/
BatteryProbe.csv       Matching.csv                  ScreenProbe.csv
FileMoverService.csv   NellBalloons5.csv             tinkerbook.csv
HardwareInfoProbe.csv  RecorderService.csv
LauncherApp.csv        RunningApplicationsProbe.csv

Now let's stage this data for transfer to git: (Note that, in cases of discrepancy, the first two arguments match how the directory will appear in git and the last argument matches the directory name in /home/ethiopia)

cscott@hydro:/home/ethiopia$ copyto-forgit.sh wonchi 2012-06-28 wonchi/wonchi_2012-06-28/

This has put the new data in $HOME/forgit:

cscott@hydro:/home/ethiopia$ cd ~/forgit/
cscott@hydro:~/forgit$ ls
wonchi
cscott@hydro:~/forgit$ ls wonchi/
2012-06-28
cscott@hydro:~/forgit$ ls wonchi/2012-06-28/
01  02  03  04  05  06  07  08  09  10  11  12  13  14  15  16  17  18  19  20

Now move this to the git dir and check it in:

cscott@hydro:~/forgit$ mv wonchi/2012-06-28 ~/ethiopia-data/wonchi/
cscott@hydro:~/forgit$ cd ~/ethiopia-data/
cscott@hydro:~/ethiopia-data$ git add wonchi/2012-06-28
cscott@hydro:~/ethiopia-data$ git commit wonchi/2012-06-28
cscott@hydro:~/ethiopia-data$ git push origin # or dev

Now repeat this for the other site (separate commits for each):

cscott@hydro:/home/ethiopia$ copyto-forgit.sh wolenchite 2012-07-01 wolonchete/wolonchete_2012-07-01/
cscott@hydro:/home/ethiopia$ cd ~/forgit/
cscott@hydro:~/forgit$ mv wolenchite/2012-07-01 ~/ethiopia-data/wolenchite/
cscott@hydro:~/forgit$ cd ~/ethiopia-data/
cscott@hydro:~/ethiopia-data$ git add wolenchite/2012-07-01
cscott@hydro:~/ethiopia-data$ git commit wolenchite/2012-07-01
cscott@hydro:~/ethiopia-data$ git push origin # or dev

Final steps

In theory we should then sync up owl and worldliteracy, then send mail to the literacy list to announce that there is new data. Something like:

rsync -a --progress wolonchete wonchi owl.laptop.org:/home/ethiopia
rsync -a --progress wolonchete wonchi worldliteracy.media.mit.edu:/home/ethiopia

You might want to run the second rsync from owl after the first is finished, rather than from hydro, since owl and worldliteracy are both in the ML and have a high-bandwidth network between them.