User:Homunq/Activity bundles v2

Design goals

Italics show options for implementation of a goal.

• UI

• activity user: The format should support associating and versioning for different versions of the "same" activity, through a private key in some way associated with that activity. Each activity has a "master key" which remains unchanged over activity versions. This is a fundamental identifier of the activity, used in many searches of the activity bundle registry, and activities with the same master key are grouped (in submenus?) in most cases in the UI.

• activity developer:

• When used on the XO, the format should, insofar as possible, lessen the burden of key management. In other words, it should not be necessary to open two separate files (a bundle and a private key) simply in order to make and sign changes in a bundle. "signature metadata": The master key is used to sign a user specific key, granting it privileges to sign the bundle. The user key is managed by sugar and signatures are available to any activity on request. To contain abuse of these privileges by malactivities, these user signatures include inseparably-signed metadata including the hash and public master key of the requesting activity. Thus, the master key signature on the user public key must also include (inseparably-signed) metadata: which activity hash and/or master key are allowed to give valid signatures on subkeys.
• it should be possible to make and test development versions of an activity, without replacing or un-defaulting the stable version. Activities without a valid signature are considered development versions.
• Creating activity bundles is practical, whatever they are developed with. Ideal if tools are useable from outside sugar. Bundlebuilder.py looks in more than one place for anything it imports, directly or indirectly, from sugar.

• security

• all algorithms should be reasonably cryptographically secure in the context of 2008. Excessive paranoia and extreme key lengths are unnecessary, but, all other things being equal, algorithms with known weaknesses should be avoided.

• The various likely developer workflows should be smooth enough that developers (and users) are not excessively tempted to take shortcuts with security. Desirable in this regard:

• possible to change translations without invalidating a signature
• possible for a user to change a whole set of journal entries with a given (mime type,creating activity) to default to a new handling activity. The UI for this should be somewhat more involved if the new activity is signed with a different key, but it should possible.
• Possible for an activity author to sign other people's patched versions in three ways: by signing them herself, by giving the other person non-delegatable authority to sign them, or by giving them delegatable authority (ie, the private key).
• other things...

• efficiency

• the format should transfer well (supports compression)
• the format should interact well with delta-based storage.
• optional: the format should support selective transfer of subfiles (see XO_updater for an example of how this might work)

These three goals are somewhat at odds. For instance, zip satisfies 1 and 3 but not 2, whereas tgz with the --rsyncable option on gzip satisfies 1 and 2 but not 3 (at least, not with standard transfer protocols, as tar format has no central index of offsets to the subfiles). I think .tgz --rsyncable is the best available option.

Signatures

An activity bundle consists of a file whose nominal extension is .xo, but whose real structure corresponds to .zip, OR .tgz (interchangeable). The difference between these two structures is auto-detected.

Zip is for backward compatibility, .tgz is a better format for differential version control because, while it is compressed for sending over a network connection, it is also easily expanded to a form that is friendly to binary diffs.

The internal structure looks something like the following:

Helloworld.activity/
    helloworld.py
    locale/
        de_DE/
            activity.linfo
        zh_CN/
            activity.linfo
    activity/
        activity.info
    icons/
        helloworld.svg
    STRUCTURE/
        MANIFEST
        TRANSLATABLES
        HASHES
        TRANSLATABLE_HASHES/
            zh_de
            de
            es
        SIGNATURES/
            deadbeef/
                deadbeef.pub
                cafeb0ef.pub.sig
                12345678.pub.sig
            12345678/
                12345678.pub
            A1A1A1A1/
                a1a1a1a1.pub
                de.sig
            cafeb0ef/
                cafeb0ef.sig
                HASHES.sig

All special files are assumed to be in UTF-8. All linefeeds are unix-style.

1. MANIFEST
   1. Mandatory
   2. list of realative paths of all files and directories (with trailing dash) in bundle, except:
      1. non-empty directories (only empty directories should be in MANIFEST)
      2. MANIFEST
      3. TRANSLATABLES
      4. HASHES
      5. any files in SIGNATURES
      6. any files matched by a pattern in TRANSLATABLES (see below)
   3. Ends with a newline
   4. Order is significant, and should be maintained
      1. When renaming a file, the renamed file should appear on the same line
      2. When deleting a file, leaving a blank line is OPTIONAL and ENCOURAGED. Blank lines are DISCOURAGED but ALLOWED in other cases.
   5. Other whitespace not part of file names is NOT ALLOWED

maintaining order makes merges, diffs, and import to source control tools easier

For the example above, MANIFEST would be exactly:

helloworld.py
icons/helloworld.svg

1. TRANSLATABLES
   1. optional file
   2. uses the same format as .gitignore
   3. indicates what files from MANIFEST should *not* be included in HASHES

Obviously, this does introduce a security risk, as an unsigned TRANSLATABLES file could theoretically cause a buffer overflow (or, indeed, be deliberately run by malicious signed code). The first case is considered avoidable with good tools, and the second is considered to be no additional risk (malicious code is malicious code)

L10n should not require bugging the original activity author - imagine having to sign multiple versions of dozens of languages for every activity version, besides the inconvenience it is a security risk because the underlying code could change between "pure-l10n" versions without the author realizing.

1. HASHES
   1. auto-generated file
   2. first line '#HASH-VERSION: 1.0; HASH-FUNCTION:sha256'.
   3. No additional whitespace
   4. The further lines of HASHES alternate
      1. one line with a path as in MANIFEST
      2. the sha1sum hash of the binary contents of the file on the line which follows.
   5. There is no limit to line length.
   6. order:
      1. [The first two lines are TRANSLATABLES and its hash, if it exists]
      2. The rest of HASHES follows MANIFEST, in the same order, excluding blank lines or any lines that match patterns in TRANSLATABLES.

1. SIGNATURES/
   1. optional directory
2. Contents
   1. activity.key
      1. public DSA key, as generated by ssh
   2. VALIDKEYS/
      1. contains pairs of files
         1. a public key, filename does not end in ".sig"
         2. a separate signature, format specified below, of that file. filename is the same with ".sig" appended.
            1. meaningful metadata:
               1. "requiredmetadata":{JSON object}. At least one key:value pair from the requiredmetadata must be in the metadata object of signatures using this key.
            2. all signatures here should be verifiable with activity.key
               1. invalid signatures are deleted on installation
   3. GENSIGS/
      1. Signatures of HASHES
         1. iff any of these are verifiable with a key from VALIDKEYS, the bundle is considered to be "signed"
         2. additional signatures are allowed, and may be respected if recognized by Sugar. (ie, a control panel for "valid global activity signers"
         3. meaningful metadata:
            1. "status":
               1. "favorite" means signor claims that this activity should be favoritted. Equivalent to "latest" unless this signature recognized by sugar.
               2. "latest" means signor claims that this version supercedes previous versions.
               3. "development" means signor claims that this version does NOT supercede previous versions
               4. any other status attribute, or none, is interpreted as "development"
            2. anything from requiredmetadata of signing key.
   4. TRANSSIGS/
      1. signatures of translation files
      2. hashes
         1. same format as HASHES, contain any subset of files which match TRANSLATABLES
         2. may contain files not in (purged for space from) bundle, as long as at least 1 of the files is still present
      3. sigs
         1. signatures on hashes
         2. any signer - this information to be used by sugar.