User:Homunq/Activity bundles v2: Difference between revisions

From OLPC
Jump to navigation Jump to search
 
(14 intermediate revisions by the same user not shown)
Line 4: Line 4:
* UI
* UI


:*activity user: The format should support associating and versioning for different versions of the "same" activity, through a private key in some way associated with that activity. ''Each activity has a "''master key''" which remains unchanged over activity versions. This is a fundamental identifier of the activity, used in many searches of the activity bundle registry, and activities with the same master key are grouped (in submenus?) in most cases in the UI.''
:*activity user: The format should support associating and versioning for different versions of the "same" activity, through a private key in some way associated with that activity. ''Each activity has a "'''master key'''" which remains unchanged over activity versions. This is a fundamental identifier of the activity, used in many searches of the activity bundle registry, and activities with the same master key are grouped (in submenus?) in most cases in the UI.''


:*activity developer:
:*activity developer:
::*When used on the XO, the format should, insofar as possible, lessen the burden of key management. In other words, it should not be necessary to open two separate files (a bundle and a private key) simply in order to make and sign changes in a bundle. '''''"signature metadata"''': The master key is used to sign a user specific key, granting it privileges to sign the bundle. The user key is managed by sugar and signatures are available to any activity on request. To contain abuse of these privileges by malactivities, these user signatures include inseparably-signed metadata including the hash and public master key of the requesting activity. Thus, the master key signature on the user public key must also include (inseparably-signed) metadata: which activity hash and/or master key are allowed to give valid signatures on subkeys. ''
::*When used on the XO, the format should, insofar as possible, lessen the burden of key management. In other words, it should not be necessary to open two separate files (a bundle and a private key) simply in order to make and sign changes in a bundle. '''''"signature metadata"''': The master key is used to sign a user specific key, granting it privileges to sign the bundle. The user key is managed by sugar and signatures are available to any activity on request. To contain abuse of these privileges by malactivities, these user signatures include inseparably-signed metadata including the hash and public master key of the requesting activity. Thus, the master key signature on the user public key must also include (inseparably-signed) metadata: which activity hash and/or master key are allowed to give valid signatures for that user.''
::*it should be possible to make and test development versions of an activity, without replacing or un-defaulting the stable version. ''Activities without a valid signature are considered "'''development versions.'''"''
::*it should be possible to make and test development versions of an activity, without replacing or un-defaulting the stable version. ''Activities without a valid signature are considered "'''development versions.'''"''
::*Creating activity bundles is practical, whatever they are developed with. Ideal if tools are useable from outside sugar. ''Bundlebuilder.py looks in more than one place for anything it imports, directly or indirectly, from sugar, and "''conditional''" imports make it fail gracefully.''
::*Creating activity bundles is practical, whatever they are developed with. Ideal if tools are useable from outside sugar. ''Bundlebuilder.py looks in more than one place for anything it imports, directly or indirectly, from sugar, and "'''conditional imports'''" make it fail gracefully.''


* security
* security
Line 20: Line 20:
::*possible for a user to change a whole set of journal entries with a given (mime type,creating activity) to default to a new handling activity. The UI for this should be somewhat more involved if the new activity is signed with a different key, but it should possible.
::*possible for a user to change a whole set of journal entries with a given (mime type,creating activity) to default to a new handling activity. The UI for this should be somewhat more involved if the new activity is signed with a different key, but it should possible.
::*Possible for an activity author to sign other people's patched versions in three ways: by signing them herself, by giving the other person non-delegatable authority to sign them, or by giving them delegatable authority (ie, the private key).
::*Possible for an activity author to sign other people's patched versions in three ways: by signing them herself, by giving the other person non-delegatable authority to sign them, or by giving them delegatable authority (ie, the private key).
::*The signatures can be validated in real-use environments, and not only on install. That is, the presence or absence of extraneous files such as *.pyc or .git/* do not break validation. Yet no file is considered "extraneous" in this sense without an explicit reason; there are no implicit ways to include an arbitrary file without breaking the signature. ''Certain files are considered "'''local'''", that is, excluded from the signature. All activities should work without these files, and there should be a command to delete all such files.''
::*other things...


* efficiency
* efficiency
Line 28: Line 28:
:*optional: the format should support selective transfer of subfiles (see [[XO_updater]] for an example of how this might work)
:*optional: the format should support selective transfer of subfiles (see [[XO_updater]] for an example of how this might work)


''These three goals are somewhat at odds. For instance, zip satisfies 1 and 3 but not 2, whereas tgz with the --rsyncable option on gzip satisfies 1 and 2 but not 3 (at least, not with standard transfer protocols, as tar format has no central index of offsets to the subfiles). I think "''.tgz --rsyncable''" is the best available option.''
''These three goals are somewhat at odds. For instance, zip satisfies 1 and 3 but, in some cases, not 2; tar is the best possible solution to 2, but fails 1 and 3 badly; whereas tgz with the --rsyncable option on gzip satisfies 1 and (mostly) 2, but not 3 (at least, not with standard transfer protocols, as tar format has no central index of offsets to the subfiles). I think "'''.tgz --rsyncable'''" is the best available option, but we should do some testing with rdiff and real activity bundles. Zip may win.''

:''After further discussion and searching the available activities, I found few examples where zip would be pathological on point 2 - that is, activities with large (>50K) files likely to be subject to frequent, small changes. Thus I think that the current format, "'''zip'''", is adequate.''


== Signatures ==
== Signatures ==
Line 38: Line 40:


Helloworld.activity/
Helloworld.activity/
MANIFEST
helloworld.py
helloworld.py
locale/
locale/
Line 49: Line 52:
helloworld.svg
helloworld.svg
STRUCTURE/
STRUCTURE/
MANIFEST
ALLOWED_EXTRA_FILES
TRANSLATABLES
HASHES
HASHES
TRANSLATABLE_HASHES/
zh_de
de
es
SIGNATURES/
SIGNATURES/
deadbeef/
deadbeef/
Line 73: Line 71:
All special files are assumed to be in UTF-8. All linefeeds are unix-style.
All special files are assumed to be in UTF-8. All linefeeds are unix-style.


#MANIFEST
=== MANIFEST ===
All bundles must have a MANIFEST file. It consists of one entry per line, with \n at the end of every line including the last. A given entry is one of the following:
##Mandatory
##list of realative paths of all files and directories (with trailing dash) in bundle, except:
###non-empty directories (only empty directories should be in MANIFEST)
###MANIFEST
###TRANSLATABLES
###HASHES
###any files in SIGNATURES
###any files matched by a pattern in TRANSLATABLES (see below)
##Ends with a newline
##Order is significant, and should be maintained
###When renaming a file, the renamed file should appear on the same line
###When deleting a file, leaving a blank line is OPTIONAL and ENCOURAGED. Blank lines are DISCOURAGED but ALLOWED in other cases.
##Other whitespace not part of file names is NOT ALLOWED


*the relative path of a signed bundle file.
''maintaining order makes merges, diffs, and import to source control tools easier''

No path may start with a character in the set [!@#$%*-+=<>]. Paths that end with the string '.py' cause the same path with a trailing 'c' (that is, X.pyc) to be considered a local file.

*the character '-' followed by the relative path of a local file

*the character '-' followed by the relative path of a local directory, ending in '/'. All files inside this directory or subdirectories thereof are considered local files.

*an empty line. This maintains the absolute line number of following entries, so that entries on the same line number in different versions of the bundle can be considered different versions of the same file.

When renaming a file, the renamed file should appear on the same line. When deleting a file, leaving a blank line is OPTIONAL and ENCOURAGED. Blank lines are DISCOURAGED but ALLOWED in other cases. Other whitespace not part of file names is NOT ALLOWED

''maintaining order makes merges, delta-based storage, and import to source control tools easier''


For the example above, MANIFEST would be exactly:
For the example above, MANIFEST would be exactly:


activity/activity.info
helloworld.py
helloworld.py
icons/helloworld.svg
icons/helloworld.svg


=== ALLOWED_EXTRA_FILES ===
#TRANSLATABLES
This is a list of files or subdirectories which can be present without invalidating the signature. Only files not in the MANIFEST and not in the STRUCTURE directory will be checked against the patterns here. If there is any file which does NOT match a pattern listed here, then the signature is invalid. An example:
##optional file
##uses the same format as .gitignore
##indicates what files from MANIFEST should *not* be included in HASHES


#Compiled python
''Obviously, this does introduce a security risk, as an unsigned TRANSLATABLES file could theoretically cause a buffer overflow (or, indeed, be deliberately run by malicious signed code). The first case is considered avoidable with good tools, and the second is considered to be no additional risk (malicious code is malicious code)''
helloworld.pyc


#Git
''L10n should not require bugging the original activity author - imagine having to sign multiple versions of dozens of languages for every activity version, besides the inconvenience it is a security risk because the underlying code could change between "pure-l10n" versions without the author realizing.''
.gitignore
.git/*


#Temporary
#HASHES
justfoolingaround.tmp
##auto-generated file

#Translations
po/*
locale/*/*.mo

=== HASHES ===
##auto-generated file
##unix-style line endings
##first line '#HASH-VERSION: 1.0; HASH-FUNCTION:sha256'.
##first line '#HASH-VERSION: 1.0; HASH-FUNCTION:sha256'.
##No additional whitespace
##No additional whitespace
Line 112: Line 120:
###the sha1sum hash of the binary contents of the file on the line which follows.
###the sha1sum hash of the binary contents of the file on the line which follows.
##There is no limit to line length.
##There is no limit to line length.
##order:
##order same as MANIFEST
###[The first two lines are TRANSLATABLES and its hash, if it exists]
###The rest of HASHES follows MANIFEST, in the same order, excluding blank lines or any lines that match patterns in TRANSLATABLES.
###The rest of HASHES follows MANIFEST, in the same order, excluding blank lines or any lines that match patterns in TRANSLATABLES.


=== SIGNATURES/ ===

#SIGNATURES/
##optional directory
##optional directory
#Contents
#Contents
#...
##activity.key
###public DSA key, as generated by ssh
##VALIDKEYS/
###contains pairs of files
####a public key, filename does not end in ".sig"
####a separate signature, format specified below, of that file. filename is the same with ".sig" appended.
#####meaningful metadata:
######"requiredmetadata":{JSON object}. At least one key:value pair from the requiredmetadata must be in the metadata object of signatures using this key.
#####all signatures here should be verifiable with activity.key
######invalid signatures are deleted on installation
##GENSIGS/
###Signatures of HASHES
####iff any of these are verifiable with a key from VALIDKEYS, the bundle is considered to be "signed"
####additional signatures are allowed, and may be respected if recognized by Sugar. (ie, a control panel for "valid global activity signers"
####meaningful metadata:
#####"status":
######"favorite" means signor claims that this activity should be favoritted. Equivalent to "latest" unless this signature recognized by sugar.
######"latest" means signor claims that this version supercedes previous versions.
######"development" means signor claims that this version does NOT supercede previous versions
######any other status attribute, or none, is interpreted as "development"
#####anything from requiredmetadata of signing key.
##TRANSSIGS/
###signatures of translation files
###hashes
####same format as HASHES, contain any subset of files which match TRANSLATABLES
####may contain files not in (purged for space from) bundle, as long as at least 1 of the files is still present
###sigs
####signatures on hashes
####any signer - this information to be used by sugar.

Latest revision as of 19:45, 26 June 2008

Design goals

Italics show options for implementation of a goal. Bolded phrases in quotes are for later use as shorthand for the italicized passage. Thus, "development versions" is just one of various possible ways to provide development versions.

  • UI
  • activity user: The format should support associating and versioning for different versions of the "same" activity, through a private key in some way associated with that activity. Each activity has a "master key" which remains unchanged over activity versions. This is a fundamental identifier of the activity, used in many searches of the activity bundle registry, and activities with the same master key are grouped (in submenus?) in most cases in the UI.
  • activity developer:
  • When used on the XO, the format should, insofar as possible, lessen the burden of key management. In other words, it should not be necessary to open two separate files (a bundle and a private key) simply in order to make and sign changes in a bundle. "signature metadata": The master key is used to sign a user specific key, granting it privileges to sign the bundle. The user key is managed by sugar and signatures are available to any activity on request. To contain abuse of these privileges by malactivities, these user signatures include inseparably-signed metadata including the hash and public master key of the requesting activity. Thus, the master key signature on the user public key must also include (inseparably-signed) metadata: which activity hash and/or master key are allowed to give valid signatures for that user.
  • it should be possible to make and test development versions of an activity, without replacing or un-defaulting the stable version. Activities without a valid signature are considered "development versions."
  • Creating activity bundles is practical, whatever they are developed with. Ideal if tools are useable from outside sugar. Bundlebuilder.py looks in more than one place for anything it imports, directly or indirectly, from sugar, and "conditional imports" make it fail gracefully.
  • security
  • all algorithms should be reasonably cryptographically secure in the context of 2008. Excessive paranoia and extreme key lengths are unnecessary, but, all other things being equal, algorithms with known weaknesses should be avoided.
  • The various likely developer workflows should be smooth enough that developers (and users) are not excessively tempted to take shortcuts with security. Desirable in this regard:
  • possible to change translations without invalidating a signature
  • possible for a user to change a whole set of journal entries with a given (mime type,creating activity) to default to a new handling activity. The UI for this should be somewhat more involved if the new activity is signed with a different key, but it should possible.
  • Possible for an activity author to sign other people's patched versions in three ways: by signing them herself, by giving the other person non-delegatable authority to sign them, or by giving them delegatable authority (ie, the private key).
  • The signatures can be validated in real-use environments, and not only on install. That is, the presence or absence of extraneous files such as *.pyc or .git/* do not break validation. Yet no file is considered "extraneous" in this sense without an explicit reason; there are no implicit ways to include an arbitrary file without breaking the signature. Certain files are considered "local", that is, excluded from the signature. All activities should work without these files, and there should be a command to delete all such files.
  • efficiency
  • the format should transfer well (supports compression)
  • the format should interact well with delta-based storage.
  • optional: the format should support selective transfer of subfiles (see XO_updater for an example of how this might work)

These three goals are somewhat at odds. For instance, zip satisfies 1 and 3 but, in some cases, not 2; tar is the best possible solution to 2, but fails 1 and 3 badly; whereas tgz with the --rsyncable option on gzip satisfies 1 and (mostly) 2, but not 3 (at least, not with standard transfer protocols, as tar format has no central index of offsets to the subfiles). I think ".tgz --rsyncable" is the best available option, but we should do some testing with rdiff and real activity bundles. Zip may win.

After further discussion and searching the available activities, I found few examples where zip would be pathological on point 2 - that is, activities with large (>50K) files likely to be subject to frequent, small changes. Thus I think that the current format, "zip", is adequate.

Signatures

An activity bundle consists of a file whose nominal extension is .xo, but whose real structure corresponds to .zip, OR .tgz (interchangeable). The difference between these two structures is auto-detected.

Zip is for backward compatibility, .tgz is a better format for differential version control because, while it is compressed for sending over a network connection, it is also easily expanded to a form that is friendly to binary diffs.

The internal structure looks something like the following:

Helloworld.activity/
    MANIFEST
    helloworld.py
    locale/
        de_DE/
            activity.linfo
        zh_CN/
            activity.linfo
    activity/
        activity.info
    icons/
        helloworld.svg
    STRUCTURE/
        ALLOWED_EXTRA_FILES
        HASHES
        SIGNATURES/
            deadbeef/
                deadbeef.pub
                cafeb0ef.pub.sig
                12345678.pub.sig
            12345678/
                12345678.pub
            A1A1A1A1/
                a1a1a1a1.pub
                de.sig
            cafeb0ef/
                cafeb0ef.sig
                HASHES.sig


All special files are assumed to be in UTF-8. All linefeeds are unix-style.

MANIFEST

All bundles must have a MANIFEST file. It consists of one entry per line, with \n at the end of every line including the last. A given entry is one of the following:

  • the relative path of a signed bundle file.

No path may start with a character in the set [!@#$%*-+=<>]. Paths that end with the string '.py' cause the same path with a trailing 'c' (that is, X.pyc) to be considered a local file.

  • the character '-' followed by the relative path of a local file
  • the character '-' followed by the relative path of a local directory, ending in '/'. All files inside this directory or subdirectories thereof are considered local files.
  • an empty line. This maintains the absolute line number of following entries, so that entries on the same line number in different versions of the bundle can be considered different versions of the same file.

When renaming a file, the renamed file should appear on the same line. When deleting a file, leaving a blank line is OPTIONAL and ENCOURAGED. Blank lines are DISCOURAGED but ALLOWED in other cases. Other whitespace not part of file names is NOT ALLOWED

maintaining order makes merges, delta-based storage, and import to source control tools easier

For the example above, MANIFEST would be exactly:

activity/activity.info
helloworld.py
icons/helloworld.svg

ALLOWED_EXTRA_FILES

This is a list of files or subdirectories which can be present without invalidating the signature. Only files not in the MANIFEST and not in the STRUCTURE directory will be checked against the patterns here. If there is any file which does NOT match a pattern listed here, then the signature is invalid. An example:

#Compiled python
helloworld.pyc
#Git
.gitignore
.git/*
#Temporary
justfoolingaround.tmp
#Translations
po/*
locale/*/*.mo 

HASHES

    1. auto-generated file
    2. unix-style line endings
    3. first line '#HASH-VERSION: 1.0; HASH-FUNCTION:sha256'.
    4. No additional whitespace
    5. The further lines of HASHES alternate
      1. one line with a path as in MANIFEST
      2. the sha1sum hash of the binary contents of the file on the line which follows.
    6. There is no limit to line length.
    7. order same as MANIFEST
      1. The rest of HASHES follows MANIFEST, in the same order, excluding blank lines or any lines that match patterns in TRANSLATABLES.

SIGNATURES/

    1. optional directory
  1. Contents
  2. ...