1. pip installing using wheels

    Written by Miguel González on 2013-12-01

    This is a quick recipe to improve dramatically the time needed to create Python virtual environments using the new wheel binary package format.

    Pre requisites

    First, make sure you have a pip version that supports wheel:

    pip install --user --upgrade pip
    

    Install wheel package:

    pip install --user --upgrade wheel
    

    I recommend to install packages in the user scheme and not system wide. This way, you don't need administrative rights, the system is not polluted with unneeded packages or broken dependencies and it's easier to reinstall if something goes wrong.

    So summing up:

    • apt-get ... or yum ... for system wide standard packages.
    • pip install --user ... for general tools like pip, wheel and virtualenv. This way you will always have the latest versions without need to wait that they will be packaged for your distribution.
    • source venv/bin/activate && pip install ... to have an isolated environment for development.

    Making wheels

    This step is common for user and virtualenv installed packages. We are creating a repository of wheel packages:

    pip wheel --wheel-dir=$HOME/.wheelhouse psycopg2
    

    It also builds its dependencies if any.

    It is possible to use a requirements.txt file:

    pip wheel --wheel-dir=$HOME/.wheelhouse -r requirements.txt
    

    Notice that you can specify package version so a different wheel file will be generated:

    $ ls .wheelhouse/psycopg2*
    .wheelhouse/psycopg2-2.4.5-cp27-none-linux_x86_64.whl
    .wheelhouse/psycopg2-2.5.1-cp27-none-linux_x86_64.whl
    

    Installing from the local wheel repository

    To use our brand new repository of wheel packages we must indicate pip to use it:

    pip install --use-wheel --no-index --find-links=$HOME/.wheelhouse psycopg2
    

    To install all requirements of a project:

    pip install -r requirements.txt --use-wheel --no-index --find-links ~/.wheelhouse
    

    Look at this screen capture, just 0.23 seconds:

    (venv) $ time pip install --use-wheel --no-index --find-links=$HOME/.wheelhouse psycopg2
    Ignoring indexes: https://pypi.python.org/simple/
    Downloading/unpacking psycopg2
    Installing collected packages: psycopg2
    Successfully installed psycopg2
    Cleaning up...
    pip install --use-wheel --no-index --find-links=$HOME/.wheelhouse psycopg2
    0.23s user 0.05s system 98% cpu 0.285 total
    

    References

  2. XSCE Moodle autologin

    Written by Miguel González on 2013-09-26

    In this document I'm going to explain in detail how Moodle autologin for XO works. This is the result of team work, I'm just joining the dots.

    TL;DR: Browse activity has a magic cookie that identifies the XO. Moodle validates cookie content against idmgr service for a previously registered XO.

    User story

    Since XS 0.6, XO users can autologin to Moodle after they registrate their laptops on the schoolserver.

    The user story is straightforward:

    • XO registers in schoolserver using sugar "Registration" option
    • XO opens moodle web page for first time
    • Moodle creates a moodle account and authenticates the user

    But how does this self defined automagic authentication mechanism really work?

    Registration

    The XO main screen, has the option of "Register". This command will send to the schoolserver XO's serial number, user's nick, a generated UUID and the pubkey:

    data = server.register(sn, nick, uuid_, profile.pubkey)
    

    If you want to analyze in depth what does the code you can review register_laptop function in schoolserver.py module in sugar.

    def register_laptop(url=_REGISTER_URL):
    
        profile = get_profile()
        client = GConf.Client.get_default()
    
        if _have_ofw_tree():
            sn = _read_mfg_data(os.path.join(_OFW_TREE, _MFG_SN))
            uuid_ = _read_mfg_data(os.path.join(_OFW_TREE, _MFG_UUID))
        elif _have_proc_device_tree():
            sn = _read_mfg_data(os.path.join(_PROC_TREE, _MFG_SN))
            uuid_ = _read_mfg_data(os.path.join(_PROC_TREE, _MFG_UUID))
        else:
            sn = _generate_serial_number()
            uuid_ = str(uuid.uuid1())
        sn = sn or 'SHF00000000'
        uuid_ = uuid_ or '00000000-0000-0000-0000-000000000000'
    
        setting_name = '/desktop/sugar/collaboration/jabber_server'
        jabber_server = client.get_string(setting_name)
        _store_identifiers(sn, uuid_, jabber_server)
    
        if jabber_server:
            url = 'http://' + jabber_server + ':8080/'
    
        nick = client.get_string('/desktop/sugar/user/nick')
    
        if sys.hexversion < 0x2070000:
            server = xmlrpclib.ServerProxy(url, _TimeoutTransport())
        else:
            socket.setdefaulttimeout(_REGISTER_TIMEOUT)
            server = xmlrpclib.ServerProxy(url)
        try:
            data = server.register(sn, nick, uuid_, profile.pubkey)
        except (xmlrpclib.Error, TypeError, socket.error):
            logging.exception('Registration: cannot connect to server')
            raise RegisterError(_('Cannot connect to the server.'))
        finally:
            socket.setdefaulttimeout(None)
    
        if data['success'] != 'OK':
            logging.error('Registration: server could not complete request: %s',
                          data['error'])
            raise RegisterError(_('The server could not complete the request.'))
    
        client.set_string('/desktop/sugar/collaboration/jabber_server',
                          data['jabberserver'])
        client.set_string('/desktop/sugar/backup_url', data['backupurl'])
    
        return True
    

    The invocation uses a remote procedure call to schoolserver host on port 8080 as default.

    But, who is listening in the other side of this connection? The answer is idmgr service, more accurately, registration-server daemon.

    In a nutshell, it does the next tasks:

    1. sanitize input
    2. create a new system user
    3. store data in the service database
    4. return information about jabber server and data backup path

    Let's see the function register.

    def register(serial, nickname, uuid, pubkey):
        try:
            #Sanitise all input
            if not serialre.match(serial):
                raise ServerError("Invalid serial: %s" % (serial,))
            if not uuidre.match(uuid):
                raise ServerError( "Invalid UUID: %s" % (uuid,))
            if "\n" in nickname:
                raise ServerError( "Invalid nickname: %s" % nickname)
            if "\n" in pubkey:
                raise ServerError("Invalid public key: %s" % pubkey)
    
            # first try creating the system user. If this fails, the
            # database will be untouched.
            _create_new_user(serial, nickname, uuid, pubkey)
            database.save_laptop({'serial':    serial,
                                  'nickname':  nickname,
                                  'full_name': '',
                                  'pubkey':    pubkey,
                                  'uuid':      uuid,
                                  })
    
        except Exception, e:
            log(syslog.LOG_ERR, str(e))
            return {'success': 'ERR',
                    'error': str(e),
                    }
    
        # OK, the laptop is registered. Now we just need to let it know.
        log(syslog.LOG_NOTICE, "Registered user %s with serial %s"
            % (nickname.encode('utf-8'), serial))
    
        response = {
            'success': 'OK',
            'backupurl': "%s@%s:%s" % (serial, config.BACKUP, config.BACKUP_PATH),
            'jabberserver': config.PRESENCE,
            'backuppath': config.BACKUP_PATH,
            }
        return response
    

    Magic cookie

    But how the client provides moodle its identity. For me this was the ticky part.

    The Browse activity elaborates a magic cookie that contains a sha1 hash of XO public key.

    Note that the cookie domain is set to jabber_server hostname so a server in other host won't be able to access.

    Here in detail:

    def _seed_xs_cookie(cookie_jar):
        """Create a HTTP Cookie to authenticate with the Schoolserver.
    
        Do nothing if the laptop is not registered with Schoolserver, or
        if the cookie already exists.
    
        """
        client = GConf.Client.get_default()
        backup_url = client.get_string('/desktop/sugar/backup_url')
        if backup_url == '':
            _logger.debug('seed_xs_cookie: Not registered with Schoolserver')
            return
    
        jabber_server = client.get_string(
            '/desktop/sugar/collaboration/jabber_server')
    
        soup_uri = Soup.URI()
        soup_uri.set_scheme('xmpp')
        soup_uri.set_host(jabber_server)
        soup_uri.set_path('/')
        xs_cookie = cookie_jar.get_cookies(soup_uri, for_http=False)
        if xs_cookie is not None:
            _logger.debug('seed_xs_cookie: Cookie exists already')
            return
    
        pubkey = profile.get_profile().pubkey
        cookie_data = {'color': profile.get_color().to_string(),
                       'pkey_hash': sha1(pubkey).hexdigest()}
    
        expire = int(time.time()) + 10 * 365 * 24 * 60 * 60
    
        xs_cookie = Soup.Cookie()
        xs_cookie.set_name('xoid')
        xs_cookie.set_value(json.dumps(cookie_data))
        xs_cookie.set_domain(jabber_server)
        xs_cookie.set_path('/')
        xs_cookie.set_max_age(expire)
        cookie_jar.add_cookie(xs_cookie)
        _logger.debug('seed_xs_cookie: Updated cookie successfully')
    

    Moodle, aid and abet

    So we have all the ingredients, here we have the final act.

    Moodle has an OLPC-XS authentication plugin.

    Moodle auth process reads the public key hash from the cookie data and searchs in its database for a user with her idnumber equals to public key.

    But, when was this user created? We are near the end... the function idmgr_sync is our last stage. It reads directly from idmgr database and creates or updates the moodle user.

    Next it's some of the code involved.

    <?php
    ...
    
    class auth_plugin_olpcxs extends auth_plugin_base {
    
            /**
             * Constructor.
             */
            function auth_plugin_olpcxs() {
                    $this->authtype = 'olpcxs';
                    $this->config = get_config('auth/olpcxs');
            }
    
            function loginpage_hook() {
            global $CFG;
            global $USER;
            global $SESSION;
    
            if ( (isguestuser() || !isloggedin())
                 && !empty($_COOKIE['xoid'])) {
                // on 1.9 and earlier, $_COOKIE is badly mangled
                // by addslashes_deep() so...
                $xoid = json_decode(stripslashes($_COOKIE['xoid']));
    
                $user = get_record('user', 'idnumber', addslashes($xoid->pkey_hash));
                if (empty($user)) { // maybe the account is new!
                    $this->idmgr_sync(true);
                    $user = get_record('user', 'idnumber', addslashes($xoid->pkey_hash));
                    if (empty($user)) {
                        // We failed to login the user
                        // even though we saw a cookie.
                        // Probably the client side is confused.
                        // Log the problem, let things continue...
                        trigger_error('auth/olpcxs: user with pkey_hash ' . $xoid->pkey_hash . ' was not found after idmgr_sync()');
                        return true;
                    }
                }
    
                // Check if this account should be aliased to a different one...
                $user_alias = get_record('user_preferences', 'name', 'olpcxs_alias',
                                         'value', "$user->id");
                if ($user_alias) {
                    $olduser = $user;
                    $user = get_record('user', 'id', $user_alias->userid);
                    add_to_log(SITEID, 'user', 'login_alias', "../user/view.php?user={$user->id}", "ids:{$olduser->id} -> {$user->id}");
                }
    
                //
                // we have the user acct, complete login dance now
                //
                $this->maybe_assign_coursecreator($user);
    
                $sitectx   = get_context_instance(CONTEXT_SYSTEM);
                $roles = $this->get_user_roles_in_context($user->id, $sitectx);
                if (in_array('coursecreator', $roles)) {
                    $this->fixup_roles();
                }
    
                // icon for the user - we cannot do this in
                // create_update_user() because it's only
                // in the cookie, and not in idmgr. grumble...
                // .. expecting something like "#FF8F00,#00A0FF"
                // rough regex
                if (!empty($xoid->color) && preg_match('/#[A-F0-9]{1,6},#[A-F0-9]{1,6}/', $xoid->color)) {
                    $iconpref = get_user_preferences('xoicon', $user->id);
                    if (empty($iconpref) || $iconpref !== $xoid->color) {
                        set_user_preference('xoicon',$xoid->color, $user->id);
                    }
                }
    
                // complete login
                $USER = get_complete_user_data('id', $user->id);
                complete_user_login($USER);
    
                add_to_log(SITEID, 'user', 'login', "../user/view.php?user={$user->id}", 'autologin');
    
                // redirect
                if (isset($SESSION->wantsurl) and (strpos($SESSION->wantsurl, $CFG->wwwroot) === 0)) {
                    $urltogo = $SESSION->wantsurl;    /// Because it's an address in this site
                    unset($SESSION->wantsurl);
                } else {
                    // no wantsurl stored or external - go to homepage
                    $urltogo = $CFG->wwwroot.'/';
                    unset($SESSION->wantsurl);
                }
                redirect($urltogo);
            }
            }
    
      ...
    
          function idmgr_sync($fast=false) {
              global $CFG;
    
              if (empty($CFG->olpcxsdb) || !file_exists($CFG->olpcxsdb)) {
                  return false;
              }
              $dbh = new PDO('sqlite:' . $CFG->olpcxsdb);
    
              //
              // new accounts to create...
              //
              $sql = 'SELECT *
                        FROM laptops';
              $tslastrun = get_config('enrol/olpcxs', 'idmgr_sync_ts');
              $tsnow   = time();
              if ($fast && !empty($tslastrun)) {
                  $sql .= " WHERE lastmodified >= '"
                      . gmdate('Y-m-d H:i:s', $tslastrun) ."'";
              }
    
              $rs = $dbh->query($sql);
              foreach ($rs as $idmgruser) {
                  $this->create_update_user($idmgruser);
              }
    
              set_config('idmgr_sync_ts', $tsnow, 'enrol/olpcxs');
              unset($dbh); unset($rs);
    
              //
              // TODO - consider cleanup account scenario...?
              //
          }
    
  3. Mock locale defaults

    Written by Miguel González on 2013-06-29

    I need to test a piece of code with different behaviour depending on system locale LANG configuration. Instead of change environment, I'm using mock

    This is an snippet (test.py):

    import unittest
    import locale
    
    from mock import patch
    
    
    class LocaleTest(unittest.TestCase):
        LOCALE = ('hi_IN', 'UTF-8')
    
        def setUp(self):
            locale_patcher = patch('locale.getdefaultlocale')
            locale_mock = locale_patcher.start()
            locale_mock.return_value = self.LOCALE
            self.addCleanup(locale_patcher.stop)
    
        def test_locale(self):
            locale_defaults = locale.getdefaultlocale()
            self.assertEqual(locale_defaults[0], 'hi_IN')
            self.assertEqual(locale_defaults[1], 'UTF-8')
    
    $ python -m unittest test
    .
    ---------------------------------------------------------------------
    Ran 1 test in 0.001s
    
    OK
    

    mock is now part of the Python standard library, available as unittest.mock in Python 3.3 onwards. It could be installed in other versions with pip install mock.

  4. My sugar development setup

    Written by Miguel González on 2013-06-20

    In my previous attempt I installed Fedora18 in a virtualbox vm.

    This time, I switched from Fedora18 to Fedora19 because it is not necessary to build webkitgtk nor node.js to build sugar environment.

    VM creation

    The F19 box is a standard one. I added "Development Tools" and python-devel to default packages.

    sudo yum groupinstall "Development Tools"
    sudo yum install python-devel.i686
    

    I couldn't run Gnome desktop. I got a crash notification screen every time gdm tried to start.

    I simply ignore this error and worked throw a ssh connection. Last time I ran in init 3 and launched Xorg manually but I had some weird issues with dbus and NetworkManager.

    Sugar build

    The next steps are documented in Sugar Labs development documentation.

    git clone git://github.com/sugarlabs/sugar-build.git
    cd sugar-build
    

    And finally, I launched the build process.

    ./osbuild build
    

    This command configured the environment installing 250 yum packages.

    After that, it pulled and built only 30 git repositories.

    Instead of several hours, I only need 30 minutes to complete the build process.

    Sugar run

    ./osbuild run
    

    image

    Great!

  5. Howto change default runlevel in Fedora18

    Written by Miguel González on 2013-06-17

    On fedora wiki about systemd:

    ... systemd uses symlinks to point to the default runlevel. You have to delete the existing symlink first before creating a new one:

    rm /etc/systemd/system/default.target
    

    Switch to runlevel 3 by default:

    ln -sf /lib/systemd/system/multi-user.target \
    /etc/systemd/system/default.target
    

    Switch to runlevel 5 by default:

    ln -sf /lib/systemd/system/graphical.target \
    /etc/systemd/system/default.target
    

    ...

    A link instead a text file... well, it is systemd way.

Links

Social