From patchwork Thu Nov 17 17:18:12 2011 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paul Larson X-Patchwork-Id: 5183 Return-Path: X-Original-To: patchwork@peony.canonical.com Delivered-To: patchwork@peony.canonical.com Received: from fiordland.canonical.com (fiordland.canonical.com [91.189.94.145]) by peony.canonical.com (Postfix) with ESMTP id 316DC24015 for ; Thu, 17 Nov 2011 17:18:16 +0000 (UTC) Received: from mail-fx0-f52.google.com (mail-fx0-f52.google.com [209.85.161.52]) by fiordland.canonical.com (Postfix) with ESMTP id 0FED2A1862A for ; Thu, 17 Nov 2011 17:18:16 +0000 (UTC) Received: by faaa26 with SMTP id a26so5649968faa.11 for ; Thu, 17 Nov 2011 09:18:15 -0800 (PST) Received: by 10.152.135.166 with SMTP id pt6mr24197570lab.26.1321550295002; Thu, 17 Nov 2011 09:18:15 -0800 (PST) X-Forwarded-To: linaro-patchwork@canonical.com X-Forwarded-For: patch@linaro.org linaro-patchwork@canonical.com Delivered-To: patches@linaro.org Received: by 10.152.41.198 with SMTP id h6cs144932lal; Thu, 17 Nov 2011 09:18:14 -0800 (PST) Received: by 10.180.105.102 with SMTP id gl6mr42050236wib.46.1321550293259; Thu, 17 Nov 2011 09:18:13 -0800 (PST) Received: from indium.canonical.com (indium.canonical.com. [91.189.90.7]) by mx.google.com with ESMTPS id fr4si12350595wbb.126.2011.11.17.09.18.13 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 17 Nov 2011 09:18:13 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of bounces@canonical.com designates 91.189.90.7 as permitted sender) client-ip=91.189.90.7; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of bounces@canonical.com designates 91.189.90.7 as permitted sender) smtp.mail=bounces@canonical.com Received: from ackee.canonical.com ([91.189.89.26]) by indium.canonical.com with esmtp (Exim 4.71 #1 (Debian)) id 1RR5bI-000672-N8 for ; Thu, 17 Nov 2011 17:18:12 +0000 Received: from ackee.canonical.com (localhost [127.0.0.1]) by ackee.canonical.com (Postfix) with ESMTP id A6EABE002F for ; Thu, 17 Nov 2011 17:18:12 +0000 (UTC) MIME-Version: 1.0 X-Launchpad-Project: lava-scheduler X-Launchpad-Branch: ~linaro-validation/lava-scheduler/trunk X-Launchpad-Message-Rationale: Subscriber X-Launchpad-Branch-Revision-Number: 95 X-Launchpad-Notification-Type: branch-revision To: Linaro Patch Tracker From: noreply@launchpad.net Subject: [Branch ~linaro-validation/lava-scheduler/trunk] Rev 95: Add some docs for lava-scheduler Message-Id: <20111117171812.24770.96178.launchpad@ackee.canonical.com> Date: Thu, 17 Nov 2011 17:18:12 -0000 Reply-To: noreply@launchpad.net Sender: bounces@canonical.com Errors-To: bounces@canonical.com Precedence: bulk X-Generated-By: Launchpad (canonical.com); Revision="14291"; Instance="launchpad-lazr.conf" X-Launchpad-Hash: e9f7b0ed3e74e3e2a4eeab755e2e2045dfef5a54 Merge authors: Paul Larson (pwlars) Related merge proposals: https://code.launchpad.net/~pwlars/lava-scheduler/lava-scheduler-docs/+merge/82452 proposed by: Paul Larson (pwlars) review: Approve - Paul Larson (pwlars) ------------------------------------------------------------ revno: 95 [merge] committer: Paul Larson branch nick: lava-scheduler timestamp: Thu 2011-11-17 11:14:38 -0600 message: Add some docs for lava-scheduler added: doc/ doc/changes.rst doc/conf.py doc/index.rst doc/installation.rst doc/process.rst doc/running.rst doc/usage.rst --- lp:lava-scheduler https://code.launchpad.net/~linaro-validation/lava-scheduler/trunk You are subscribed to branch lp:lava-scheduler. To unsubscribe from this branch go to https://code.launchpad.net/~linaro-validation/lava-scheduler/trunk/+edit-subscription === added directory 'doc' === added file 'doc/changes.rst' --- doc/changes.rst 1970-01-01 00:00:00 +0000 +++ doc/changes.rst 2011-11-16 21:02:31 +0000 @@ -0,0 +1,11 @@ +Version History +*************** + +.. _version_2011.11: + +Version 2011.11 +=============== + +.. todo:: + High level-changelog should be placed here. We should use intersphinx to + link to changelogs for particular components === added file 'doc/conf.py' --- doc/conf.py 1970-01-01 00:00:00 +0000 +++ doc/conf.py 2011-11-16 21:02:31 +0000 @@ -0,0 +1,204 @@ +# -*- coding: utf-8 -*- +# +# LAVA Dashboard documentation build configuration file, created by +# sphinx-quickstart on Mon Dec 27 16:39:47 2010. +# +# This file is execfile()d with the current directory set to its containing dir. +# +# Note that not all possible configuration values are present in this +# autogenerated file. +# +# All configuration values have a default; values that are commented out +# serve to show the default. + +import sys +import os + +# If extensions (or modules to document with autodoc) are in another directory, +# add these directories to sys.path here. If the directory is relative to the +# documentation root, use os.path.abspath to make it absolute, like shown here. +sys.path.append(os.path.abspath('..')) + +# -- General configuration ----------------------------------------------------- + +# Add any Sphinx extension module names here, as strings. They can be extensions +# coming with Sphinx (named 'sphinx.ext.*') or your custom ones. +extensions = ['sphinx.ext.autodoc', 'sphinx.ext.doctest', 'sphinx.ext.intersphinx', 'sphinx.ext.todo', 'sphinx.ext.coverage'] + +# Configuration for sphinx.ext.todo +todo_include_todos = True + +# Add any paths that contain templates here, relative to this directory. +templates_path = [] + +# The suffix of source filenames. +source_suffix = '.rst' + +# The encoding of source files. +#source_encoding = 'utf-8' + +# The master toctree document. +master_doc = 'index' + +# General information about the project. +project = u'LAVA Scheduler' +copyright = u'2010-2011, Linaro Limited' + +# The version info for the project you're documenting, acts as replacement for +# |version| and |release|, also used in various other places throughout the +# built documents. +# +# The short X.Y version. +import versiontools +import lava_scheduler_app +version = "%d.%d" % lava_scheduler_app.__version__[0:2] +# The full version, including alpha/beta/rc tags. +release = versiontools.format_version(lava_scheduler_app.__version__) + +# The language for content autogenerated by Sphinx. Refer to documentation +# for a list of supported languages. +#language = None + +# There are two options for replacing |today|: either, you set today to some +# non-false value, then it is used: +#today = '' +# Else, today_fmt is used as the format for a strftime call. +#today_fmt = '%B %d, %Y' + +# List of documents that shouldn't be included in the build. +#unused_docs = [] + +# List of directories, relative to source directory, that shouldn't be searched +# for source files. +exclude_trees = [] + +# The reST default role (used for this markup: `text`) to use for all documents. +#default_role = None + +# If true, '()' will be appended to :func: etc. cross-reference text. +#add_function_parentheses = True + +# If true, the current module name will be prepended to all description +# unit titles (such as .. function::). +#add_module_names = True + +# If true, sectionauthor and moduleauthor directives will be shown in the +# output. They are ignored by default. +#show_authors = False + +# The name of the Pygments (syntax highlighting) style to use. +pygments_style = 'sphinx' + +# A list of ignored prefixes for module index sorting. +#modindex_common_prefix = [] + + +# -- Options for HTML output --------------------------------------------------- + +# The theme to use for HTML and HTML Help pages. Major themes that come with +# Sphinx are currently 'default' and 'sphinxdoc'. +html_theme = 'default' + +# Theme options are theme-specific and customize the look and feel of a theme +# further. For a list of options available for each theme, see the +# documentation. +#html_theme_options = {} + +# Add any paths that contain custom themes here, relative to this directory. +#html_theme_path = [] + +# The name for this set of Sphinx documents. If None, it defaults to +# " v documentation". +#html_title = None + +# A shorter title for the navigation bar. Default is the same as html_title. +#html_short_title = None + +# The name of an image file (relative to this directory) to place at the top +# of the sidebar. +#html_logo = None + +# The name of an image file (within the static path) to use as favicon of the +# docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32 +# pixels large. +#html_favicon = None + +# Add any paths that contain custom static files (such as style sheets) here, +# relative to this directory. They are copied after the builtin static files, +# so a file named "default.css" will overwrite the builtin "default.css". +html_static_path = [] + +# If not '', a 'Last updated on:' timestamp is inserted at every page bottom, +# using the given strftime format. +#html_last_updated_fmt = '%b %d, %Y' + +# If true, SmartyPants will be used to convert quotes and dashes to +# typographically correct entities. +#html_use_smartypants = True + +# Custom sidebar templates, maps document names to template names. +#html_sidebars = {} + +# Additional templates that should be rendered to pages, maps page names to +# template names. +#html_additional_pages = {} + +# If false, no module index is generated. +#html_use_modindex = True + +# If false, no index is generated. +#html_use_index = True + +# If true, the index is split into individual pages for each letter. +#html_split_index = False + +# If true, links to the reST sources are added to the pages. +#html_show_sourcelink = True + +# If true, an OpenSearch description file will be output, and all pages will +# contain a tag referring to it. The value of this option must be the +# base URL from which the finished HTML is served. +#html_use_opensearch = '' + +# If nonempty, this is the file name suffix for HTML files (e.g. ".xhtml"). +#html_file_suffix = '' + +# Output file base name for HTML help builder. +htmlhelp_basename = 'LAVADocumentation' + + +# -- Options for LaTeX output -------------------------------------------------- + +# The paper size ('letter' or 'a4'). +#latex_paper_size = 'letter' + +# The font size ('10pt', '11pt' or '12pt'). +#latex_font_size = '10pt' + +# Grouping the document tree into LaTeX files. List of tuples +# (source start file, target name, title, author, documentclass [howto/manual]). +latex_documents = [ + ('index', 'LAVAScheduler.tex', u'LAVA Scheduler Documentation', + u'Linaro Validation Team', 'manual'), +] + +# The name of an image file (relative to this directory) to place at the top of +# the title page. +#latex_logo = None + +# For "manual" documents, if this is true, then toplevel headings are parts, +# not chapters. +#latex_use_parts = False + +# Additional stuff for the LaTeX preamble. +#latex_preamble = '' + +# Documents to append as an appendix to all manuals. +#latex_appendices = [] + +# If false, no module index is generated. +#latex_use_modindex = True + + +# Example configuration for intersphinx: refer to the Python standard library. +intersphinx_mapping = {'http://docs.python.org/': None} === added file 'doc/index.rst' --- doc/index.rst 1970-01-01 00:00:00 +0000 +++ doc/index.rst 2011-11-16 21:02:31 +0000 @@ -0,0 +1,49 @@ +========================= +LAVA Scheduler Documentation +========================= + +.. warning:: + This document is *work in progress*. + +Features +======== +The LAVA Scheduler is an extension to LAVA Server that handles +scheduling of test resources, accepts jobs, and dispatches them on +systems while monitoring the job. + + +Indices and tables +================== + +.. toctree:: + :maxdepth: 2 + + installation.rst + running.rst + usage.rst + process.rst + changes.rst + +* :ref:`search` + + +TODO List +========= + +This documentation is not finished (not even close yet). The following list +contains items that need more work. + +.. note:: + The source code for this document can be found in the lp:lava-project + branch. Please contribute patches to make the TODO list shorter. + +.. todolist:: + + +Questions +^^^^^^^^^ +.. _questions: + +If you have any questions, including to the content of this document, feel free +to ask them here: https://answers.launchpad.net/lava-project + === added file 'doc/installation.rst' --- doc/installation.rst 1970-01-01 00:00:00 +0000 +++ doc/installation.rst 2011-11-16 21:02:31 +0000 @@ -0,0 +1,101 @@ +Installation +^^^^^^^^^^^^ + +LAVA can be installed in several different ways. As with any open source +project that does source distribution the end user has all the freedom to do +what they want. We support certain installation methods more than others. You +can always ask for support using Launchpad support tracker (see +:ref:`questions`) + +To install LAVA Scheduler, you will first need to install LAVA Server. +For more information about LAVA Server, see +http://lava-server.readthedocs.org + +Using virtualenv +****************** + +Python Virtualenv is a useful tool for creating a sandbox for working +with python modules. In Ubuntu, you can get it by installing +*python-virtualenv* using apt-get. For source and pypi installations of +non-production systems, it is highly recommended. + +Example usage :: + + $ virtualenv sandbox + $ cd sandbox + $ . bin/activate + +Once activated, the environment for that session will be set up so that +subsequent commands will use the virtual environment settings. + +Installation from source +************************ + +This is the most complicated and error prone installation method. It requires +the user to download source release tarballs. Unpack them and install them in +the correct order. Depending on the exact set of components that are installed +(especially client or server side components) some additional steps are +necessary. This may include setting up the web application host (one of many +possible configurations here), setting up the database (again multiple possible +options, our recommendation is to use the latest stable version of PostgreSQL). + +For installing from source, it's normally much simpler to install from +pypi first, then update using the source. This is useful if you want +to use it for development against your own branch. For instance, after +installing from pypi (see directions below) you could do the following. + +Updating in a virtualenv using source :: + + $ bzr branch lp:lava-scheduler + $ cd lava-scheduler + $ ./setup develop + +Installation from PypI +********************** + +PyPi is the python package index (http://pypi.python.org/pypi). It is +maintained by the python community and is the preferred distribution method +used by many open source projects written in the python programming language. + +Here a front-end program, such as pip (http://pypi.python.org/pypi/pip) is used +to install packages, and their python dependencies. Pip finds the required set +of packages, downloads their source releases and does the hard work of figuring +out the right way to put them together. + +This is the best compromise between wide system support (any flavour of Linux, +OS X and Windows), simplicity, upgrade and availability. The downside is that +it does not handle, by itself, the last mile. This method does not handle +things like setting up and running the application server. It also requires the +user to have additional development packages, such as python header files, +database server header files, the C compiler and more. + +To install using pypi (For development only, not for production):: + $ pip install lava-scheduler + $ lava-server manage --development syncdb + $ lava-server manage --development migrate + +You will need to answer a few questions during the syncdb step. This +will use a simple sqlite database, and should normally only be used for +testing or hacking on lava-server. + +.. todo:: + Installation instructions for production installations against + postgresql using pypi + +Installation from PPA +********************* + +This method is only suitable for users running Ubuntu 10.04 or later. Here LAVA +is pre-compiled and packaged as Debian packages (debs). The installation +scripts embedded in the packages take care for setting up additional services +so usually this is the best method to quickly have a self-contained running +installation. The downside is longer release period as packaging takes +additional time after each release. Another downside is that our support is +limited to Ubuntu. + +To install using the ppa :: + + $ sudo add-apt-repository ppa:linaro-validation/ppa + $ sudo apt-get update + $ sudo apt-get install lava-scheduler + === added file 'doc/process.rst' --- doc/process.rst 1970-01-01 00:00:00 +0000 +++ doc/process.rst 2011-11-16 21:02:31 +0000 @@ -0,0 +1,64 @@ +Development process +=================== + +LAVA development process is based on Launchpad. If you are not familiar with +that system you should read the https://help.launchpad.net/ guide first. This +guide also includes the basics of Bazaar, our version control system of choice. + +Most of the work is done by the members of the Linaro Validation Team (you can +learn more about this team, in particular here: +launchpad.net/~linaro-validation). Having said that, the code is free and open +source software, we welcome third party contributions and new team members. + +Our team is spread geographically around the world, with some members in +Europe, America, Asia and Oceania. We are usually talking on our IRC channel +#linaro. + + +Release process +^^^^^^^^^^^^^^^ + +LAVA is being developed on a monthly release schedule. Each release is tagged +around 20th of each month. We publish all our releases on pypi (for actual +consumption, packaging, installation, etc.) and Launchpad (for reference). + +Launchpad release tarballs are following our YYYY.MM (year, month) pattern. +Should we need to release an upgrade to any existing release (such as a +critical bug fix) we append a sequential number preceded by a dash +(YYYY.MM-NN). + +Our PyPi releases use sensible version numbers instead. In general we use +MAJOR.MINOR.MICRO pattern (where MICRO is omitted when zero). Some components +are post 1.0, that is they have a major version greater than zero. For such +components we take extra care to ensure API stability, with sensible transition +periods, deprecation warnings and more. For other components (that have zero as +a major release number) our strategy is to keep them compatible as much as +possible but without ensuring a third party developer code would still work on +each upgrade. + + +Reporting Bugs +^^^^^^^^^^^^^^ + +New bugs can be reported here https://bugs.launchpad.net/lava/+filebug. + +If you are not sure which component is affected simply report it to any of the +LAVA sub-projects and let us handle the rest. As with any bug reports please +describe the problem and the version of LAVA you ware using. + +If you were using our public LAVA instance, the one used by Linaro for daily +activities (http://validation.linaro.org) try to include a link to a page +that manifests the problem as that makes debugging easier. + + +Patches, fixes and code +^^^^^^^^^^^^^^^^^^^^^^^ + +If you'd like to offer a patch (whether it is a bug fix, documentation update, +new feature or even a simple typo) it is best to follow this simple check-list: + +1. Download the trunk of the correct project +2. Add your code, change any existing files as needed +3. Commit in your local branch +4. Push to launchpad (to the public copy of your branch) +5. Propose a merge request === added file 'doc/running.rst' --- doc/running.rst 1970-01-01 00:00:00 +0000 +++ doc/running.rst 2011-11-16 21:02:31 +0000 @@ -0,0 +1,35 @@ +Running LAVA Scheduler +^^^^^^^^^^^^^^^^^^^^^^ + +LAVA Scheduler has two main components, the web application and the +daemon. To process jobs, the scheduler daemon must be running. Jobs +are accepted via the xmlrpc API on the web application. + +Adding Devices +************** +Before jobs can be submitted or processed, devices must exist to run +them on. To do this, login as an admin user in LAVA Server. + +First, create a device type unless you are just adding a device for +which you have already created a type. To create a device type from the +admin console, click the *Add* button next to *Device types* under the +*Lava_Scheduler_App* section. You only need to provide the name. Other +attributes of the device type such as default boot parameters will be +defined in the LAVA Dispatcher configuration files. + +Once you have at least one device type, devices can be added from the +admin console as well. To add a device, click the *Add* button next to +*Devices* under the *Lava_Scheduler_App* section. Select the device +type and add the name of the device you wish to add. The name given +here needs to correspond to the name of the device in the LAVA +Dispatcher config. + +Running the Scheduler Daemon +**************************** +If you installed LAVA Scheduler on an Ubuntu system using the debian +packages, you should be able to start the daemon by using :: + + $ sudo start lava-scheduler + +If you installed from source or from pypi, you can start it manually +by simply running *lava-scheduler*, or by adding an init script for it. === added file 'doc/usage.rst' --- doc/usage.rst 1970-01-01 00:00:00 +0000 +++ doc/usage.rst 2011-11-16 21:02:31 +0000 @@ -0,0 +1,51 @@ +Using LAVA Scheduler +^^^^^^^^^^^^^^^^^^^^ + +Submitting Jobs +*************** +Jobs can currently be submitted to the scheduler in one of two ways: +through the *lava-scheduler-tool* command line tool, or directly via +xmlrpc API. + +Generating a Token +================== +Before a job can be submitted, a token must be generated. Logged in as +a user with *lava_scheduler_app | test job | Can add test job* and +*linaro_django_xmlrpc | auth token | Can add auth token* permissions +enabled, select *API* from the menu at the top, then *Authentication +Tokens*. From this page, click on *Create a new token*. Once you have +created at least one token, you can click *Display this token* to show +it. The token string can be copied from the browser for pasting into a +tool later, or saved to a file. + +Using lava-scheduler-tool +========================= +LAVA Scheduler Tool is actually a plugin to LAVA Tool. It can be +installed from debian packages, source, or pypi in the same way +described for installing the scheduler in the installation section. + +To submit jobs using the scheduler, you should first set up the server +to which you will be submitting jobs. +With lava-scheduler-tool installed, run :: + + $ lava-tool auth-add https://user@example.com/lava-server/RPC2/ + +In this example, *user@example.com* should be replaced with your userid +and webserver. Using https is *highly* recommended since it will ensure +the token is passed to the server using ssl, but http will work if your +web server is not configured for ssl. + +When entering this command, you will be prompted to enter the token. +Copy/paste the text of the token from your browser window here; it will +not be echoed to the screen. Alternatively, you can also save the token +to a file and use the --token-file parameter to specify the file +containing your token. + +Once the auth-add step is complete, you can submit jobs by running :: + + $ lava-tool submit-job http://user@example.com/lava-server/RPC2/ + jobfile.json + +.. todo:: + Add link to information about constructing a job - lava-project might + be a better place to put usage information in general