OpenStack

Is open source software for building private and public clouds.

Projects

Programs/Horizontal Efforts

Release Management

• Time Based Releases
• Six Month Cadence
  • Similar to Ubuntu and Fedora Release Schedules
• Design summits each cycle
• Continuously Open Trunk
  • Develop directly on master
• Intra-Cycle Milestone Releases
• Post-Release Stable Branches

Contributors

• Unaffiliated individuals
• Commercial entities
• Nonprofit organizations
• National and local governments
• Number, quality, and area of contributions can change daily

Consistent Tooling

• Minimize meta-development
• Process divergence == wasted developer time
• Lowers onboarding time
• Consolidate tool development
• Minimize project-specific weird build crud

Developer Infrastructure

System Administration is in Git

• All system administration performed via code review
• Anyone on the internet can look at our changes
• Anyone can sign up for a code review account
• Anyone can write and propose changes
• https://git.openstack.org/cgit/openstack-infra/config/tree/

Development Environment

• Python
  • CentOS (2.6), Ubuntu LTS (2.7, 3.3, pypy)
  • PEP-8 standards
  • Oslo (common libraries)
  • virtualenv/pip/tox
• Freenode IRC (#openstack-dev, #openstack-meeting)
• DevStack
• Tests run on all newly submitted changes
• Code merges are gated on tests

Project Gating

• Ensures Code Quality
• Protects developers
  • Devs always start from working code
• Protects tree
  • Bad code doesn't land
• Egalitarian
  • Process is the same for everyone
  • Process is transparent
  • Process is automated

Everything Is Automated

CI Workflow

Process Flow

Gerrit

• Developed by Google for Android
• Stand-alone patch review system
• Integration points: hooks, JSON queries, event-stream
• Extensible review categories, default: Verified, Code-Review
• REST API

Gerrit

Gerrit Diff View

Bug Integration - Launchpad

Approved Reviews

Git Review

• External Git subcommand
• Developers can easily incorporate code review into git workflow
• Zero-configuration
• Can be used for any project, being adopted by other projects

corvus@shiprock:~/git/neutron$ git commit -a
[new-versionpy ddf1dce] Base version.py on glance.
 3 files changed, 28 insertions(+), 107 deletions(-)
 delete mode 100644 version.py

corvus@shiprock:~/git/neutron$ git review
remote: Resolving deltas:   0% (0/3)
remote:
remote: New Changes:
remote:   https://review.openstack.org/3072
remote:
To ssh://corvus@review.openstack.org:29418/openstack/neutron.git
 * [new branch]      HEAD -> refs/for/master/bug/916018

Types of Tests

• Unit tests
  • Should be run in virtual environments, may run in operating systems
  • Fairly quick and easy for a developer to run
• Integration tests
  • May be able to run on virtual servers, should run on real servers
  • Difficult and time-consuming for a developer to run

Specific Challenges/Solutions

• Testing effect of merging change
• Using cloud builders
• Large numbers of similar projects
• Disparate hardware configurations

Gerrit Git Prep

• Test the result of the change, not the change
• openstack-infra/config:modules/openstack_project/files/slave_scripts/gerrit-git-prep.sh
• For each gerrit change:
  • grabs target branch
  • cleans tree
  • merges change to be tested

Interrelated Integration Testing

• devstack-gate & Nodepool
  • boot fresh servers
  • run DevStack
  • run integration tests
• Gate proposed change against current state of other projects

Devstack-Gate Problems

• Tests are slow
• Cloud API calls can fail
• External services are unreliable

Devstack-Gate Solutions

• Create a new node
• Pre-fetch all needed packages, repos
• Snapshot to cloud image
• Maintain a pool of cloud nodes
• Slave can only be used for one test run
• Python and shell scripts run as jobs
• Partition tests and run them in parallel

Zuul

• A general purpose trunk gating system
• Interfaces with Gerrit and Jenkins
• Flexible configuration allows for many kinds of project automation
• Allows parallel testing of serialized changes

Bottlenecking

• Serialize changes across all projects
• Speculative execution of tests
• Run in parallel in order triggered
• Assume success
• Start over on failure

Zuul Simulation

Zuul Gate Pipeline

pipelines:
  - name: gate
    manager: DependentPipelineManager
    precedence: high
    trigger:
      gerrit:
        - event: comment-added
          approval:
            - approved: 1
    start:
      gerrit:
        verified: 0
    success:
      gerrit:
        verified: 2
        submit: true
    failure:
      gerrit:
        verified: -2

Zuul Project Configuration

projects:
  - name: openstack/nova
    check:
      - gate-nova-pep8
      - gate-nova-python26
      - gate-nova-python27
      - gate-tempest-devstack-vm
    gate:
      - gate-nova-pep8
      - gate-nova-python26
      - gate-nova-python27
      - gate-tempest-devstack-vm
    post:
      - nova-branch-tarball
      - nova-docs
      - nova-upstream-translation-update
    release:
      - nova-tarball
      - nova-docs

Templated Jobs

• TONS of consistent projects
• Manage everything through Git and code review
• Jenkins Job Builder
• Manage jobs as YAML files in Git
• Jobs updated via Jenkins API
• groupings of jobs supported

Example Job Template

- job-template:
    name: 'gate-{name}-pep8'

    builders:
      - gerrit-git-prep
      - pep8:
          github-org: '{github-org}'
          project: '{name}'

    publishers:
      - console-log

Apply Template

- project:
    name: nova
    github-org: openstack
    tarball-site: tarballs.openstack.org
    doc-publisher-site: docs.openstack.org

    jobs:
      - python-jobs
      - gate-{name}-pep8
      - translation-jobs

Logs

• Massive amount of logs generated
  • Terabytes of compressed logs per development cycle
• Filtered and normalized by LogStash
• Recent logs indexed by ElasticSearch
  • Searchable through Kibana and ElasticSearch API
• Driving automatic failure classification

Scaling Hardware Needs

• Different vendors have different combinations of configurations
• http://ci.openstack.org/third_party.html#the-jenkins-gerrit-trigger-plugin-way
• Run their own Jenkins
• Consume Gerrit event stream
• Their Jenkins votes VRFY +1/-1
• Our Zuul votes VRFY +2/-2
• They can run their lab - we don't have to know anything about it

Multi-Master Jenkins

• Horizontal Scaling
• Redundancy
• Uses the Jenkins Gearman Plugin
• Masters: jenkins01.openstack.org, jenkins02.o.o, jenkins03.o.o, etc..
• 600+ test nodes

Jenkins Slaves

Elastic test nodes

• Nodepool
• Automate management of test nodes
• Registers nodes with Jenkins masters
• Uses Jenkins ZeroMQ events for build status
• Manages multiple openstack clouds

Elastic test nodes

Thanks!

These slides available at: http://docs.openstack.org/infra/publications/