'; ?> Scaling OpenStack Development: Continuous Integration Overview
OpenStack logo

Scaling OpenStack Development
Continuous Integration Overview


Monty Taylor <mordred@inaugust.com>
James E. Blair <jeblair@openstack.org>
Jeremy Stanley <fungi@yuggoth.org>
Clark Boylan <cboylan@sapwetik.org>
Elizabeth K. Joseph <lyz@princessleia.com>

OpenStack

Is open source software for building private and public clouds.

Projects

https://wiki.openstack.org/wiki/Projects

Programs/Horizontal Efforts

Release Management

Contributors

Consistent Tooling

Developer Infrastructure

  • Code review and VCS
    • Gerrit
    • Git
    • git-review
    • reviewday
  • Test/build automation
    • devstack-gate
    • Nodepool
    • Jenkins
    • JJB
    • Zuul
  • Repository mirrors
    • Apache
    • Cgit
    • GitHub
    • PyPI packages
  • Job logs, artifacts
    • logs site
    • Logstash
  • Documentation/publication
    • Sphinx
    • Wiki
  • Releases/pre-releases
    • PyPI uploads
    • tarballs
  • IRC
    • eavesdrop
    • gerritbot
    • meetbot
    • statusbot
  • Project blogs
    • Foundation
    • Planet
  • Collaboration
    • Mailing lists
    • Lodgeit paste
    • Etherpad
  • Authentication
    • CLA
    • membership
    • Launchpad SSO
  • Feature/bug tracking
    • LP blueprints
    • LP bugs

System Administration is in Git

Development Environment

Project Gating

Everything Is Automated

CI Workflow

Process Flow

Gerrit

Gerrit

Gerrit Diff View

Bug Integration - Launchpad

Approved Reviews

Git Review

corvus@shiprock:~/git/neutron$ git commit -a
[new-versionpy ddf1dce] Base version.py on glance.
 3 files changed, 28 insertions(+), 107 deletions(-)
 delete mode 100644 version.py

corvus@shiprock:~/git/neutron$ git review
remote: Resolving deltas:   0% (0/3)
remote:
remote: New Changes:
remote:   https://review.openstack.org/3072
remote:
To ssh://corvus@review.openstack.org:29418/openstack/neutron.git
 * [new branch]      HEAD -> refs/for/master/bug/916018

Types of Tests

Specific Challenges/Solutions

Gerrit Git Prep

Interrelated Integration Testing

Devstack-Gate Problems

Devstack-Gate Solutions

Zuul

Bottlenecking

Zuul Simulation

Zuul Gate Pipeline

pipelines:
  - name: gate
    manager: DependentPipelineManager
    precedence: high
    trigger:
      gerrit:
        - event: comment-added
          approval:
            - approved: 1
    start:
      gerrit:
        verified: 0
    success:
      gerrit:
        verified: 2
        submit: true
    failure:
      gerrit:
        verified: -2

Zuul Project Configuration

projects:
  - name: openstack/nova
    check:
      - gate-nova-pep8
      - gate-nova-python26
      - gate-nova-python27
      - gate-tempest-devstack-vm
    gate:
      - gate-nova-pep8
      - gate-nova-python26
      - gate-nova-python27
      - gate-tempest-devstack-vm
    post:
      - nova-branch-tarball
      - nova-docs
      - nova-upstream-translation-update
    release:
      - nova-tarball
      - nova-docs

Templated Jobs

Example Job Template

- job-template:
    name: 'gate-{name}-pep8'

    builders:
      - gerrit-git-prep
      - pep8:
          github-org: '{github-org}'
          project: '{name}'

    publishers:
      - console-log

Apply Template

- project:
    name: nova
    github-org: openstack
    tarball-site: tarballs.openstack.org
    doc-publisher-site: docs.openstack.org

    jobs:
      - python-jobs
      - gate-{name}-pep8
      - translation-jobs

Logs

Scaling Hardware Needs

Multi-Master Jenkins

Jenkins Slaves

Elastic test nodes

Elastic test nodes

o

Thanks!

These slides available at: http://docs.openstack.org/infra/publications/