*Ceilometer Integration Testing

*The session will include the following items:
    
1. Tempest integration
2. Devstack + Ceilometer testing

The main questions:
    1. Do we have any issues that block tempest integration?
      - test plan
      - break tests into event generator + check part
      - API tests vs backend-specific tests
      - tempest tests run in parallel -- need to repeat tests to catch race conditions
      https://blueprints.launchpad.net/tempest/+spec/basic-tempest-integration-for-ceilometer
    2. What issues do happen during devstack's tests and how we can avoid with it? From emails: http://logs.openstack.org/68/52768/1/check/check-tempest-devstack-vm-full/76f83a4/logs/screen-ceilometer-collector.txt.gz?level=TRACE
      - postgresql integration with devstack
      - collector doesn't work in gate
      - mongodb 2.3 or 2.4 conflicts with libvirt in Cloud Archive
    3. Do we need all backends for Tempest?
      - we want them all
      - mysql and postgresql to be fixed in infra first
      - mongodb blocked by Ubuntu packaging issue
    4. Now there are unit tests for all DB backends. For Mongo-DB and DB2 these tests are half-integration (user is forced to use real Mongo ) but for HBase in-Memory db is used. Are we going to change anything here?
     - testing against real DB is good, need to solve mongodb deployment for testing
     - the same for H-Base
     - DB2 tests could run against mongodb, similar API
     - sqlite for testing SQL backends
    5. In addition to API-level tests, it would be ideal to have specific load tests for the storage drivers and gate on performance regressions. 
      - don't fit as Jenkins check/gate tests
      - could be implemented as tests for external envs, like other stress tests in Tempest
      - highly useful to share use cases, data sets and performance profiles
    6. Analyze test coverage percentage as a part of check/gate testing


Here's a hacky piece of code to simulate Nova events for instance create/delete/modify. Generates N-events-per-minute
https://github.com/SandyWalsh/twobillion ... mostly for mongo now, but could be refactored to work as a source for integration/load testing.

Start by adding test job to tempest's experiemental queue in zuul

Actions:
    - fix gate issues
    - umbrella blueprint for integration testing stays in Tempest
    - add integration testing as a recurring part of meeting agenda
    - separate unittests and integration tests as different modules