Date & Time: Thursday November 7, 2013 9:00am - 9:40am 
Venue: AWE Level 2, Room 201A
Session Proposer: Julie Pichon
    
Topic
Integration Testing for Horizon

Session Abstract: http://icehousedesignsummit.sched.org/event/9e7981d66e65c886a94c9c68c7b95423

We mock a lot and end up relying heavily on manual testing to find out when one of the many APIs we depend on has changed in a backwards incompatible way, and other surprises. Horizon would greatly benefit from a wider set of integration tests that also exercises all of our interfaces to other projects.

Suggestions and issues to work through: 

 * How to write useful, durable tests that won't need to be modified all the time because e.g. a button's location has been adjusted?
   Probably depends on the chosen tool. Any previous experience to share?
	* Danger: id changes (rare in Horizon?)
	* Needing a browser = some flakiness
	* Splinter may help a little
	* Having the tests in-tree make them easier to modify
   
 * Implementation ideas:
     * Separate, self-contained new tests?
     * (Optional) part of the Django Selenium tests?
	     => probably not a good idea, do it separately so it doesn't set up the "mocked" Django env
     * A "mocking" switch that could turn mocking on or off: in e.g. a fully set up devstack environment, run the tests without mocking?
       How would that work?
       -> Can we change our current tests to be able to ignore the mocks? (investigate that)
		* Issues with pre-included data (Tempest does that)
     * Tempest scenarios?
        Would adding dependencies on Selenium or another tool be acceptable?
		* May be fine, show the code
		* Would impact Horizon workflow possibly significantly (needing to update Tempest before changing UI)
			* Infra: cross-project dependencies, someday
     * Other ideas?
 
 * Where should they live: in our tree to be easily modified alongside the main code, in Tempest, elsewhere?
   * YES: In-tree: can be kept in line with the code
  * Maybe later: In Tempest/elsewhere: easier to gate on for all projects? (We can gate on it anyway)
 
 * Tooling: Selenium, PhantomJS, CasperJS, Splinter, other...? Any previous experience to share?
	* Avoid the Javascript stuff, easier to contribute, run (subunit emitter), etc
 
 Session notes:
     Two types of integration tests:
		*          Kind of the unit tests we have currently, but without unit tests
		*          Integration tests driven by the UI

	* should be usable to run the function tests like for swift in a devstack environment
	* UI tests can be flaky, it could be tricky to gate on that
	* tempest  gates exclusively on stable APIs

	* Add the tests to Horizon, where they can be maintained more easily, and gate all projects on it
		* Bonus: client library testing
		* Of interest: Keystone client session in QA track tomorrow
	* Consensus: Start in the Horizon repo, see later on how that goes, if it should be moved etc
	* Note: devstack sets up a number of default users, etc we can use


	* testr?
		* How does this handle UI interactions - can't be used with the Django tests at the moment
		* To be dealt with later
		* To try and figure out this cycle: testr and Django
			* https://github.com/openstack-infra/storyboard/blob/master/tox.ini -> does it already