*Preemptively Integrate the Universe
*Session proposed by Sean Dague
*Thursday November  7, 2013  5:20pm -  6:00pm
*Design Summit 1
*Infrastructure

We are currently doing preemptive integration on all  the openstack projects needed to run an OpenStack cloud. But we depend  on a hundred upstream python packages, which have very mixed track  records on their ability to upgrade smoothly. This creates problems all  the time during the release, and two weeks of hell right before pycon,  as everyone pushes out new major versions.

We can do better.

I'd  like to come up for a plan of attack to preemptively integrate the  world. Starting with all the python libraries that have moved to  stackforge, and expanding to any upstream library that we've had issues  with in the past. For non stackforge things this wouldn't be gating, but  be informative, and help us understand when problems are coming, and  hopefully let us catch bugs in upstream packages before they hit  released versions.

*Scenario #1 - pecan / wsme - OpenStack BFFS
Projects that are in stackforge, want to run devstack gate to ensure they don't break OpenStack on every commit.

Issues we experienced in getting them ready for this:
	* nodepool prep scripts require explicit listing of all repos (not very dynamic to adding new things)
		* cache everything in gerrit --- WIN!
	* devstack needed to be modified to support setting up these trees (hidden behind a flag)
		* can we make devstack prepare everything in gerrit....?
		* /opt/git ?
		* pre-gate hook for setup.py  - jeblair - icehouse-1
		* pre-devstack hook in wrap script
		* extras hook for devstack 
	* new devstack jobs need to be created for this
		* clarkb - icehouse-1

The process we go through to set up devstack is too intensive to do this for non BFF projects

Plan of Attack for Stackforge Projects:

*Scenario #2 - testr, pip, tox
Project that's important to OpenStack, used in all our infrastructure, and has maintainers that are close to the team.

How do we provide integration from these projects as a test environment?
How do we provide useful feedback to project like this?
Should we do it?
How often?

Plan of Attack for testr (and projects like it):
	* trigger on pull request
	* test post merge
mordred - icehouse-1
trial balloon would be pip

*Scenario #3 - sqlalchemy
Historically problematic package, requires a pin to ever keep it working. 

How do we track upstream to see issues coming?
How do we periodically attempt to try exposing the unpinned versions?
Tell us how you are going to increase your numbering scheme for incompatible changesg

*Version Exploration?
Can we use this approach to version explore the requirements space?
Report bugs for pins
Attempt to unpin requirements periodically, and if succeed, propose new requirements patch to uncap

Questions (ttx):
This is all about preventively watching upstream to prevent bad things from happening before they break us, but there will always be bad guys releasing crap.
Could we detect disruption when the new version of a dependency is about to be used in the gate ?
	* sounds a good idea if only to actually measure the amount of pain
	* yes, autofile bug on daily job
	* Does it have to be a daily job vs. a per-version_bump job ?
If yes, could we gate the dependency upgrade and prevent it to land if it breaks stuff ? (would avoid randomly breaking unrelated changes)
	* no?