I flew up to Boston on Monday to lead some workshops on Continuous Integration. I checked into my room at the Hyatt and then went downstairs to see who was around. I ran into a couple of speakers milling about, and eventually joined one of them, and we headed over to the MIT Press bookstore, me to look for my Solr book. I wasn’t too sure of the name of the other speaker I was with (I asked once, but couldn’t quite remember what it was…). So we got to the book shop, I ask my fellow speaker again: James Bach. The name I was familiar with, but couldn’t quite place it… I ended up buying Parentonomics, and then we go for coffee.

So, over coffee, he asks me about what my topic is, and I gave him the brief summary of my two CI related workshops. Wow.. Little did I realize that I was sitting with the guy who rails against the “fetish” that Agile folks have for automated testing! That his entire approach to “testing” is to use skilled, motivated folks who do “sapient testing”. And I’m the guy who’s selling an approach that REQUIRES automated tests! That encourages expanding the use of automated testing!

He actually walked me through a process of talking about how to “think like a tester”, and it was really great mini-workshop.. He definitely subscribes to the socratic approach, and believes in his message, I was sweating at the end of it! That chat probably sparked more ideas in less time over that coffee then anything else this week. I also heard a lot of ideas and phrases that were echoed in Michael Bolton’s keynote later on in the week. Clearly a lot of collaboration between the two!

Probably the biggest idea that James and chatted about was the idea that automated tests really aren’t automated tests, they are automated checks. They verify that the expected behavior of the code was met. His argument that if you want to do testing, real testing, then computers, automated processes can’t meet that need, only people can.

Now, I don’t know if I believe that is completely true, but I am very aware that the “manual testing” where long test scripts written as Word documents are executed by human beings by hand are really a waste of human potential. And that those test scripts are really, to use James terms, “check scripts” because the people are not using any creativity! In fact, a lot of my interest in CI comes from the idea that people should not do monkey testing, that machines can do it much better, and my frustation with the perception that testing is a low value activity and can be easily shipped off to low skilled folks.

I think that this shift away from the term “test” for automated tests is actually happening in many places. In the Ruby world, we have libraries like Shoulda that are moving from using words like assert to other words like should. A Cucumber test really shows how controlled the space that a test needs to be to work well in an automated fashion:

Scenario: See all vendors
Given I am logged in as a user in the administrator role
And There are 3 vendors
When I go to the manage vendors page
Then I should see the first 3 vendor names

So while I don’t know if I am bought in on the idea that only people can do “testing”, and machines can only do “checking”. Tools like Heckle try to simulate aspects of what a human can do. While not suggesting that we can automate the “does my website look okay after someone changed the CSS” type of work today, in the future our automated testing will be more capable then just “checking” because we will move beyond the very constrained tests we have today to ones that mimicing the richness of the simulators that Airline Pilots use. Instead of testing the training given to pilots, we’ll be testing the robustness of software via simulations!

At any rate, James Bach, while taking a rather provocative approach to sharing his ideas, does subscribe to my favorite bullet in the Agile Manifesto: Individuals and interactions over processes and tools.

Here is him giving a great presentation with the subversive title of “How to Fake a Project” that was incredible entertaining, and also quite thought provoking:

What do you think? Automated testing is a fetish of the Agile community?

However, recently we’ve had two projects were the usual CI solutions such as CruiseControl etc haven’t worked out well, and we had to develop our own internal CI tool that we are ready to publish to the world called MonkeyCI.

On the first project, which was a PHP based application with 5 full time developers, we used CruiseControl with the phpUnderControl addon. However, we were running CruiseControl on what turned out to be an underpowered hosted Windows server, and we kept getting build failure errors related to environmental difficulties. Now, if you’ve seen my talk about CI, you know how big I am on speccing a beefy server for CI, and this experience reinforced that lesson. We decided that migrating the CI environment to a bigger server was something that we felt was in the “nice to have” category, and that it could wait till the next iteration. But we needed something immediate. Enter MonkeyCI.

The heart of CI is all about building the code, running the tests, and publishing the results frequently. Everything else, the reports, the red/green lava lamps, the pretty JavaDocs etc are all gravy. To meet the needs of your developers, you need to know if the “bar is green”. So MonkeyCI does that in a decidedly low tech way:

Everytime someone runs the full suite of unit tests they record the day and time, and put their initials. If the build is failing then they immediately fixed it. We’ve played with writing the results in red for failing builds and green for successful builds as well. Then, each day at the standup someone highlights how the CI is doing, and verifies that multiple folks are initialing, which means that the tests are running on multiple systems successfully.

While this does mean you have an additional manual process, it’s also really easy to do, requiring just a whiteboard! And for small project teams, the overhead of maintaining a reliable CI system is too much.

We’re doing another two developer project right now, and at least so far MonkeyCI has been great. We haven’t seen integration issues yet such as database scripts that don’t run, or busted code being checked in. I’ll post a picture of our whiteboard once we have a bunch of checkoffs recorded!

We call this simple low tech process MonkeyCI because typically we refer to anything manual, such as testing by pounding keyboards as Monkey testing. Also, somewhat of a reference to the great developers at the Primate Programming Institute who I am sure would use this approach to CI!:

More information and registration is available at http://www.richmondspin.org/home22332.

I’m looking forward to a good crowd, lots of questions, and drinks afterwords!

Last week I posted a photo and asked people to comment with how many Agile practices they could count in the picture. There were a couple of very good replies – thanks!

As promised, and long awaited, here is the list I made when posting those photos:

8 Agile Practices

1. Burndown chart on wall – Kind of hard to see, but there is a print out of past burndown charts on the wall. Good for seeing the burndown change over time.
2. Sprint goals on wall – Prominently displayed to remind us what we’re really trying to accomplish in the current sprint.
3. CI Server on Michaels’ monitor
4. Product owner in meeting – We’ve got a great product owner on this project who is highly engaged in the project and participates in all stand up meetings. I’ve been on projects where the product owner is not as engaged in the stand ups, and it really takes away a lot of the value.
5. Burndown/Task List on Google Docs on Arin’s monitor – In every stand up, I keep the current burndown displayed on my laptop for us to go over.
6. Remote team member in meeting on video chat (Stefan in Finland). While video chat is not strictly agile, having all your team members involved is. In this project, even though one of the client’s key developers is in Helsinki, he participates in the daily stand ups via video or audio chat, and we also use TeamViewer to pair program with him.
7. Team working on a single table, pairing up as needed
8. Scrum poker cards – Nobody guessed this one, but that’s my fault. You really can’t see them in the picture because they’re white and next to a white roll of paper towels. But there are scrum poker cards there that we used in task estimation.

In addition to those Agile specific parts of the project, here’s a few other things to note about the photo that are not necessarily specific to Agile, but you will often see on an OSC project.

1. User stories on wall – as was pointed out by James in the comments of my previous post, these are actually color coded to represent stories that relate to different types of users for the website we are working on.
2. Large monitors – they just make life so much easier, and are great for pair programming and demos.
3. Entity modeling on wall and whiteboard – We went through some great exercises with the clients modeling the entities in their business. We were originally going to be more dependent on a CMS in the project, but the modeling exercise made it clear some of their needs were too unique for the CMS, and so we scaled back (though didn’t eliminate) the use of the CMS in the system.
4. Mountain Dew in Michael’s drink – Michael’s trying to be sneaky by putting his Mountain Dew in a travel mug, but I know that he was actually drinking the Dew as a breakfast drink. Very common to see on an OSC project.
5. OSC schwag on Arin – The world famous bright red OpenSource Connections logo shirt. Guaranteed to be a collector’s item someday.
6. Windows and Mac living happily together. Since I don’t have a big monitor like Michael and Youssef, I try to compensate by bringing two laptops. Sure, I could just use Parallels on my Mac, but then I don’t get to have twice the computing power at my disposal.

I count 8 items in this photo that are specifically Agile. How many can you find? I’ll post my “solution” in a couple days, and I encourage you to post your own answers in the comments to this post. In addition to 8 Agile-specific items in this picture, I also count 6 other more general common traits of an OpenSource Connections project.

For bonus points, can you think of anything missing from this photo?

What sort of prize does the winner get, you ask? Hmmm, I don’t really have one. How about free bowling lessons from our newly found OSC official “Bowling Kingpin” Jim Nist?

4:00 PM – 5:00 PM March 19, 2009
Location: Clemons Library room 407

Please come hear Eric Pugh of Open Source Connections
(http://www.opensourceconnections.com/) discuss Continuous Integration (CI).
The goal of CI is to have “a fully automated and reproducible build [of your
project], including testing, that runs many times a day.” (I stole that from
Eric’s online slides.)

Eric is an active participant in the open source world, and is a contributor
to CruiseControl, a widely used open source CI solution. Eric has presented
for beTech several times in the past, and his talks are always excellent and
informative. He has also been quite active in organizing beCamp in the past,
so we might also use this meeting for some brief preliminary planning about
this year’s beCamp event.

I mentioned that I would publish my slides, and if anyone has questions, please let me know at epugh@opensourceconnections.com.

The slides are online at http://www.slideshare.net/o19s/ci-presentacion-presentation/.

Also, a couple people pinged me about beCamp, and I’ll be getting started on it! Any volunteers to setup the 2009 wiki page?

.

Come to the CVREG presentation at Strategy Cafe on January 13th. More information, and a map, is available on CVREG’s upcoming events page.

I’ve posted the slides along with the transcription of the talk in Spanish on SlideShare:

A huge round of thanks needs to go out to Raynald Korchia and the rest of the Sogeti crew for putting together this conference. It was wonderful to see such a large and diverse testing community in Spain, with what I guessed as around 250 people attending the conference, and a great set of vendors as well.

I really enjoyed spending time with the attendees and speakers, I learned a lot from your individual experiences in the fields of testing. I’m looking forward to getting more plugged into the world of software testing in Europe, and especially some of the upcoming conferences in Spain that people mentioned.

And lastly, a huge amount of thanks needs to go to my wife Kate, with whom I spent hours and hours over bottles of Riojan wine transcribing my badly formed thoughts in Spanish into proper Castellano, and generally making sure my presentation was a success!

A lot of folks have been using Apache and mod_proxy and mod_auth to try and wrap CruiseControl.rb, but you still need to add firewall rules so people don’t just go directly to the server on port 3333. There has been quite a bit of traffic on the CC.rb mailing list lately about this.

However, I found that HTPasswd was a much simpler solution. HTPasswd is one of those little know gem’s of a plugin. It makes it dirt simple to add both Basic and Digest authentication to Rails apps.

I just SSH’ed into the server running CC.rb, browsed to the CC webapp directory and ran

./script/plugin install http://wota.jp/svn/rails/plugins/branches/stable/htpasswd/.

I added the line htpasswd :user=>"MYUSER", :pass=>"MYPASS" to application.rb, restarted CC.rb, and now have a secure CruiseControl.rb instance!

Much simpler then hacking around with Apache configuration files. Please go to the HTPasswd page on AgileWebDevelopment and vote for this great plugin!

A couple year’s ago I worked on a LIMS system, and we had extensive regression tests written using Selenium. We used the Selenium HTML script, which was great, but maintaining them was a real challenge as over time we renamed buttons, reorganized menus, and were constantly fixing broken regression tests. One thing that I would disagree with Elisabeth on is that the maintenance burden for regression tests goes down if you are running them constantly. If you run them constantly, and the developers are responsible for them passing they same as unit tests, then that keeps them up to date, and adding more value then they might otherwise. But I agree, they are a burden, and sometimes really only worth it for certain high risk/high error portions of your application. Unfortunately putting something like Mercury Interactive based tests under CI is hard, and Selenium isn’t that much easier…

A lot of my interest about writing “Stories” using the RSpec framework comes directly from trying to make the tests easier for people to read, and separate out the logic of clicking on a button from the desire to perform an action that would require clicking on that button!

I met Elisabeth at CITcon 2007, and I still wear the Test Obsessed wrist band she gave me!

What is Continuous Integration? What should be including in Continuous Integration 2.0?

This session was focused on an introduction of continuous integration and a discussion on what should be included in the next version. The following contains a few bullet points of what was discussed at the session:

• Modularize the code to create smaller faster tests and quicker builds
• Source control packages should provide an option to enable automatic updates on commit
• Source control packages should verify build is green before allowing a commit to the code
  
• Commit frequently to shorten the feedback cycle to discover errors or problems quickly
• A good roll back process can eliminate downtime when a build is broken

How to make builds faster?

A person responsible for the builds at a company has run into the problem with large build times on their project. They were looking for ways in which they could reduce the time for the build. The ideas that were suggested for reducing the time of the build were as follows:

• A common practice is to restart the application server after a successful build to setup for the next build.
• Modularize the build so that only the modified code may be built and not the whole project. Maven has a built in method for modularizing the code.
  
• Add or upgrade the build server hardware. The build server should have as good of hardware as the developers hardware.
• Implement a distributed computing model to offload the build onto multiple machines
  
• Analyze the build to determine which part of it is inefficient and determine if the increase in build time is justified, specifically related to tests
  
• Tests are a common culprit in extending build times
• Break tests up into unit, function, integration and system tests. Consider only running each type of tests when it is appropriate
• Assign a member of the team to monitor the build time using metrics to determine if the build has exceeded the set metrics and analyze the change made to determine if the build time increase is acceptable.

Test data setup techniques

A participant of the conference was interested in discovering the different ways people setup data when they run their tests. The following items were discussed at the session:

• Add code to the tests that will add data and later remove the data when the tests have completed. This solution can slow down the build and does not allow the tests to be run independently.
• Write a script to duplicate the production database with minimal or no data and run update scripts on the newly created database. This gives you a clean database to start testing the application.
  
• Maintain scripts to create the database with minimal or no data and run update scripts on the newly created database.