So, is James Bach (@jamesmarcusbach) the bad boy of testing?

I flew up to Boston on Monday to lead some workshops on Continuous Integration. I checked into my room at the Hyatt and then went downstairs to see who was around. I ran into a couple of speakers milling about, and eventually joined one of them, and we headed over to the MIT Press bookstore, me to look for my Solr book. I wasn’t too sure of the name of the other speaker I was with (I asked once, but couldn’t quite remember what it was…). So we got to the book shop, I ask my fellow speaker again: James Bach. The name I was familiar with, but couldn’t quite place it… I ended up buying Parentonomics, and then we go for coffee.

So, over coffee, he asks me about what my topic is, and I gave him the brief summary of my two CI related workshops. Wow.. Little did I realize that I was sitting with the guy who rails against the “fetish” that Agile folks have for automated testing! That his entire approach to “testing” is to use skilled, motivated folks who do “sapient testing”. And I’m the guy who’s selling an approach that REQUIRES automated tests! That encourages expanding the use of automated testing!

He actually walked me through a process of talking about how to “think like a tester”, and it was really great mini-workshop.. He definitely subscribes to the socratic approach, and believes in his message, I was sweating at the end of it! That chat probably sparked more ideas in less time over that coffee then anything else this week. I also heard a lot of ideas and phrases that were echoed in Michael Bolton’s keynote later on in the week. Clearly a lot of collaboration between the two!

Probably the biggest idea that James and chatted about was the idea that automated tests really aren’t automated tests, they are automated checks. They verify that the expected behavior of the code was met. His argument that if you want to do testing, real testing, then computers, automated processes can’t meet that need, only people can.

Now, I don’t know if I believe that is completely true, but I am very aware that the “manual testing” where long test scripts written as Word documents are executed by human beings by hand are really a waste of human potential. And that those test scripts are really, to use James terms, “check scripts” because the people are not using any creativity! In fact, a lot of my interest in CI comes from the idea that people should not do monkey testing, that machines can do it much better, and my frustation with the perception that testing is a low value activity and can be easily shipped off to low skilled folks.

I think that this shift away from the term “test” for automated tests is actually happening in many places. In the Ruby world, we have libraries like Shoulda that are moving from using words like assert to other words like should. A Cucumber test really shows how controlled the space that a test needs to be to work well in an automated fashion:

Scenario: See all vendors
Given I am logged in as a user in the administrator role
And There are 3 vendors
When I go to the manage vendors page
Then I should see the first 3 vendor names

So while I don’t know if I am bought in on the idea that only people can do “testing”, and machines can only do “checking”. Tools like Heckle try to simulate aspects of what a human can do. While not suggesting that we can automate the “does my website look okay after someone changed the CSS” type of work today, in the future our automated testing will be more capable then just “checking” because we will move beyond the very constrained tests we have today to ones that mimicing the richness of the simulators that Airline Pilots use. Instead of testing the training given to pilots, we’ll be testing the robustness of software via simulations!

At any rate, James Bach, while taking a rather provocative approach to sharing his ideas, does subscribe to my favorite bullet in the Agile Manifesto: Individuals and interactions over processes and tools.

Here is him giving a great presentation with the subversive title of “How to Fake a Project” that was incredible entertaining, and also quite thought provoking:

What do you think? Automated testing is a fetish of the Agile community?

At OSC, we have a well developed methodology that we apply to our client work, and one of the core tenets is using Continuous Integration to ensure our code behaves the way we intend it to.

However, recently we’ve had two projects were the usual CI solutions such as CruiseControl etc haven’t worked out well, and we had to develop our own internal CI tool that we are ready to publish to the world called MonkeyCI.

On the first project, which was a PHP based application with 5 full time developers, we used CruiseControl with the phpUnderControl addon. However, we were running CruiseControl on what turned out to be an underpowered hosted Windows server, and we kept getting build failure errors related to environmental difficulties. Now, if you’ve seen my talk about CI, you know how big I am on speccing a beefy server for CI, and this experience reinforced that lesson. We decided that migrating the CI environment to a bigger server was something that we felt was in the “nice to have” category, and that it could wait till the next iteration. But we needed something immediate. Enter MonkeyCI.

The heart of CI is all about building the code, running the tests, and publishing the results frequently. Everything else, the reports, the red/green lava lamps, the pretty JavaDocs etc are all gravy. To meet the needs of your developers, you need to know if the “bar is green”. So MonkeyCI does that in a decidedly low tech way:

Everytime someone runs the full suite of unit tests they record the day and time, and put their initials. If the build is failing then they immediately fixed it. We’ve played with writing the results in red for failing builds and green for successful builds as well. Then, each day at the standup someone highlights how the CI is doing, and verifies that multiple folks are initialing, which means that the tests are running on multiple systems successfully.

While this does mean you have an additional manual process, it’s also really easy to do, requiring just a whiteboard! And for small project teams, the overhead of maintaining a reliable CI system is too much.

We’re doing another two developer project right now, and at least so far MonkeyCI has been great. We haven’t seen integration issues yet such as database scripts that don’t run, or busted code being checked in. I’ll post a picture of our whiteboard once we have a bunch of checkoffs recorded!

We call this simple low tech process MonkeyCI because typically we refer to anything manual, such as testing by pounding keyboards as Monkey testing. Also, somewhat of a reference to the great developers at the Primate Programming Institute who I am sure would use this approach to CI!:

I have the honor of speaking to the Richmond SPIN group on Continous Integration next week Wednesday, May 13. SPIN is the Software and Systems Process Improvement Network, and are the local groups sponsored by Carnegie Mellon University’s Software Engineering Institute.

My presentation is going to be a bit different from some of the previous topics that have talked about process improvement, whereas I am talking about a specific improvement that enhances your process. A CI system can provide the base framework for layering on much more then just the basic automatic code/compile/test cycle, and we’ll talk about what else it can be.

More information and registration is available at http://www.richmondspin.org/home22332.

I’m looking forward to a good crowd, lots of questions, and drinks afterwords!

This picture may not tell a thousand stories, but I count at least eight Agile related stories, as well as a few others about OpenSource Connections.

Last week I posted a photo and asked people to comment with how many Agile practices they could count in the picture. There were a couple of very good replies – thanks!

As promised, and long awaited, here is the list I made when posting those photos:

8 Agile Practices

1. Burndown chart on wall – Kind of hard to see, but there is a print out of past burndown charts on the wall. Good for seeing the burndown change over time.
2. Sprint goals on wall – Prominently displayed to remind us what we’re really trying to accomplish in the current sprint.
3. CI Server on Michaels’ monitor
4. Product owner in meeting – We’ve got a great product owner on this project who is highly engaged in the project and participates in all stand up meetings. I’ve been on projects where the product owner is not as engaged in the stand ups, and it really takes away a lot of the value.
5. Burndown/Task List on Google Docs on Arin’s monitor – In every stand up, I keep the current burndown displayed on my laptop for us to go over.
6. Remote team member in meeting on video chat (Stefan in Finland). While video chat is not strictly agile, having all your team members involved is. In this project, even though one of the client’s key developers is in Helsinki, he participates in the daily stand ups via video or audio chat, and we also use TeamViewer to pair program with him.
7. Team working on a single table, pairing up as needed
8. Scrum poker cards – Nobody guessed this one, but that’s my fault. You really can’t see them in the picture because they’re white and next to a white roll of paper towels. But there are scrum poker cards there that we used in task estimation.

In addition to those Agile specific parts of the project, here’s a few other things to note about the photo that are not necessarily specific to Agile, but you will often see on an OSC project.

1. User stories on wall – as was pointed out by James in the comments of my previous post, these are actually color coded to represent stories that relate to different types of users for the website we are working on.
2. Large monitors – they just make life so much easier, and are great for pair programming and demos.
3. Entity modeling on wall and whiteboard – We went through some great exercises with the clients modeling the entities in their business. We were originally going to be more dependent on a CMS in the project, but the modeling exercise made it clear some of their needs were too unique for the CMS, and so we scaled back (though didn’t eliminate) the use of the CMS in the system.
4. Mountain Dew in Michael’s drink – Michael’s trying to be sneaky by putting his Mountain Dew in a travel mug, but I know that he was actually drinking the Dew as a breakfast drink. Very common to see on an OSC project.
5. OSC schwag on Arin – The world famous bright red OpenSource Connections logo shirt. Guaranteed to be a collector’s item someday.
6. Windows and Mac living happily together. Since I don’t have a big monitor like Michael and Youssef, I try to compensate by bringing two laptops. Sure, I could just use Parallels on my Mac, but then I don’t get to have twice the computing power at my disposal.

My sons often get activity books with games in them like “Can you find the 10 different things between these two photos” or “Find these 10 animals hidden in this picture.” So rather than just post a picture of our uber-cool Agile development space with our new client Newswise, I figured instead I’ll post the photo without a description of what we’re doing and see if you can figure it out.

I count 8 items in this photo that are specifically Agile. How many can you find? I’ll post my “solution” in a couple days, and I encourage you to post your own answers in the comments to this post. In addition to 8 Agile-specific items in this picture, I also count 6 other more general common traits of an OpenSource Connections project.

For bonus points, can you think of anything missing from this photo?

What sort of prize does the winner get, you ask? Hmmm, I don’t really have one. How about free bowling lessons from our newly found OSC official “Bowling Kingpin” Jim Nist?

I’ll be speaking on CI for the beTech group March 19th:

4:00 PM – 5:00 PM March 19, 2009
Location: Clemons Library room 407

Please come hear Eric Pugh of Open Source Connections
(http://www.opensourceconnections.com/) discuss Continuous Integration (CI).
The goal of CI is to have “a fully automated and reproducible build [of your
project], including testing, that runs many times a day.” (I stole that from
Eric’s online slides.)

Eric is an active participant in the open source world, and is a contributor
to CruiseControl, a widely used open source CI solution. Eric has presented
for beTech several times in the past, and his talks are always excellent and
informative. He has also been quite active in organizing beCamp in the past,
so we might also use this meeting for some brief preliminary planning about
this year’s beCamp event.

Last night I had the privilege of talking about Continous Integration (CI) to the fine folks of CVREG, and had a wonderful time. CVREG had good turnout, and it was a great discussion of CI, and how CI can be part of a virtuous circle of becoming better software developers.

I mentioned that I would publish my slides, and if anyone has questions, please let me know at epugh@opensourceconnections.com.

The slides are online at http://www.slideshare.net/o19s/ci-presentacion-presentation/.

Also, a couple people pinged me about beCamp, and I’ll be getting started on it! Any volunteers to setup the 2009 wiki page?

.

Missed my presentation on CI in Spain? Fortunately I’ll be in our backyard, Richmond, next month to talk about CI with a Ruby flavor to CVREG: Central Virginia Ruby Enthusiasts Group.

Come to the CVREG presentation at Strategy Cafe on January 13th. More information, and a map, is available on CVREG’s upcoming events page.