I love CruiseControl.rb, but one of the things I’ve found is that it exposes via the web interface a lot of what is potentially confidential data. From the cruise_config.rb containing accounts on our SVN server, to being able to browse source files, which would include database.yml, again, exposing usernames and passwords.
(more…)

I don’t normally post about another blog article, but Agile-Friendly Test Automation Tools/Frameworks by Elisabeth Hendrickson is spot on about the challenges of Test Automation in an Agile world and well worth reading.

A couple year’s ago I worked on a LIMS system, and we had extensive regression tests written using Selenium. We used the Selenium HTML script, which was great, but maintaining them was a real challenge as over time we renamed buttons, reorganized menus, and were constantly fixing broken regression tests. One thing that I would disagree with Elisabeth on is that the maintenance burden for regression tests goes down if you are running them constantly. If you run them constantly, and the developers are responsible for them passing they same as unit tests, then that keeps them up to date, and adding more value then they might otherwise. But I agree, they are a burden, and sometimes really only worth it for certain high risk/high error portions of your application. Unfortunately putting something like Mercury Interactive based tests under CI is hard, and Selenium isn’t that much easier…

A lot of my interest about writing “Stories” using the RSpec framework comes directly from trying to make the tests easier for people to read, and separate out the logic of clicking on a button from the desire to perform an action that would require clicking on that button!

I met Elisabeth at CITcon 2007, and I still wear the Test Obsessed wrist band she gave me!

I attended the Continuous Integration and Testing Conference (CITCON) in Denver Colorado last weekend and I have to say that I was impressed by the effectiveness of the the OpenSpace conference format. The following are highlights from the sessions I attended:

What is Continuous Integration? What should be including in Continuous Integration 2.0?

This session was focused on an introduction of continuous integration and a discussion on what should be included in the next version. The following contains a few bullet points of what was discussed at the session:

• Modularize the code to create smaller faster tests and quicker builds
• Source control packages should provide an option to enable automatic updates on commit
• Source control packages should verify build is green before allowing a commit to the code
  
• Commit frequently to shorten the feedback cycle to discover errors or problems quickly
• A good roll back process can eliminate downtime when a build is broken

How to make builds faster?

A person responsible for the builds at a company has run into the problem with large build times on their project. They were looking for ways in which they could reduce the time for the build. The ideas that were suggested for reducing the time of the build were as follows:

• A common practice is to restart the application server after a successful build to setup for the next build.
• Modularize the build so that only the modified code may be built and not the whole project. Maven has a built in method for modularizing the code.
  
• Add or upgrade the build server hardware. The build server should have as good of hardware as the developers hardware.
• Implement a distributed computing model to offload the build onto multiple machines
  
• Analyze the build to determine which part of it is inefficient and determine if the increase in build time is justified, specifically related to tests
  
• Tests are a common culprit in extending build times
• Break tests up into unit, function, integration and system tests. Consider only running each type of tests when it is appropriate
• Assign a member of the team to monitor the build time using metrics to determine if the build has exceeded the set metrics and analyze the change made to determine if the build time increase is acceptable.

Test data setup techniques

A participant of the conference was interested in discovering the different ways people setup data when they run their tests. The following items were discussed at the session:

• Add code to the tests that will add data and later remove the data when the tests have completed. This solution can slow down the build and does not allow the tests to be run independently.
• Write a script to duplicate the production database with minimal or no data and run update scripts on the newly created database. This gives you a clean database to start testing the application.
  
• Maintain scripts to create the database with minimal or no data and run update scripts on the newly created database.