To briefly recap, I pointed out that Agile team estimation and Planning Poker (or Scrum Poker) is great, but one big problem I have with it is the reliance on single point estimates in the burndown.

That’s why we have built range estimation into our Scrum Poker and burndowns at OpenSource Connections, and we find it to be more effective. By holding two cards instead of one, each team member is indicating a range estimate and communicating lot more about the risk and complexity that they see in that project.

So how does it work? In this blog post I’m going to describe how we do it using Google Spreadsheets.

How to build a burndown using ranges

Ingredients needed:

• Spreadsheet of any sort (I like google spreadsheets for easy sharing)
• Scrum Poker cards
• An engaged team!

Steps:

• Build a normal Scrum spreadsheet with a list of tasks on it.
• Estimate each task using Scrum poker. Have each participant hold up two cards to indicate the range of their estimate.
• The Scrummaster should encourage debate about the estimates when they vary widely in the size of the estimates or the width (ie risk) of the estimates.
• Once the team agrees on a range, the low and high values are entered in the spreadsheet.
• The spreadsheet calculates an estimate which is 2/3 of the way between the low and high estimate.
• Use the “2/3 estimate” as your ideal burndown to zero and to see how many tasks fit in your sprint.
• Use the low and high estimates as a band of reasonable deviation from the ideal burndown. They should both slope to zero at the same rate as the 2/3 estimate.
• Use the ranges to encourage more conservative adoption of tasks into a sprint, and to communicate risk and progress more clearly with customers.

In the end, we come up with a burndown similar to this, which represents the low and high estimates, and is a representation of the risk of the sprint or as a band of acceptable divergence from the estimate.

Make Google Spreadsheets (or Excel) do the Hard Work!

The next thing to show is how we represent this in a Scrum spreadsheet, and derive the formulas to build that pretty burndown. The vertical image is an example scrum spreadsheet like we use at OpenSource Connections, and the following sections will describe the major formulas being used in that spreadsheet.

This is a little tricky the first time, but then just copy the sheet from then on.

Low Estimate (D6-D8) and High Estimate (E6-E8) are entered by the team as they agree on estimates for each task in sprint planning.

Initial Estimate per task (F6-F8) is calculated by the spreadsheet, and is a single integer weighted 2/3 of the way from the low estimate to the high estimate:
F6 = (D6*0.33)+(E6*0.67)

2/3 Estimate (F2 and duplicated in F3) is just a sum of the initial estimates across all the tasks.
F3 = sum(F6:F8)

2/3 Estimate per day (G3-K3) is a calculated field, which decreases the estimate for each day down to zero by the end of the sprint. This provides the “ideal burndown” line for your chart at a 2/3 estimate.
G3 = F3-($F3/[Days in Sprint])
H3 = G3-($F3/[Days in Sprint]) … and so on

Low Estimate (F4) is a sum of all the low estimates: F4 = sum(D6:D8)

Low Estimate per day (G4-K4) is a calculated line showing the path down towards zero for our low estimates. It uses the slope of the 2/3 line.
G4 = IF((F4-($F3/[Days in Sprint]))>0,F4-($F3/[Days in Sprint]),0) … and so on

High Estimate (F5) is a sum: F5 = sum(E6:E8)

And High Estimate per day (G5-K5) is calculated similar to the low estimate per day:
G5 = F5-($F3/[Days in Sprint]) … and so on

Then developers update estimates each day as usual, and this makes the “Actual” line in the burndown.

That’s really all there is to it!

Once you’ve got your spreadsheet setup, you can just copy it from sprint to sprint, and then it really doesn’t take any more time in sprint planning to use ranges instead of single points. But using ranges with a 2/3 weighting like we do will help force your team to think a little more about the risk involved in their estimates and to estimate more conservatively. This allows you to better communicate with your customer and more reliably predict what you can commit to in a sprint.

Arin Sime is a Senior Consultant with OpenSource Connections, specializing in Agile process consulting, Solr search, and development consulting. You can follow Arin’s tweets at ArinSime

Don’t take it so personally – I’m certainly not saying I’m perfect at it either! But you’re among friends here, so let’s have an honest discussion.

While you will never be perfect at estimation (it is after little more than prediction of the future), you can get better at building and communicating estimates if you remember a few key reasons why you’re so bad at it. You’re still human after all, and so I guarantee you will fall into some of these traps.

Reason #1: You’re doing it alone

Too often, estimates are done alone. Typical scenarios include one person (such as the team lead) estimating an entire project, or a project manager going to a specific developer and asking them for an estimate. One person giving an estimate by themselves is one of the most unreliable ways to give an estimate, because there is no one to check you against common biases. And since all initial estimates tend to be very optimistic, you are almost guaranteed to set unrealistic expectations.

Reason #2: You’re doing it for others

When the team lead gives an estimate on behalf of their team, several problems ensue. First, you are relying solely on the judgment of one person (the team lead), and no matter how smart that person thinks they are, they are susceptible to biases when they go it alone. Second, when the rest of the team is told what the estimate for the project is, they had no voice in it, and therefore, they have no sense of ownership in that estimate. A lack of ownership means they will have no sense of accountability, and they will not go the extra mile to stick to the estimate.

Reason #3: You’re being way too optimistic

Very few of us give pessimistic estimates. For one reason, we want to please the customer. When they say they want something in two weeks, we make mental leaps in how we will build the software to try and get them what they want. Or we discount the likelihood of obstacles, and hope for the best case scenario.

Reason #4: You think you’re an expert

There’s a great chapter in the book “The Black Swan” about the problems with predictions, and one problem described is “the expert problem.” Studies have shown that experts tend to underestimate their own margins of error on predictions, because they are overconfident in their expert status. As software developers, we fall into this trap regularly when we say things like “oh yeah, that will be easy, I’ve done something similar before.” When we do remember that task last time was not as easy as we thought it would be, we often discount those obstacles we encountered as outliers. We assume that “now that we know how to do it”, we won’t encounter those obstacles again, and no other obstacles will arise. We are usually wrong.

Reason #5: You’re using single point estimates

Using single point estimates makes all of these previous reasons worse. Single point estimates encourage you to go with your gut and the first number you think of, or the number that the customer wants to hear. Think how ridiculous it would be someone asked you “How long does it take to get to your house?”, and you answered “Oh, about 25.6735 minutes”. By including all those extra decimal points, you are creating a false impression of accuracy. Single point estimates are the same idea. If I say “it takes 25 minutes to get home”, then you will expect to be there in exactly 25 minutes. But what about traffic, or other variables? A more accurate answer is to say that it takes “25-30 minutes, depending on traffic.”

A solution to estimation biases: Range estimation in Scrum Poker

Using a range instead offers a lot of benefits. By saying “that will take me 2-5 days” instead of “I can get that to you in two days”, you are communicating the uncertainty that is inherent in all software development. The size of the range you provide communicates risk. And it prepares the customer for a more realistic schedule, instead of setting you up for failure when they expect completion on the most optimistic schedule possible.

Combining range estimation with the Agile practices of team estimation is doubly effective. In Agile teams we often play “Scrum Poker.” I’ll avoid the long description of it here, but in short, each team member gets a deck of cards. When you are estimating a story or a task, each team member privately chooses a card (or range of cards) from their deck. When everyone has chosen, they all show their cards at the same time.

Showing the cards at the same time combats a number of the reasons I list above, because you get a true sense of the variety of estimates across the team without introducing the biases of the team lead or architect. Those who picked the lowest or highest estimates are not discounted as outliers, but encouraged to describe to the team why they chose that number. This encourages the team to discuss the idea until they have reached a common vision for the complexity of that story or task.

Agile team estimation is great, but one big problem I still have with it is the reliance on single point estimates. That’s why we use range estimation in our Scrum Poker at OpenSource Connections, and we find it to be more effective. By holding two cards instead of one, each team member is indicating a range estimate and communicating lot more about the risk and complexity that they see in that project.

Next Tuesday (August 10th, from 11am-noon in room Asia5) I’ll be presenting at the Agile 2010 conference on “Building a More Accurate Burndown: Using Range Estimation in Scrum”. If you’re going to be in Orlando, I hope you’ll stop by to hear more! You can also see my slides on slideshare.

Arin Sime is a Senior Consultant with OpenSource Connections, specializing in Agile process consulting, Solr search, and development consulting. You can follow Arin’s tweets at ArinSime

http://www.slideshare.net/o19s/intro-to-agile-practices-and-values

The discussion covered a lot of overview topics in Agile, in fact too many topics to fit into our one hour time slot! But true to form, the students in the class were great – they asked a lot of questions and we had a very engaging two-way conversation. I much prefer being “agile” in my presentations and having to skip material because the audience is engaged. It’s a lot better than a command and control presentation where only the speaker is talking!

Among the many great conversations were two that I want to highlight on the blog.

Tip #1: Failing Fast does not mean sacrificing quality

Agile’s value comes in iterations

One of the big values of agile development methods is the emphasis on short, iterative development cycles. This allows you to push changes to the customer more often, get their feedback, and incorporate that feedback into future releases. Iterative development allows you to get some incremental value from your work right away, instead of waiting until the “big bang deployment” when all features have been completed and are all released at once. Big bangs are riskier because you haven’t received any customer feedback yet or seen how the customer will use your software.

Failing Fast means knowing what the customer really wants
In agile, we talk about “Failing Fast” because if a customer doesn’t like or doesn’t use a feature we are working on, we want to know that as soon as possible. That way we don’t expend any unnecessary effort on developing a feature that the customer will never use (also referred to in some cases as “gold plating”).

I like the term “Failing Fast”, but it’s important to understand (as one of the MSMIT students pointed out) that the failure we are willing to accept in agile is failure of the business value. In other words, if we are building something that doesn’t deliver the expected value to our customers or our business, we want to know about that failure sooner rather than later, so we can adjust to it.

Failure does not mean failure in the quality of the code.
Code we deploy in agile development methods should still be well tested (preferably using Test Driven Development methods and Continuous Integration), and we should be highly confident that what we are deploying will work as designed. We just aren’t confident yet if the customers will like what we have designed – and that is what we want to find out as soon as possible!

Tip #2: Avoiding Bad Standups

Another particularly interesting discussion we had was about what makes a good “standup.” Many teams believe they are agile because they have standup meetings. The reality is there’s a lot more to agile than daily standups, and if that’s all you’re doing, you’re probably not even doing the standup correctly in an agile way.

Basic rules to follow when conducting your standups

• Stand ups should be no more than 15 minutes.
• Each developer only discusses the three items (What did I do yesterday, What am I doing today, Am I encountering any obstacles)
• All other items of discussion should be taken off line after the stand up, and only involve interested parties, not the whole team.
• Product owners are always invited to the stand ups, but they should be a listener only. If they have questions they can bring them up with individual developers offline.
• As I saw posted on a discussion board recently, standups should be “by developers and for developers.” They should be run by developers, not project management or clients.

Make sure everyone on the team values the standups
One team lead in the class asked how rigidly I believe you should adhere to the 15 minute length. He said that his team is spread out remotely, and so this is often their only form of communication daily, and that their meetings are more typically 30 minutes, but that it seems to be going well.

My first question was “Who thinks it’s going well at 30 minutes? You as the team lead? Or have you asked your team?” I was intentionally being slightly confrontational to make sure he realized that the point of the stand up is developer communication, and to see if he had a true sense of what his developers thought of these daily meetings.

The pitfalls of standups “in name only”
Often times when the stand ups are run by traditional project managers, they quickly become long daily meetings, and the only person who actually likes them are the project managers or team leads. The developers are bored, get distracted and annoyed that they aren’t doing something more productive (like developing code!), and they tune out. Because they tune out, they aren’t even listening to what the other developers are saying, and they lose all benefit of the conversation. The conversation becomes a hierarchical conversation between each individual developer and the project manager running the meeting, not a conversation between developers.

Those type of “standups” lose their value because when developers aren’t talking to each other or paying attention, they will not offer to help remove obstacles for each other or offer advice. It becomes more of a traditional command and control situation and de-emphasizes the collaborative nature of productive Agile teams.

The particular team lead who asked the question was perhaps a bit taken aback by my response, and paused a moment to consider it. Ultimately he said that his team does also see value in the 30 minute daily standups, but I hope that my response gave him pause to consider if everyone really saw the same value in longer meetings that he does. It’s always worth asking your team!

Dealing with remote teams in a stand up

With remote teams, it’s understandable that you might have longer phone calls so people can socialize a little bit (just a little though please!). But in those cases, as I recommended to the team lead, at least still focus on keeping the stand up itself to only 15 minutes. If people want to stay on the line and discuss things for another 15 or 30 minutes, that’s fine.

But if you get the real standup done first within your 15 minute time limit, then at least those who are not interested in socialing or further discussing that day will feel free to leave the call. By incorporating more detailed discussions and socialization into the stand up itself, you are forcing everyone to stay around for the whole call every single day. And that is definitely not going to be efficient for everyone.

In conclusion…

It’s often said that you can understand the basic concepts of agile in a couple of days, but it takes a lifetime to master. The concepts of iterative development, failing fast, and stand ups really are pretty simple if you stick to a few simple rules. But if you don’t master the underlying values of why you are doing Agile, and trying to empower your whole team (not just project management!), then you will inevitably revert back to a control & command environment and lose the benefits of whatever agile practices remain.

I’m pleased to be doing some follow up agile process consulting work this week for one of the students’ teams. If OpenSource Connections can help you improve your adoption of agile, please don’t hesitate to contact us for a consultation!

The Agile CIO is a series of blog posts advising IT leaders on how best to incorporate Agile techniques into their organizations. For more information about OpenSource Connections’ Agile process consulting services, please contact the author at ASime@OpenSourceConnections.com.

Where are Eric and Youssef?

This year two OSC folks are working remotely for extended periods of time. Youssef spent 6 weeks in Lebanon earlier this year, and I am spending the month of July in the mountains of Western North Carolina. We had talked about some of the rewards as well as challenges of working effectively when you are in a new/fun/exciting/[insert superlative here] place. Youssef and I came up with a few tips to make balancing work/play easier!

Tip 1: Establish a Routine

Youssef says: There will be no shortage of distractions, specially during the World Cup season. A trip to the mountains, a music festival or a friend’s unannounced (or announced) visit are all good reasons to drop what you’re doing and enjoy the place you are in. The only way to both get your work done and have fun is to know when is the appropriate time for each. I found that establishing a routine worked best for me. I woke up early every morning and took advantage of the early quiet hours of the day. Started work at 8am and got plenty done early on. Then I gave myself a couple of hours of break time for unscheduled events (a friend’s visit, a trip to the store to get souvenirs, or even a couple of hours at the beach). Then I resumed work till the evening when I had a conference call scheduled every day at 5:45pm. After the call I was free to do whatever I wanted and that gave me plenty of time, specially in a city like Beirut that does not sleep.
Eric says: I know that I sometimes struggle to get started working in the morning. And being in the office makes it simpler because everyone else is working. So having a specific schedule of when I am working helps me shift my brain out of family/vacation mode and into work mode!

Tip 2: Fast Internet Connection

Eric says: When we looked for cottages to rent, one of our requirements was that they have something better then dial up! We got lucky that one of our top choices was just installing a HughesNet satellite link at the cottage.
Youssef says: It is embarrassing to be a on a call with a client and have that call be constantly dropped because of a bad connection. Skype requires a decent bandwidth and fast connection to work properly. You do not want a 15 minute stand up call to turn into a half an hour of you asking the people on the other end “Can you hear me now?” (Verizon does not operate in foreign countries!). So make sure you get yourself a decent internet connection even if you have to spend a little extra on it. And always take any install CDs or DVDs you think you might use with you. Some countries do not have the proper infrastructure and thus impose a cap on the download/upload limit you may use, which makes you think twice about how you use that limit (downloading the new Xcode from apple’s website is not an option).
Eric chimes in: I ran into the same Skype issue, turns out Skype doesn’t work well with satellite internet due to the latency issue! Fortunately, if I am upstairs on the west side of the cottage, standing on one foot, with my finger in my ear, my iPhone gets 1 bar of signal! Phew. I did notice that I needed to talk a little slower and not move about too much and folks can hear me.

Tip 3: Regular Checkins

Youssef says: Stand up calls in Scrum keep the developers honest. When you are working remotely, you do not want to lose touch with the work environment, specifically in a field that evolves constantly such as software development. Having a point of contact in the “real world” or the “work world” helps you stay in touch and not lose focus.
Eric says: When you are in the office everyday, you pick up all the water cooler gossip. But working remotely means you lose that contact. I find that having a regular series of checkins helps stay connected. I put them on my calendar so they don’t get skipped!

Tip 4: Track your time!

Eric says: Don’t forget that you are working remotely is because you are doing something special. In my case, it’s taking my family for a month to the mountains. So I want to make sure that I don’t miss out on the experience. Over the four weeks I am here, I am taking a week of vacation, spread out a bit. So I am tracking my hours both to make sure I don’t shortchange work, but also don’t shortchange the experience of being 200 feet from the Appalachian Trail!
Youssef says: I definitely did not travel for 24 hours, crossed an ocean, a continent and a sea to do the exact the same things I was doing in Charlottesville. It also does not mean I should use that as an excuse to ignore my responsibilities. Tracking your time helps you balance out the experience and establishing a routine like we mentioned above goes well with this concept.

Tip 5: Stay Agile

Youssef says: Working remotely does not mean you should change the way you work, but flexibility is key. Working on the east coast while your client is on the west coast requires some coordination. Working in Lebanon (GMT + 2) while your coworker is on the east coast and your client is on the west coast, well that requires some bending and flexibility. The key part about dealing with this situation is an agreement with all parties involved about which approach to follow. The method I followed included three chunks of time. A chunk where I was working ahead of everyone else (during which I could be dealing with issues that came up the day before), a chunk of time where everyone was available at the same time and third chunk when I was off the clock while everyone else was still working. That allowed for ample communication and work time without sacrificing my fun time. Others might prefer to synchronize the working time, or might prefer to just deal with the delays. Balance worked best for me.
Eric says: I agree with Youssef, being flexible on your approach to work can make being remote a very enjoyable and productive experience!

Looking out of my window at NC Mountains

But multi-disclipinary skill sets are not the only key to fluidly switching people between tasks. You must also be able to change technology or development environments easily as well, and this is an area that I find almost all development environments suffer from.

As an IT leader, when you bring in a team of consultants to augment your team and give their velocity a shot in the arm, you expect them to integrate quickly and be productive right away. And that’s what the consultant wants too. But are you giving them the tools they need to be effective right away? Or do you expect each of them to go through the same development environment setup that you ask new hires to do?

When you hire a new person, perhaps it’s worth the time to have them setup everything about their development environment. They learn about the nuances of your system, and a few days or a week of their time is nothing compared to the long term investment you expect from having them work for you for several years.

When you bring in a consultant, it’s different. You may only expect to have them for a few months, and their time is more expensive then your internal staff. And they are not going to benefit much from the learning experience of setting up a dev environment.

And even for your full time staff, what happens when they have to upgrade their machine a year from now, or their system dies and they have to rebuild from scratch? Do you really want to have them spend another week trying to remember how to get past that one install bug that they encountered a year or more before?

One answer to this problem is to develop code using virtual machines. Instead of having your first developer start by setting up the toolsets and plugins on their machine, have them start with a clean virtual image, using a tool like VMWare Player. Keep a bare bones image around with your basic development tools, or start with one from a similar project. After the developer gets the code started and everything basically working, then they should make a copy of that image and distribute it to the rest of the development team.

A backup copy of this image should also be kept on a network share or external drive so that when you add in consultants, or bring in a new hire, they can get started on code just as soon as they copy the image over to their machine. Instead of spending a week getting everything installed and picking the brain of your development staff, or Googling on obscure error messages, the only time delay is how long it takes the image to copy to their local machine (probably no more than an hour in most cases).

Now you can spend their time where it’s valuable: Bringing them up to speed on the nuances of your code itself, not the right service pack or magic incantations to get the development tools working.

You might be thinking that with cloud computing and languages like Ruby on Rails, shouldn’t this be a non issue? I just pull down the code from svn or github, and I’m ready to go! In my experience, while that is usually the case with relatively small projects or with projects started in the last year or two, it’s not always the case with lots of other code that exists in the world and that we have to work on.

I was reminded of this because of a client I recently worked with is still using version 1.1 of the .NET framework, and so is required to run Microsoft Visual Studio 2003. As part of the project we’ll probably end up doing an upgrade to more recent versions, but we still have work we have to do first. And unfortunately I discovered on my first day on the project that Windows 7 does not play well with Visual Studio 2003. Once it installed, it wouldn’t connect to IIS in debug mode. After doing some googling and seeing that this is a common problem with no easy solution (I tried many of the recommended solutions to no avail), I decided it was time to just get an image of Windows XP going. The rest of the dev team is on XP, but they are not using virtual machines to do it unfortunately.

So I started creating a new XP image under Virtual PC, only to have it started failing on certain files. Even though the install continued, I ended up with an image that could not talk to the outside world and was clearly half-baked.

So then I started over with VMWare Player, and as I write this, my XP image with Visual Studio 2003 already installed is nearly done. 3 days later. Before I start changing code on it, I’ll be setting the image aside to give to another consultant joining the team in a few weeks, so he doesn’t have to go to the same level of pain. I’ll also make sure the client has a copy of that image to make their life easier in the future the next time they bring us in or make a new hire.

Before you attribute the delays I experienced to Microsoft, I’ve seen similar challenges on Php projects and to a lesser degree on Groovy on Grails projects.

Virtualization is great for reducing the size of your server room, cutting down on power bills, and becoming more green. But don’t let it stop there. A little bit of time spent up front on virtualizing your development environment will save you a lot of heartache and expenses later down the road, and make your IT shop much more agile.

The Agile CIO is a series of blog posts advising IT leaders on how best to incorporate Agile techniques into their organizations. For more information about OpenSource Connections’ Agile process consulting services, please contact the author at ASime@OpenSourceConnections.com.

AgileCville
Date: Tuesday, June 15th, 2010
Time: 6 PM to 7:30 PM
Venue: OpenSpace
455 Second Street SE, Suite 100
Charlottesville, VA 22902

Your development team will likely respond to your calls with heroic effort. And you might actually make the deadline. But if you do it regularly, your reputation as a leader may actually suffer, and your team will probably burn out unless you can pour on enough accolades and spot-bonuses.

At some point, almost all IT leaders, including myself, have done it. That is unfortunately not much of a secret and it’s why IT teams often develop a bad reputation. The bigger secret is that a lot of developers not only regularly take on the hero role, they actually like it.

This creates a dangerous situation, where IT leaders ask for heroic efforts because it is the path of least resistance. And the development teams respond with grumbles and late nights, but also with the misconception of job security and the expectation that they will receive a lot of praise when they come in on Monday bleary-eyed (but with at least part of the job done).

Why is there so much danger in encouraging your developers to be heroes? Do a google search on “cowboy coders” or “programmer heroes” and you’ll see other blog posts that describe the whys and the pitfalls of reliance on hero development. In short, you’ll get inconsistent and buggy code that may not fit your architecture, won’t meet the customer’s needs, is not very flexible, creates technical debt you will complain about for years, and will probably need to be reworked eventually.

So how do you avoid heroes? As a CIO or technical lead, the tools of Agile offer many solutions:

• Sprint planning – restricting the work that is done to the current sprint helps discourage sudden changes of priority. Most tasks can in reality wait until the next sprint to be added, especially if the customer understands that it means the current sprint has to be put on hold or canceled.
• Daily standups and regular customer interaction – Chronic hero programmers often like to hideaway on their work, and so they can be more easily detected when you are interacting with the customer each day. There is less opportunity for the developer to hide away and not give status updates, and the standups provide the customer more immediate feedback on everything that is being held up because they sent the hero on a distracting quest.
• Team estimation – When the team estimates tasks for a sprint together, before they are assigned to people, that encourages open discussion and debate on how long something will take.
• Continuous Integration – Maintaining rules about daily code check-ins and regular integration and unit testing prevents the hero from going too far away from the mainstream code base, and gives visible signs of progress. When done properly, it also helps to reduce the bugs delivered in the rushed code.

In addition, in our OpenApproach implementation of Agile and Scrum, we use range estimation of tasks. Asking people to give you a single-point estimate of how long something will take encourages them to be overly optimistic. “Oh, I can get that done in about 2 hours.”

Really? What if it’s not as simple as you or the developer thought? Then the only solution is to put in a heroic effort if the timeline must be met.

A simple range estimate will provide a much more realistic view, and encourages a more thorough thought process: “It should only take about 2 hours of actual coding, but I’ll have to switch projects and get that site running again on my machine. Plus, sometimes those java script issues can be really though to debug, so it could take as long as 8 hours of work.”

As an Agile CIO, it’s your job to not only be understanding of your developers when they give you a range estimate, but encourage it. Don’t settle for the heroic answer of “I’ll just put in some extra time this weekend.” Get to the root cause of the problems, and make sure the customer or your boss is aware of the delays earlier rather than later. It will make all of your lives easier, and help you to avoid the pitfalls of programmer heroes.

The Agile CIO is a series of blog posts advising IT leaders on how best to incorporate Agile techniques into their organizations. For more information about OpenSource Connections’ Agile process consulting services, please contact the author at ASime@OpenSourceConnections.com.

B.D.D (Behavior Driven Development) which is a spin off of T.D.D (Test Driven Development) focuses on the language of testing and and quality assurance of software, in hopes to make creating tests easier as well and specifications.

The concept was coined by Dan North. Since then new libraries like rspec, cumcumber, specflow, nbehave, machine spec have come about.

I can agree that sometimes changing the business language can help people with making associations and understanding of processes easier. And using “Behavior” instead of “test” does help in understanding that you need to validate how the software or code is supposed to behave.

Questioning The Boilerplate Human Readable Specifications

Where people need to be more skeptical is the question of is there real business value in getting code to generate boilerplate systematic specifications?

For an example, I’m pulling this from a cucumber example from github.

# language: en
Feature: Addition
  In order to avoid silly mistakes
  As a math idiot
  I want to be told the sum of two numbers

  Scenario Outline: Add two numbers
    Given I have entered <input_1> into the calculator
    And I have entered <input_2> into the calculator
    When I press <button>
    Then the result should be <output> on the screen

  Examples:
    | input_1 | input_2 | button | output |
    | 20      | 30      | add    | 50     |
    | 2       | 5       | add    | 7      |
    | 0       | 40      | add    | 40     |

We live in a time surrounded by constant news, media, following four subplots instead of one. Its hard to get people concentrate for long periods of time or sit still, much less to get them to read technical or requirement specifications.

Now I was to write this scenario for a requirement out as a spec, it would read like the following.

The mad scientist2 with his destructo calculator enters input 1
and input2. He manically presses said red button . The output
must be the sum of input 1 and input 2.

(i1 + i2 = output)

So why are we creating specifications that repeat words like I,As,When,Given,Then like bad boilerplate code? They might be human readable, but the questions of being understandable, easy to read, enticing to read, is a whole different story.

I have to honestly question that even if a developer takes their time and makes sure the code is up to spec. The question is are the specs kept up to day (requirements change). And are lay people going to want to sit through and actually read these things?

Is this a real business value?

I don’t see any at this point. I see no real return on time invested to write boring systematic specs and just to have them match up in some way with code.

Specifications, like any documentation worth it’s salt, needs to be able to:

• be engaging and appeal to more than the logical side.
• provide the meat of information that is digestible.
• avoid templating that read like government contracts.
• be terse and yet expressive.

We have been here before

Their have been tools like this before, (Fit, Finesse) before Dan North was really doing this thing. Yet you don’t see much usage from them across projects.

I think the reason why cucumber has caught on, is because the rails community has been keeping momentum from going from one “next big thing” onto the next one. Cucumber is the new pink.

But the end result is the same. Less than specular on par with boring. Hard to digest boiler plate specifications.

Don’t get me wrong, having specifications to work from is important. It helps to drive development and allows others to collaborate and work together. But this doesn’t mean B.D.D is on the right track with its story/QA methodology.

I think there is value of translating specs into code to some degree and testing against requirements.

I think the value is lost when:

• coders who might not fully understand the business are writing specs.
• business analysts who have no clue to code write specifications that affect how code is written.
• specs written  in a templated manner that is the antithesis of being terse.

Should vs Must

Also the domain language is for B.D.D. a bit off.

If you’ve read technical specs, you know there is a vital difference between “Should” and “Must”. Should is something software could do, but doesn’t have to. Must is an absolute requirement. If you don’t believe me, check out the W3 specs sometime.

Yet, we’ve chosen “should” for some reason to put everywhere inside of our “specification” code. If we’re going to write specifications, we must chose words that fit.

Conclusion

B.D.D. isn’t evil or bad, but still needs to evolve, be debated, thought on, expounded.

I think the software community as a whole needs to watch out from getting caught up too much in trends for the sake of being first to do something and buying the t-shirt to say he/she was there.

This time, we decided to mix up the way we do hackathons. We added in a theme, voting on the best project, celebrity judges, a gift certificate to Best Buy, and even a trophy! The usual elements of time pressure, creativity, fun, caffeine, and a beer at the end of the day were still present.

At the end of the day: Scott Stults was crowned the inaugural OSC “Code Ninja”. Congratulations Scott! In a moment I will describe a little more about the hackathon, but first, let’s all sit back and enjoy this impressive photo of Scott with his prized trophy. Scott is making a feeble attempt to strike the same pose as the karate-guy on the trophy.

For the hackathon, we started the day by meeting at the Nook for breakfast. Then we headed over to a conference room at CitySpace where we set up camp for the day.

The theme of the hackathon was “time”, which meant that you could build any application you wanted, as long as there was some element of time in it. The app had to be built in the 7 or so hours of the hackathon, and judging began at 4:30pm. The applications were judged on three axes: Innovation, Business Utility/Usefulness, and Completeness. Each person voted on all the projects except for their own, giving 1-5 stars for each app on each of the three axes. The project with the highest total number of points was the winner.

To add another dynamic to the voting, we invited “celebrity judges” to join us at 4:30 and they had votes as well. Our celebrity judges were Glenn Wasson and Brian Wheeler. Glenn is an architect at SAIC, and is perhaps most famous for being one of the founders of “The Oracle of Bacon” website. Brian is famous for the Charlottesville Tomorrow website.

Both judges were excellent, and very gracious with their time since we kept them there until nearly 6pm on a Friday night. Thanks Glenn and Brian!

We had a variety of applications built. Caleb worked with Html5 to build a dynamic graph for tracking stocks. Michael worked on extending unit testing frameworks in Visual Studio. Eric built a website where you can send your server logs to, and it will stream out audio sounds that you can listen to which indicate what types of messages are being logged (errors, info, etc). Youssef and I teamed up to build a droid app and deploy it to my phone. We had some cool ideas, but ended up scaling them down to building an Agile standup meeting timer. It made my phone vibrate, which was of course extremely cool.

Despite that coolness, and despite a strong challenge from Eric, the inaugural code ninja is Scott! Scott didn’t just win because he was the only one of us to prepare a powerpoint presentation about his app (that actually lost him a few votes I’m guessing – yuk yuk), but because he made wiki’s exciting! Scott used SPARQL in Semantic Media Wiki SMW+ to show how we could track our projects at OSC and see timelines of who is working on what project.

Thanks to everybody for a great time – I’m looking forward to the next hackathon! I understand that Scott has removed family photos in order to place the trophy in a place of honor at his home, and so I wish the next Code Ninja the best of luck in wresting it away from him.

Building a More Accurate Burndown: Using Range Estimation in Scrum

Traditional Scrum burndowns are based on single point estimates of how long a task will take. However, single point estimates are inherently faulty and inaccurate, and they encourage underestimation. Learn how to incorporate range based estimation techniques into your Scrum burndown, and better communicate to your boss or clients what a project is really going to take. Arin will back up this thesis with academic and industry research, real world examples, and an engaging presentation style. Participants will leave with concrete tips & templates for using range estimates in their projects.

Process/Mechanics

The first part of the session will focus on the pitfalls of traditional estimation techniques, in particular single point estimates. This will involve audience interaction as well as citing industry and academic research. Next the session will move into a discussion of range estimation and how that leads to better accuracy. Then I will present specific advice for how to use range estimates in common Scrum practices like Scrum poker and the burndown. Along the way I will show examples of how our company has been using this practice with clients over the last year, and some best practices that have come out of it. The techniques will be fairly simple and easy to adopt, but provide power results in estimating more accurately and better communication with managers and clients.

Potential pitfalls and best practices of range estimation will also be discussed. For example, some managers are reluctant to accept range estimates, but they can be convinced when a demonstration is made to them of how it better communicates risk and allows them to make better financial decisions on the viability of a project.

This topic started for me with a paper I wrote in my Masters in Management of IT program in 2009 at the University of Virginia, and now has become regular practice at OpenSource Connections. I am also pursuing further research on the topic with two professors from the University of Virginia (Professors Ryan Nelson and Mike Morris of the McIntire School of Commerce). Any preliminary findings or papers from that research will also be woven into the presentation.

I have proposed this as a 60 minute talk, but it could also be made into a 90 minute Tutorial with relative ease. In that scenario, I would bring more hands on activities to encourage participants to really see the value of range estimation in their projects. I have experience with those types of sessions from helping to teach corporate education classes at Virginia Commonwealth University, and I would use that experience to make sure things are kept interesting.

It’s my intention for this session to take the relatively easy-to-use but underutilized practice of range estimation, and make it so compelling that participants will want to apply it right away on their projects. This will be done through a combination of good research, real world examples, engaging presentation style, and practical tips.

Learning outcomes

* Why single point estimates lead to underestimation
* Why range estimation reduces natural biases
* How range estimates better communicate a project’s impact
* How to use range estimation in Scrum Poker
* How to incorporate range estimates into a Scrum burndown
* How range estimates impact project decision making

For those registered on the Agile 2010 site, you can also see this session here.

After much faffing about, I finally figured it out.  On the server I would ask for the set of timezones that matched the offset, and grab the first one and put that in the session:

[sourcecode]
result = ActiveSupport::TimeZone.all.select{|t|t.utc_offset == gmtoffset}.first
session[:time_zone] = result.name
[/sourcecode]

The list of named time zones returned when the browser is in Charlottesville, Virginia are: Bogota, Eastern Time (US & Canada), Indiana (East), Lima, Quito.

So when I use Bogota as the timezone, and ask Rails to show the time localized:

[ruby]
Time.now.in_time_zone(session[:time_zone])
[/ruby]

I get back the time without taking into account daylight savings wrong. I started trying to figure out if the browser was in a DST zone using this JavaScript code: http://www.michaelapproved.com/articles/daylight-saving-time-dst-detect/ and while it seems very promising, it still wasn’t quite giving me what I want.

Finally, I realized it…. By arbitrarily grabbing the first time zone in the list, I was showing time in Bogota, Columbia. But if I chose Eastern Time (US & Canada) then I do get a localized time that takes into account day light savings!

So right now I have this method:

[ruby]
result = ActiveSupport::TimeZone.all.select{|t| t.utc_offset == gmtoffset && t.name.include?("US")}.first
session[:time_zone] = result.name
[/ruby]

Obviously this is pretty hardcoded to just work in the US, and isn’t a real solution. I’d love to hear other ideas! Part of me wonders if I should just display all times in UTC in HTML, and have some sort of client side JavaScript that localizes the time display?

My full set of code:
Javascript in my index.html.erb view:
[javascript]
// Calls the server and sets the user’s time.
Event.observe(window, ‘load’, function(e) {
var now = new Date();
var gmtoffset = TimezoneDetect();
//use ajax to set the time zone here.
var set_time = new Ajax.Request(‘<%=url_for :controller => "home", :action => "gmtoffset"%>?gmtoffset=’+gmtoffset, {
onSuccess: function(transport) {
//alert("Response" + transport.responseText);
}
});
});

// http://www.michaelapproved.com/articles/daylight-saving-time-dst-detect/

function TimezoneDetect(){
var dtDate = new Date(’1/1/’ + (new Date()).getUTCFullYear());
var intOffset = 10000; //set initial offset high so it is adjusted on the first attempt
var intMonth;
var intHoursUtc;
var intHours;
var intDaysMultiplyBy;

//go through each month to find the lowest offset to account for DST
for (intMonth=0;intMonth < 12;intMonth++){
//go to the next month
dtDate.setUTCMonth(dtDate.getUTCMonth() + 1);

//To ignore daylight saving time look for the lowest offset.
//Since, during DST, the clock moves forward, it’ll be a bigger number.
if (intOffset > (dtDate.getTimezoneOffset() * (-1))){
intOffset = (dtDate.getTimezoneOffset() * (-1));
}
}

return intOffset;
}

[/javascript]

home_controller.rb action:
[ruby]
def gmtoffset
gmtoffset = params[:gmtoffset].to_i*60 if !params[:gmtoffset].nil? # notice that the javascript version of gmtoffset is in minutes

result = ActiveSupport::TimeZone.all.select{|t| t.utc_offset == gmtoffset && t.name.include?("US")}.first
session[:time_zone] = result.name

render :update do |page|
page.replace_html ‘time_of_chat_starting’, :partial=> ‘super_short_time’
page.visual_effect :highlight, ‘time_of_chat_starting’
end
end
[/ruby]

Rendered partial helper view _super_short_time.erb:
[ruby]
<%= super_short_time(Time.now.in_time_zone(session[:time_zone])) %>
[/ruby]

If you’re a geek in or around the Charlottesville metroplex or even if you’re merely tech-curious, this is the event you don’t want to miss. beCamp is Charlottesville’s version of the BarCamp unconference phenomenon—organized on the fly by attendees, for attendees. Realizing that the most energizing parts of any tech conference are the ad hoc conversations that take place in the hallways between the sessions, beCamp facilitates these types of interactions for an entire event.

As of this writing, we are at 87 campers! To participate, just add your name to the wiki page!

A big thank you to all our sponsors, including at this point,  Hotelicopter, Google, Perrin Quarles and Associates, NRAO, and University of Virginia ITC.  Interested in supporting the Cville tech community?  Check out our needs at http://barcamp.org/sponsor-beCamp-2010.

Annotation is the process of selecting portions of an original document and adding additional information. These additions are not necessarily saved with the original and are commonly used by editors and scholars to include background citations. In Computer Science, annotations are used in Semantic Web and work collaboration technologies. The Comment function in MS Word is an example of using annotations for collaboration.

Business uses

Aside from these functions I think that there is a whole class of business processes that could benefit from annotation technology. For example, we often copy line items from a Request For Proposal (RFP) and paste them into our proposal. Later, these line items may get pasted into a Product Backlog, Production Readiness Checklist, or ticketing system.

As a programmer, I shudder whenever I see so much copy-and-pasting. If a block of text is important enough to be present in multiple documents, it should be extracted as its own entity and reused. This eliminates the problem of maintaining that block in multiple places and ensures that each instance of the block is accurate. More importantly, it helps you concentrate on adding value rather than repeating yourself (see the DRY principle.)

Categories of technologies

Annotation technologies can be split into two broad categories based on where the annotations are saved: In a server or in the document. Annotations that are saved centrally in a server lend themselves to collaboration because multiple people can update the annotations without having to modify a master document. On the other hand, document control may be more important than collaboration, in which case storing the annotations within the document would be more appropriate.

Example technologies

While Googling for what’s around the web regarding Annotations, I ran across Ian Lumb’s blog.  He’s got a number of excellent posts on the subject.  I also found a number of different technologies listed over at the Semantic Web portal, but the list appears to be dated (several were private projects, there were a couple broken links, and a few of the projects appear to be unmaintained.) However, there were enough working projects there for me to get an idea of what people are doing in the field of Annotations. The promising projects I saw there were:

• Annotea
• Annozilla
• GATE Teamware
• Semantic MediaWiki+
• KIM Semantic Annotation Platform
• Nuxeo Document Management

Annotea is more of a protocol than software, but there were a number of client and server implementations listed (Annozilla was one of them). The Zope server product ZAnnot was a breeze to install on top of a fresh download of Zope, and I was able to get it working with Annozilla pretty quickly. Annozilla itself, though, needs a little TLC before I can incorporate it into a working system.

First of all, Annozilla hasn’t been updated in about nine months and requires FireFox 3.5 (current version is 3.6 and I reinstalled with version 3.5 just to try it out.) Secondly, the annotations themselves are free-form and cannot be constrained or reused (such as with an ontology). This is a well-known problem in the world of tagging content, where search-as-you-type tagging helps you avoid multiple tags that are almost but not exactly the same (such as the tags “annotations” and “annotation”.)

GATE (General Architecture for Text Engineering) is an amazing collection of projects centered around doing things with text. I found only one technology in there that was relevant to annotations, and that was regarding automatic annotations (examining text and feeding annotations into Annotea.) There were a lot of other interesting libraries in there and I hope to check them out later.  Specifically, they announced a Teamware application would be forthcoming which would incorporate annotation technology and workflows.

Likewise, the KIM Semantic Annotation Platform seemed to be oriented more toward automatic annotation rather than streamlining human annotation.

Nuxeo Document Management is a Java-based product very similar to Alfresco.  It has a document preview feature that shows you an HTML version of Word or PDF documents it stores, and also has an annotation module that lets you annotate that preview.  [edit:  When I originally installed it I rushed through and misread some of the documentation.  Stefane Fermigier, the founder of Nuxeo saw this post and corrected me, but I have yet to revisit Nuxeo.  In the interim I've edited this post to remove my misinterpretations.  Hopefully soon I'll be able to follow up with a more in-depth look at Nuxeo.]

Lastly I gave SMW+ a try.  I’ve used Semantic MediaWiki before and I think it has great promise.  I especially like the rich report formats that you can generate with semantic queries (like showing a SIMILE timeline as the result of a time series query.)  There was a long list of extensions to install, but I eventually got to the point where I could copy and paste a whole RFP as a wiki page and annotate it.  The annotation tool was AJAX-based and a little clunky, but once I was done I had a nice report of each annotation in the original document.  It wasn’t clear how to bind an ontology to it, so I have the same complaint with it as I do with Annozilla.

Parting Thoughts

The kind of annotation I want to do seems possible using some of the WYSIWYG editors embedded in most blogs and CMS (like kupu or TinyMCE). A colleague of mine noted that you could simply supply a custom CSS style that would be included in the menu of available styles, but refrain from adding any style changes to it. So in theory any CMS which provides document preview ala Nuxeo should be able to supply an annotation editor which wouldn’t change the original. That to me is the most promising direction.

If you have any experience with similar technology please let me know in the comments.

Opensource software on the Microsoft stack has tons of stored potential, even some movement, but it is still left wanting. The evil empire has embraced opensource software, releasing jQuery with Visual Studio, starting codeplex.com and the Codeplex Foundation.

The are tons of abandoned projects or ones that have gone stale (log4net anyone?). Compared to java or even the new kid ruby, we’re lagging behind. We don’t even have a fully managed opensource enterprise web server, compared to Java’s n-th variety of containers to pick and use. Even rails has a built in server.

Of course there is kayak http web server framework and webserver on codeplex, but they’re new, far from mature and their not an industry defacto standard like jboss or tomcat.

Don’t get me wrong there are some amazing opensource projects out there. Gallio, db4o, Subtext, blogengine.net, facebook developer toolkit to name a few.

However, there are gaps in having a full opensource Microsoft stack. We have plenty of unit testing and mock testing libraries, but with NDoc gone, left to SandCastles release schedule, libraries left to rust like log4net wit not even .net 4.0 beta build or silverlight build. CruiseControl.Net is in dire need of revamp and version 2, at the very least it needs some decent competition that isn’t java.

With plenty of single person projects out there who just end up getting burnt out, seemingly stagnated public dialog from the likes of the code plex foundation, its hard to really get developers to rally and get some much needed things done.

The community need some decent leaders, organizers, and some company backing. Most software vendors and clients get a great productivity boost from these projects. It would only make sense to invest in their growth, even pool resources for joint projects.

Organize some hack-a-thons days with some cool prizes for work top-notch work. Even put together a small guild of programmers, just have 20 or so companies pitch in, put their banners on a website and churn out some decent open source projects that everyone can use.

.Net isn’t going away and its time the community and companies invest more into the opensource community instead of letting all that potential go to waste.

Want to use a favicon.ico but don’t want to put it at the root as favicon.ico?  Then add <LINK REL=”SHORTCUT ICON” href=”<%=image_path(‘limelight.ico’)%>”>.   The use of image_path means that you get all the goodness of Rails routing to generate a complete image path that will work.  Even if you deploy under some sub URL, like a war file in JBoss!

Are you working with ISO-8859-1 encoded text, (more info at  at http://www.w3schools.com/tags/ref_entities.asp).

ISO 8859-1 Characters

And look at the character À  (should be a capital A with a caret on top called a grave accent).  That will kill Solr, regardless of container deployed in, like Jetty versus Tomcat.  But the other ways of representing this entity work great:

Of course, there is a bit of confusion on this, as supposedly if the XML document posted to Solr is UTF-8 encoded, then Solr shouldn’t have any issues.  So, still some digging to do!

Solritas and JBoss and Velocity Oh My

I recently ran into a Java Classloader issue between JBoss and Solr when loading Velocity. If you are getting in the browser:

TTP Status 500 – loader constraint violation: when resolving method “org.apache.velocity.Template.merge

or messages like “SEVERE: java.lang.LinkageError: loader constraint violation: when resolving method “org.apache.velocity.Template.merge” in the logs, then that means a conflict in the velocity jars.  Oddly enough, I could not actually find a velocity.jar anywhere in my JBoss app.  However, the fix was to copy the velocity jar from Solr into my JBoss ./lib/ directory.

Solr

Sunspot is the up and coming solution for integrating Solr into Ruby on Rails, and fortunately enough, the 1.0 release (followed quickly by 1.0.1!) has just come out last week.  Between acts_as_solr and Sunspot, Sunspot wins hands down for it’s support of a master/slave Solr configurations, embedded Solr for testing, richer indexing semantics, and not being tied to ActiveRecord.  The companion sunspot_rails gem does give wonderful ActiveRecord integration however.

Solr cores are the bees knees!  We’ve built a simple RoR webapp using HTTParty and the Solr API that allows you to perform all the admin functions for cores, and allows you to quickly clone a core for your own nefarious purposes!  Simplifies hacking around with a new schema or configuration without having a local copy of Solr running.  Allows multiple QA environments to potentially share a single Solr infrastructure.

Solr master and slave setup in a single VM.  While pointless from a scaling perspective, it’s a really great way to work out the kinks!  It’s funny to see a slave core polling the same Solr VM its in for updated segments!

JRuby

Doesn’t suck after all.  Actually, maybe I should say that JBoss, when combined with JRuby, means that JBoss doesn’t suck so much.  I had the aforementioned Solr core admin tool bundled up as a WAR file with JRuby, and was able to deploy it to an existing environment that had JBoss installed!  I didn’t have to install ruby on the box, (or JRuby for that matter!)  I just deployed the WAR file and bamn, off to the races.  Ops folks get the JBoss they love, I get the Ruby on Rails that I love.

And on a related note, Warbler was the key to thinking JRuby is cool.  I’d never actually had to package up a RoR app, so Warbler came to the rescue.  And you know what?  It was nice to build a single file that I knew had everything that I needed in it that could be scp’ed around!  And thanks to some cool code in the environment.rb, my app was able to load up the right configuration file for the environment based on an environmental variable set in JBoss.

Virtual Machines

I recently migrated a Linux VPS based RoR + Solr app (see a trend in tech choices )  to a Windows environment.  And to deliever the new Windows environment, I used VirtualBox to host the Windows Vista environment on my Mac laptop.

A couple of notes:

• VirtualBox may not have all the snazzy integration points of Parallels with the host computer like seamless application sharing, but it seems to be much lighter weight.  Starts up quicker, and I don’t get the spinning beach ball of death as much.
• If you are shipping a 11 GB file, you can’t use a 16 GB USB Memory Stick…  Turns out the biggest file is 4 GB.  (Although I never tried formatting the stick as NTFS, maybe that would have allowed a single 11 GB file???)
• Uploading 11 GB to a remote out on the internet server will take a long long long time.  Even on a really fast network. connection.
• If you need to format an external USB hard drive as NTFS on a Mac, it is possible!  Just fire up your trusty Windows Vista image in Parallels, plug the USB drive in, download and install the correct USB drivers so the drive doesn’t show up as a network share mapped to the Mac, and then use the built in reformatting tools!  Warning: This will take a loooong time!
• Lastly, if you are using VirtualBox, and you attempt to create a Windows XP machine, and attach a Windows Vista hard disk image to it, VirtualBox will let you!  And then Windows won’t start.  sigh.