Agile teams should be based on talented development staff with multi disclipinary skill sets. This allows team members to easily trade tasks and keep the iteration moving forward. As a consultant, I have seen and worked in many environments where development staff are very silo’d in their skill sets. In most cases they are very good at what they do, but at various times each team member can become a bottleneck because they are the only person who knows how to do a particular task or work on a certain legacy system.

But multi-disclipinary skill sets are not the only key to fluidly switching people between tasks. You must also be able to change technology or development environments easily as well, and this is an area that I find almost all development environments suffer from.

As an IT leader, when you bring in a team of consultants to augment your team and give their velocity a shot in the arm, you expect them to integrate quickly and be productive right away. And that’s what the consultant wants too. But are you giving them the tools they need to be effective right away? Or do you expect each of them to go through the same development environment setup that you ask new hires to do?

When you hire a new person, perhaps it’s worth the time to have them setup everything about their development environment. They learn about the nuances of your system, and a few days or a week of their time is nothing compared to the long term investment you expect from having them work for you for several years.

When you bring in a consultant, it’s different. You may only expect to have them for a few months, and their time is more expensive then your internal staff. And they are not going to benefit much from the learning experience of setting up a dev environment.

And even for your full time staff, what happens when they have to upgrade their machine a year from now, or their system dies and they have to rebuild from scratch? Do you really want to have them spend another week trying to remember how to get past that one install bug that they encountered a year or more before?

One answer to this problem is to develop code using virtual machines. Instead of having your first developer start by setting up the toolsets and plugins on their machine, have them start with a clean virtual image, using a tool like VMWare Player. Keep a bare bones image around with your basic development tools, or start with one from a similar project. After the developer gets the code started and everything basically working, then they should make a copy of that image and distribute it to the rest of the development team.

A backup copy of this image should also be kept on a network share or external drive so that when you add in consultants, or bring in a new hire, they can get started on code just as soon as they copy the image over to their machine. Instead of spending a week getting everything installed and picking the brain of your development staff, or Googling on obscure error messages, the only time delay is how long it takes the image to copy to their local machine (probably no more than an hour in most cases).

Now you can spend their time where it’s valuable: Bringing them up to speed on the nuances of your code itself, not the right service pack or magic incantations to get the development tools working.

You might be thinking that with cloud computing and languages like Ruby on Rails, shouldn’t this be a non issue? I just pull down the code from svn or github, and I’m ready to go! In my experience, while that is usually the case with relatively small projects or with projects started in the last year or two, it’s not always the case with lots of other code that exists in the world and that we have to work on.

I was reminded of this because of a client I recently worked with is still using version 1.1 of the .NET framework, and so is required to run Microsoft Visual Studio 2003. As part of the project we’ll probably end up doing an upgrade to more recent versions, but we still have work we have to do first. And unfortunately I discovered on my first day on the project that Windows 7 does not play well with Visual Studio 2003. Once it installed, it wouldn’t connect to IIS in debug mode. After doing some googling and seeing that this is a common problem with no easy solution (I tried many of the recommended solutions to no avail), I decided it was time to just get an image of Windows XP going. The rest of the dev team is on XP, but they are not using virtual machines to do it unfortunately.

So I started creating a new XP image under Virtual PC, only to have it started failing on certain files. Even though the install continued, I ended up with an image that could not talk to the outside world and was clearly half-baked.

So then I started over with VMWare Player, and as I write this, my XP image with Visual Studio 2003 already installed is nearly done. 3 days later. Before I start changing code on it, I’ll be setting the image aside to give to another consultant joining the team in a few weeks, so he doesn’t have to go to the same level of pain. I’ll also make sure the client has a copy of that image to make their life easier in the future the next time they bring us in or make a new hire.

Before you attribute the delays I experienced to Microsoft, I’ve seen similar challenges on Php projects and to a lesser degree on Groovy on Grails projects.

Virtualization is great for reducing the size of your server room, cutting down on power bills, and becoming more green. But don’t let it stop there. A little bit of time spent up front on virtualizing your development environment will save you a lot of heartache and expenses later down the road, and make your IT shop much more agile.

The Agile CIO is a series of blog posts advising IT leaders on how best to incorporate Agile techniques into their organizations. For more information about OpenSource Connections’ Agile process consulting services, please contact the author at ASime@OpenSourceConnections.com.

Everybody loves employees and consultants who can get stuff done. Especially if you’ve made a commitment to a client or your boss, it can be very difficult to ask for an extension to that timeline. It’s much easier to ask your team to put in the extra hours to get the job done, even if that means the job won’t be done well. Sometimes leaders will even resort to claiming the product is “done”, when it was never truly done at all, and needs a lot more work.

Your development team will likely respond to your calls with heroic effort. And you might actually make the deadline. But if you do it regularly, your reputation as a leader may actually suffer, and your team will probably burn out unless you can pour on enough accolades and spot-bonuses.

At some point, almost all IT leaders, including myself, have done it. That is unfortunately not much of a secret and it’s why IT teams often develop a bad reputation. The bigger secret is that a lot of developers not only regularly take on the hero role, they actually like it.

This creates a dangerous situation, where IT leaders ask for heroic efforts because it is the path of least resistance. And the development teams respond with grumbles and late nights, but also with the misconception of job security and the expectation that they will receive a lot of praise when they come in on Monday bleary-eyed (but with at least part of the job done).

Why is there so much danger in encouraging your developers to be heroes? Do a google search on “cowboy coders” or “programmer heroes” and you’ll see other blog posts that describe the whys and the pitfalls of reliance on hero development. In short, you’ll get inconsistent and buggy code that may not fit your architecture, won’t meet the customer’s needs, is not very flexible, creates technical debt you will complain about for years, and will probably need to be reworked eventually.

So how do you avoid heroes? As a CIO or technical lead, the tools of Agile offer many solutions:

• Sprint planning – restricting the work that is done to the current sprint helps discourage sudden changes of priority. Most tasks can in reality wait until the next sprint to be added, especially if the customer understands that it means the current sprint has to be put on hold or canceled.
• Daily standups and regular customer interaction – Chronic hero programmers often like to hideaway on their work, and so they can be more easily detected when you are interacting with the customer each day. There is less opportunity for the developer to hide away and not give status updates, and the standups provide the customer more immediate feedback on everything that is being held up because they sent the hero on a distracting quest.
• Team estimation – When the team estimates tasks for a sprint together, before they are assigned to people, that encourages open discussion and debate on how long something will take.
• Continuous Integration – Maintaining rules about daily code check-ins and regular integration and unit testing prevents the hero from going too far away from the mainstream code base, and gives visible signs of progress. When done properly, it also helps to reduce the bugs delivered in the rushed code.

In addition, in our OpenApproach implementation of Agile and Scrum, we use range estimation of tasks. Asking people to give you a single-point estimate of how long something will take encourages them to be overly optimistic. “Oh, I can get that done in about 2 hours.”

Really? What if it’s not as simple as you or the developer thought? Then the only solution is to put in a heroic effort if the timeline must be met.

A simple range estimate will provide a much more realistic view, and encourages a more thorough thought process: “It should only take about 2 hours of actual coding, but I’ll have to switch projects and get that site running again on my machine. Plus, sometimes those java script issues can be really though to debug, so it could take as long as 8 hours of work.”

As an Agile CIO, it’s your job to not only be understanding of your developers when they give you a range estimate, but encourage it. Don’t settle for the heroic answer of “I’ll just put in some extra time this weekend.” Get to the root cause of the problems, and make sure the customer or your boss is aware of the delays earlier rather than later. It will make all of your lives easier, and help you to avoid the pitfalls of programmer heroes.

The Agile CIO is a series of blog posts advising IT leaders on how best to incorporate Agile techniques into their organizations. For more information about OpenSource Connections’ Agile process consulting services, please contact the author at ASime@OpenSourceConnections.com.

This is like the state of the union address, except in mid march, and the only thing I’m president of is my current residence. If you have ever studied Science, you know about potential energy vs kinetic, Or maybe a better metaphor for the reality TV generation, is the Swan.

Opensource software on the Microsoft stack has tons of stored potential, even some movement, but it is still left wanting. The evil empire has embraced opensource software, releasing jQuery with Visual Studio, starting codeplex.com and the Codeplex Foundation.

The are tons of abandoned projects or ones that have gone stale (log4net anyone?). Compared to java or even the new kid ruby, we’re lagging behind. We don’t even have a fully managed opensource enterprise web server, compared to Java’s n-th variety of containers to pick and use. Even rails has a built in server.

Of course there is kayak http web server framework and webserver on codeplex, but they’re new, far from mature and their not an industry defacto standard like jboss or tomcat.

Don’t get me wrong there are some amazing opensource projects out there. Gallio, db4o, Subtext, blogengine.net, facebook developer toolkit to name a few.

However, there are gaps in having a full opensource Microsoft stack. We have plenty of unit testing and mock testing libraries, but with NDoc gone, left to SandCastles release schedule, libraries left to rust like log4net wit not even .net 4.0 beta build or silverlight build. CruiseControl.Net is in dire need of revamp and version 2, at the very least it needs some decent competition that isn’t java.

With plenty of single person projects out there who just end up getting burnt out, seemingly stagnated public dialog from the likes of the code plex foundation, its hard to really get developers to rally and get some much needed things done.

The community need some decent leaders, organizers, and some company backing. Most software vendors and clients get a great productivity boost from these projects. It would only make sense to invest in their growth, even pool resources for joint projects.

Organize some hack-a-thons days with some cool prizes for work top-notch work. Even put together a small guild of programmers, just have 20 or so companies pitch in, put their banners on a website and churn out some decent open source projects that everyone can use.

.Net isn’t going away and its time the community and companies invest more into the opensource community instead of letting all that potential go to waste.

Last week was the crucial week on my current Lucene -> Solr project for making our goals.  A lot of work the previous couple of weeks came together.  I wanted to take a couple of minutes and just record some of the little things that I’ve been learning about:

Solr

Sunspot is the up and coming solution for integrating Solr into Ruby on Rails, and fortunately enough, the 1.0 release (followed quickly by 1.0.1!) has just come out last week.  Between acts_as_solr and Sunspot, Sunspot wins hands down for it’s support of a master/slave Solr configurations, embedded Solr for testing, richer indexing semantics, and not being tied to ActiveRecord.  The companion sunspot_rails gem does give wonderful ActiveRecord integration however.

Solr cores are the bees knees!  We’ve built a simple RoR webapp using HTTParty and the Solr API that allows you to perform all the admin functions for cores, and allows you to quickly clone a core for your own nefarious purposes!  Simplifies hacking around with a new schema or configuration without having a local copy of Solr running.  Allows multiple QA environments to potentially share a single Solr infrastructure.

Solr master and slave setup in a single VM.  While pointless from a scaling perspective, it’s a really great way to work out the kinks!  It’s funny to see a slave core polling the same Solr VM its in for updated segments!

JRuby

Doesn’t suck after all.  Actually, maybe I should say that JBoss, when combined with JRuby, means that JBoss doesn’t suck so much.  I had the aforementioned Solr core admin tool bundled up as a WAR file with JRuby, and was able to deploy it to an existing environment that had JBoss installed!  I didn’t have to install ruby on the box, (or JRuby for that matter!)  I just deployed the WAR file and bamn, off to the races.  Ops folks get the JBoss they love, I get the Ruby on Rails that I love.

And on a related note, Warbler was the key to thinking JRuby is cool.  I’d never actually had to package up a RoR app, so Warbler came to the rescue.  And you know what?  It was nice to build a single file that I knew had everything that I needed in it that could be scp’ed around!  And thanks to some cool code in the environment.rb, my app was able to load up the right configuration file for the environment based on an environmental variable set in JBoss.

Virtual Machines

I recently migrated a Linux VPS based RoR + Solr app (see a trend in tech choices )  to a Windows environment.  And to deliever the new Windows environment, I used VirtualBox to host the Windows Vista environment on my Mac laptop.

A couple of notes:

• VirtualBox may not have all the snazzy integration points of Parallels with the host computer like seamless application sharing, but it seems to be much lighter weight.  Starts up quicker, and I don’t get the spinning beach ball of death as much.
• If you are shipping a 11 GB file, you can’t use a 16 GB USB Memory Stick…  Turns out the biggest file is 4 GB.  (Although I never tried formatting the stick as NTFS, maybe that would have allowed a single 11 GB file???)
• Uploading 11 GB to a remote out on the internet server will take a long long long time.  Even on a really fast network. connection.
• If you need to format an external USB hard drive as NTFS on a Mac, it is possible!  Just fire up your trusty Windows Vista image in Parallels, plug the USB drive in, download and install the correct USB drivers so the drive doesn’t show up as a network share mapped to the Mac, and then use the built in reformatting tools!  Warning: This will take a loooong time!
• Lastly, if you are using VirtualBox, and you attempt to create a Windows XP machine, and attach a Windows Vista hard disk image to it, VirtualBox will let you!  And then Windows won’t start.  sigh.

Last week, the U.S. Federal District Court of Northern California found a software developer liable for unattributed use of a Java Model Railroad Interface.   The lack of attribution, as required in the license, the Court found, violated the Digital Millennium Copyright Act.  While the Court’s finding has limited legal jurisdiction, it does set a precedent which will surely be cited in future cases looking for both damages and to set further legal precedent and define more fully the cases in which stare decisis will apply.

While, analogously, no different than any other license violation, I predict that this finding will have a ripple effect in government adoption of open source projects in their applications, further dampening the adoption rates.

• Many government agencies are skeptical of open source in the first place.  Open source projects, rightly or wrongly, have a blanket aura of the two high school kids in a garage image.  Most government agencies like controls and process, which they don’t perceive exist in open source projects.  If some of them tried to become committers on major projects, perhaps the perception would change.
• Very few open source projects have sales representatives.  The sales representatives at major software companies such as Microsoft do an excellent job of the appropriate education and shepherding to provide assurances to the government buyers.
• Walking through a purchase process gives the perception of license compliance.  A contracting officer can say that they have made a purchase and have a license and feel like they are in compliance, although audits oftentimes reveal shortcomings.  With an open source project, anyone can download source code and deploy the software onto a system, creating manifold increases in risk of license violations.
• Most legal counsel isn’t familiar with open source licensing requirements and restrictions.  Rather than digging into and understanding GPL or “copy-left” restrictions, it’s easier to just stay away from it.  Again, there are probably as many, if not more, EULAs than accredited and accepted open source licenses, but there are also sales representatives to walk through EULAs whereas the open source projects do not have such champions.
• It’s easier to create blanket restrictions than to make judgments on each case based on the specific merits of the case.  The finding in Jacobsen v. Katzer makes it easier to point to a known and potentially quantifiable risk to deny use.  The Government is, and in most cases rightly so, risk-averse, and this finding creates a potentially big risk.  The risk may be a black swan, but it is very hard to measure and, in a perfect (in the Government’s view probably an imperfect) storm, could be potentially exceptionally large.

While it won’t have an immediate effect, the end result, I expect, will be an even more rigorous set of processes to complete for approval of the use of most open source software within the government.  I hope this is not the case, because there are many cases where open source software is the best solution.  Furthermore, I truly believe that:

• By and large, open source developers are not a litigious group.  They want to write software which makes life easier, and as long as they get the credit and attribution they want, they’re happy to share.  It’s what’s written in the licenses.  Plus, legal paths are expensive, and very few can afford to go down that path.  Compliance with a very few legal requirements or selection of a different platform will avoid any potential legal pratfalls.

Thanks to Onlyopensource for pointing out this article!

Today, Microsoft announced that it was abandoning development of the FAST search engine for Linux and Unix.  Given that Microsoft paid $1.2 billion for FAST, the move is an apparent revelation of a strategy to get non-Windows based users to move to an enterprise Windows platform rather than to continue to support FAST.

This move seems to be risky.  The Microsoft bet is that its FAST customers are more loyal to FAST than they are the operating platform, and the perception of switching costs are higher for moving from FAST to another enterprise search engine rather than the opposite–a loyalty to the operating system and a perception that search engines are interchangeable.

Microsoft might be right for most of its customers, but this announcement will certainly be grist for the mill in IT departments over the coming weeks.  Many companies built their IT infrastructure around a Linux-based platform, and being forced to change to a Windows environment may be a pill that is too hard to swallow.  The alternative will be to look to other search engines, which can do nothing but help Solr and Lucene.  With an established user base, enterprise grade support packages from companies like Lucid Imagination, and a significantly lower total cost of ownership than the FAST + Windows package, Solr will appeal to many a CTO who might otherwise have continued to gladly pay the licensing costs for FAST but is now forced to reconsider his or her decision.

Rather than supporting FAST on both platforms at the cost of a few developers, Microsoft may lose many more customers and revenues because of its insistence on one platform.  It will be interesting to see how companies like Lucid respond to the new opportunity.

Yesterday, the House of Representatives introduced S.2989, the Small Business Revitalization Act, which punishes government prime contractors for failing to pay their subcontractors after the government has paid them for work performed.

While noble in trying to ensure that small businesses get compensated for work performed, the bill seems to miss the mark on the bigger issue with small business subcontracting.  I admit, I cannot find the text of the bill (and would love it if someone could point it out to me), so I could be wrong, but based on the article linked above, it appears that a couple of key issues are not addressed:

• Prime contractors actually meeting their subcontracting requirements.  In most large contracts, prime contractors are required to have a subcontracting plan and meet certain thresholds of work sharing amongst disadvantaged groups.  Very few enforcement mechanisms exist to ensure these thresholds are met, and this act does not address the issue.
• Prime contractors actually using the teams that they proposed in RFP responses.  If a prime wins a contract by describing teaming arrangements with certain companies, then it should actually utilize them as proposed to the government originally, or replace like-for-like on the team.  Again, no enforcement mechanism exists unless a contracting officer writes a constraining and binding contract.

As I have written before, set-asides and subcontracting goals go against my libertarian nature, as I want the most value for my money as a taxpayer; however, if they are going to exist, then the least the government can do is make sure that the rules are enforced.  Unfortunately, the Small Business Revitalization Act, it appears, fails to help meet those goals.

In a memorandum dated October 16, 2009, the CIO of the Department of Defense published guidance on the use of open source software within the DoD.  The CIO’s assessment of the perception of open source sums up quite well the obstacles we have seen time and time again:

“Unfortunately, there have been misconceptions and misinterpretations of the existing laws, policies and regulations that deal with software and apply to OSS, that have hampered effective DoD use and development of OSS.”

I wonder how long it will take for contracting officers to get the memorandum (so to speak) and to try to find ways around the guidance, spelled out in Attachment 2:

“In almost all cases, OSS meets the definition of commercial computer software and shall be given appropriate statutory preference in accordance with 10 USC 2377 (reference (b)) (see also FAR 2.101(b), 12.000, 12.101 (reference (c)); and DFARS 212.212, and 252.227-7014(a)(1) (reference (d))).”

We also eagerly await broader acceptance within the DoD information assurance community, but do not expect rapid change, despite the following guidance:

“DoD Instruction 8500.2, Information Assurance (IA) Implementation, (reference (g)) includes an Information Assurance Control, DCPD-1 Public Domain Software Controls, which limits the use of binary or machine-executable public domain software or other software products with limited or no warranty, on the grounds that these items are difficult or impossible to review, repair, or extend, given that the Government does not have access to the original source code and there is no owner who could make such repairs on behalf of the government. This control should not be interpreted as forbidding the use of OSS, as the source code is available for review, repair and extension by the government and its contractors.”

In sum, this is good news for those of us who believe that open source has a place in the Department of Defense.  However, word spreads slowly in the Government, and I expect that this policy will take many months, if not years, before gaining significant traction in DoD contracting policy.  A good place to start would be providing correct links in the policy memo.  Clicking on the link currently leads to a 404 error.

Edit: This is the correct link: Clarifying Guidance Regarding Open Source Software (OSS)

Thanks to the Powdermonkey blog and Dr. Mark Drapeau for sharing this!