Where are Eric and Youssef?

This year two OSC folks are working remotely for extended periods of time. Youssef spent 6 weeks in Lebanon earlier this year, and I am spending the month of July in the mountains of Western North Carolina. We had talked about some of the rewards as well as challenges of working effectively when you are in a new/fun/exciting/[insert superlative here] place. Youssef and I came up with a few tips to make balancing work/play easier!

Tip 1: Establish a Routine

Youssef says: There will be no shortage of distractions, specially during the World Cup season. A trip to the mountains, a music festival or a friend’s unannounced (or announced) visit are all good reasons to drop what you’re doing and enjoy the place you are in. The only way to both get your work done and have fun is to know when is the appropriate time for each. I found that establishing a routine worked best for me. I woke up early every morning and took advantage of the early quiet hours of the day. Started work at 8am and got plenty done early on. Then I gave myself a couple of hours of break time for unscheduled events (a friend’s visit, a trip to the store to get souvenirs, or even a couple of hours at the beach). Then I resumed work till the evening when I had a conference call scheduled every day at 5:45pm. After the call I was free to do whatever I wanted and that gave me plenty of time, specially in a city like Beirut that does not sleep.
Eric says: I know that I sometimes struggle to get started working in the morning. And being in the office makes it simpler because everyone else is working. So having a specific schedule of when I am working helps me shift my brain out of family/vacation mode and into work mode!

Tip 2: Fast Internet Connection

Eric says: When we looked for cottages to rent, one of our requirements was that they have something better then dial up! We got lucky that one of our top choices was just installing a HughesNet satellite link at the cottage.
Youssef says: It is embarrassing to be a on a call with a client and have that call be constantly dropped because of a bad connection. Skype requires a decent bandwidth and fast connection to work properly. You do not want a 15 minute stand up call to turn into a half an hour of you asking the people on the other end “Can you hear me now?” (Verizon does not operate in foreign countries!). So make sure you get yourself a decent internet connection even if you have to spend a little extra on it. And always take any install CDs or DVDs you think you might use with you. Some countries do not have the proper infrastructure and thus impose a cap on the download/upload limit you may use, which makes you think twice about how you use that limit (downloading the new Xcode from apple’s website is not an option).
Eric chimes in: I ran into the same Skype issue, turns out Skype doesn’t work well with satellite internet due to the latency issue! Fortunately, if I am upstairs on the west side of the cottage, standing on one foot, with my finger in my ear, my iPhone gets 1 bar of signal! Phew. I did notice that I needed to talk a little slower and not move about too much and folks can hear me.

Tip 3: Regular Checkins

Youssef says: Stand up calls in Scrum keep the developers honest. When you are working remotely, you do not want to lose touch with the work environment, specifically in a field that evolves constantly such as software development. Having a point of contact in the “real world” or the “work world” helps you stay in touch and not lose focus.
Eric says: When you are in the office everyday, you pick up all the water cooler gossip. But working remotely means you lose that contact. I find that having a regular series of checkins helps stay connected. I put them on my calendar so they don’t get skipped!

Tip 4: Track your time!

Eric says: Don’t forget that you are working remotely is because you are doing something special. In my case, it’s taking my family for a month to the mountains. So I want to make sure that I don’t miss out on the experience. Over the four weeks I am here, I am taking a week of vacation, spread out a bit. So I am tracking my hours both to make sure I don’t shortchange work, but also don’t shortchange the experience of being 200 feet from the Appalachian Trail!
Youssef says: I definitely did not travel for 24 hours, crossed an ocean, a continent and a sea to do the exact the same things I was doing in Charlottesville. It also does not mean I should use that as an excuse to ignore my responsibilities. Tracking your time helps you balance out the experience and establishing a routine like we mentioned above goes well with this concept.

Tip 5: Stay Agile

Youssef says: Working remotely does not mean you should change the way you work, but flexibility is key. Working on the east coast while your client is on the west coast requires some coordination. Working in Lebanon (GMT + 2) while your coworker is on the east coast and your client is on the west coast, well that requires some bending and flexibility. The key part about dealing with this situation is an agreement with all parties involved about which approach to follow. The method I followed included three chunks of time. A chunk where I was working ahead of everyone else (during which I could be dealing with issues that came up the day before), a chunk of time where everyone was available at the same time and third chunk when I was off the clock while everyone else was still working. That allowed for ample communication and work time without sacrificing my fun time. Others might prefer to synchronize the working time, or might prefer to just deal with the delays. Balance worked best for me.
Eric says: I agree with Youssef, being flexible on your approach to work can make being remote a very enjoyable and productive experience!

Looking out of my window at NC Mountains

After much faffing about, I finally figured it out.  On the server I would ask for the set of timezones that matched the offset, and grab the first one and put that in the session:

[sourcecode]
result = ActiveSupport::TimeZone.all.select{|t|t.utc_offset == gmtoffset}.first
session[:time_zone] = result.name
[/sourcecode]

The list of named time zones returned when the browser is in Charlottesville, Virginia are: Bogota, Eastern Time (US & Canada), Indiana (East), Lima, Quito.

So when I use Bogota as the timezone, and ask Rails to show the time localized:

[ruby]
Time.now.in_time_zone(session[:time_zone])
[/ruby]

I get back the time without taking into account daylight savings wrong. I started trying to figure out if the browser was in a DST zone using this JavaScript code: http://www.michaelapproved.com/articles/daylight-saving-time-dst-detect/ and while it seems very promising, it still wasn’t quite giving me what I want.

Finally, I realized it…. By arbitrarily grabbing the first time zone in the list, I was showing time in Bogota, Columbia. But if I chose Eastern Time (US & Canada) then I do get a localized time that takes into account day light savings!

So right now I have this method:

[ruby]
result = ActiveSupport::TimeZone.all.select{|t| t.utc_offset == gmtoffset && t.name.include?("US")}.first
session[:time_zone] = result.name
[/ruby]

Obviously this is pretty hardcoded to just work in the US, and isn’t a real solution. I’d love to hear other ideas! Part of me wonders if I should just display all times in UTC in HTML, and have some sort of client side JavaScript that localizes the time display?

My full set of code:
Javascript in my index.html.erb view:
[javascript]
// Calls the server and sets the user’s time.
Event.observe(window, ‘load’, function(e) {
var now = new Date();
var gmtoffset = TimezoneDetect();
//use ajax to set the time zone here.
var set_time = new Ajax.Request(‘<%=url_for :controller => "home", :action => "gmtoffset"%>?gmtoffset=’+gmtoffset, {
onSuccess: function(transport) {
//alert("Response" + transport.responseText);
}
});
});

// http://www.michaelapproved.com/articles/daylight-saving-time-dst-detect/

function TimezoneDetect(){
var dtDate = new Date(’1/1/’ + (new Date()).getUTCFullYear());
var intOffset = 10000; //set initial offset high so it is adjusted on the first attempt
var intMonth;
var intHoursUtc;
var intHours;
var intDaysMultiplyBy;

//go through each month to find the lowest offset to account for DST
for (intMonth=0;intMonth < 12;intMonth++){
//go to the next month
dtDate.setUTCMonth(dtDate.getUTCMonth() + 1);

//To ignore daylight saving time look for the lowest offset.
//Since, during DST, the clock moves forward, it’ll be a bigger number.
if (intOffset > (dtDate.getTimezoneOffset() * (-1))){
intOffset = (dtDate.getTimezoneOffset() * (-1));
}
}

return intOffset;
}

[/javascript]

home_controller.rb action:
[ruby]
def gmtoffset
gmtoffset = params[:gmtoffset].to_i*60 if !params[:gmtoffset].nil? # notice that the javascript version of gmtoffset is in minutes

result = ActiveSupport::TimeZone.all.select{|t| t.utc_offset == gmtoffset && t.name.include?("US")}.first
session[:time_zone] = result.name

render :update do |page|
page.replace_html ‘time_of_chat_starting’, :partial=> ‘super_short_time’
page.visual_effect :highlight, ‘time_of_chat_starting’
end
end
[/ruby]

Rendered partial helper view _super_short_time.erb:
[ruby]
<%= super_short_time(Time.now.in_time_zone(session[:time_zone])) %>
[/ruby]

If you’re a geek in or around the Charlottesville metroplex or even if you’re merely tech-curious, this is the event you don’t want to miss. beCamp is Charlottesville’s version of the BarCamp unconference phenomenon—organized on the fly by attendees, for attendees. Realizing that the most energizing parts of any tech conference are the ad hoc conversations that take place in the hallways between the sessions, beCamp facilitates these types of interactions for an entire event.

As of this writing, we are at 87 campers! To participate, just add your name to the wiki page!

A big thank you to all our sponsors, including at this point,  Hotelicopter, Google, Perrin Quarles and Associates, NRAO, and University of Virginia ITC.  Interested in supporting the Cville tech community?  Check out our needs at http://barcamp.org/sponsor-beCamp-2010.

Want to use a favicon.ico but don’t want to put it at the root as favicon.ico?  Then add <LINK REL=”SHORTCUT ICON” href=”<%=image_path(‘limelight.ico’)%>”>.   The use of image_path means that you get all the goodness of Rails routing to generate a complete image path that will work.  Even if you deploy under some sub URL, like a war file in JBoss!

Are you working with ISO-8859-1 encoded text, (more info at  at http://www.w3schools.com/tags/ref_entities.asp).

ISO 8859-1 Characters

And look at the character À  (should be a capital A with a caret on top called a grave accent).  That will kill Solr, regardless of container deployed in, like Jetty versus Tomcat.  But the other ways of representing this entity work great:

Of course, there is a bit of confusion on this, as supposedly if the XML document posted to Solr is UTF-8 encoded, then Solr shouldn’t have any issues.  So, still some digging to do!

Solritas and JBoss and Velocity Oh My

I recently ran into a Java Classloader issue between JBoss and Solr when loading Velocity. If you are getting in the browser:

TTP Status 500 – loader constraint violation: when resolving method “org.apache.velocity.Template.merge

or messages like “SEVERE: java.lang.LinkageError: loader constraint violation: when resolving method “org.apache.velocity.Template.merge” in the logs, then that means a conflict in the velocity jars.  Oddly enough, I could not actually find a velocity.jar anywhere in my JBoss app.  However, the fix was to copy the velocity jar from Solr into my JBoss ./lib/ directory.

Solr

Sunspot is the up and coming solution for integrating Solr into Ruby on Rails, and fortunately enough, the 1.0 release (followed quickly by 1.0.1!) has just come out last week.  Between acts_as_solr and Sunspot, Sunspot wins hands down for it’s support of a master/slave Solr configurations, embedded Solr for testing, richer indexing semantics, and not being tied to ActiveRecord.  The companion sunspot_rails gem does give wonderful ActiveRecord integration however.

Solr cores are the bees knees!  We’ve built a simple RoR webapp using HTTParty and the Solr API that allows you to perform all the admin functions for cores, and allows you to quickly clone a core for your own nefarious purposes!  Simplifies hacking around with a new schema or configuration without having a local copy of Solr running.  Allows multiple QA environments to potentially share a single Solr infrastructure.

Solr master and slave setup in a single VM.  While pointless from a scaling perspective, it’s a really great way to work out the kinks!  It’s funny to see a slave core polling the same Solr VM its in for updated segments!

JRuby

Doesn’t suck after all.  Actually, maybe I should say that JBoss, when combined with JRuby, means that JBoss doesn’t suck so much.  I had the aforementioned Solr core admin tool bundled up as a WAR file with JRuby, and was able to deploy it to an existing environment that had JBoss installed!  I didn’t have to install ruby on the box, (or JRuby for that matter!)  I just deployed the WAR file and bamn, off to the races.  Ops folks get the JBoss they love, I get the Ruby on Rails that I love.

And on a related note, Warbler was the key to thinking JRuby is cool.  I’d never actually had to package up a RoR app, so Warbler came to the rescue.  And you know what?  It was nice to build a single file that I knew had everything that I needed in it that could be scp’ed around!  And thanks to some cool code in the environment.rb, my app was able to load up the right configuration file for the environment based on an environmental variable set in JBoss.

Virtual Machines

I recently migrated a Linux VPS based RoR + Solr app (see a trend in tech choices )  to a Windows environment.  And to deliever the new Windows environment, I used VirtualBox to host the Windows Vista environment on my Mac laptop.

A couple of notes:

• VirtualBox may not have all the snazzy integration points of Parallels with the host computer like seamless application sharing, but it seems to be much lighter weight.  Starts up quicker, and I don’t get the spinning beach ball of death as much.
• If you are shipping a 11 GB file, you can’t use a 16 GB USB Memory Stick…  Turns out the biggest file is 4 GB.  (Although I never tried formatting the stick as NTFS, maybe that would have allowed a single 11 GB file???)
• Uploading 11 GB to a remote out on the internet server will take a long long long time.  Even on a really fast network. connection.
• If you need to format an external USB hard drive as NTFS on a Mac, it is possible!  Just fire up your trusty Windows Vista image in Parallels, plug the USB drive in, download and install the correct USB drivers so the drive doesn’t show up as a network share mapped to the Mac, and then use the built in reformatting tools!  Warning: This will take a loooong time!
• Lastly, if you are using VirtualBox, and you attempt to create a Windows XP machine, and attach a Windows Vista hard disk image to it, VirtualBox will let you!  And then Windows won’t start.  sigh.

Installer

The first thing you’ll notice is that an installer (and uninstaller) is provided that walks you through the basic steps of installing Solr. Now Solr itself is pretty darn simple to work with already, but you do need to compile the code, which means you need Ant to be installed. The Lucid installer avoids that need, and  adds support for running Solr in Tomcat as well as Jetty. And, assuming you have a support agreement with Lucid, it supports downloading plugins from Lucid to extend your Solr platform. Right now the only free plugin is the Reference Guide PDF. Having an installer available definitely checks a box for the systems type folks who may be installing Solr, but it doesn’t really do anything crazy special. Also, one nit is that if you install into /opt/dirA, and then want to install into /opt/dirB, you have to delete ~/.LucidWorks/ directory as the install dir is cached!  But it does demonstrate what might be coming from Lucid in future updates!

Installer Targets Screen

Another enhancment from Lucid is a Tray Application for managing your Solr instances. However, this turns out to just be a basic (on OSX at least!) menubar application that allows you to start/stop a local Solr server. There doesn’t seem to be any options to stop and start remote servers, or monitor the health of running Solrs, so I think this is something you use once and never again! Hey Lucid, it would be great though if the Tray App integrated stoplight monitoring of Solr instances and popped open web pages to admin pages to perform various tasks on your collection of Solr servers!

Directory Layout

The directory that you’ve installed Solr into should look very familiar. In fact, too familiar to me! I’ve gone back and forth on the way that Solr is distributed with source code as well as compiled jars. While Solr used to be a tool that only Java centric shops would look at, it’s now gone mainstream, to where many, if not most, organizations that use Solr are not traditional Java shops! I really wish I could download a version of Solr that didn’t have the src directory, was just a stripped down ready to go application. Admittedly, the example application that is part of the source functions as a template, but it has been bemoaned by myself and others that folks just use and abuse the configuration of what was meant as an example app, to their detriment!

So I was hoping that the LucidWorks distros’ Installer would function as that smart template by walking me through including/excluding various extensions like DIH, Clustering, and Extraction. But at least in this first version, no such luck. The support though for for picking either Tomcat or Jetty as a container shows what could be in the offing though!

While the LucidWorks distro still ships with the hoary old example directory is still there, there is now a lucidworks directory. When you run the new toplevel start.sh shell script it starts Solr with solr.solr.home=lucidworks/solr directory. Something to note is that the start.sh has complete paths defined in it from the installer:

[sourcecode language="text"]
cd /Users/epugh/solr/solr2/LucidWorks/lucidworks/jetty/../
[/sourcecode]

It really should at least have a single variable at the top that you can changing depending on what environment you are in.

The lucidworks project is also setup as a single index project.  Since the future is multicore configurations, I’d like to see that as the default in more examples.  (The example app needs a bit of work as well to better show off multicore as a first class feature!)

solrconfig.xml

Doing a diff on the example and lucidworks versions of solrconfig.xml shows its pretty much the same as the one from the example app, but with the correct configurations for DataImportHandler and the Velocity based search UI called Solritas. Solritas is a nice tool for helping you “wedge” Solr into places by providing a simple Velocity template based translation layer, and even build a GUI, within your Solr environment. Solritas hasn’t received a lot of buzz, so it’s nice seeing it turned on by default! The clustering functionality is also specified, but not sure if the solr.cluster.enabled=true startup parameter is actually required or not.

The other oddity is that the Lucid monitoring product for Solr, SolrGaze, isn’t enabled by default! Doesn’t seem like the most ringing endorsement for the software. I’m excited by the prospect of better visiblity into the internals of Solr, so I enabled it.

schema.xml

Diffing the two schema.xml files reveals the addition of the Lucid KStemmer com.lucidimagination.solrworks.analysis.LucidKStemFilterFactory for fast non-aggresive text stemming. According to Lucid it provides:

Large field performance shows a 220% performance increase, while small fields show a 1140% increase compared to the original UMASS code.

SolrGaze

SolrGaze promises to make it easier to see what is going on inside of Solr. Anything that makes it simpler for operations folks instead of developers to manage Solr is good in my book. I ran into one nit which was I opened up SolrGaze using the url http://localhost:8983/gaze/index.html. It barfed connecting to Solr to display gathered metrics, but if I used http://127.0.0.1:8983/gaze/index.html then everything was fine.

I haven’t had to chance to really play with Gaze yet, so I’ll post a more in-depth review soon.

Summary

All in all, the Lucid distro would be what I would recommend for a first timer to download, or someone doing a spike of development and needing a quick install of Solr.  Not requiring Ant to be installed is a wonderful thing, and being pre-configured for Clustering, DIH, and Solritas means you get to see a working Solr install, complete with a full featured GUI, right out of the box.  In terms of using for a production deploy, there is less to recommend it since you’re going to want to strip down to just the bits and bobs that your require for your specific needs.  I haven’t delved down into what SolrGaze provides, so that feature may be the tipping point for deciding to use the Lucid distribution.

He has written a very indepth review, that I’ll admit I was nervous to read! But he summed it up as:

Grand Finale
I spelled out a lot of fiddly feedback above, and I expect the great addendum wiki page will factor in any keepers from this review. Of course most of the review points out mistakes or differences of opinion, that’s what a review is for, though this is a solid, useful book. So, if you’re considering using Solr, this book is for you. If you’re already using Solr, you’ll likely pick up a useful trick or three. Go get it!

As you can see from the level of detail in his post, when we come out with a second version of the Solr book, updating it for changes between when we published it and the final release of Solr 1.4 will be very easy!

url = "http://#{file[path_ends_at,file.size]}"
title, content = parse_html(file, title, content)

puts "Bad Content:#{!page_content.blank?} #{url} #{title}"

begin
solr.add :id=> url, :url=>url, :mimeType=>"text/html", :title => title, :docText => page_content
solr.commit
count = count + 1
rescue RSolr::RequestError
puts "<strong>Could not index #{file}</strong>"
end
end
end
puts "Imported #{count} webpages successfully."
solr.optimize
redirect_to root_path

end[/code]
This worked great, but I realized that indexing over 10,000 documents takes a long time, and meanwhile the user is staring at the browser slowly loading, wondering if things had frozen or not! So I wondered if I could somehow stream some info back to the user. Fortunately Rails has already solved that problem! ActionController has the ability to render as text a proc object, and stream the output:
[code lang="ruby"] # Renders "Hello from code!"
render :text => proc { |response, output| output.write("Hello from code!") }[/code]

So I quickly wrapped my existing code in a large proc, changed the puts to output.write, and now stream out to the browser constant progress reports:
[code lang="ruby"]
def index_bulk_html
solr = RSolr.connect :url=>SOLR_URL
count = 0
files = Dir.glob("/Users/epugh/Documents/code/www.somesite.com/**/*.{html,htm}")
render :text => proc { |response, output|
files.each do |file|
path_ends_at = file.index("www.somesite.com")
unless path_ends_at.nil?
output.write("<strong>Processed #{count} of #{files.size}</strong>") if count % 100 == 0

url = "http://#{file[path_ends_at,file.size]}"
title, content = parse_html(file, title, content)

output.write "Bad Content:#{!page_content.blank?} #{url} #{title}"
output.flush

begin
solr.add :id=> url, :url=>url, :mimeType=>"text/html", :title => title, :docText => page_content
solr.commit
count = count + 1
rescue RSolr::RequestError
output.write "<strong>Could not index #{file}</strong>"
output.flush
end
end
end
output.write "Imported #{count} webpages successfully."
}
solr.optimize

end
[/code]
Thank you Rails, Hpricot, and RSolr for making life so simple!

I flew up to Boston on Monday to lead some workshops on Continuous Integration. I checked into my room at the Hyatt and then went downstairs to see who was around. I ran into a couple of speakers milling about, and eventually joined one of them, and we headed over to the MIT Press bookstore, me to look for my Solr book. I wasn’t too sure of the name of the other speaker I was with (I asked once, but couldn’t quite remember what it was…). So we got to the book shop, I ask my fellow speaker again: James Bach. The name I was familiar with, but couldn’t quite place it… I ended up buying Parentonomics, and then we go for coffee.

So, over coffee, he asks me about what my topic is, and I gave him the brief summary of my two CI related workshops. Wow.. Little did I realize that I was sitting with the guy who rails against the “fetish” that Agile folks have for automated testing! That his entire approach to “testing” is to use skilled, motivated folks who do “sapient testing”. And I’m the guy who’s selling an approach that REQUIRES automated tests! That encourages expanding the use of automated testing!

He actually walked me through a process of talking about how to “think like a tester”, and it was really great mini-workshop.. He definitely subscribes to the socratic approach, and believes in his message, I was sweating at the end of it! That chat probably sparked more ideas in less time over that coffee then anything else this week. I also heard a lot of ideas and phrases that were echoed in Michael Bolton’s keynote later on in the week. Clearly a lot of collaboration between the two!

Probably the biggest idea that James and chatted about was the idea that automated tests really aren’t automated tests, they are automated checks. They verify that the expected behavior of the code was met. His argument that if you want to do testing, real testing, then computers, automated processes can’t meet that need, only people can.

Now, I don’t know if I believe that is completely true, but I am very aware that the “manual testing” where long test scripts written as Word documents are executed by human beings by hand are really a waste of human potential. And that those test scripts are really, to use James terms, “check scripts” because the people are not using any creativity! In fact, a lot of my interest in CI comes from the idea that people should not do monkey testing, that machines can do it much better, and my frustation with the perception that testing is a low value activity and can be easily shipped off to low skilled folks.

I think that this shift away from the term “test” for automated tests is actually happening in many places. In the Ruby world, we have libraries like Shoulda that are moving from using words like assert to other words like should. A Cucumber test really shows how controlled the space that a test needs to be to work well in an automated fashion:

Scenario: See all vendors
Given I am logged in as a user in the administrator role
And There are 3 vendors
When I go to the manage vendors page
Then I should see the first 3 vendor names

So while I don’t know if I am bought in on the idea that only people can do “testing”, and machines can only do “checking”. Tools like Heckle try to simulate aspects of what a human can do. While not suggesting that we can automate the “does my website look okay after someone changed the CSS” type of work today, in the future our automated testing will be more capable then just “checking” because we will move beyond the very constrained tests we have today to ones that mimicing the richness of the simulators that Airline Pilots use. Instead of testing the training given to pilots, we’ll be testing the robustness of software via simulations!

At any rate, James Bach, while taking a rather provocative approach to sharing his ideas, does subscribe to my favorite bullet in the Agile Manifesto: Individuals and interactions over processes and tools.

Here is him giving a great presentation with the subversive title of “How to Fake a Project” that was incredible entertaining, and also quite thought provoking:

What do you think? Automated testing is a fetish of the Agile community?

We’ll look at the thriving Ruby ecosystem that has grown up around integrating with Solr. From Ruby gems that integrate with Solr like solrb and rsolr, to general search solutions like acts_as_solr and sunspot. We’ll also look at a complete “shrink wrapped” catalog solution for Solr using BlacklightOPAC.

You’ll lean the basics of getting started with Solr, and an understanding of what Ruby solutions are available to simplifying adding great search to your site!

As usual, food and beverages will be provided.

OSC is proud to announce that we will attend and sponsor this year’s EdUI Conference 2009 which is bein held at the University of Virginia on 21st-22nd September 2009. A number of folks from the OSC team will be attending, and stop by our booth in the Vendor Hall on the second day and introduce yourself!

EdUI 2009 boasts a powerhouse lineup of renowned and popular headliner speakers, most often found at the Web industry’s premier events. In addition to these, it features a series of presentations, selected through a proposal process, to allow peers, colleagues, and geek kindreds to enlighten one another with their expertise and ideas. Our very own Arin Sime will be speaking on The Facebook API: Thinking About UI in a Social Way.

I am very proud to annouce the first book on Solr has been published by Packt. This has been a labor of love for myself and my co-author David Smiley, and we are excited to see the book now “in the wild!”. Below is a copy of the email sent to the Solr community:

Fellow Solr users,

I’ve finally finished the book “Solr 1.4 Enterprise Search Server” with my co-author Eric. We are proud to present the first book on Solr and hope you find it a valuable resource. You can find full details about the book and purchase it here:
http://www.packtpub.com/solr-1-4-enterprise-search-server/book
It can be pre-ordered at a discount now and should be shipping within a week or two. The book is also available through Amazon. You can feel good about the purchase knowing that 5% of each sale goes to support the Apache Software Foundation. For a free sample, there is a portion of chapter 5 covering faceting available as an article online here:
http://www.packtpub.com/article/faceting-in-solr-1.4-enterprise-search-server

By the way, we realize Solr 1.4 isn’t out [quite] yet. It is feature-frozen however, and there’s little in the forthcoming release that isn’t covered in our book. About the only notable thing that comes to mind is the contrib module on search result clustering. However Eric plans to write a free online article available from Packt Publishing on that very subject.

“Solr 1.4 Enterprise Search Server” In Detail:

If you are a developer building a high-traffic web site, you need to have a terrific search engine. Sites like Netflix.com and Zappos.com employ Solr, an open source enterprise search server, which uses and extends the Lucene search library. This is the first book in the market on Solr and it will show you how to optimize your web site for high volume web traffic with full-text search capabilities along with loads of customization options. So, let your users gain a terrific search experience

This book is a comprehensive reference guide for every feature Solr has to offer. It serves the reader right from initiation to development to deployment. It also comes with complete running examples to demonstrate its use and show how to integrate it with other languages and frameworks

This book first gives you a quick overview of Solr, and then gradually takes you from basic to advanced features that enhance your search. It starts off by discussing Solr and helping you understand how it fits into your architecture—where all databases and document/web crawlers fall short, and Solr shines. The main part of the book is a thorough exploration of nearly every feature that Solr offers. To keep this interesting and realistic, we use a large open source set of metadata about artists, releases, and tracks courtesy of the MusicBrainz.org project. Using this data as a testing ground for Solr, you will learn how to import this data in various ways from CSV to XML to database access. You will then learn how to search this data in a myriad of ways, including Solr’s rich query syntax, “boosting” match scores based on record data and other means, about searching across multiple fields with different boosts, getting facets on the results, auto-complete user queries, spell-correcting searches, highlighting queried text in search results, and so on.

After this thorough tour, we’ll demonstrate working examples of integrating a variety of technologies with Solr such as Java, JavaScript, Drupal, Ruby, XSLT, PHP, and Python.

Finally, we’ll cover various deployment considerations to include indexing strategies and performance-oriented configuration that will enable you to scale Solr to meet the needs of a high-volume site

Sincerely,

David Smiley (primary-author)
dsmiley@mitre.org
Eric Pugh (co-author)
epugh@opensourceconnections.com

A huge round of thanks goes to David for bringing me into this project and being such a great partner on it! With 5% of the proceeds going to the Apache Software Foundation, here’s hoping it’s a great success!

However, recently we’ve had two projects were the usual CI solutions such as CruiseControl etc haven’t worked out well, and we had to develop our own internal CI tool that we are ready to publish to the world called MonkeyCI.

On the first project, which was a PHP based application with 5 full time developers, we used CruiseControl with the phpUnderControl addon. However, we were running CruiseControl on what turned out to be an underpowered hosted Windows server, and we kept getting build failure errors related to environmental difficulties. Now, if you’ve seen my talk about CI, you know how big I am on speccing a beefy server for CI, and this experience reinforced that lesson. We decided that migrating the CI environment to a bigger server was something that we felt was in the “nice to have” category, and that it could wait till the next iteration. But we needed something immediate. Enter MonkeyCI.

The heart of CI is all about building the code, running the tests, and publishing the results frequently. Everything else, the reports, the red/green lava lamps, the pretty JavaDocs etc are all gravy. To meet the needs of your developers, you need to know if the “bar is green”. So MonkeyCI does that in a decidedly low tech way:

Everytime someone runs the full suite of unit tests they record the day and time, and put their initials. If the build is failing then they immediately fixed it. We’ve played with writing the results in red for failing builds and green for successful builds as well. Then, each day at the standup someone highlights how the CI is doing, and verifies that multiple folks are initialing, which means that the tests are running on multiple systems successfully.

While this does mean you have an additional manual process, it’s also really easy to do, requiring just a whiteboard! And for small project teams, the overhead of maintaining a reliable CI system is too much.

We’re doing another two developer project right now, and at least so far MonkeyCI has been great. We haven’t seen integration issues yet such as database scripts that don’t run, or busted code being checked in. I’ll post a picture of our whiteboard once we have a bunch of checkoffs recorded!

We call this simple low tech process MonkeyCI because typically we refer to anything manual, such as testing by pounding keyboards as Monkey testing. Also, somewhat of a reference to the great developers at the Primate Programming Institute who I am sure would use this approach to CI!:

At Sun, 3 May 2009 11:19:17 -0400,
Eric Pugh wrote:
Cool!

It’s one of those things that seems like everybody wants it, but no
one has quite figured out. And the various “services” like
thumbshots all feel kinda “seedy”, I am always expecting to see
advertisements for viagra stamped on top of the screenshots and
other questionable business practices.

It seems like you should be able to have the pages be render inside
of a library such as WebKit, but I guess rendering is very
intertwined with monitor displays and resolutions etc.

I have a research projects that aggregates info about people,
events, and organizations and I’d love a better solution for linking
in screenshots of the organizations and individuals site. Here is an
example using the thumbshot service for now..:
http://www.hightechcville.com/organizations/318-worrell-water-technologies

Here is the text (thanks to Mark Phillips for this):

Khtml2png – http://khtml2png.sourceforge.net/ “Khtml2png is a
command line program to create screenshots of webpages. It uses
libkhtml (the library that is used in the KDE web browser Konqueror).
In khtml2png 2.0.5 to 2.5.0, “convert” from the ImageMagick graphic
conversion toolkit is used to create the output files in various
image file formats. 2.6.0 and future development will use the built-in
conversion of the Qt library.” – from the Khtml2png website

Pearl Crescent Page Saver -
http://pearlcrescent.com/products/pagesaver/ “Pearl Crescent Page
Saver” is an extension for Mozilla Firefox that lets you capture
images of web pages. These images can be saved in PNG format or (with
Firefox 2) in JPEG format. The entire page or just the visible portion
may be captured. Options let you control whether images are captured
at full size (which is the default) or scaled down to a smaller size.
Page Saver uses the canvas feature that was introduced in Firefox 1.5.”
– from the Pearl Crescent Page Saver website

Webkit2png – http://www.paulhammond.org/webkit2png/ “Webkit2png is a
command line tool that creates PNG screenshots of webpages. …
webkit2png makes use of webkit, the rendering engine used in Safari.”
– from the Webkit2png website This utility is only available for Mac
OSX because of the dependence on Safari.

Webshot – http://www.websitescreenshots.com/ “WebShot is a program
that allows you to take screenshots and thumbnails of web pages or
whole websites. It comes with a command line interface for advanced
users. The following image formats are supported: JPG, GIF, PNG, BMP.”
– from the WebShot website WebShot uses Internet Explorer as the
engine for creating thumbnails of HTML files.

best,
Erik Hetzner

More information and registration is available at http://www.richmondspin.org/home22332.

I’m looking forward to a good crowd, lots of questions, and drinks afterwords!

I was attending the presentation with a certain sense of trepidation… I consider myself a hard core developer( despite my “Test Obsessed” armband) who doesn’t have time or patience for pointy headed manager paperwork. But I am also someone who focuses on process improvement and honing my craft of software development, so where can metrics inspired by CMMI be useful to a fast moving Agile team cranking out functioning software every three weeks? So I listened to Kris and tried to think about what he said in the context of Scrum.

The first slide pointed out that there are three kinds of lies: “lies, damn lies, and statistics”. You need to be careful about what your data says. When you are measuring velocity in Scrum, you need to have a couple of sprints, at least 3 to have any sense of your progress. If you say “we get X done” based on the first sprint, well often the first one is very conservative sprint. And, as you look at your average burndown, you need to have a couple of days of information before you can get a sense of average burndown as the first days often you find as many tasks as you accomplish. And, over the 15 days of your iteration, progress can be pretty spiky… Many teams have pretty flat burn down at the beginning, and then some steep drops… Ideally you are looking to see progress per day become flatter, less spiky, which would indicate that your estimating is improving. Or your team is being less affected by external factors that might hamper their productivity.

He talked about whether to use average or median numbers in looking at series of numbers, such as looking at your burndown.. If you have a lot of outliers, then use median to get a better view, otherwise use average. So if you burndown 20, 25, 22, 40, 25 then using the average is good. But if you have 8, 20, 25, 22, 40, 12 then maybe median would be better.

Kris talked about the cultural challenge of convincing people to provide the data required to build metrics. People need to know why the numbers matter, and even better why it will help then. It’s why developer’s hate filling out TPS reports! And why I like Scrum and it’s low overhead, as well as more passive measurement tools like HackyStat and 6thSense(now part of RallyDev). I feel like mandating one metric, say your basic time tracking is viable, but if you add more on you start getting more push back or gaming of the metrics.

He talked about getting metrics like Project Start Date and Project End Date. A local company splits up it’s year as 3 week iterations, and then apportions iterations across competing projects. These iterations then feed the project start and end dates.

He stressed that you need to have a shared vision on metrics, and a shared vision of what “on time, in budget, with quality” really means. I know we the other day had a scrum team debate how to track found tasks. And this was a set of people that had worked together previously on different project having different visions of tracking found tasks and how they should affect the “ideal burndown” line! Covering periodically what that shared vision, and ensuring your team and stakeholders are all on the same page is very valuable.

He stressed that your metrics need to be actual “measurable” things. You need to be able to quantify the metrics that you are using using a shared basis so that when you compare two things using the same metric that you are doing an apples to apples comparison, not an apples to kiwi comparison! For Scrum, it means you need to use the same time frame for iterations, and you can compare one sprint to another for a specific team on a specific project, but not across teams or projects.

Scrum for us provides a very standarized metrics across the OSC organization, regardless of client or specific technology. As long as we are sharing the same vision for our metrics!

When we do a retrospective, and look back at our burndowns, we are doing “Progress Indicators” that are lagging indicators. One of the things that the speaker was advocating was to look at forward looking indicators that predict into the future where we will be. But of course, identifying and seeing a leading indicator is difficult, and takes a lot more analysis. In scrum we would have to tie our tasks to various sprint goals which are appropriately estimated against to provide our velocity.

Did highlight that you need multiple projects happening to be able to gather the variety of data points to be able to compare data points. Compare two scrum teams together and it’s tough to compare them because you can’t see the outliers. But, if you have 10 scrum teams, who have a shared vision of the metrics, then you can start comparing them together. You may have to normalize across the teams, but with enough iterations you can compare and predict.

So some things to show would be a histgram of how much the team burns down a day. Highlight what kind of deviation we have in our progress per day. Over multiple sprints you can maybe see what the first third, middle third, and final third look like. Can we characterize “At OSC, we typically see this kinda of result in the thirds of the project?” Hey, does this feed into our Waterfall projects in 3 weeks?

First week is requirements solidification. Second week is development. Third week is testing and polish.

Can we figure out how to predict the results for sprint 3? We could do this for sprint 1 and 2 and predict sprint 3.

We measure progress per day as a ratio, and then sum it over a week. With that progress per week, then we can see what a sprint 3 would do.