Adventures in Puppetry!
My first attempt at using Puppet to automate a server build was not as easy as I’d hoped. The goal seemed simple enough: Build a RedHat server running Solr 4. What I didn’t know was that this particular path is not as well-traveled as, say, running WordPress on Ubuntu. You’ll notice I wrote “running Solr 4″ rather than Tomcat or Jetty. When I began this quest I didn’t care which container held Solr. After a quick discussion with Eric Pugh I found out that, unofficially at least, the core Lucene and Solr developers recommend Jetty 6 for running SolrCloud. The reasoning was easy enough to understand: Jetty 6 is packaged with Solr and that’s what gets tested the most. Since SolrCloud is a very new feature I was inclined to go with what’s tested. So from that point on I focused on installing Jetty 6. Getting Puppet installed was easy. We use Amazon Web Services a lot, so the first thing I did was log into AWS and instantiate a RedHat Enterprise Linux 6 (RHEL6) virtual machine. That’s the RHEL version that shows up in the quick launch screen, so that’s what I picked. Latest and greatest, right? Here’s the Puppet install distilled:
sudo rpm -ivh \ http://yum.puppetlabs.com/el/5/products/i386/puppetlabs-release-5-6.noarch.rpm sudo vi /etc/yum.repos.d/puppetlabs.repo sudo yum update sudo yum install puppet sudo puppet resource cron puppet-apply ensure=present user=root minute=1 \ command='/usr/bin/puppet apply $(puppet --configprint manifest)'
Well that was easy! In fact, I could have gotten that far by using one of the sample Cloud Front templates AWS provides. My next step was to see if PuppetForge had a Jetty 6 module: Nope! But I did see that there was a Solr module, but it was only working on Ubuntu. To see why, I went to the project’s GitHub repository and browsed the init.pp to see if I could find anything Ubuntu-specific. The only thing I found was a reference to a package called “solr-jetty”. That meant that there was an Ubuntu package called solr-jetty available, but not one for RHEL6. To make some forward progress I decided to try this out on Ubuntu so at least our development environments could be automated. What I found out was that the solr-jetty package uses jsvc to launch Jetty, and jsvc is a binary executable that’s dynamically linked to a shared library called “libcap.so”. My Ubuntu VM only had something called “libcap-ng.so”, and no, it wasn’t compatible. Fine. I didn’t want to run on Ubuntu anyway. From there I spent some time searching for and checking out the various Jetty RPMs available for RHEL6, but they only included the APIs. What I really wanted was an RPM that did operational things like create a Jetty user and init.d service. The Red Hat Yum facilities don’t have it, but I found a package on JPackage that fit the bill: Plain ol’ jetty6 RPM. But JPackage doesn’t work on RHEL6! This might have been a temporary bug in their site mirrors, but it kept me from getting what I wanted. Fine. I’ll run RHEL5. And then something magical happened: Things started working! RHEL5 doesn’t include wget, so I tried installing it through puppet:
puppet module search wget puppet module install maestrodev-wget puppet apply -e 'include wget'
And it worked! Next I hooked up JPackage:
wget http://www.jpackage.org/jpackage50.repo vi jpackage50.repo
I had to enable the distro-specific repository, but when I automate all this I’ll just install a pre-edited jpackage50.repo as part of the manifest. The Jetty Puppet modules I found were only available on GitHub and not PuppetForge. I briefly tried using one of the git modules available on PuppetForge. I chose one that seemed the simplest and installed it to my locally available modules:
puppet module search git puppet module install puppetlabs-git puppet apply -e 'include git'
This didn’t work, but that was probably due to my newbie approach in the last line rather than putting something in site.pp. I decided to uninstall that module and just pull down the zip file I needed manually:
puppet module uninstall puppetlabs-git wget https://github.com/puzzle/puppet-jetty/archive/master.zip mv master master.zip unzip master.zip mv puppet-jetty-master /etc/puppet/modules/jetty puppet apply -e 'include jetty'
This spat out a bunch of errors related to the particular Java package RHEL5 came with: 1.4.2-gcj. Wow: I didn’t realize RHEL5 was that old! Now, the GCJ Java package is a dependency for a few other packages so Puppet couldn’t uninstall it without some help. It didn’t look like I needed those other packages, so rather than risk running into dependency hell by having Puppet uninstall them and then repeating for every other dependency, I decided to use Yum to uninstall:
yum erase java-1.4.2-gcj-compat puppet resource package java-1.4.2-gcj-compat ensure="absent"
I’m not sure if the last line did anything to change the state of that Java package in Puppet’s internals, but it did show me that Puppet believed Java was absent. Finally:
puppet apply -e 'include jetty'
Puppies and rainbows! It worked! Then the icing on the cake:
puppet resource service jetty6 enable="true" ensure="running"
I poked around the running processes, etc and var/log folders, just to see if it looked like Jetty was installed properly. Then I hit localhost:8080 with curl and got a nice 404 error to stdout and a log entry in the request log. So the road was rocky and filled with dead-ends, but it worked. I saw a few comments along the way from people who were hosting their own Yum repository for deployments. At the time I didn’t want to add that complexity to our environment, but now it looks like the best way to proceed if we want to run Jetty on RHEL6. As for running it on Ubuntu, well, that’s a challenge I can safely pass on for now.
1/25/2013 Edit: It looks like there’s an even easier way to install JPackage with Puppet:
puppet module install yguenane-jpackage puppet apply -e "include jpackage" puppet resource yumrepo jpackage-rhel enabled="1"