This blog is one of a series of blogs on the challenges of learning Chef, ops, and the technical world of IT all in one motion. Jeff Carapetyan is an Ops consultant concentrating on basic Chef work, and is available for contracts in the Austin area.
I probably shouldn't be starting this.
What I am about to introduce is a controversial topic. If given only the prompt "community cookbooks", a room full of Chef users would be able to talk for several hours on the best approaches, stating their opinions on how things should be run.
For me, where I stand is that community cookbooks are not for me, or not as they currently exist. This position is not where I started out.
Others may say that they are excellent and an integral part of their workflow. And I don't want to disagree with them as it clearly works for them, just not for me. However, I think that everybody can agree on a couple of issues that do pop up when using them.
Credits
Much of the thinking that went into this article came from extended conversations involving Matt Ray at our local Austin Chef Meetup.
Other clarity on this thinking came from this great talk by Noah Kantrowitz.
Cookbooks are highly opinionated in their design
Those who are new to Chef probably won't understand the distinction of a library vs. application vs. wrapper cookbook (and some experienced people who don't either. Go here for a better understanding), but anybody with about 10 minutes of Chef experience will tell you that no two cookbooks are written alike.
There are so many options in fact, that it gets tough at first to grasp how you should be writing and debugging a cookbook. None of this matters unless you write a cookbook for everyone. Which means, by definition, that almost everyone would have written it a different way than it got written.
A Case of Over-reach
In the defense of the cookbooks, it really isn't their fault that they got this way. The goal of a community cookbook is to have one cookbook that does everything you could ever want to do for an application or service. The problem is the degree of difficulty in accomplishing such a thing. What starts as a simple cookbook for one OS very quickly becomes a hydra with every special use-case and OS version that you have to account for. In doing so you have add attribute values and libraries and write custom ruby code well beyond the scope of normal chef, and it makes things far more difficult to understand. This is a problem that has gone away over time, but is one that may appear as the most offensive and egregious.
Noah Kantrowitz does a fantastic job of defining the challenges of reusable cookbooks in this great talk. The two biggest takeaways I find in this talk are
- A good re-usable cookbook is not twice as much work, it is 10 times as much work
- Good cookbook design is like good code design - it takes a lot of discipline, art, and sticking to well worn patterns.
They still require you to know a lot about the stuff you are installing.
I know how bad this sounds, but it is an important point. The downside to all these configurable options on version numbers and types and configuration options is that you have to know which ones to tweak to get your version. The "default" settings that come with the cookbook might be dramatically different than what you need. Therefore, you have to know the difference between the two and how to get everything running. For senior level people who are looking for an easy way to set up an nginx server like they have 1000 times before this isn't a big deal, but it still remains a barrier to entry for the entry level people.
For example, if there are 8 key attribute to configure properly, and 2 of these 8 must be changed in tandem with each other, how would the random user even know that?
There are robust cookbooks, but there isn't a way to tell which is which is which
Matt Ray comes right out and says it. "Maybe [6?] community cookbooks are really great cookbooks that anyone could and should use. The rest....nah" He then enumerates the ones that fits this rare category. My experience has validated his comments as being correct.
Matt also prescribes the required solution: Namespacing and versioning. If you only have one hadoop cookbook no one is going to be happy. But if they are namespaced and versioned then there's a better chance that multiples can exist, one of which might fit your use case.
Matt also speaks of an antipattern - the rating system. A 1-5 rating on a generic hadoop cookbook might be meaningless: Two hadoop cookbooks written for different purposes could both be 5 rated, but only for their intended purpose. Using the other 5 rated cookbook for the wrong use case would make it a 1 for that use case.
Because of these challenges, the rating system to tell if a cookbook is good or not is almost entirely ignored, and so people usually just go with the most downloaded cookbook of that type. That type of thinking may be ignoring a cookbook which is way better written and easier as well.
Sometimes, they don't even work right
One thing that I found out quickly was that not all community cookbooks are readily maintained after being created. Someone who reserved the namespace may have written one version and then utterly forgotten about it, making it obsolete and useless. And don't think because it is readily maintained that it is bug free. I have had more than my fair share of issues with some cookbooks not installing properly on a new version (again, looking at you Java).
The Future
There are a few things coming on the horizon that are making community cookbooks look more promising, but nothing released currently. The first is the introduction of Chef-supermarket, which will address a couple of the major issues.
The second is that people are realizing community cookbooks are too complicated, they are doing what they can to simplify them properly.
Bottom line, this is an area which can only get better from here. I'm hoping for great things.
Here Is What I Do:
- I write my own cookbooks. Seems crazy to re-write what others have already done, but my cookbooks are really small and for a very specific use case. Listen at the conferences and you will find that this is very common advice.
- I always steal ideas and insights from community cookbooks. This is where they really shine. Great stuff.
- I'm super into testing and versioning and modularizing my cookbooks. Like the well worn Unix Philosophy "Only do one thing and do it well".
You can see the cookbooks we use here for our internal demos at our github site.