I like this post from Jon Marks at JonOnTech. It’s about questions you should be asking your CMS vendor you might not have thought to ask. The first five are especially good (see his post for the explanation of each question and the rest of the list):

1. Who was the last vendor to beat you in the last round of a selection exercise? Why do you think they won?
2. If, in a few years time, we decided to move away from your product, how would I go about migrating all my content into a new system?
3. How many active developers do you have on your developer forums?
4. All of these are important, but please rate these in order of your priority: a) Product Features b) Performance and Stability c) Usability d) Security
5. How much would I expect to pay a contractor developer that is skilled with your CMS, and are they easy to find?

I am consistently disappointed with how companies evaluate and choose software vendors. Part of the problem is when companies use RFP processes that handle software purchases the same way that factory equipment purchases are handled, but that’s another post (see Making RFP’s More Effective).

The other part of the problem is the questions that never get asked during the vendor pitch. To Jon’s list, I would add:

1. How long and how many resources did it take to build this demo? You’re looking for closeness of fit, effort to customize, and skillsets involved.
2. What are the top three technical resources my team should have at the ready during the implementation? You’re looking for availability and helpfulness of documentation. How much of it is vendor-produced versus community-produced? It’s not necessarily bad if the majority of the resources are community-produced–it’s just a data point.
3. If it makes sense depending on the kind of software, ask do you use your own software in-house. If they don’t, that’s certainly a data point. If they do, ask, as an end-user, what are your top-three headaches when using the software? This is sort of a “what is your biggest area for improvement” kind of question–watch out for turn-your-weakness-into-a-positive kind of answer (“The software is just too powerful!”). Every piece of software has idiosyncrasies. They should be able to name a few.
4. Tell us about the last implementation that just went completely sideways for reasons attributable to the technology, not to project mis-management, political, or other issues. Obviously, the vendor scores points for honesty on this one, but it’s also interesting to hear how much/little the vendor was involved in salvaging the deal (if it was able to be salvaged).
5. What is your maintenance renewal rate? I’ve never heard this one asked, but I would think this would be a very telling stat. Customers have all sorts of reasons for not renewing maintenance, but the obvious one is that they feel like the vendor isn’t giving them enough support value for the expense. For commercial open source vendors, support may be their sole source of revenue (excluding professional services, hosting, etc.), so for them you’d think this would be a very high number, otherwise, what’s the point?

By the way, giving your vendors a good grilling isn’t limited to software companies. Picking a services firm also deserves a good set of probing questions, but that’s also another post.

What about you? Got any good questions to ask CMS or other software vendors?