It’s hard for me to lend much credence to Forrester’s Vendor Scorecard for Alfresco, recently released as part of its Forrester Wave report on ECM Suites. The reason is that I don’t believe too many people are looking to buy an “ECM Suite” at least as Forrester defines it.

Forrester doesn’t seem to care that most of us have moved past the legacy definition of ECM as an all-encompassing, single-vendor software suite. But as the “Suite” term implies, Forrester’s report is slanted toward gigantic vendors who offer every piece of technology even remotely related to ECM.

This is in direct contrast to what Optaros clients ask us for. They want agility, speed of implementation, product innovation, and freedom of choice. Many are implementing Web 2.0 and Enterprise 2.0 solutions which work better with the concept of “ECM as a service” rather than “ECM as giant, complex software offered by a single vendor who has cobbled together their offering from acquired and poorly integrated products”.

Given that, it is odd that Alfresco would be measured with the same scorecard used to evaluate an “ECM suite”, but that’s what they did. They judged Alfresco unfairly in a couple of places and generously in others. Really, though, the report made me realize Forrester needs to come up with a better way to segment and evaluate the market because I’m not sure people are in the market for the product that would score well against their scorecard.

Let’s take a look at the report and you’ll see what I mean. Specifically, I’m looking at the Vendor Scorecard spreadsheet for Alfresco. I’ll start by calling out what they got right.

Alfresco scored a perfect 5 for “Platform readiness”. Unfortunately for Alfresco, it’s only worth 2.5% of the total score. This set of criteria favors suites that accomplish more with fewer disparate sub-systems or products so it makes sense that Alfresco would do well here.

Alfresco took a ding on E-forms and Email archiving scoring 1.0’s for both criteria, which together count toward 2.5% of the total points possible. Alfresco can bump up their score by extending current XForms functionality to the DM side of the house which is long overdue. Email archiving should get a better rating when the new “email addressable objects” functionality is released.

This starts a theme that repeats itself throughout the report: In keeping with the “Suite” approach, Forrester expects an ECM vendor to offer better E-forms and Email archiving, for example, than pure-play vendors. This automatically favors the largest of the ECM vendors because they simply acquire and integrate pure-play offerings. So don’t expect a “5” in these areas for Alfresco any time soon using the “Suite” standard.

“Core capability architecture” scored well, which is a measure of Alfresco’s ability to work in an “enterprise” IT environment, but, strangely, this score is only worth 5% of the total. That’s not a typo. The ability for the product to leverage a client’s application and security infrastructure as well as its extensibility is not as important, in Forrester’s view, as the ability of the ECM vendor to offer Business Intelligence software (see “Differentiation Strategy”).

Forrester’s expectations for the breadth of an ECM Suite become clear when you look at what a vendor would have to do to get a perfect score. And this is where I start to question the value of their definition of the ECM Suite, let a lone the scorecard itself. Let’s look at a few examples…

Under “Core ECM capability breadth”, the “Content Integration” category is interesting. On the face of it, I would think products that support open standards and have many API’s available for integrating the repository with other systems would do well here. On the contrary, Forrester gave Alfresco a 1.0. Here’s what a vendor must offer to get a 5:

“Comprehensive, independent content integration offering supporting central, federated bi-directional access and management of content in heterogeneous, disparate repositories and content information sources.”

It’s like the “E” in ECM stands for “Excessive”. Seriously, I’m sure certain clients will have a legitimate need for this functionality but there are more common, more practical integration-related concerns vendors should be striving for.

Let’s turn our attention to “Extended capabilities” which is worth 10% of the total points possible. Alfresco got an overall score of 1.05. Alfresco’s lack of an Enterprise Search offering (Score of 0 with a weight of 15% for the category) and, in my opinion, a mis-rated BPM capability (Score of 1.0 with a weight of 20% for the category) were the primary culprits here.

The inclusion of Enterprise Seach as part of an ECM Suite evaluation floors me. Does anyone really want to buy an enterprise search engine from their ECM vendor? I suppose if you are IBM you are happy it is included. It is about this point in the report I start to form a mental image of Forrester’s Dream ECM Suite as the giant, lumbering Imperial Star Destroyer shown in the opening sequence of Star Wars (or, better yet, the parody of the same in Spaceballs in which the enormous ship has a bumper sticker that reads, “We Brake for Nobody”).

The BPM score is simply a bad rating. Either the criteria for BPM is so overblown as to not be relevant to the majority of clients who need workflow within an ECM suite or the analysts simply did not do enough research into the capabilities of the embedded JBoss jBPM engine. Forrester doesn’t offer any details as to why it considers the functionality to be “basic” so we’re forced to guess.

Collaboration is another example where Forrester’s threshold is set way too high. Alfresco scored a 2. To get a “3” in this criteria–an average score–you have to offer real-time web conferencing. Huh?

When you get to “Strategy” and “Market Presence” you realize who Forrester values: Huge vendors that offer everything to everybody and have more than $10 billion in annual revenue (that’s what it takes to earn a 5). And to be successful, the vendor must not focus exclusively on ECM. Take a look at “Differentiation strategy”. This one’s important to Forrester. At 12.5% of the total score it represents the single most heavily weighted item in the scorecard. To score the max, here’s what a vendor has to do:

“Vendor differentiates by offering ECM as part of a broader information management portfolio (data, master data management, BI, content, BPM, portal, and collaboration technologies) with a promise of lower TCO.”

Hmm. Who could offer broader information management, Business Intelligence apps, Portal, AND collaboration technologies? Wait, don’t answer yet. The vendor musn’t stop there. It’s not quite broad enough. To score max points on “Vendor’s business”, another 5% of the total, the vendor must provide “…IT application infrastructure solutions including application infrastructure, developer tools, middleware, and information management.” Basically, if your name is IBM, Oracle, or Microsoft, give yourself a 5.

So Alfresco took a couple of knocks here and there it probably shouldn’t have. The bigger issue is that Forrester’s vision of ECM is out of step with what clients want and their scorecard reflects it. No one’s in the market for Imperial Star Destroyers anymore.