“Shadow IT” is a term to describe what happens in large organizations when an IT department is non-responsive to their customers’ (the Business Units) needs. The catalyst is usually some critical business problem that needs to be solved that the IT department is too busy to respond to or simply deems unimportant. The business unit (or department or division) responds by either hiring, training, or outsourcing their own technical talent, throwing a server or two under a desk, and implementing a solution. Repeat this a couple of times and you’ve got a fledgling IT department that is essentially living “off the grid”.

There are different techniques for dealing with Shadow IT. Overbearing CIOs see these “Cowboys” as a real impediment to Enterprise-wide standardization, integration, and their ability to drive down technology costs. They seek to squash these maverick groups. Tactics include applying political pressure to division heads to reassign projects to IT or shut them down altogether, denying physical access to critical infrastructure (like the DMZ for external access to an application), or smothering initiatives with draconian methodologies or heavy-handed project management oversight. The result is a build-up of resentment between IT and the business which, of course, spawns new and strengthens existing Shadow IT outfits.

On the other end of the spectrum, weak CIOs live with their heads in the sand–naively or purposefully ignoring that crazy guy over in Sales that’s building a Next Generation Internet application for his national sales force. In this case the result is that Shadow IT departments grow in number and strength while the IT department weakens and becomes irrelevant, ultimately relegated to keeping the spam filter updated on the email server.

The smart CIO has a strategy for dealing with Shadow IT. But before we get to that, let’s look at why this is something that every CIO needs to be thinking about.

The first wave of Shadow IT

The 1990’s were boom times for Shadow IT. Companies like Microsoft and Lotus offered platforms that business users could leverage to create complex applications quickly and cheaply. For less than $10,000 a department could get a Notes server, a few “designer” clients, and enough Notes clients for their department. Or maybe the company had already purchased an enterprise license which meant there were zero license costs involved. The platform delivered tremendous value and functionality out-of-the-box. If IT wasn’t interested in running or supporting Notes (or Exchange), the business could do it themselves with very little effort.

Over a short period of time the complexity of the platform, the out-of-the-box value it delivered, and the number of disparate implementations within large enterprises increased at a rapid rate. For most, costly consolidation efforts were inevitable as departmental implementations grew in scope to include integration with enterprise systems, huge volumes of data and users, and concerns about high availability and disaster recovery. Departmental developers–some having started as business folks who liked to dabble in technology–found themselves in the deep end of the pool as the platforms began to rely less on macro or scripting languages and more on mainstream, object-oriented development languages.

Open source fuels the second wave of Shadow IT

Today, as open source continues to move up the stack, business organizations are again able to implement incredibly complex solutions without relying on centralized IT. Enterprise-ready application servers, portal servers, collaborative platforms, content management systems, web content management, relational databases, and reporting solutions are all freely-available. Divisions which previously may have had to pool their requirements with other parts of the business in order to better cost-justify licenses for proprietary technology can now focus on “fit” rather than “cost” as a primary value driver. The result is a growth in Shadow IT.

Smart IT organizations partner

So what is a smart CIO’s response? Smart CIO’s embrace the reality of today’s technology environment: (1) They know that open source technology will find its way into their organization and (2) They know they cannot possibly keep up with the rate of change demanded by their customers without creating massive, inefficient organizations. This leads to the conclusion that CIO’s must embrace Shadow IT as a partner in the delivery of open source solutions to the business units.

Essentially the idea is this: Do everything you can to help the business units help themselves. Leverage the strengths inherent to a centralized IT organization (scalable infrastructure, backups/disaster recovery, help desk, security, regular maintenance rythyms) and leave the rest to the IT partners (aka Shadow IT) in the business units. This is the IT-as-utility-company mindset.

With that mindset in place, IT should:

• Ask their customers about the solutions they are looking to build in the next 6 to 18 months. This information can be gathered for all business units to help figure out where IT needs to focus first.
• Collaboratively select and implement major components of the infrastructure such as a JCR-compliant CMS like Alfresco, a portal like Liferay, a web content management solution, and smaller components like chat, wikis, and blogs. The actual types of components needed in your organization will depend heavily on the results of the first bullet. Also, note that this must be a collaborative effort. The Architects are actually going to have to leave the ivory tower and work hand-in-hand with the businesses on selection and implementation.
• Support these components like you would other significant pieces of your infrastructure (databases, application servers, etc.). That means having dedicated teams that use the community to track and implement bug fixes and security patches (or paying someone like SpikeSource to do that for you).
• Centralize expertise but decentralize major project efforts. IT should have a handful of experts on hand that can be loaned to the business units for significant project efforts, but the business ought to run and staff the projects. Otherwise, the needs of each business unit get prioritized along with everyone else’s and IT will be seen as unresponsive.
• Provide guidelines (not standards, necessarily) or blueprints that show some possible ways in which the infrastructure can be leveraged to build business solutions.
• Be tolerant (and even learn from and offer assistance to) business units who still choose to go it on their own, particularly if they choose one or more of the IT-recommended open source technologies.

So if you are in the IT department of a large organization, start now. Partnering with Shadow IT on their open source initiatives will help prevent today’s pilot projects scattered throughout your organization from becoming tomorrow’s problem children left on your door step.

And if you are part of a Shadow IT organization, give me a call. I’ve got plans for all of those Lotus Notes applications you’ve got sitting around.