The Diminishing Science of Research in Technology Analyst Firms

The technology analyst industry has ended its barrage of exuberant predictions for the new year. After digesting all of these commentaries (some were a challenge, I admit), we need to stop our predicting and instead start a discussion about what all of us ought to be doing. From my perspective, that should be an annual statement of direction, what we here call the research agenda. I have decided it is time to remind all of us what this important activity is and why it is really important to see from industry analysts.

It should be a standard practice in our business for each firm to issue a research agenda that is supposed to guide its analysts or practices in the coming year. Ours have been available to clients and now will become public record as they are annually. Unfortunately, most analyst firms have excused themselves from performing this important step; instead, they just bring forward predictions and what I have heard from many of my colleagues as educated guesses from their opinion and inquiries from dozens of clients that are looking to advance their technology. The absence of a research agenda is not a good idea, for the simplest of reasons: Clients pay for services, and they have a right to expect that the analysts whose time they pay for will set out what their knowledge and experience leads them to conclude are the areas on which they should focus so the clients can evaluate how it matches their own objectives.

At Ventana Research we are guided in setting our research agenda by what our clients need to advance their competency and maturity. In practice, for each market segment this requires comprehensive research across hundreds of organizations. We do this using our benchmarking methodology, which provides a framework for conducting, analyzing and presenting the research. Using this approach we are able to evaluate the level of maturity of an organization in a particular realm and also to drill into the maturity of its specific people, process, information and technology competencies. Another required element of how we establish competency and maturity is that we conduct research in both the business and IT sides of an organization. Researching only one of these two audiences can significantly skew the research findings and thus guidance provided to clients and the industry.

We’re hardly the only firm that claims to conduct research. But we are unique in how our research is conducted within a consistent framework and is guided by clearly stated overall objectives. I’ve pointed out years ago (See: “Why Bad Research Could Jeopardize Your Business”) that conducting surveys without a methodology and not as part of an overall research agenda can easily rob the results of actionable value. Our benchmark research, in contrast, provides us with a strong foundation on which to build our education efforts and to use in advising business and IT buyers.

Our research and the insights it yields makes it possible for us to determine best and worst practices, providing a perspective on what to do and what to avoid. Most analyst firms fall far short in identifying these because they do not conduct primary research, depending instead only on client inquiries and in most cases exclusively from IT. Unfortunately, this can bias their view of the market. Recently one of the largest IT analyst firms, one that does not do primary research across business and IT, admitted publicly that it was late to the table in covering mobile technologies. It did not see that the business side is driving the adoption of mobile technologies in the enterprise because it only advises the IT side, which has not yet been pushing to adopt these in more than a corporate standard for electronic mail.

Nor is just doing buyer research sufficient. One needs to understand the technology itself in order to evaluate its fit for the people and process needs of an organization. Doing so requires not just supplier/vendor briefings and attending product demonstrations and reading product reviews but actually engaging into trying the combination of software and technology. It still amazes me that the majority of analysis who do not like to actually see, try and evaluate the technology they offer advice. It is important that analyst firms offering advice and guidance have undertaken a detailed assessment of different vendors and their products to assess their merits in order to be able to match them to buyer needs and evaluate how well they address the overall market.

We do this using an approach that is framed as a balanced RFP (request for proposal). It includes evaluation categories such as usage, capabilities and management as well as cost, benefit and an examination of other customers’ deployments. Some approaches spend more time looking at the vision of the vendor and the future of their products or looking at their financials and number of customers. While these are important, they should not comprise much more than 20 percent of your evaluation review of vendors and products. If you are weighting these vendor criteria much more than this amount, you could be doing your business and technology environment a disservice.

Why I am starting a conversation on the science of research and what you should expect from technology analyst firms? The answer is to let you, the buyer, know that there is a wide spectrum of research methods across analyst firms, and to ensure that you understand this to be able to determine the value of the services and advice you are receiving. Does the analyst with which you work have sufficient experience in the segment or technology that matters to you? Does that analyst derive knowledge from in-depth buyer research, supplier research, or both? Does his or her perspective come from being a counselor to organizations or only from reading research and education that is already published? Does the advice you’re given speak directly to your current competencies and those of your organization, or does it just regurgitate advice drawn from a spread of technology adoption laggards to leaders?

All of these are questions you should ask yourself to ensure you get the advice you need – and, more importantly, the quality of knowledge transfer that can enable you to decide your needs and to act on them quickly. It is time for you to be more knowledgeable on what you are hearing and reading to ensure you can trust the information and advice you are utilizing for your business and IT organizations. I may be biased in my framing of this dialogue, of course, but after 20 years of seeing the good, the bad and the ugly of analyst firms and worked for one other than Ventana Research, I thought it would be good to start a discussion on what you as a buyer should expect in analyst firms and their research to provide you with the service you need and deserve.

Let me know your thoughts or come and collaborate with me on Facebook, LinkedIn and Twitter.


Mark Smith – CEO & EVP Research

The Business Technology Revolution in 2011

The new year has brought a revolution in business technologies that allows enterprises to roll out new applications, processes and services in record time. What used to take a business six months to a year to develop and deploy – and not too long ago that was considered fast! – can now be done in a week or two, making enterprise application and business intelligence deployments of the recent past seem more like ancient history.

Businesses can thank five new technologies for this: business collaboration; social media; analytics; business mobility; and cloud computing.  Perhaps the most important of these is what I call business mobility technologies – that is, smart phones and tablet computers. In 2011, businesses can deploy an Apple iPad, rent new analytics and collaboration applications as part of a software as a service (SaaS) offered in the cloud, configure and load initial systems within days, and introduce new advancements within a week. This cycle time validates a point I have been making for a long time: that ever-more-rapid innovation is of the keys to business success today.

This business technology revolution means that businesses can finally make a long-held dream a reality – that with the right leadership, resources, time and working capital, they can deploy technology effectively and use their human capital efficiently. I, for one, am excited that after a decade of limited innovation in business technology, important enterprise issues can now be rapidly be addressed. Adopting these new business mobility platforms is disruptive, of course, but it will introduce innovation to organizations that need it. Too many are currently operating under the lock of Windows-based computers running Microsoft Web browsers to connect to local Windows-based servers running applications that took months or years to develop, customize and install, and which are in some cases almost a decade old. Innovating in this environment takes too long. With the new business mobility platforms, companies operate in cloud computing-based collaboration environments that enable people to interact to address a range of human capital issues and to evolve business processes as needs arise. This has created a new tone of optimism in business.

The challenge for business now is to assess these new mobile computing platforms and their portfolios of applications, information and services. It is vital to understand how they can support the  range of activities required for each area of a business. Any portfolio being considered should go well beyond the classic trio of e-mail, instant messaging and basic Internet browsing, as they are merely the basics required of any business mobility technology. Although this may seem simple and obvious, it actually is not. Since we are at the moment so early in the evolution of smart phones and tablets, companies need to examine what is available to meet their business needs and what they will need to develop.

Many organizations will be challenged to come up with the budgets and skills to step up the business to the next level. Nothing comes without investment; therefore the business case needs to be based partly on a cost-and-benefit analysis but also on the need to innovate and advance technologically for the organization to stay competitive. As I see it, the combined effects of these business technology investments can have a dramatic impact for every organization no matter what size and industry. The challenge is yours – you need to look at your organization’s technology with new eyes to see to what extent you can renovate existing approaches, or if you need to innovate with something that is more fresh, more interesting, and especially more relevant for your company’s future.

Let me know your thoughts or come and collaborate with me on Facebook, LinkedIn and Twitter.


Mark Smith – CEO & EVP Research