Splunk Provides Operational Intelligence in the Cloud

Splunk’s innovated ability to access and use machine data for targeted operational insights can help improve IT and enhance business operational efficiency. Its work to capitalize on big data was part of my last analysis, while my colleague Tony Cosentino looked at its focus on search and operational analytics. Splunk also was a recipient of the 2012 Ventana Research Technology Innovation Award for IT Performance for Splunk Enterprise.VR_2012_TechAward_Winner_Logo

The latest Splunk release, version 5, advances its ability to provide operational intelligence to organizations by using data both from existing IT systems and in Hadoop. A new SDK offering supports Java, JavaScript, PHP and Python as part of its API. It can help developers get at data from social media and online applications and combine it with machine-generated data, which our research found to be the top data source according to 62 percent of organizations, followed by application data (53%) and historical data warehouse sources (43%).

Splunk also now provides a cloud computing offering called Splunk Storm that helps companies take advantage of machine data in the cloud or on-premises. It lets users quickly create projects and analyze machine data with charts that can be shared with others. I had a chance to go through the new offering and found it to be simple and quick to analyze data and present analytics based on events, IP addresses and other machine data. Our latest benchmark research into operational intelligence found that activity or event monitoring is a top need in 62 percent of organizations; Splunk Storm can address this through its search and analyze approach.

Splunk has made the pricing of the new offering simple. You select the storage volume you need in the cloud and get the pricing quickly. You can sign up for free and work with the product on data up to 1GB. Organizations can also analyze cloud data with the on-premises version of the product, but for many who need to quickly assess data without the hassle of using internal resources and systems, the software-as-a-service version is an easy way to get started. This is important, as the top barriers for operational intelligence are lack of resources (41%) and no budget (40%), and the on-demand approach is now preferred in 21 percent of organizations, which allows Splunk to address an expanding opportunity.

With its on-premises product and its latest cloud computing offering, Splunk provides customers a good array of options. The company is moving quickly to add alerting and APIs that can be used to integrate to other offerings to upcoming Splunk Storm releases. Splunk has little competition when it comes to combining machine data with other data from business and IT to help organizations in cloud and enterprise approaches. It lets businesses harvest existing sources without having to establish a specialized data store first. Organizations should take a look at what Splunk is providing and how it can help address a class of operational and analytic needs across IT and business data.


Mark Smith
CEO & Chief Research Officer

Splunk: Big Data Machine for Operational Intelligence

Splunk recently entered the financial markets as a publicly traded company (NASDAQ: SPLK) and also entered a new phase in its corporate growth. Splunk combines the power of search and discovery with analytics on data generated by IT systems, that they call machine data, and provide insight for a new generation of operational intelligence that helps everyone in IT including the CIO determine the efficiency of its systems that support business. The company has built a platform that can index data on a large scale (“big data”) for rapid analysis and search. They also through its analytics provide the ability to perform visual and data discovery which is critical to reduce the time to determining unknown issues in existing IT systems. This helps IT staff ascertain not just the performance but the efficiency of systems that operate on a 24-by-7 basis. Splunk’s software operates in real time, surpassing the traditional methods of applying business intelligence against a data warehouse – a practice that’s ineffective for use in IT, where time is not the CIO’s friend when it comes to understanding issues or opportunities for improvement. Splunk has grown rapidly, partly because it’s simple to download and try, and then to license for use in production. It has more than 3,300 licensed customers in 75 countries. The management team is led by CEO Godfrey Sullivan, who has experience and a track record at companies such as Hyperion.

Where is Splunk finding its opportunity? IT organizations have been like the cobbler’s children who have no shoes. While they deliver applications and technology to business, IT departments seldom have time to analyze and optimize their own processes, systems, applications and projects. While this might sound counterintuitive, it is true. Almost seven years ago, I started investigating whether performance management being applied elsewhere in a business was occurring in IT. I discovered that CIOs lack critical tools to define goals and track the performance of their people and machines. Our IT Performance Management benchmark research identified lack of resources, budget and awareness as major contributing factors to this situation. It also showed that organizations that deploying appropriate tools were increasing IT effectiveness and supporting the needs of business more efficiently. Machine data or files were the top data source within 43 percent of IT organizations, and balancing loads between networks and servers, doing root cause analysis and doing impact analysis were key tasks in at least 52 percent of them. Splunk, which addresses this niche, has had significant growth over the last three years even though economic conditions have been shaky.

Last year we came out with new benchmark research on IT Analytics that investigated analytics related to IT organizations managing systems that support business. This research found that only 15 percent of IT organizations are at the highest level of maturity in IT analytics in the key categories of people, process, information and technology; the bulk of organizations are still struggling to function effectively. Interestingly, 91 percent of IT organizations want their analytics to be simpler – understandably so, as 79 percent are using spreadsheets universally or regularly and more than half are dissatisfied with their existing technology. The top issues with current efforts are that information is not readily actionable (for 46%) and not adaptable or flexible to change (44%); many organizations have reports or files but can’t assess and act on the data. Splunk also addresses that need of more than half (52%) that are not able to easily get the data they need for IT analytics. Considering that IT spends more than two-thirds of its time in the analytics process in data-related tasks as opposed to actual analysis, Splunk’s ability to get to machine data and index it for search and analysis is a welcome sign. Usability was identified as the top-ranked evaluation criteria, followed by the need to search and navigate machine data, which is exactly what Splunk offers.

Our analysis of Splunk’s latest release of softwareversion 4.3, found a series of advancements, including supporting access from tablets such as Apple’s iPad, support for data in cloud-based applications and the ability to operate in VMware environments, making it easier to deploy Splunk in the cloud. This version expands the enterprise scale of the platform, supporting more concurrent users and faster search. Users can preview machine data then start indexing against it for further discovery and analytics. These are all critical tasks as Splunk becomes a platform for harvesting machine data for search and discovery across the IT organization. Splunk must continue to expand its footprint with more collaborative capabilities to help IT understand issues and opportunities where analytics on the machine data are the starting point for dialogue and action. Introducing more collaborative capabilities and the ability to correlate data toward actions and notifications will help Splunk further differentiate its technology.

Since our analysis, Splunk has continued to support new machine data environments, including Hadoop. It can run MapReduce queries against data in Hadoop and then bring the data into Splunk for analysis. Our benchmark research into Hadoop and Information Management found that users of Hadoop are most often focused on integrating machine data from application and Web logs with the events and network monitoring data that Splunk is designed to address. Hadoop users need to analyze data much faster than others, with 77 percent of organizations indicating they need analysis within a day and 25 percent within an hour. One challenge for organizations using Hadoop is staffing and training; Splunk addresses those obstacles with a tool that is easier to use than creating custom programs, and it provides real-time and integration features, each sought by two-thirds of organizations. The growth of Hadoop comes from the need to address big data needs but also the simple access to download the software and then use it which is also what Splunk does with its download to use its software.

Splunk has a promising opportunity in the market, considering the size of IT spending on technology every year and the legacy of applications and systems that must be optimized for business use. Also, as IT organizations continue to advance their IT security and compliance and increase their use of cloud computing, the need not just to monitor but to analyze and act on issues and opportunities grows larger. Our last benchmark research on Operational Intelligence in IT found that IT has prioritized to manage IT performance through metrics in 71 percent of organizations, and more than half (58%) plan to improve their IT organization’s responsiveness. The research found more than half of organizations have at least 11 information sources to routinely interact with, which is part of why Splunk fits these needs on machine data. Splunk has little competition, with IT management software vendors like Computer Associates, Hewlett-Packard, IBM and Oracle providing little innovation or capabilities to address operational intelligence needs through discovery and analysis of machine data in IT. Eventually they or others will realize their shortcomings, but at that point I would expect to see Splunk be significantly larger with an expanded portfolio of capabilities for meeting a CIO’s need to truly become an information officer of his own IT organization.


Mark Smith – CEO & Chief Research Officer