You are currently browsing the tag archive for the ‘splunk’ tag.

Splunk’s annual gathering, this year called .conf 2015, in late September hosted almost 4,000 Splunk customers, partners and employees. It is one of the fastest-growing user conferences in the technology industry. The area dedicated to Splunk partners has grown from a handful of booths a few years ago to a vast showroom floor many times larger. While the conference’s main announcement was the release of Splunk Enterprise 6.3, its flagship platform, the progress the company is making in the related areas of machine learning and the Internet of Things (IoT) most caught my attention.

Splunk’s strength is its ability to index, normalize, correlate and query data throughout the technology stack, including applications, servers, networks and sensors. It uses distributed search that enables correlation and analysis of events across local- and wide-area networks without moving vast amounts of data. Its architectural approach unifies cloud and on-premises implementations and provides extensibility for developers building applications. Originally, Splunk provided an innovative way to troubleshoot complex technology issues, but over time new uses for Splunk-based data have emerged, including digital marketing analytics, cyber security, fraud prevention and connecting digital devices in the emerging Internet of Things. Ventana Research has covered Splunk since its establishment in the market, most recently in this analysis of mine.

Splunk’s experience in dealing directly with distributed, time-series data and processes on a large scale puts it in position to address the Internet of Things from an industrial perspective. This sort of data is at the heart of large-scale industrial control systems, but it often comes in different formats and its implementation is based on different formats and protocols. For instance, sensor technology and control systems that were invented 10 to 20 years ago use very different technology than modern systems. Furthermore, as with computer technology, there are multiple layers in stack models that have to communicate. Splunk’s tools help engineers and systems analysts cross-reference these disparate systems in the same way that it queries computer system and network data, however, the systems can be vastly different. To address this challenge, Splunk turns to its partners and its extensible platform. For example, Kepware has developed plug-ins that use its more than 150 communication drivers so users can stream real-time industrial sensor and machine data directly into the Splunk platform. Currently, the primary value drivers for organizations in this field of the industrial IoT are operational efficiency, predictive maintenance and asset management. At the conference, Splunk showcased projects in these areas including one with Target that uses Splunk to improve operations in robotics and manufacturing.

For its part, Splunk is taking a multipronged approach by acquiring companies, investing in internal development and enabling its partner ecosystem to build new products. One key enabler of its approach to IoT is machine learning algorithms built on the Splunk platform. In machine learning a model can use new data to continuously learn and adapt its answers to queries. This differs from conventional predictive analytics, in which users build models and validate them based on a particular sample; the model does not adapt over time. With machine learning, for instance, if a piece of equipment or an automobile shows a certain optimal pattern of operation over time, an algorithm can identify that pattern and build a model for how that system should behave. When the equipment begins to act in a less optimal or anomalous way, the system can alert a human operator that there may be a problem, or in a machine-to-machine situation, it can invoke a process to solve the problem or recalibrate the machine.

Machine learning algorithms allow event processes to be audited, analyzed and acted upon in real time. They enable predictive capabilities for maintenance, transportation and logistics, and asset management and can also be applied in more people-oriented domains such as fraud prevention, security, business process improvement, and digital products.  IoT potentially can have a major impact on business processes, but only if organizations can realign systems to discover-and-adapt rather than model-and-apply approaches. For instance, processes are often carried out in an uneven fashion different from the way the model was conceived and communicated through complex process documentation and systems. As more process flows are directly instrumented and more processes carried out by machines, the ability to model directly based on the discovery of those event flows and to adapt to them (through human learning or machine learning) becomes key to improving organizational processes. Such realignment of business processes, however, often involves broad organizational transformation.Our benchmark research on operational intelligence shows that challenges associated with people and processes, rather than information and technology, most often hold back organizational improvement.

Two product announcements made at the conference illuminate the direction Splunk is taking with IoT and machine learning. The first is User Behavior Analytics (UBA), based VR2015_InnovationAwardWinneron its acquisition of Caspida, which produces advanced algorithms that can detect anomalous behavior within a network. Such algorithms can model internal user behavior, and when behavior deviates from the specified norm, it can generate an alert that can be addressed through investigative processes usingSplunk Enterprise Security 4.0. Together, Splunk Enterprise Security 4.0 and UBA won the 2015 Ventana Research CIO Innovation Award.The acquisition of Caspida shows that Splunk is not afraid to acquire companies in niche areas where they can exploit their platform to deliver organizational value. I expect that we will see more such acquisitions of companies with high value ML algorithms as Splunk carves out specific positions in the emergent markets.

The other product announced is IT Service Intelligence (ITSI), which highlights machine learning algorithms alongside of Splunk’s core capabilities. The IT Service Intelligence App is an application in which end users deploy machine learning to see patterns in various IT service scenarios. ITSI can inform and enable multiple business uses such as predictive maintenance, churn analysis, service level agreements and chargebacks. Similar to UBA, it uses anomaly detection to point out issues and enables managers to view highly distributed processes such as claims process data in insurance companies. At this point, however, use of ITSI (like other areas of IoT) may encounter cultural and political issues as organizations deal with changes in the roles of IT and operations management. Splunk’s direction with ITSI shows that the company is staying close to its IT operations knitting as it builds out application software, but such development also puts Splunk into new competitive scenarios where legacy technology and processes may still be considered good enough.

We note that ITSI is built using Splunk’s Machine Learning Toolkit and showcase, which currently is in preview mode. The vr_Big_Data_Analytics_08_top_capabilities_of_big_data_analyticsplatform is an important development for the company and fills one of the gaps that I pointed out in its portfolio last year. Addressing this gap enables Splunk and its partners to create services that apply advanced analytics to big data that almost half (45%) of organizations find important. The use of predictive and advanced analytics on big data I consider a killer application for big data; our benchmark research on big data analytics backs this claim: Predictive analytics is the type of analytics most (64%) organizations wish to pursue on big data.

Organizations currently looking at IoT use cases should consider Splunk’s strategy and tools in the context of specific problems they need to address. Machine learning algorithms built for particular industries are key so it is important to understand if the problem can be addressed using prebuilt applications provided by Splunk or one of its partners, or if the organization will need to build its own algorithms using the Splunk machine learning platform or alternatives. Evaluate both the platform capabilities and the instrumentation, the type of protocols and formats involved and how that data will be consumed into the system and related in a uniform manner. Most of all, be sure the skills and processes in the organization align with the technology from an end user and business perspective.


Ventana Research

A few months ago, I wrote an article on the four pillars of big data analytics. One of those pillars is what is called discovery analytics or where visual analytics and data discovery combine together to meet the business and analyst needs. My colleague Mark Smith subsequently clarified the four types of discovery analytics: visual discovery, data discovery, information discovery and event discovery. Now I want to follow up with a discussion of three trends that our research has uncovered in this space. (To reference how I’m using these four discovery terms, please refer to Mark’s post.)

The most prominent of these trends is that conversations about visual discovery are beginning to include data discovery, and vendors are developing and delivering such tool sets today. It is well-known that while big data profiling and the ability to visualize data give us a broader capacity for understanding, there are limitations that can be vr_predanalytics_predictive_analytics_obstaclesaddressed only through data mining and techniques such as clustering and anomaly detection. Such approaches are needed to overcome statistical interpretation challenges such as Simpson’s paradox. In this context, we see a number of tools with different architectural approaches tackling this obstacle. For example, Information Builders, Datameer, BIRT Analytics and IBM’s new SPSS Analytic Catalyst tool all incorporate user-driven data mining directly with visual analysis. That is, they combine data mining technology with visual discovery for enhanced capability and more usability. Our research on predictive analytics shows that integrating predictive analytics into the existing architecture is the most pressing challenge (for 55% or organizations). Integrating data mining directly into the visual discovery process is one way to overcome this challenge.

The second trend is renewed focus on information discovery (i.e., search), especially among large enterprises with widely distributed systems as well as the big data vendors serving this market. IBM acquired Vivisimo and has incorporated the technology into its PureSystems and big data platform. Microsoft recently previewed its big data information discovery tool, Data Explorer. Oracle acquired Endeca and has made it a key component of its big data strategy. SAP added search to its latest Lumira platform. LucidWorks, an independent information discovery vendor that provides enterprise support for open source Lucene/Solr, adds search as an API and has received significant adoption. There are different levels of search, from documents to social media data to machine data,  but I won’t drill into these here. Regardless of the type of search, in today’s era of distributed computing, in which there’s a need to explore a variety of data sources, information discovery is increasingly important.

The third trend in discovery analytics is a move to more embeddable system architectures. In parallel with the move to the cloud, architectures are becoming more service-oriented, and the interfaces are hardened in such a way that they can integrate more readily with other systems. For example, the visual discovery market was born on the client desktop with Qlik and Tableau, quickly moved to server-based apps and is now moving to the cloud. Embeddable tools such as D3, which is essentially a visualization-as-a-service offering, allow vendors such as Datameer to include an open source library of visualizations in their products. Lucene/Solr represents a similar embedded technology in the information discovery space. The broad trend we’re seeing is with RESTful-based architectures that promote a looser coupling of applications and therefore require less custom integration. This move runs in parallel with the decline in Internet Explorer, the rise of new browsers and the ability to render content using JavaScript Object Notation (JSON). This trend suggests a future for discovery analysis embedded in application tools (including, but not limited to, business intelligence). The environment is still fragmented and in its early stage. Instead of one cloud, we have a lot of little clouds. For the vendor community, which is building more platform-oriented applications that can work in an embeddable manner, a tough question is whether to go after the on-premises market or the cloud market. I think that each will have to make its own decision on how to support customer needs and their own business model constraints.


Tony Cosentino

VP and Research Director

As I listened to the keynote address at, conf2012, the annual Splunk user conference, my initial impression was that the company was spreading itself too thin. The company highlighted four rather formidable areas of organizational focus: Enterprise 5.0, the company’s flagship data platform, which is now in beta; Development, which is support for building applications and integrating Splunk within the broader IT infrastructure; Content, the continued development of core applications and use cases in areas such as systems management and security; and Cloud, based on the recent Splunk Storm product, which targets a new class of customer – namely those developers who use services for everything. Is this broad-based vision a realistic goal, or merely an attempt to appease Wall Street pressure given the company’s relatively recent IPO?

The key to answering this question lies in Splunk’s second objective, Development. Splunk sees its software as a platform upon which the developer community can build applications and create value. This business model has worked for such venerable companies as Microsoft, Apple, Amazon and Facebook.  The key is gaining enough traction, and the main driver of such traction is Splunk’s extensible data fabric, with its 170+ REST-based interfaces and SDKs for Python, Java, JavaScript in beta, and PHP as a public preview. Such an approach allows Splunk to develop many use cases outside its core areas of systems and security, and cover the last mile to the business user with personal productivity tools such as Microsoft Excel and visual discovery tools such as Tableau.

In talking with Splunk customers, I found a real passion for the product. Every customer I spoke with was either in the process of expanding their implementation or planning to do so. When asked about Splunk’s competition, customers couldn’t provide any quick answers; most said that there really was not any competition for Splunk.

Some customers were going as far as having an autonomous Splunk team within their organization with an internal chargeback structure. This got me thinking about the fluidity of the software buying process within today’s organizations, as customer analytics and technology spending shifts toward LOB budgets. In this new age, Splunk is potentially a secret weapon of IT, since it gives the IT department the ability to go to other departments and show its business value.

The trick for Splunk will be to quickly expand the use and usability of its product. Systems management and security have catapulted the company over the proverbial chasm of technology adoption, but in order to live up to expectations, it will have to address the needs of the business users – and Splunk should be able to do that, based on the presentations I saw and executive discussions I had at the conference. A sampling of innovative use cases includes:

  • Expedia using Splunk to improve customer experience through increased response times, as well as optimize its search engine bidding process;
  • Intuit using Splunk to increase customer intimacy by looking at actual site behavior rather than having to get the stated experience from an already frustrated customer;
  • Comcast linking customer search behavior, location and billing information to customize content and usability;
  • Bosch using Splunk to tie together sensor data from a network of medical devices to increase usability, satisfaction and uptime for in-home care patients.

These are just a few public cases; I also spoke with large telecommunication companies and government organizations that are using Splunk in innovative ways to use data from systems at any velocity.

At the end of the day, the killer application for Splunk is the same as it was for Google: search. It solves the same problem of expanding sets of distributed unorganized data, and it solves it in much the same way; but note that Splunk uses its own implementation of MapReduce. Our benchmark research into information management shows that data spread across too many applications and systems is the number-one barrier to managing data, and as Google did, Splunk helps to solve this problem with a relatively simple and elegant solution. The only real difference is that Splunk targets the business market instead of the consumer market, and the engine returns machine language instead of human language.

Of course, the Google-like search function is a double-edged sword. The ease of use of search and iterative discovery analytics that Splunk provides its users can add tremendous value. At the same time, the interface is not a business user interface. It will need to expand further with data and visual discovery to provide stimulating presentation of the data, many business users or analysts will not be inclined to use the tool. This is one reason I’m excited to see the integration with Excel and visualization toolsets; as Splunk can roll out these tools, it should be able to grow share with business users. Splunk has made it easy to access the tool with a free download on their website.

My recommendation for clients already using Splunk is to look at the expanded use cases above and see if any of these might apply to your organization. As you build your business case, don’t go to business managers (especially high-level ones) and show them a bunch of machine data. Splunk is definitely able to apply search and analytics on large volumes of data for what my colleague refers to them as big data machine for operational intelligence. Give them the right type of analytics and metrics and show them how Splunk helps solve issues such as the 360-degree view of the customer. For those currently without Splunk, there are many use cases including systems management and security along with customer interactions and experience in which it provides value and can combine systems and business data together dynamically.


Tony Cosentino

VP and Research Director

RSS Tony Cosentino’s Analyst Perspectives at Ventana Research

  • An error has occurred; the feed is probably down. Try again later.

Tony Cosentino – Twitter

Error: Twitter did not respond. Please wait a few minutes and refresh this page.


  • 72,942 hits
%d bloggers like this: