New Generation of Enterprise Messaging Supports Digital Transformation

Enterprise messaging is the technology backbone of communications for applications and systems within and between organizations. Both its importance and its complexity are growing as organizations increasingly have to provide real-time responses to business customers and consumers as well as their own business professionals who support them and their internal supply chains. The variety of use cases for enterprise messaging also is growing rapidly, expanding to the Internet of Things (IoT) market of sensors and devices including wearable technology; to new generations of applications and services for consumers and customers; to cloud computing and the shift to platform or infrastructure as a service (PaaS or IaaS); and to real-time big data and analytics. All of these innovations will enable these types of transformation to digital business that is impacting organizations around the world.

Enterprise messaging is closely related to message-oriented middleware (MoM) technology that consumes and publishes messages as part of applications and services as well as to other middleware and integration technologies. Through acquisitions and partnerships many middleware and some integration technology providers have blended their interfaces with enterprise messaging to VentanaResearch_IoT_BenchmarkResearch-250ensure they are part of real-time networks across IT and business. As for their customers, our previous research into operational intelligence found that almost half (48%) of organizations evaluated alternatives in their messaging middleware throughout 2015. Our new research into The Internet of Things and Operational Intelligence, currently under way, is assessing the role of enterprise messaging in the changing technology landscape.

Today’s enterprise messaging is advancing beyond the design of message-oriented middleware; those queuing or brokering approaches are struggling to keep pace with millisecond and even faster transmission of data across internal and external fiber networks that in some cases needs to support guaranteed delivery like that found in financial trades and other commerce. The decades-old technology in existing enterprise messaging systems and MoM spans multiple generations of brokering and queuing approaches, a range of protocols and standards, and dozens of vendors. This technology enables exchange and transport of messages between applications and systems. The messages can be processed asynchronously or synchronously, in the publish and subscribe method and in secure encrypted formats. The variance across approaches ranges in level of latency from low to very low, which performs in subsecond times between points across a network.

In recent years the necessity of processing messages fast has placed extreme pressure on message queue and broker approaches that were not designed to meet such low latency demands or efficiently process the huge volumes of data found in the emerging generation of enterprise and consumer-focused applications and services. The diversity of these new systems challenges the most experienced enterprise architects, who have to rationalize complex legacy environments and determine where to simplify them to become more cost-effective and in some cases more secure. These challenges push many organizations to reassess their architectures for messaging and examine alternative approaches.

The middleware technology approach to messaging is challenged further by the externalization of enterprise systems from on-premises to private and public cloud computing. As middleware-related markets have consolidated, the transition to platform or infrastructure as a service has necessitated new middleware for enterprise architectures as messaging and APIs are becoming more virtual, in what is called microservices. Simultaneously, businesses demand more real-time functionality as they discover that their underlying transaction and information architectures are ineffective for rapid communications and processing of data to meet new requirements.

Reliability, performance and scalability of the messaging technology and the infrastructure and resources required to support it are focus points of re-evaluation for organizations. Part of that review involves addressing the requirement that messaging must interface to the middleware or PaaS that is being used to develop new applications. For many organizations the messaging API they use depends on the middleware they’re using for applications; it may be provided by IBM, Microsoft, Oracle, Red Hat or another vendor that has a stake in binding the organization’s infrastructure to its technology. At minimum these providers influence developers to look first at the messaging that is part of their middleware or PaaS. Further complicating the issue, messaging between applications and systems is not controlled by one vendor, and depending on the history, biases or preferences of individuals and the use case, evaluations may not make it to the RFP or RFI stage. That could be risky for organizations that are not keeping up with the technological and architectural changes that have occurred.

In addition, standards often play a role in selecting technologies. One recent standard is Advanced Message Queuing Protocol (AMQP), which evolved from financial markets and operates across the wire on TCP to facilitate a robust approach to messaging. Another, Message Queue Telemetry Transport (MQTT), is being used for IoT and connecting machines to messaging on the internet. Other approaches such as Java Message Service (JMS) have gained traction through enterprise familiarity with Java and middleware such as Red Hat’s. Even the cloud computing offerings from Amazon, IBM, Microsoft and Oracle have added integrated messaging to their environment.

Architecturally, organizations are also examining microservices, which embed independent services that bind into applications, typically through an API that separates logic and communications from messaging. This technology approach provides a pattern for development and does not preclude the interface to APIs and messages that communicate with enterprise messaging.

The new focus on enterprise messaging has organizations examining their legacy approaches for messaging middleware. Users of IBM and Tibco, for example, have had to increase spending on hardware and resources to scale out and support the reliability required for their growing messaging volumes. It is no surprise that our research has found that messaging middleware is insufficient in almost one-quarter (23%) of organizations that want to use it for other applications and tools in the enterprise. This lack of overall reliability places pressure on the management and monitoring of servers to ensure that they scale adequately; many struggle to meet the requirements for very low latency and guaranteed messaging. Many organizations feel forced to re-evaluate their architecture and approach to enterprise messaging to find one that is more cost-effective, more efficient and more reliable. Some organizations are working with commercialized open source messaging approaches such as Apache ActiveMQ, RabbitMQ and StormMQ.

The next generation of enterprise messaging now in the market includes virtualized messaging across cloud-based platforms and the internet and the use of appliances and related tools for networking. Enterprise messaging appliances are attractive because they are able to handle extremely large volumes of messaging but can be managed by software already in operation at data centers and network operations centers. These appliances can be placed into the data center or hosted on the internet in a distributed computing approach. One such enterprise messaging appliance provider is Solace Systems, which has been operating its appliances for years in large global deployments. More recently IBM and Tibco introduced appliances into the market to address the shortcomings in their software-based approaches, which, as I have already mentioned, are challenging and costly for companies to maintain.

The advances in intelligent communications across devices and machines demand reliable throughput, which enterprise messaging can provide. Messaging appliances and virtualized messaging are part of the emerging future in which digital technologies operate in real time and support how consumers and business operate. I will write more about these tools in 2016. If you have not examined your organization’s messaging and infrastructure, look into enterprise messaging to better understand what you will need to be successful in the new digital business that is interconnected and happens in real time.


Mark Smith

CEO and Chief Research Officer

Qlik Makes Sense of its Analytics and Business Value

At the 2015 technology analyst summit in Austin, Texas, analytics and business intelligence software vendor Qlik discussed recent market and product developments and explained its roadmap and strategy for 2016. Discussion topics included its Qlik Analytics Platform and QlikView 12.0, Qlik Sense and Qlik DataMarket, applications built on the platform but also how it is expanding its analytics experience for business.

The engine of Qlik Analytics Platform is Qlik Indexing Engine (QIX) provides sophisticated correlation analysis, or what the company calls an “associative experience.” Qlik uses a simple but powerful visual approach to help users understand and explore data. For instance, values in a data set can be highlighted in green, directly linked values are white, excluded but still related values are light gray, and completely unrelated values are dark gray. The key differentiator from other tools in the market is in the light gray areas, which lead users to natural next steps in investigation of the data or what we call guided analytics. Put another way, this associativeVR_AnalyticsandBI_VI_HotVendor_2015 experience enables users to explore all of the data relationships and avoid confirmation bias, which results from having a predefined idea of how data should be related. Most other visual discovery tools are still beholden to confirmation bias since the data model is based on a specific hypothesis of data relationships put forward by a developer or business user. Neither visual exploration nor human intellect alone can discover all patterns, especially the most complex ones. To make sense of large data sets, however, data mining and statistical techniques such as correlation can find patterns, relationships and anomalies in the data. The associative experience helped contribute to Qlik earning a Hot Vendor rating in our 2015 Analytics and Business Intelligence Value Index.

Qlik Sense is the company’s flagship visual analytics software that combines the associative functionality with easy-to-manage, high-performing and scalable approach. Its aim is to give business users a simplified visual analytic experience that takes advantage of modern technologies that can operate in cloud computing environment. The Qlik Sense architecture allows a choice to do visual design on the server or on the client and provides users with streamlined management capabilities and an intuitive user experience. Other ease-of-use functions include probabilistic search capabilities and automatic linking of objects within newly created dashboards. The user experience in both design and in consumption is important; our benchmark research on analytics software in the last four years consistently finds that usability is the most important product evaluation criterion for companies. But the importance of manageability and reliability is critical to which is why organizations should broaden its evaluation to not just be about the features of the product.

Qlik Sense, at time of this analysis is version 2.1, is complemented by announcements Qlik made over the last year. They include Qlik Sense Cloud, which provides cloud hosting services for Qlik Sense, and Qlik DataMarket, which offers enriched analytic data sets that are integrated and ready for consumption by analytic users. This service, often called data as a service (DaaS), provides information to be used in both descriptive and predictive analytic scenarios. My colleague Robert Kugel has written about this in what is called Cryptic Data and where Qlik is removing the hassle of integrating relevant external data into analytics.  Using a cloud-based architecture, vr_DAC_07_importance_of_external_data_sourcesQlik is able to easily bring together data from external sources, which according to our data and analytics in the cloud benchmark research include cloud-based business applications (61%), social media (49%), Internet information (48%), government sources (33%), market sources (29%) and data brokers (27%). Preintegration of data sets is an emerging trend that helps address the most formidable challenge in cloud computing, which is data preparation (cited by 55% of organizations).

Qlik was a pioneer in the visual discovery market with QlikView. More recently, it took time to rearchitect Qlik Sense to take advantage of technological advances such as cloud computing. That move temporarily disrupted the company’s momentum in the market but has rapidly accelerated forward and is approaching to be a billion dollar provider of analytics software. Now Qlik appears to have gotten much of its momentum back in company revenue growth but in the adoption and deployment of Qlik Sense.

Qlik’s partner strategy continues to advance where they have been embedded in a significant number of applications across industries. Having a modern architecture, the company can take several directions, such as entry into various platform-as-a-service (PaaS) ecosystems and be part of those environments. The Qlik Branch tool provides resources for embedding Qlik Sense directly into applications, enabling developers to build extensions using modern RESTful approaches. The site provides developer tools, community efforts such as d3.js integration and synchronization with Github for sharing and branching of designs. This provides an advantage to Qlik as these community assets can be used by innovative enterprises and independent software vendors (ISVs). Furthermore, Qlik can use these development efforts to decide where to invest its own resources in product development and support. The ability to also combine DaaS into its efforts reinforces the company’s competitive position and unique differentiation with these innovative efforts for enterprises and application assemblers and developers including ISVs.

Organizations considering self-service visual analytics software should put Qlik on their shortlist. For companies that already use QlikView, recently released version 12.0 is a logical path especially if they are exploring accompanying deployment of Qlik Sense. With the release of QlikView 12.0, both Qlik Sense and QlikView will utilize a common QIX engine to provide the associative experience. For first-time implementations of Qlik, Qlik Sense will likely be the best option except in some industry-specific cases in which QlikView offers sophisticated tailored solutions. Qlik Sense Cloud provides sharing of Qlik applications in a hosted and managed environment. While the company has made improvements in developing its capabilities and partner ecosystem for Qlik Sense, these should be closely examined in conjunction with the specific business objectives in mind. For developers, Qlik Sense provides a fully featured cloud platform on which to build and well-documented APIs to create extensions and customize the product. Partners, content providers and ISVs should consider the Qlik platform and Qlik Branch for embedding resources directly into applications. Every class of user can download Qlik Sense for free and test it directly on the desktop.

Overall, we find that Qlik Sense’s flexible approach can support various technology directions for analytics and is a strong choice especially for analysts and also application-oriented approaches that are needed.  Qlik has focused on the broader analytic experience and what is required to streamline analysis but also to ensure that it is comprehensive and easily shared with others in the enterprise. If you have not looked at Qlik lately, now is the time to try it out.


Mark Smith

CEO and chief research officer