Digital Business Innovation and Enterprise Messaging Work Well Together

Organizations are facing a digital transformation, as I have written, that is rapidly changing the applications and services that businesses use to operate and deliver information. This new digital generation addresses the expectations of consumers and business partners for information and service in real time. One example of it is enterprise messaging. Recently I wrote about the shift to this technology and the challenges it poses for organizations that lack sufficient skills. However, new messaging appliances and virtualized messaging can carry some of this burden. By interconnecting them, organizations can be more confident in their ability to support the range of information and applications that operate in real time, not only for people but on devices and machines.

There are four key areas in which digital business innovation for enterprise messaging is essential to the future of almost every organization. Let’s look at each of them.

The Internet of Things (IoT) is a rapidly evolving technology segment, in which real-time messages are transmitted from machines, devices and sensors to other points or gateways for processing through a messaging infrastructure. The conventional approach to connecting devices and sensors uses various VentanaResearch_IoT_BenchmarkResearchproprietary system interfaces and protocols that do not always provide sufficient security and flexible interoperability and may not be designed for the scale that large organizations require. Major corporations in most industries now have internal groups led by digitally proficient officers at the C or VP level who are responsible for investigating and selecting new technology tools like the ones that my colleague has pointed out. They are most common in companies that provide services but also in manufacturing and product-related ones.

IoT devices range from appliances in the home to wearables such as watches or clothing to sensors in factories and warehouses. The Internet of Things is gaining momentum and will support a range of services for both business-to-consumer and business-to-business interactions. It tracks and reports on activity or usage of devices or services (electricity, for example), and connected sensors can indicate the need for maintenance or impending failure. To transmit information efficiently across the internet, messages can communicate with servers of service providers or manufacturers. Efficiency requires real-time processing and capabilities to utilize information in the messages for further actions, monitoring or analysis. This range of requirements is at the center of our new research into IoT, which is currently under way.

The second key area is the new generation of consumer and customer applications and services that has emerged through social media, virtualized contact centers or customer vr_NGCE_Research_10_benefits_of_engagement_technologiesportals, and commerce systems that require real-time information and action. Existing methods of digital engagement do not exist or are too slow, so many organizations are investing in new applications that can drive customer growth and retention. These systems rely on enterprise messaging across the internet and company networks. The huge volume of customer data and the rapid velocity of its flow could overwhelm networks and databases that are not architected to handle them. In addition these applications in many cases are operating on mobile devices. Their uses include finding and subscribing to new services and communicating customer demands. Again, they must run in real time to communicate from individuals to services on the internet. Our research into next-generation customer engagement reveals benefits of these types of investments, most often improved customer service, increased customer satisfaction and improved customer choice.

These applications often operate in cloud computing environments or use enterprise messaging that is hosted on the internet. Cloud computing, already widespread, is the third area. The advent of platform or infrastructure as a service (PaaS or IaaS) has changed the overall software market as many organizations now prefer to subscribe to or rent software rather than purchase it and pay for maintenance on-premises. Over the past five years the markets for business applications, tools and now databases have shifted to the cloud. Organizations can choose to be a single tenant in a private cloud or one among multiple tenants operating in a public cloud, and most ISVs have shifted new software development to cloud platforms. An even more transformational trend is application development done in the cloud; Amazon, IBM, Microsoft, Oracle and others offer PaaS or IaaS for this purpose. These platforms feature embedded messaging that can run within the applications or connect to enterprise messaging systems.

The fourth area, big data and analytics, deals with the volume, velocity, variety and granularity of data that organizations must manage and derive value from. The latest big data management technology appears in commercialized versions of the open source Apache Hadoop from providers such as Cloudera, Hortonworks and MapR. Established data management providers such as IBM, Microsoft, Oracle and Teradata have embraced Hadoop and expanded their architecturalvr_BDI_14_factors_driving_big_data_integration and technological approaches to a new generation of big data technologies. All these developments increase the need for directly connecting data management technologies and avoiding the silos of so-called data lakes. The most dynamic aspect of big data is transferring it in real time, which requires tremendous computing power and sophisticated messaging.

Big data is not new to messaging providers, nor is streaming of events in real time; this is often referred to as big data in motion. But most Hadoop-based technologies have not been able to scale their processing and compute operations sufficiently to deal with the rapid velocity of such data in motion. Some hope lies in an open source project called Kafka, which is a part of the Hadoop architecture designed to process streams of data. Fairly primitive in its initial design, it is now seen in the Apache and open source developer community as the path to handle real-time data with Hadoop. However, it will have to be connected via messaging to support the uses discussed above. The need for it is shown in our research into big data integration, which finds that business improvement initiatives are the top reason for investment in more than half (54%) of organizations.

These examples emphasize the importance of real-time processing and messaging in digital business innovation. Organizations that do not have the skills for enterprise messaging but know they need to advance their support and responsiveness to consumers, services and their own operations should assess what it will take to make this critical technology part of their future.

Regards,

Mark Smith

CEO and Chief Research Officer

New Generation of Enterprise Messaging Supports Digital Transformation

Enterprise messaging is the technology backbone of communications for applications and systems within and between organizations. Both its importance and its complexity are growing as organizations increasingly have to provide real-time responses to business customers and consumers as well as their own business professionals who support them and their internal supply chains. The variety of use cases for enterprise messaging also is growing rapidly, expanding to the Internet of Things (IoT) market of sensors and devices including wearable technology; to new generations of applications and services for consumers and customers; to cloud computing and the shift to platform or infrastructure as a service (PaaS or IaaS); and to real-time big data and analytics. All of these innovations will enable these types of transformation to digital business that is impacting organizations around the world.

Enterprise messaging is closely related to message-oriented middleware (MoM) technology that consumes and publishes messages as part of applications and services as well as to other middleware and integration technologies. Through acquisitions and partnerships many middleware and some integration technology providers have blended their interfaces with enterprise messaging to VentanaResearch_IoT_BenchmarkResearch-250ensure they are part of real-time networks across IT and business. As for their customers, our previous research into operational intelligence found that almost half (48%) of organizations evaluated alternatives in their messaging middleware throughout 2015. Our new research into The Internet of Things and Operational Intelligence, currently under way, is assessing the role of enterprise messaging in the changing technology landscape.

Today’s enterprise messaging is advancing beyond the design of message-oriented middleware; those queuing or brokering approaches are struggling to keep pace with millisecond and even faster transmission of data across internal and external fiber networks that in some cases needs to support guaranteed delivery like that found in financial trades and other commerce. The decades-old technology in existing enterprise messaging systems and MoM spans multiple generations of brokering and queuing approaches, a range of protocols and standards, and dozens of vendors. This technology enables exchange and transport of messages between applications and systems. The messages can be processed asynchronously or synchronously, in the publish and subscribe method and in secure encrypted formats. The variance across approaches ranges in level of latency from low to very low, which performs in subsecond times between points across a network.

In recent years the necessity of processing messages fast has placed extreme pressure on message queue and broker approaches that were not designed to meet such low latency demands or efficiently process the huge volumes of data found in the emerging generation of enterprise and consumer-focused applications and services. The diversity of these new systems challenges the most experienced enterprise architects, who have to rationalize complex legacy environments and determine where to simplify them to become more cost-effective and in some cases more secure. These challenges push many organizations to reassess their architectures for messaging and examine alternative approaches.

The middleware technology approach to messaging is challenged further by the externalization of enterprise systems from on-premises to private and public cloud computing. As middleware-related markets have consolidated, the transition to platform or infrastructure as a service has necessitated new middleware for enterprise architectures as messaging and APIs are becoming more virtual, in what is called microservices. Simultaneously, businesses demand more real-time functionality as they discover that their underlying transaction and information architectures are ineffective for rapid communications and processing of data to meet new requirements.

Reliability, performance and scalability of the messaging technology and the infrastructure and resources required to support it are focus points of re-evaluation for organizations. Part of that review involves addressing the requirement that messaging must interface to the middleware or PaaS that is being used to develop new applications. For many organizations the messaging API they use depends on the middleware they’re using for applications; it may be provided by IBM, Microsoft, Oracle, Red Hat or another vendor that has a stake in binding the organization’s infrastructure to its technology. At minimum these providers influence developers to look first at the messaging that is part of their middleware or PaaS. Further complicating the issue, messaging between applications and systems is not controlled by one vendor, and depending on the history, biases or preferences of individuals and the use case, evaluations may not make it to the RFP or RFI stage. That could be risky for organizations that are not keeping up with the technological and architectural changes that have occurred.

In addition, standards often play a role in selecting technologies. One recent standard is Advanced Message Queuing Protocol (AMQP), which evolved from financial markets and operates across the wire on TCP to facilitate a robust approach to messaging. Another, Message Queue Telemetry Transport (MQTT), is being used for IoT and connecting machines to messaging on the internet. Other approaches such as Java Message Service (JMS) have gained traction through enterprise familiarity with Java and middleware such as Red Hat’s. Even the cloud computing offerings from Amazon, IBM, Microsoft and Oracle have added integrated messaging to their environment.

Architecturally, organizations are also examining microservices, which embed independent services that bind into applications, typically through an API that separates logic and communications from messaging. This technology approach provides a pattern for development and does not preclude the interface to APIs and messages that communicate with enterprise messaging.

The new focus on enterprise messaging has organizations examining their legacy approaches for messaging middleware. Users of IBM and Tibco, for example, have had to increase spending on hardware and resources to scale out and support the reliability required for their growing messaging volumes. It is no surprise that our research has found that messaging middleware is insufficient in almost one-quarter (23%) of organizations that want to use it for other applications and tools in the enterprise. This lack of overall reliability places pressure on the management and monitoring of servers to ensure that they scale adequately; many struggle to meet the requirements for very low latency and guaranteed messaging. Many organizations feel forced to re-evaluate their architecture and approach to enterprise messaging to find one that is more cost-effective, more efficient and more reliable. Some organizations are working with commercialized open source messaging approaches such as Apache ActiveMQ, RabbitMQ and StormMQ.

The next generation of enterprise messaging now in the market includes virtualized messaging across cloud-based platforms and the internet and the use of appliances and related tools for networking. Enterprise messaging appliances are attractive because they are able to handle extremely large volumes of messaging but can be managed by software already in operation at data centers and network operations centers. These appliances can be placed into the data center or hosted on the internet in a distributed computing approach. One such enterprise messaging appliance provider is Solace Systems, which has been operating its appliances for years in large global deployments. More recently IBM and Tibco introduced appliances into the market to address the shortcomings in their software-based approaches, which, as I have already mentioned, are challenging and costly for companies to maintain.

The advances in intelligent communications across devices and machines demand reliable throughput, which enterprise messaging can provide. Messaging appliances and virtualized messaging are part of the emerging future in which digital technologies operate in real time and support how consumers and business operate. I will write more about these tools in 2016. If you have not examined your organization’s messaging and infrastructure, look into enterprise messaging to better understand what you will need to be successful in the new digital business that is interconnected and happens in real time.

Regards,

Mark Smith

CEO and Chief Research Officer