New Generation of Enterprise Messaging Supports Digital Transformation

Enterprise messaging is the technology backbone of communications for applications and systems within and between organizations. Both its importance and its complexity are growing as organizations increasingly have to provide real-time responses to business customers and consumers as well as their own business professionals who support them and their internal supply chains. The variety of use cases for enterprise messaging also is growing rapidly, expanding to the Internet of Things (IoT) market of sensors and devices including wearable technology; to new generations of applications and services for consumers and customers; to cloud computing and the shift to platform or infrastructure as a service (PaaS or IaaS); and to real-time big data and analytics. All of these innovations will enable these types of transformation to digital business that is impacting organizations around the world.

Enterprise messaging is closely related to message-oriented middleware (MoM) technology that consumes and publishes messages as part of applications and services as well as to other middleware and integration technologies. Through acquisitions and partnerships many middleware and some integration technology providers have blended their interfaces with enterprise messaging to VentanaResearch_IoT_BenchmarkResearch-250ensure they are part of real-time networks across IT and business. As for their customers, our previous research into operational intelligence found that almost half (48%) of organizations evaluated alternatives in their messaging middleware throughout 2015. Our new research into The Internet of Things and Operational Intelligence, currently under way, is assessing the role of enterprise messaging in the changing technology landscape.

Today’s enterprise messaging is advancing beyond the design of message-oriented middleware; those queuing or brokering approaches are struggling to keep pace with millisecond and even faster transmission of data across internal and external fiber networks that in some cases needs to support guaranteed delivery like that found in financial trades and other commerce. The decades-old technology in existing enterprise messaging systems and MoM spans multiple generations of brokering and queuing approaches, a range of protocols and standards, and dozens of vendors. This technology enables exchange and transport of messages between applications and systems. The messages can be processed asynchronously or synchronously, in the publish and subscribe method and in secure encrypted formats. The variance across approaches ranges in level of latency from low to very low, which performs in subsecond times between points across a network.

In recent years the necessity of processing messages fast has placed extreme pressure on message queue and broker approaches that were not designed to meet such low latency demands or efficiently process the huge volumes of data found in the emerging generation of enterprise and consumer-focused applications and services. The diversity of these new systems challenges the most experienced enterprise architects, who have to rationalize complex legacy environments and determine where to simplify them to become more cost-effective and in some cases more secure. These challenges push many organizations to reassess their architectures for messaging and examine alternative approaches.

The middleware technology approach to messaging is challenged further by the externalization of enterprise systems from on-premises to private and public cloud computing. As middleware-related markets have consolidated, the transition to platform or infrastructure as a service has necessitated new middleware for enterprise architectures as messaging and APIs are becoming more virtual, in what is called microservices. Simultaneously, businesses demand more real-time functionality as they discover that their underlying transaction and information architectures are ineffective for rapid communications and processing of data to meet new requirements.

Reliability, performance and scalability of the messaging technology and the infrastructure and resources required to support it are focus points of re-evaluation for organizations. Part of that review involves addressing the requirement that messaging must interface to the middleware or PaaS that is being used to develop new applications. For many organizations the messaging API they use depends on the middleware they’re using for applications; it may be provided by IBM, Microsoft, Oracle, Red Hat or another vendor that has a stake in binding the organization’s infrastructure to its technology. At minimum these providers influence developers to look first at the messaging that is part of their middleware or PaaS. Further complicating the issue, messaging between applications and systems is not controlled by one vendor, and depending on the history, biases or preferences of individuals and the use case, evaluations may not make it to the RFP or RFI stage. That could be risky for organizations that are not keeping up with the technological and architectural changes that have occurred.

In addition, standards often play a role in selecting technologies. One recent standard is Advanced Message Queuing Protocol (AMQP), which evolved from financial markets and operates across the wire on TCP to facilitate a robust approach to messaging. Another, Message Queue Telemetry Transport (MQTT), is being used for IoT and connecting machines to messaging on the internet. Other approaches such as Java Message Service (JMS) have gained traction through enterprise familiarity with Java and middleware such as Red Hat’s. Even the cloud computing offerings from Amazon, IBM, Microsoft and Oracle have added integrated messaging to their environment.

Architecturally, organizations are also examining microservices, which embed independent services that bind into applications, typically through an API that separates logic and communications from messaging. This technology approach provides a pattern for development and does not preclude the interface to APIs and messages that communicate with enterprise messaging.

The new focus on enterprise messaging has organizations examining their legacy approaches for messaging middleware. Users of IBM and Tibco, for example, have had to increase spending on hardware and resources to scale out and support the reliability required for their growing messaging volumes. It is no surprise that our research has found that messaging middleware is insufficient in almost one-quarter (23%) of organizations that want to use it for other applications and tools in the enterprise. This lack of overall reliability places pressure on the management and monitoring of servers to ensure that they scale adequately; many struggle to meet the requirements for very low latency and guaranteed messaging. Many organizations feel forced to re-evaluate their architecture and approach to enterprise messaging to find one that is more cost-effective, more efficient and more reliable. Some organizations are working with commercialized open source messaging approaches such as Apache ActiveMQ, RabbitMQ and StormMQ.

The next generation of enterprise messaging now in the market includes virtualized messaging across cloud-based platforms and the internet and the use of appliances and related tools for networking. Enterprise messaging appliances are attractive because they are able to handle extremely large volumes of messaging but can be managed by software already in operation at data centers and network operations centers. These appliances can be placed into the data center or hosted on the internet in a distributed computing approach. One such enterprise messaging appliance provider is Solace Systems, which has been operating its appliances for years in large global deployments. More recently IBM and Tibco introduced appliances into the market to address the shortcomings in their software-based approaches, which, as I have already mentioned, are challenging and costly for companies to maintain.

The advances in intelligent communications across devices and machines demand reliable throughput, which enterprise messaging can provide. Messaging appliances and virtualized messaging are part of the emerging future in which digital technologies operate in real time and support how consumers and business operate. I will write more about these tools in 2016. If you have not examined your organization’s messaging and infrastructure, look into enterprise messaging to better understand what you will need to be successful in the new digital business that is interconnected and happens in real time.


Mark Smith

CEO and Chief Research Officer

Mastering Marketing Mayhem in a Meaningful, Meticulous Manner

I hope this title captures your attention; I’m trying to make a point about the chaos going on in managing and operating marketing. What marketing needs in 2016 is to manage and optimize its efforts in a more unified manner. This perspective kicks off a new series on the challenges for marketing to automate or execute tasks and manage toward maximum performance. We all know that the craft of marketing is in need of significant transformation, from the CMO throughout the entire marketing organization and all the way out to the experience of consumers and customers. But this may be a fanciful mission, as applications and technology does not really automate marketing let alone manage it. Most marketing automation products are specialized applications that are not used by marketing management, let alone front-line marketing managers; they are for specialized needs in demand generation or digital marketing that personalizes inbound and outbound interactions with contacts for the purpose of advancing dialogue and creating relationships. Marketing automation, like its cousin sales force automation, has been a placeholder category that describes only a narrow slice of marketing, and the term has been co-opted by the industry for its own purposes. Though some observers predict that CMOs will outspend CIOs and other leaders of the business in technology investments, I have debunked this ludicrous idea; even if it were true, that would not make marketing departments much more efficient in their management and operations. To counterbalance the silliness of the marketing automation dialogue, I plan to bring you a series on key areas for investment to start the conversation. Evaluating them should help Marketing demonstrate its commitment to promoting effectively its organization and its products and services. Here is an overview of the many issues in the landscape.

Marketing encompasses a complex set of functions that have been and continue to be separate operations rather than methodically aligned to a common set of goals. Few organizations have an integrated plan that measures these efforts with a common set of metrics or key indicators of marketing performance. Recently, however, newer vendors have begun to advance marketing performance management (MPM) and focus on using a common departmental budget with historical spend and focus on creating forward-looking plans that can integrate a common library of metrics. But there is still much to do to manage Marketing’s portfolio of resources and assets to achieve desired goals and expected outcomes. The first step in such a transformation is to establish a common set of goals and objectives with a continuous plan that links to metrics and monitors progress and becomes the basis for a common dialogue on achievements and utilization of the budget. So far few marketing organizations have taken this step, most are adopting software for marketing analytics yet without it they cannot demonstrate the level of governance and commitment required for marketing excellence. Finding software that can provide consistency and a sharedvr_NGBP_06_dedicated_applications_help_users_look_ahead environment for metrics and collaboration is critical; it also should be easy to integrate with marketing budgeting and planning systems. This will be worth the effort: In our next-generation business planning benchmark research more than half (55%) of organizations said that a dedicated application enables them to explore all implications of scenarios; many fewer (40%) said this of spreadsheets. In addition only one-fifth (22%) of marketing organizations are able to drill down to details during a meeting. Thus a capable tool can help optimize execution and facilitate collaboration in the marketing team. At this point, however, only 15 percent of participants said that they manage the marketing planning process very well.

It also is time for Marketing to embrace the digital experience in which consumers and customers interact with websites, communities and even commerce on the Internet. This must be done carefully, as lack of consistency in the digital experience can lead to bad customer experiences and degrade customer satisfaction. Over the past year in reviewing technologies (even the ones we at Ventana Research use) I have been amazed to find how little they are interconnected; lack of integration creates significant impediments to streamlined user experiences in accessing information about products and services along with their supporting content. It impedes not just personalization of the experience but the ability to go from browsing to purchasing of goods and services. Obviously this impacts a company’s revenue and bottom line. The disconnect is not easy to resolve. Some website software includes a content management system (CMS), and demand generation systems offer digital marketing capabilities. Vendors of these technologies talk about simple integration and their partnerships, but almost all require significant customization and consulting resources to actually work.

And for organizations building a community and looking to reduce the inbound load of inquiries and sometimes confusion with their FAQ, the use of a learning management system (LMS) can help their marketers communicate information related to products and services. Our next-generation learning management system research shows that an LMS can help in responding to customer service complaints and issues, which more than one-third (35%) of organizations said is important. Innovative new approaches that use virtualization and video can enable an actual employee to converse with customers are now available and can help reduce questions into the organization.

Underlying the marketing investments for website and community or commerce environment must be intelligence for supporting the inbound and outbound digital marketing efforts that have evolved from demand generation systems and engaging visitors to the sites. Part of which is to directly link to active promotion and targeting of pay-per-click (PPC) and ensure effective search engine optimization (SEO). In practice many organizations interact with customers through separate systems that each manage their own environments such as Bing and Google; Facebook, LinkedIn, Twitter and other social media channels take specialized approaches to create presence and lure visitors into the marketing channels. But we are beginning to see more integrated approaches to digital marketing to simplify automation and oversight, to extract data to apply analytics to and to create metrics to determine effectiveness and return on investment. This is important because, according to our data and analytics in the cloud benchmark research, social media is the second-most important external data source in almost half (48%) of organizations, following cloud-based business applications (61%). And for many organizations they are beginning to use social media ratings of individuals to determine their relevance and for establishing a better approach to responding to consumers and customers as part of multi-channel contact center efforts. The amount of time and resources companies spend should motivate them to look for more efficient methods to manage digital interactions, especially across social media channels.

This point also applies to demand generation, which tries to turn suspects into prospects that can be passed along to sales and is a key part of what Sales expects from Marketing. While marketing organizations have experimented with a variety of email and automation-based approaches over the past few years, not many have succeeded in finding full value from their efforts, often losing some trust from sales and even corporate and finance management in their ability to generate the quantity of viable marketing qualified leads. For this activity many marketing organizations depend on a key person or two who may not have the experience in best practices or systems to know how to apply continuous improvement or when to ditch one technological approach for a better one. This often results in stale legacy approaches that produce a decline in the quantity of quality leads being generated by the marketing organization. Lacking target personas and aligned positioning and messaging embedded in automated nurturing, demand generation is part of why they are not fulfilling its promise.

In some cases, sales organizations seeing this decline have been slowly taking responsibility for achieving the quotas for which they are compensated and evaluating a new generation of inside sales that nurture and interact with prospects directly. I highlighted this in my recent diatribe on the state of sales organizations that outlines the increasing divide between marketing and sales in many organizations. If we expect marketing teams to be more progressive, then it probably will be necessary to link a portion of their incentives and rewards to revenue objectives. In addition most demand generation systems as systems of record are ineffective in applying and presenting analytics and metrics; it requires more software capabilities and people skills to get the appropriate data into the marketing analytics and performance management. Marketing will have to turn elsewhere for analytics; it is one the largest users of data and analytics in the cloud, as more than one-third (37%) use them today, according to that research. Similarly our recent next-generation predictive analytics research finds that Marketing is most often the user of these advanced analytics in almost half (48%) of organizations, and almost one-third (29%) are evaluating it for use into 2017. Predictive analytics for example can help determine the probability of individuals to respond to offers or becoming customers, and it can be articulated in the form of a score or a grade that can be useful for helping nurture or directly engage particular individuals.  Marketing information is used by 44 percent of organizations for predictive analytics, and one-third (32%) plan to use it which indicates its potential.

A positive digital experience is not complete without consistent information about products and services available across the website and commerce systems to other channels of marketing, sales and operations that could range from print, store and channel efforts to the demand and supply chain. Product information management (PIM) systems can support these efforts and can be managed by the marketing and product teams without depending on IT, even though these information assets must be shared and in many cases syndicated across business channels. In organizations where IT has established master data management, this is an infrastructure function and not the focal point of PIM efforts, which require a product master that can be managed within the application environment. For more detail on this issue, see my perspective on PIM Trumps MDM. Having a diverse set of product information that can include images, videos, descriptions, diagrams and even feedback and ratings from others is essential to a full digital experience that can facilitate answers to questions or accelerate browsing to purchasing in commerce or other channels. Many organizations have instituted digital asset management (DAM) to support their website and commerce, but usually these systems are not flexible for managing the entirety of product information. This issue has led to a new generation of PIM applications that blend the technologies. The unification of PIM and commerce systems was critical to almost two-thirds (63%) of organizations participating in our previous PIM benchmark research. It makes sense for commerce and PIM providers to forge partnerships since they have common customers in marketing organizations and businesses. PIM is necessary to eliminate errors and mistakes in product information as well to improve the customer experience; these were two key benefits of a dedicated approach to PIM in our previous research. We expect it to be a key point in the new research we are conducting this year on the next generation of PIM, which must adapt to the roles and specific business processes of marketing and sales organizations.

As business models evolve, many industries need to find ways to support subscription billing and recurring revenue processes. Most commerce systems are good at single transactions and shopping basket experiences but not as good at renting and periodic licensing of services that require subscriptions. In recurring revenue effective dating of when the service begins or fees change and when additional services are added can be complex. Customers also may have a variety of billing methods that can be applied differently for services. Even the largest software companies providing software as a service often lack effective billing to support online payment transfer, which half of organizations (52%) use, second only to credit cards (77%), as found in our recurring revenue benchmark research. Improvingvr_Recurring_Revenue_05_multiple_departments_interact_with_customers the customer experience is the second-most important business objective (68%) of recurring revenue, and half (51%) of organizations see it as a driver for introducing recurring revenue. But once you engage customers, you must continue to engage and promote other relevant services that can enhance the customer relationship and potential revenue contributions. Our research finds that Marketing is interacting with customers in almost two-thirds (63%) of organizations that use recurring revenue, and 20 percent or more of marketing organizations are involved with defining, identifying or evaluating subscription billing systems. Overall the research finds integrating systems to be the top technology challenge to recurring revenue business in half (51%) of organizations.

For organizations seeking to renovate their digital commerce environment in the presentation of information related to the organization and its specific products and services, addressing many of the previous points is essential. Rather than simply replacing the current system, I recommend considering the needs of the consumer in terms of PIM and the potential of subscription billing and recurring revenue. Organizations that have not developed a refined plan and unified approach to the digital experience before selecting technology may find themselves involved in customization chores and consulting expenses; they should rather look for applications designed to adapt to each other. It is clear to me that marketing leaders will need to be vocal and lead in demanding better approaches to support this mission, as many vendors are focused less on what is needed to simplify their systems than on selling their add-on modules and work with only a few products that come from vendors with whom they have relationships.

I have pointed out how the digital technology disruption is changing business processes and methods for operating, and this is true for marketing. In fact, the majority of new systems operate as software as a service in a cloud computing approach that is rented and available on the Internet. Almost all new software for marketing released in the past five years, including analytics, operates in the cloud. The large numbers of users and data vr_DAC_15_spreadsheets_unreliable_for_analyticssources in dispersed locations are best served with this approach, especially compared to copying and pasting in spreadsheets: More than half (58%) of organizations see them as unreliable for analytics.

Marketing cannot afford to overlook the advances in mobile technologies and the need to make the digital experience operate seamlessly and responsively for users of smartphones and tablets. This might seem simple, but variances across Apple and Android devices and across three or four browsers make it complex for digital experiences in marketing. In efforts to make Internet-based efforts more responsive, marketing organizations should explore developing native applications on the mobile devices or how to operate across a variety of social and digital platforms, which is important for cross-promotion and -selling. To offer a complete digital experience requires using all internal and external information assets about visitors and customers, including third-party big data sources combined with analytics to provide market intelligence and cryptic data that was not previously available to integrate with marketing data. To manage such a large portfolio will require project management that can encompass all the assets and resources and make them accessible for the entire marketing organization.

It’s obvious that Marketing has a lot to do to compete and prosper in the digital age. My analysis is that CMOs and other marketing leaders will have to get more engaged, take risks and even step outside their comfort zones in exploring and learning. Take a broad view of the processes, information and technology for the entire organization and its people and identify where working together and having consistency of projects, plans, analytics and results will help engage employees and inspire the best possible performance. Marketing is a team sport, and CMO need to not just be a captain but also equip the team to work together and be effective in their roles. As I noted in beginning, Marketing is under pressure to excel in performance and provide tangible value to the business. These steps can help it do that.


Mark Smith

CEO and Chief Research Officer