Big Data Search is Getting Better with LucidWorks

LucidWorks addresses the growing volume of information now being stored in the enterprise and in big data with two products aimed at the enterprise with search technology. Though you may not be familiar with LucidWorks (previously known as Lucid Imagination), the company has for many years contributed to Apache Lucene, an open source search project, and commercialized and supported for it for business.

Search is a necessary enterprise application, but it is often not deployed successfully, for many reasons, some to do with the lack of focus and priority by IT and in some cases because a given application is not designed to work well across an organization. Our technology innovation benchmark research finds that search is a critical capability for 38 percent of organizations, and was a top-five capability for business intelligence, ranked as very important in 29 percent of organizations in our next-generation business intelligence research. But few of today’s solutions in BI and business analytics have well-integrated search capabilities that can work against the analytics, let alone with a broader set of information Technology Considerationssources. Inadequate search capabilities hurt applications’ usability, and our latest research finds that usability is the top technology consideration in 64 percent of organizations.

LucidWorks has two product offerings in the search market. LucidWorks Search provides the ability to rapidly set up search and index content using Apache Solr. The company not only provides full commercial-grade support and services and a security framework, but has also improved on Solr’s usability for developers and business users. Solr, built on top of Lucene, is an enterprise platform that provides full-text search, dynamic clustering, geospatial search and other enterprise-class capabilities.

Second, LucidWorks Big Data, which was released late last year, uses the compute and storage capabilities of Hadoop to support larger-scale deployments. LucidWorks also announced more formalized integration with MapR to support it from within its software. LucidWorks has built a big data framework to take advantage of key Apache technologies such as HBase for storage and access, Kafka for distributed publish and subscribe, Mahout for scalable machine learning, Pig for Map-Reduce scripting and ZooKeeper for distributed coordination. Using Hadoop technologies to support big data search could be the tipping point that brings enterprise and Internet-class search to every organization.

LucidWorks provides its products via either on-premises or cloud computing deployment. LucidWorks has embraced cloud computing through Amazon Elastic Compute Cloud (EC2) and Microsoft Windows Azure, making it simple to integrate LucidWorks with existing approaches. As big data deployments evolve, our technology innovation research finds that 43 percent of organizations prefer big data on-premises, while 26 percent prefer it via cloud computing and 19 percent have no preference. Providing choices for organizations is essential Big Data Capabilities Not Availableto meeting the broadest range of needs. Our big data research found searching for data is one of the top big data capabilities not available today in 20 percent of organizations and is needed. In addition, our research found the need for search on mobile technology to be important in 29 percent of organizations, so demonstrations of how LucidWorks can operate across smartphones and tablets will be essential.

According to our research, the market potential for providing search to business is quite high. Business understands the value of time and needs to get information. Unfortunately, the market for enterprise search software has been hampered by lack of interest from IT, and on the part of analyst firms that follow the IT industry but do not measure the needs of business. Historically, search has been relegated to IT systems and not seen as software to help business be more productive.

As organizations begin to build a new generation of applications to leverage big data they should not forget that search is a required component to get to information not only from within an application but also from any related information beyond it. LucidWorks has priced its offering to be affordable for any midsize to large organization and made it easy to set up through its cloud computing offering. Keeping it simple to set up is important, as lack of resources is the highest barrier for use of innovative technology in 51 percent of organizations.

Last year LucidWorks added a new CEO, Paul Doscher, a veteran in the enterprise software and search technology industry, to join its chief technology officer and founder, Grant Ingersoll. Going forward I expect to see LucidWorks search results further integrated into business applications and even business analytics. I hope that the company expands its big data offering beyond just Hadoop. LucidWorks should also get more of its customers to promote the software by talking about their deployments. I will be further researching the need for search as part of our big data and information optimization research in 2013. LucidWorks is definitely a vendor to examine if you are looking to bring enterprise-class search to your organization and big data deployments.


Mark Smith

CEO & Chief Research Officer

Information Builders Advances Business Analytics and Big Data

More than 1,000 people attended the 32nd annual Information Builders Summit conference this week to learn about the company’s advances in big data, business analytics, cloud computing, mobile technology and social media, which CEO and founder Gerald Cohen announced and demonstrated during his keynote address. With WebFOCUS version 8, Information Builders has made significant strides in a range of technology areas to support analytics and visualization since my analysis after last year’s conference.

In big data, Information Builders has improved in adaptability and integration of data in underlying technologies such as from 1010data and Teradata. At the conference it demonstrated WebFOCUS Hyperstage, an in-memory technology for managing the caching of analytics results, which was released earlier this year. This technology provides a simple way to increase performance and scalability. In addition, WebFOCUS now has its own information connector to Facebook for processing text to determine specific sentiments and potential intentions of individual users. In a demonstration using a Walmart beef promotion from Memorial Day weekend, the software found potential issues in how people perceive buying meat at the retailer. Through the software’s tag cloud visualization users can identify and interact with sentiment. Thus Information Builders addresses what our big data benchmark research found is the top benefit of big data technology, which is to retain and analyze more data, and it also increases the speed and accuracy of the results.

Information Builders is moving from its Developer Studio to a new Application Studio product for creating WebFOCUS applications. The new technology provides a simpler user experience for analysts and IT, who can assemble a range of applications, dashboards or portals, and it helps empower a rapid cycle of information and analytics deployment while bypassing tools that are too developer-centric for business users.

WebFOCUS Performance Management Framework (PMF) integrates analytics, visualizations and alerts to uncover root causes for why goals and objectives are not being met. It’s designed to help organizations manage performance, not just improve it – a practice that is slowly gaining momentum as organizations establish a single foundation for business analytics and metrics.

Along similar lines, Information Builders has built on its BI platform the embedding and extension of R, a statistical modeling environment that provides predictive analytics capabilities. At the conference presenters demonstrated a pricing optimization application assembled from predictive analytics that allows finance, operations or sales professionals to input specific variables to see the impacts of adjusted pricing. Integrating analytics with business intelligence tops the priorities in 58 percent of organizations according to our predictive analytics benchmark research and addresses one of the top obstacles, which is the difficulty of integrating predictive analytics into the information architecture.

Information Builders also introduced a new generation of visualization, from business charts to statistical methods. The company has integrated the Moonbeam HTML 5 catalog to support rendering from Web browsers on computers and tablets, making it adaptable to future needs to consume analytics across platforms and tools. Information Builders also provides a native deployment application in the Apple App Store that WebFOCUS can publish to.

The company also demonstrated new capabilities in its Magnify search platform that can scale against specific big-data stores and assess sentiment within text. Our benchmark research on big data found that search is one of the top five capabilities not available in business today.

Information Builders announced Omni-Patient, its packaged information solutions to deliver consistent and quality data related to patients for healthcare. This uses the Omni Framework, which is built on IBI’s information management technologies for data governance, data integration, data quality and master data management. The product helps advance the use of information in healthcare by reducing the risk and time of having IT or consultants build solutions. Our benchmark research on information management found obstacles in staffing, according to 68 percent of organizations, and also insufficient training and skills, both of which are addressed with Omni-Patient’s prebuilt information management frameworks.

To ensure it can meet the growing demand for cloud computing models and software as a service, Information Builders provides subscription pricing and supports multitenant computing with the required level of security. It also has expanded its support for to ensure it can be transparently accessed when embedded within its application framework. Its information management capabilities can support data-related activities within and between cloud environments and the enterprise; our data in the cloud benchmark research found that accessing, ensuring quality of and governing cloud-based data are important to more than three-quarters of organizations. Information Builders is beginning to see more demand for these features from customers who are dealing with a technological shift from on-premises to cloud computing.

Throughout its 37 year history Information Builders has demonstrated an ability to innovate without making acquisitions. The company has substantial growth in businesses and industries where the risk of not having information readily available can negatively impact business. I would like to see the company support more business and social collaboration in its products, to let customers share and capture the knowledge surrounding their information to reduce the time they need to make the best possible decision. With version 8 WebFOCUS supports key business technology trends. If you have not looked at how Information Builders has brought business intelligence and information management together to support business analytics, I recommend that you take the opportunity to see what its products can accomplish.


Mark Smith – CEO & Chief Research Officer