You are currently browsing the category archive for the ‘Big Data’ category.

Organizations today create and collect data at ever faster rates, and this introduces challenges in ensuring that data is not just managed but used in a consistent manner for a range of operational and analytic tasks. This is made more difficult by new data sources whose definitions vary from standard and vr_Info_Optimization_04_basic_information_tasks_consume_timewidely used formats. Making all information available and consistent is essential to support business processes and decision-making. A key technology tool for this effort is master data management (MDM). Every business area needs MDM, whether it deals with customers, products, employees, finance or others individually or collectively in what is called multidomain MDM. It is an essential tool for data governance across an organization, which has become a focal point for improvement as many organizations spend significant time in data-related tasks. Our benchmark research on information optimization shows that preparing data for analysis (47%) and reviewing data for quality and consistency issues (45%) are the two information tasks that consume the most time. Properly used MDM enables data stewards and other IT professionals to improve the consistency and quality of departmental and enterprise data.

Profisee is a software vendor that seeks to provide consistent definitions across data through master data management. It has been part of the MDM industry since its origins when the company’s founders created what is now part of Microsoft SQL Master Data Services. Profisee offers dedicated technology to help manage data across applications, databases and systems. Its data management platform and tools, Master Data Maestro Suite, provides the ability to establish and maintain consistent business definitions that help normalize any kind of data. It supports analysts, data stewards and others responsible for making data suitable for a range of operational and business tasks by helping them establish “golden” records – the agreed-upon data definitions and formats that users can trust.

Maestro Desktop provides the core functions of MDM, from modeling of data to definitions and mapping of data from applications and systems. It enables management of hierarchies that align, map and roll up data. In addition it offers Golden Record Management in which data quality, matching and survivorship contribute to establishing golden records for operational and analytical needs. Profisee handles various data quality tasks from simple ones such as address validation to location and contact services that help streamline tasks that can be difficult and time-consuming. The product has been designed to enable analysts across business units and IT to share in the tasks that ensure that master data is being applied properly. Because MDM should involve collaboration across roles in the organization, Profisee include workflow so the right people can share and approve tasks. Our information optimization research finds that in 42 percent of organizations business and IT work together; this is essential because business users are most familiar with the data and its use in business processes and resulting outcomes.

Profisee Master Data Maestro also provides analytics that help users understand the state of data governance and track data processing to determine issues to be resolved. It supplies adapters to help integrate data to databases and applications from Microsoft, Oracle, Salesforce.com and many others. It also provides a developer’s kit to help design Web services and message-based interfaces needed for application and system integration. It also extends the use of Microsoft SQL Server and Master Data Services by providing broad support for data governance and MDM across an enterprise.

Our research confirms the growing number of data sources: A significant number of organizations that have more than 16 sources said it is very important to simplify information for business. This situation is creating increased pressure to improve vr_Info_Optimization_17_with_more_sources_easy_access_more_importantdata-related processes which MDM can help. Advances in so-called big data now enable organizations to take in huge volumes of data at rapid cycle times, and big data is an important technology in 41 percent of organization, according to our information optimization benchmark research. As big data becomes widely adopted, MDM becomes critical to ensure that data is used in the proper context. Our research finds that establishing master data has become a top concern in two-thirds of organizations. Profisee has expanded its support for big data environments including Hadoop, ensuring it can process data from various databases and platforms.

Another rising technology, cloud computing, also can benefit from MDM. Cloud deployment of applications and data increases the need to manage data outside the enterprise. Profisee recently announced MDM for Microsoft Azure Cloud, which enables Master Data Maestro to support Microsoft SQL Server Enterprise Edition and Microsoft Master Data Services in the company’s Azure cloud. Supporting cloud-based systems is nothing new for Profisee, which has developed several adapters such as it did in 2014 for Salesforce.com.

Addressing the need for quality and consistency of data is essential to business processes and decision-making that rely on business analytics. Profisee is contributing to this effort not only through its products but also through its range of services and industry models which help perform proof-of-concepts and establish a roadmap that will help ensure smooth adoption of MDM, avoid lengthy consulting engagements and save time and resources. Profisee is making MDM practical and easy to use, and its product does not require a large budget to get started and maintain data governance in small and midsize companies. If you are looking to improve your data governance process through MDM whether across the organization or within a line of business, Profisee is a supplier that you should examine closely.

Regards,

Mark Smith

CEO and Chief Research Officer

Big data has great promise for many organizations today, but they also need technology to facilitate integration of various data stores, as I recently pointed out. Our big data integration benchmark research makes it clear that organizations are aware of the need to integrate big data, but most have vr_BDI14_performance_01_overallyet to address it: In this area our Performance Index analysis, which assesses competency and maturity of organizations, concludes that only 13 percent reach the highest of four levels, Innovative. Furthermore, while many organizations are sophisticated in dealing with the information, they are less able to handle the people-related areas, lacking the right level of training in the skills required to integrate big data. Most said that the training they provide is only somewhat adequate or inadequate.

Big data is still new to many organizations, and they face challenges in integrating big data that prevent them from gaining full value from their existing and potential investments. Our research finds that many lack confidence in processing large volumes of data. More than half (55%) of organizations characterized themselves as only somewhat confident or not confident in their ability to accomplish that task. They have even less confidence in their ability to process data that arrives at high velocity: Only 29 percent said they are somewhat confident or not confident in that. In dealing with the variety of big data, confidence is somewhat stronger, as more than half (56%) declared themselves confident or very confident. Assurance in one aspect is often found in others: 86 percent of organizations that said they are very confident in their ability to integrate the variety of big data are satisfied with how they manage the storage of big data. Similarly 91 percent of those that are confident or very confident with their data quality are satisfied with the way they manage the storage of big data.

Turning to the technology being used, we find only one-third (32%) of organizations satisfied with their current data integration technology, but twice as many (66%) are satisfied with their data integration pro­cesses for loading and creating big data. A substantial majority (86%) of those very confident in their ability to integrate the needed variety of big data are vr_BDI_03_plans_for_big_data_technologysatisfied with their existing data integration processes. Those that are not satisfied said the process is too slow (61%), analytics are hard to build and maintain (50%) and data is not readily available (39%). These findings indicate that making a commitment to data integration, for big data and other­wise, can pay off in confidence and satisfaction with the processes for doing it. Additionally, organizations that use dedicated data integration technology (86%) are satisfied much more often than those that don’t use dedicated technology (52%).

New types of big data technologies are being introduced to meet expanding demand for storage and use of information across the enterprise. One of those fast-growing technologies is the open source Apache Hadoop and commercial enterprise versions of it that provide a distributed file system to manage large volumes of data. The research finds that currently 28 percent of organizations use Hadoop and about as many more (25%) plan to use it in the next two years. Nearly half (47%) have Hadoop-specific skills to support big data integration. For those that have limited resources, open source Hadoop can be affordable, and to automate and interface with it, adopters can use SQL in addition to its native interfaces; about three in five organizations now use each of these options. Hadoop can be a capable tool to implement big data but must be integrated with other information and operational systems.

Big data is not found only in conventional in-house information environments. Our research finds that data integration processes are most often applied between systems deployed vr_BDI_07_types_of_data_integration_processeson-premises (58%), but more than one-third  (35%) are integrating cloud-based systems, which reflects the progress cloud computing has made. Nonetheless, cloud-to-cloud integration remains least common (18%). In the next year or two 20 to 25 percent of organizations plan additional support for all types of integration; those being considered most often are cloud-to-cloud (25%) and on-premises-to-cloud (23%), further reflecting movement into the cloud. In addition, nearly all (95%) organizations using cloud-to-cloud integration said they have improved their activities and proces­ses. This finding confirms the value of inte­gration of big data regardless of what types of systems hold it. With a growing number of organi­za­tions using cloud computing, data inte­gra­tion is a critical requirement for big data projects; more than one-quarter (28%) of organizations are deploying big data integration into cloud computing environments.

Because of the intense need of business units and process for big data, integration requires IT and business people to work together to build efficient processes. The largest percentage of organizations in the research (44%) have business analysts work with IT to design and deploy big data integration. Another one-third assign IT to build the integration, and half that many (16%) have IT use a dedicated data integration tool. The research finds some distrust in involving the business side. Almost one in four (23%) said they are resistant or very resistant to allowing business users to integrate big data that IT has not prepared first, and the majority (51%) resist somewhat. For more than half (58%) the IT group responsible for BI and data warehouse systems also is the key stakeholder for designing and deploying big data integration; no other option is used by more than 11 percent.

It is not surprising that IT is the department that most often facilitates big data and needs integration the most (55%). The most frequent issue arising between business units and IT is entrenchment of budgets and priorities (in 42% of organizations). Funding of big data initiatives most often comes from the general IT budget (50%); line-of-business IT budgets (38%) are the second-most commonly used. It is understandable that IT dominates this heavily technical function, but big data is beneficial only when it advances the organization’s goals for information that is needed by business. Management should ensure that IT works with the lines of business to enable them to get the information they need to improve business processes and decision-making and not settle for creating a more cost-effective and efficient method to store it.

Overcoming these challenges is a critical step in the planning process for big data. My analysis that big data won’t work well without integration is confirmed by the research. We urge organizations to take a comprehensive approach to big data and evaluate dedicated tools that can mitigate risks that others have already encountered.

Regards,

Mark Smith

CEO and Chief Research Officer

Mark Smith – Twitter

Top Rated

Stats

  • 173,464 hits
Follow

Get every new post delivered to your Inbox.

Join 18,549 other followers

%d bloggers like this: