Dashboards: The new face of BI

Dashboards are being increasingly adopted as the new face of Business Intelligence, because they can communicate complex information quickly, translating information into visually presentations, and they are also easy to use by business users, permitting to view the performance of business metrics at a glance and make decisions effectively and quickly. The dashboards transform BI from tools used by executives, information workers and power users to tools used by everyone in the company.

A performance dashboard is a multilayered application built on a business intelligence and data integration infrastructure that enables organizations to measure, monitor, and manage business performance more effectively. The performance dashboards are defined as three applications in one, woven and working together: monitoring application, analysis application and management application.

The performance dashboards have three layers or views of information: summarized graphical view (top layer), multidimensional view (middle layer) and detailed reporting view (bottom layer):
- Summarized graphical view - provides a summarized view, usually graphical, of the status of key performance metrics and exception conditions. This layer is where users monitor information. The technologies used are dashboards, scorecards and portal interface.
- Multidimensional View - provides the data behind the graphical metrics and alerts. Using multidimensional analysis tools, users navigate the data by dimension and hierarchies (slice and dice, drill-down, drill-up). The technologies used are online analytical processing (OLAP), parameterized reporting and advanced visualization tools.
- Detailed reporting view - lets users view detailed reports and transaction records. This layer connects users to existing operational reports or dynamically queries a data warehouse or operational system to obtain the records.

There are three types of dashboards: operational, tactical and strategic. The operational dashboards monitor core operational process and deliver detailed information that is only lightly summarized. The tactical dashboards are used in process or projects by a limited people. They are usually updated daily or weekly and have detailed and summarized data. The strategic dashboards are used to monitor strategic objectives. The objective of strategic dashboards are to align the organization, and is frequently implemented using the concepts of Balanced Scorecard.

To integrating performance dashboards, the companies can to use two kinds of approaches: centralized or federated. The centralized is the best way to integrate performance dashboards, automatically generating custom dashboards to the organization. The federate approach is used to link existing and incompatible performance dashboards.

The main benefits of performance dashboards are to communicate and refine strategy, give a consistent view of the business and deliver actionable information.

P.S. - The book Performance Dashboards: Measuring, Monitoring, and Managing Your Business, written by Wayne W. Eckerson, is a reference on the subject.

The use of agile techniques in DW/BI projects

The companies are increasingly more interested in DW/BI solutions that can be implemented and that shows results quickly, that is why the use of agile techniques in DW/BI projects are becoming important nowadays.

The Modern software development processes are iterative and incremental, it means evolutionary.

The concept of agile software development started in the 90's and emerged in 2001 when some software engineers came together to discuss ways to creating better software. They created the agile manifesto and defined some principles.

The Principles behind the Agile Manifesto are:
- Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.
- Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage.
- Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.
- Business people and developers must work together daily throughout the project.
- Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.
- The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.
- Working software is the primary measure of progress.
- Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.
- Continuous attention to technical excellence and good design enhances agility.
- Simplicity-the art of maximizing the amount of work not done--is essential.
- The best architectures, requirements, and designs emerge from self-organizing teams.
- At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.

You can use agile techniques in DW/BI

Although the approach to develop DW/BI projects be different of software development, the most of principles of agile can be apply throughout DW/BI lifecycle.

Some important steps that you should to use during the DW/BI lifecycle development are:
- Stakeholder involvement - The participation of stakeholder during all the project is crucial.
- Use an evolutionary approach - Envision the requirements and architecture at a high-level to start, and define the details during the process when they are required, using iterations. You can work from a broad requirement, with capabilities that are identified and determined through a prototyping process.
- Usage-centered approach - use cases and/or usage scenarios and/or use stories.
- Prove the DW/BI architecture works early - you need to prove that the ETL strategy works, you are accessing the major data sources and your front-end tools can access the DW.
- Use requirements to organize your work, not technical issues.
- Deliver some product working in each iteration.
- Check the data quality problems early.
- Test throughout the DW/BI lifecycle - The database regression test is necessary to assure your data, using a test suite to validate your data.
- DW/BI development is complex, you need to use good tools to data modeling, data quality, ETL process and front-end.

The power of Analytics

The scenario of Business Intelligence has shifted drastically, with news concepts, approaches and tools, and one of the most interesting and powerful is the concept of analytics.

A good definition of analytics, according Thomas Davenport and Jeanne Harris, in their excellent book Competing on Analytics: The New Science of Winning: "By Analytics we mean the extensive use of data, statistical and quantitative analysis, explanatory and predictive models, and fact-based management to drive decisions and actions. The analytics may be input for human decisions or may drive fully automated decisions. Analytics are a subset of what has come to be called Business Intelligence: a set of technologies and processes that use data to understand and analyze business performance."

Although the idea of analytics is not so new, only lately are appearing new approaches and tools to use. Analytical cultures and processes are appearing in any business that can use extensive data, fact-based decision making and complex statistical processing. Analytics don't only will support competitive strategies, but increasingly will perform a primary role.

The important factors to apply a successful analytics in the companies are:
- The companies need to have committed executives, that should be believers in analytical and fact-based decision making, appreciate the methods and analytical tools, and willing to act on the results of analyses.
- Build a sustainable pipeline of projects and analytical technologies to analytical groups.
- Bridging business analysis and technical development
- The companies need the availability of sufficient volumes of high-quality data, to have conditions to do sophisticated analytics.

The future of Analytics

The future of Analytics will happen through three type of changes: technology-driven, human-driven and strategy-driven. The technology-driven change include pervasive BI software, more real-time analytics , more automated decisions and increasing use supercomputers optimized for business intelligence applications, more visual analytics, more mining of texts, and more prediction with less reporting. The human-driven change is mainly because the companies need greater numbers of analytically oriented people. The strategy-driven change is because the companies will need to push the boundaries of analytics in their products, service and business models. The number of changes in the analytical environment will be driven by business strategies.

The importance of alignment between IT and the business

Nowadays, one of the most important issues in the companies is the alignment between IT and the business. It is a very difficult issue. First of all, it is necessary that the IT people know more about business, and it is also necessary the business people to know more about IT.

The big issue is because the IT and the business work with different value perspectives, on the whole, the IT uses the data to value perspective and the business area uses the goals to value perspective. Neither of these perspectives are wrong, but both are incomplete. The company needs to align the value perspective, sharing the business view with the IT view, basing IT actions on business strategies, making the IT results and business results converge to the same point of view.

The IT people need to learn more about business, know the business issues. For example, reading the documents from the company, like strategic planning, annual and strategic reports; also is important to develop interpersonal skills.

The business people need to learn what the role and the objective of the IT to the company are. Also they need to know what technologies are being used and how to use the technologies.

For the IT and the business to work together effectively, it is important define some guidelines:
- Define a business strategy and communicate to all business units of the company.
- Define a technology strategy and communicate to all business units of the company.
- Align IT and business in the levels strategic and tactic.
- Synchronize business plans and IT initiatives.
- Define a process to a continuous alignment, looking for identify gaps in IT and business process.

The BI can help the alignment

The companies should use the Business Intelligence to help the alignment between business and IT. When the company is starting a BI project, it is necessary understand the business strategy. The BI team need to study the business’ goals, and they can do that together with the business team, defining and showing how BI can help the business objectives and building together a business strategy plan.

The company also can use the concepts of operational BI to help the alignment. Operational BI can align the overall strategy with business process, using a process-centric perspective, unifying strategic and tactical business initiatives.

The IT and the business working aligned is a big challenge to the board of the companies. Consequently, the companies that solve this difficulty which is an important issue certainly achieve better results in their business.

The next generation of Business Intelligence (BI 2.0)

After the Web 2.0, it's time of BI 2.0. Everyone of BI area agrees that a lot of changes are happening, although several important analysts and professionals of industry try to resist the use of the name Business Intelligence 2.0.

Although Business Intelligence 2.0 is a buzzword, with a convenient marketing of the version number, the concepts proposed are quite interesting. By the way, there is nothing new about most of the concepts, what is changing is how to integrate with BI, through new approaches. The BI 2.0 is an extension of the traditional BI, not replacing it only complementing. Some concepts of new BI are based on the concepts of Web 2.0.

The BI 2.0 includes the concepts and technologies of data mining, statistical analysis, advanced visualization, mobile technology, alert notifications, rich reports, predictive analytics, BI search, text analytics and semantic data model.

The BI 2.0 also takes advantage with the concepts, processes, architectures and technologies that improve business process and event processing, as Service-Oriented Architecture(SOA), Business Process Management(BPM), Business Activity Monitoring (BAM), Complex Event Processing(CEP) and Event Stream Processing(ESP).

While the main function of traditional BI (sometimes called BI 1.0) is to analyze the past, the BI 2.0 is to predict the future, using the concepts of traditional BI together the concepts of BI 2.0 to build a smart picture. The BI 2.0 is more pro-active than reactive. The traditional BI is closed loop, unable outside input data, because it occurs after the events, and BI 2.0 is open loop because enable input data when the events occur, and its real time analysis.

The BI 2.0 is more intuitive, interactive, pervasive, collaborative and process-driven.

The main effects of BI 2.0 in the companies are: Data integration and comprehension (using Master Data Management), roles (people with many roles at the same time), lower cost of BI licensing (many people using BI), convergence of structured and unstructured data, convergence with the rest of operations.

But one of the most important concepts of BI 2.0 is not about technology, is about people. The technical innovation is the easy part, the hard is to align the organization. It is important to focus on people, remember that with the continuous technical innovation, people are taking cultural shifts, and people will use insights, a new way of thinking and the concepts of social networking and collaboration to work more effectively.

The BI 2.0 allows a BI for everyone, not only for executives, power users or information workers, and extends beyond corporate boundaries, to suppliers and customers.

The traditional BI makes the deal, is useful and keeps alive, but BI 2.0 will provide a new way for the companies improve their results.

Best Practices to implement BI in your company

Currently, with the complexity of business and the necessity to make decisions quickly, implement a Business Intelligence project on the companies is crucial, but not easy.

Because this, you need to adopt some best practices to achieve implement a successful BI.

First of all, you need understand your business issues before implement your BI, aligning the business goals with the BI strategy, creating a solid partnership with business and IT.

When you start the process to implement BI in your company, it is essential that you have an executive sponsor that has influence on all divisions and business units of the company, because it is necessary to show and persuade the business that BI is not just another IT project, is a continuous process to delivery better information to the company make better decisions.

It is interesting to consider the creation of a Business Intelligence Steering Committee. The BI Steering Committee includes the senior representatives of the principal areas of the company and the BI manager. The BI Steering Committee will work like a gear to BI project.

It is interesting also to consider the creation of a Business Intelligence Competency Center (BICC). The BICC must be a center of expertise for BI, sharing resources, best practices and support to maximize its use in the company. The BICC can be physical or virtual (a team with defined roles and tasks).

When you are gathering business requirements, you should to talk with persons in different levels of the company, from directors, managers to analysts. If possible, use two persons from requirements team in each meeting, and always invite more than a single user to represent the business in each meeting, even when you think that the subject is easy. You should to document what you learned in the meeting and give feedback with the results. Remember that you should keep the scope of your DW/BI requirements process. During the process, you should use many approaches, from the traditional method to ask the people what they want, also looking for study how the processes work through the company, how the people work, how they make decisions, which information they need to make decisions and when they need those information.

The IT and the business need to develop together a business-focused metadata that provide all the business requirements.

It is very important to check the data quality in the source systems before start the BI development, because one of the main keys of success to a well implemented BI is the data quality. (I wrote about this in a previous post called The concern about data quality).

Choose carefully your BI Tools, looking for standardization. Of course, you can and should choice a BI vendor to front-end, and others to the back-end as data warehouse platform or ETL tool. The standardization is mainly in the BI front-end tools (query, reporting and analysis); you should choice only one BI suite of tools that attend all the front-end requirements. What you shouldn't buy different tools from the different vendors to the same area of BI, like to buy multiple reporting or query tools.

Ongoing training the users on the BI tools and how to use better the data in the tools, also offering the right BI tool to the right user group, recognizing the importance of BI front-end tools in attracting the users.

If you are migrating from the old DW/BI to a new project, it is very important define before start the new project, a group to identify and solve all problems about data before and during the migration of environment. This group should be consisting of business and IT people, and define together guidelines about data quality, to ensure that the right data are move to new DW/BI environment.

Always keep in your mind that Business Intelligence has to delivery the right information when it is required, in the right format, in the right time to the right people.

The Real-Time Enterprise and the Right-Time Business Intelligence

In the beginning of Business Intelligence, a several years ago, when the BI projects were defined only to use in the strategy level, the data warehouse was normally updated monthly, and sometimes weekly, in some specific areas.

Nowadays, the companies need to make decisions quickly, and with the expansion of BI for operational applications, update the data warehouse has become increasingly in near real time.

We are in the era of real-time enterprise, and the data need to be updated according to the business requirements, in some cases, less than a second after after a business event has happened in a transactional system.

The process to get information almost without latency in the business event is called Real-Time Business Intelligence and the process to delivery information when it is required is called Right-time Business Intelligence. The difference is Right-Time BI can not be necessarily in real time.

Although the latency depends of the kind of business, and all BI systems have some latency, it is important define which time is better to update the Data Warehouse, that can be less than a second, seconds, minutes, hours or daily, according to serve the business requirements. For define this time, you should know that as more you reduce the time of data latency, you will reduce the time to take action.

Some areas where is very interesting to apply the concepts of Real-Time BI are: financial markets (stock exchange), fraud detection, application performance monitoring and risk management.

Some kinds of application don't need of real-time BI, because don't need to update the data warehouse frequently. There are cases where if you implement the Real-Time BI, this will be a hindrance, because the data updated quickly can affect the analysis. For example, a sales application where the customers normally emit orders once a week, if you update the data warehouse daily, and try to analyze the results in 3 or 4 days, you can make wrong decisions, because sometimes several orders are emitted in the last days of the week.

I think the concept of Right-Time BI is comprehensive and interesting, you delivery information when it is required, being real time or not.

P.S. - The term Right-Time Business Intelligence was coined by Colin White, in a report of The Data Warehousing Institute (TDWI), entitled “Building the Real-Time Enterprise”.

The concern about data quality

Lately, the concern about data quality in the companies is growing, the called dirty data or bad data.

The dirty data are the data considered incomplete, inaccurate, incorrect, outdated, redundant or misleading.

All the companies around the world have problems with the data quality, to a greater or lesser degree, and a lot of companies ignore the issue of data quality or have underestimated the importance.

The poor data quality takes significant costs to the companies, and obviously, this reflected in the profit.

Nowadays, in the age of globalization, with a lot of companies acquiring others companies around the world, those problems are aggravated. Culture different and the complication to integrated data of companies, resulted in numerous ways of entering shipping, billing, invoice and other key data. This kind of problem with inaccurate data is typical across all industries.

Especially when you are talking about Business Intelligence and Data Warehouse. To be a successful BI, you need to have data accurate, because you can have a Data Warehouse correctly modeled , a well implemented BI using good tools, but if your data are not accurate, you will make decisions using unreliable information.

Data Governance

It is very important the companies create a policy of Data Governance, but is not so easy.

First of all, you have to know your data: checking what you have, if have IT projects duplicating information, which data is inaccurate, which data is reliable and which is not. It is important to communicate to stakeholders about the value of create a policy of data governance and the cost of not having it to the company. Successful data governance depends on sponsorship in management level, preferably the CIO. You need to treat your data as a valuable corporate asset.

Focus on create data quality processes, a master data management solution and establish a culture of data governance. Ongoing training both IT and users, always showing that the measures of data governance are helping to improve your data quality.

Define a data cleaning process, considering that to clean the data is very expensive to the companies. The better way to clean your data are define strategically, IT together with the business users, which data need to be clean based on the business impact. If the cost of dirty data on the business impact are greater than the effort and cost to clean, this data certainly should be clean.

To keep the data cleaned, is very important controlling who accesses the data, and which level of manipulation have, allowing only each person responsible for each process has access to input, update and delete the data in this process.

After all, the companies need to use data quality tools, to create mechanism of standardization and validation, so they can ensure that the new data entering in the systems are clean.

Ten Business Intelligence Trends and Forecasts

1. The companies will be more interested in BI solutions that can be implemented and that shows results quickly.

2. The concept that BI is not just another IT project, BI is a continuous process to provide accurate information to help the business users to take better decisions, will continue to grow within companies.

3.The globalization has increased the concern about access data in real time. The companies have data distributed across worldwide, in different locations, becoming the data integration in an essential issue. To fill this gap between the demand and the current reality, the companies will need to use increasingly analytic applications with real-time monitoring, near real-time dashboards, and other applications that use the concepts of Complex Event Processing (CEP) and Event Stream Processing (ESP).

4. The concern with data quality and the utilization of Master Data Management (MDM) will grow and also how make the convergence of structured and unstructured data.

5. IT and Business increasingly will work together. The business users will have more participation in the decision about the BI solutions, from the choice of technology and BI vendors to definition of business model.

6. BI will continue improving to be more intuitive, interactive, pervasive, collaborative and process-driven.

7. BI will have a continuing growth in small and medium-sized companies.

8. The BI vendor consolidation happened in 2007, will cause more interest in open source BI.

9. The BI as a Service will continue to grow, because offers many benefits over the traditional BI vendors, like no required purchase of hardware, significant faster and lower cost in implementation.

10. The BI will be more integrated with Business Process Management (BPM) for analyzing historical process data and support decision business processes.

2007 was a good year for Business Intelligence

This was the year that the big IT Companies bought the BI Companies. In less than one year, from March to November, 3 big acquisitions happened, all for billions of dollars. Oracle acquired Hyperion, SAP acquired Business Objects and IBM acquired Cognos. Few areas are in a process of consolidation as the Business Intelligence industry. This consolidation is happening because technology is essential for develop any company and the deeper knowledge about information is increasingly fundamental to create business strategy for the future, and that is the main role of BI companies. For those companies, those acquisitions are important to support a competition in a consolidated scenario.

The pressure for low-cost of tools helped strengthen some trends, like Software as a Service (SaaS) and open source BI (OSBI). The SaaS came definitely to Business Intelligence. New BI vendors appeared in the marketplace offering products using SaaS model and some traditional BI vendors are offering their own products as SaaS.

The open source BI (OSBI) also received more attention in 2007. The success of open source and collaborative development in the other areas of IT industry, like Operational System (Linux), Database (MySql and PostgreSql) and Application Server (Jboss), are making the companies look to OSBI with attention. There are several companies (Pentaho, Talend, Actuate, JasperSoft and others) developing OSBI products, providing a different approach to the marketplace.

I think 2007 was a good year for Business Intelligence and 2008 promise much more.

PS - This post was originally published in 1/31/2008, in my old blog: http://mjfb.wordpress.com/