Sometimes, when I communicate with product companies, it seems to me that they are too much obsessed with data.

In recent months, with colleagues from various fields, we have been discussing the role of data analysis in decision making in product development. And in the process of these discussions an interesting thought was born, which I want to share. In various projects, I notice that the main problem with decision-making based on data is that, in fact, decision-making does not take place, but only an endless analysis and discussion of its results. Teams and companies to which analytics brought significant results at the start of the project noticeably obsessed with data and focused on the product internals so much that they forgot about the world around them. Work on the product in such cases resembles the endless improvement of a push-button telephone, while smartphones appears around.

Why didn’t the data please me?

At the initial stage of product development, any sane data analytics certainly gives a result that helps to develop the product and move the business. To scale the result, obviously, it is necessary to scale the analytics as well: hire more analysts and invest in the integration of different analytical systems into a single data-warehouse. And at some point, usually the moment when the “superanalytics” system, covering all facets of the business and managed by the best graduates, is ready, it turns out that the product in its current form is already “done”, that is, all those changes that could give a significant ROI already contributed.

And what happens next?

1. The product does not get better

Analysts need to do something; systems were also developed for good reason. All those beautiful stories begin when the buttons move on one pixel, and all changes are justified by the research. Even articles are written about this on Medium and are upvoted on HackerNews, but revenue for some reason does not grow or grows as part of astrological fluctuations. And the product remains the same.

2. Changes are implemented slowly

Serious changes are not implemented or are implemented more slowly than competitors, because they fail on all tests. Old users love what is familiar, but on new products it will perform worse than the old one, because the old concept has been updated over the years by live users, and the new one is not yet. It takes a lot of time to constantly test hypotheses. The product gradually stagnates, although 1–2% of improvements in the moment can be observed, and they are proudly reported to the management and in press releases. The employees who were responsible for the success of the project, at the same time, lose their motivation, because for years engaging in changing the color of buttons is boring.

3. Analytics becomes an instrument of justification

Data analysis involves interpretation, and it depends heavily on the goals of the interpreter. For management, a position is very beneficial when each decision is supported by research. This allows in case of any problem to make a reference to the study, and not to its incompetence. Justification of those responsible is reminiscent of the classic: “the trading strategy successfully works on historical data.” Perhaps there is still not enough data. Or more analysts required.