A Phoenix from the Ashes. Behold the Power of Data!
October 7th, 2016
A Phoenix from the Ashes. Behold the Power of Data!
If I had asked people what they wanted, they would have said faster horses OR How Apteligent achieved a culture of Innovation.
Greatness often comes from resilient teams seeking a transformative approach to a problem; Apteligent created a rich platform for innovation from the ashes of a failed project. A safe innovation environment, data driven thinking, reduced red tape, and a time allowance for experimentation form the basis of this creative cultural shift. Its first product is a game-changer in the crash reporting industry.
How an imperfect project contained the seed of a radical culture change
Consider this quote from Henry Ford: “If I had asked people what they wanted, they would have said faster horses”. Henry Ford was no doubt a pioneer in innovative thinking, something that Apteligent has been trying to foster for several years. It’s not easy – there doesn’t seem to be any hard and fast rules to make people innovate. However, the frequency of innovative ideas at Apteligent seems to have exploded in recent months. It all started with one long and tedious project: “Better Crash Grouping”.
The crash reporting industry is about collecting crash data from consumers’ smartphone apps and reducing that data into information that the developer can consume, referred to as “crash groups”. Crash reporting is one of Apteligent’s features so we are no strangers to this problem. However, the information is often overwhelming, the app developer can be confronted with hundreds of thousands of crash groups, each one representing a problem that many of their app users experienced.
In a data-driven business such as ours, an obvious place for innovative opportunities is in our data.
Across the entire crash reporting industry, developers have been crying out for “better crash grouping”, the process in which the vast multitude of crash reports are reduced into a representative group to help focus developer effort. Here’s generally how it works (here’s how it works in much more detail):
A stack trace is extracted from a crash report and compressed into a key using a process called “hashing” – which is the basis of the group. If there is a subtle difference in the stack trace, a different key will be produced and a new group will be formed.
There have been surprisingly few deviations or innovations for crash grouping in the industry, which either points to a really hard problem or an industry that doesn’t care.
Well, Apteligent cares, and we don’t shy away from hard problems (what good engineer would?)
During a long and tedious project (“Better Crash Grouping” / BCG) to modify our grouping algorithms, we spent a LOT of time looking at our crash data with the intention of solving the grouping problem. The challenge was to find an algorithm that could be applied to all customers’ data and strike a balance between being aggressive enough to reduce the number of groups, and being too aggressive where we collapsed unrelated crashes into fewer, larger groups.
Despite the best efforts of our engineers, the data spoke for itself; in the end we were unsatisfactorily forced to err on the side of caution and not be aggressive with grouping. And that was absolutely the correct thing to do. It was essential to preserve and protect the integrity and atomicity of the crash group. However, most of our customers will disagree – their justification being that if you look at the stack trace you can SEE it’s obviously the same as many other groups. In other words, it’s obvious to a human but not to our grouping algorithm. The project left our customers frustrated and our engineers disheartened.
But wait… a phoenix from the ashes. Behold the power of data!
Though the BCG project was super expensive and not very effective, we made an observation during the study of a single customer’s data: The same suspect line occurs in many crashes and is a natural way to join groups. This gave us an unexplored path to find crash groups.
A safe space to innovate
Having an insight into a new innovative opportunity doesn’t count for much if there is no room or flexibility to incubate these insights. Alongside the new platform and data opportunities, we made a fairly significant cultural shift: instead of a centrally-controlled schedule, we moved responsibility to individual teams for scheduling and committing to regular work.
This initiative came from the top down, and the teams felt trusted that they could get the assigned work done and also embark on innovative projects. The important change in mindset gave the teams the opportunity to shift the workload and simultaneously spend some time on “the secret projects”.
We found that if people are insecure, unhappy, or feel that they are being watched, measured, and tracked, they lose the incentive to innovate and will focus only on delivering the prescribed work. People who are excited about the work they do, and the opportunities to create something phenomenal, will happily brainstorm and work on innovative projects outside working hours (the shower seems to inspire ideas?!).
It’s very important to acknowledge innovative ideas, as well as the people who suggest and work on them, regardless of eventual inclusion in the product. We have found that including innovative thinking and ideas into the sprint demo proliferates the culture of innovation and spreads excitement. “We did it first! We’re doing things other companies haven’t done!”
BCG gave us the access to the data, but the culture and excitement over new possibilities is what drove us forward. It was the sharing, recognition, and encouragement of innovative thinking and the space created in the schedule, even, dare we say it, by “under loading” staff, that drove innovation.
A solution worth finding a problem for
The classical way of creating a product is to find a problem and then a solution. The product might be great and useful for the users, it might even be just what they want (faster horses anyone?). Innovation shows people what they don’t even know they want. Finding the problems that can be solved with a given solution becomes the innovation.
Now, back to our new idea. Remember that we discovered we could access crashes by the suspect line in the stack trace. We pondered how we could use this to improve the product. It didn’t take long to realize two things…
The file name is available in the suspect line
Problem solved: Find all groups that are rooted from a given file!
There is often a correlation between the suspect line and one or more other lines of the stack trace.
Problem solved: From a given group, find other crash groups that share the same characteristics AND sort them by relevance!
Behold the power of Innovation! We don’t want to change the atomic basics of grouping, we will give you multiple attack vectors on your crash reports, giving the ability to reduce hundreds of thousands of groups down to the essential causes of issues – by the same file name, or by similarity of the stack trace!
Reduce the red tape
An important part of innovation is getting feedback from the people who will be using it: our customers. But the only way to get real and valuable feedback is to get the implementation deployed to production.
Initially this was a huge blocker for testing new ideas. We had a heap of red tape with regard to what was presented to our customers, like the UX had to be polished and branded, it couldn’t cause errors, it couldn’t be slow, it had to pass quality checks, etc. These are important criteria for a fully-fledged product feature. Our customers have the right to expect a quality product, but if you just want to know if it’s a good idea or not, then all this makes it too hard.
Clearly we had a good idea, but on the legacy platform it was slow to calculate, and the practical use of Similar Crash Groups (like bulk status changes of groups) were not safe operations to perform on the old system. There was simply far too much effort required to get it to “production quality” just to get it out to our users for evaluation.
Solve policy issues with policy. We wrote a policy called “Apteligent Labs”. Here is a quote from the document:
…Scrappy style. Wild Wild West. Just one driving principle: Protect user experience.
Be clear about level of polish of any given ‘ingenuity’ feature or area as well as the level of support (i.e. we may choose whether or not to fix bugs, no commitment nor obligation)
Allow users to just ignore it if they so wish, i.e. not use it without having any of their flows disrupted
Make sure the feature does not take down the product if it doesn’t work (allow you to be flexible with respect to test coverage at no risk to production users)
Be consistent in how we present these features (whatever visual cue we use) so they are easy to identify without the user having to be retrained every time
If we choose to ‘kill’ a lab feature, we just tell the users it’s not there anymore and move on. Or we leave it and mark it as ‘unsupported’.
If you are an existing customer, you might have noticed features appear with a “Labs” icon next to them, certainly “Similar Crash Groups” was one of them. It was received extraordinarily well. Our customers loved it. The Product Development Team loved it. Yes we would show you the similar groups, but the only issue was that restrictions in the old processing platform prevented us from doing anything with it.
So Similar Crash Groups was in production and our customers were using it. What happened to it? Our customers loved it so we decided to graduate it from Labs to a full feature. We ran the experiment, gathered the feedback and then rolled a production quality version of the project into the product roadmap. A first pass of the project is nearing completion and you’ll see it as part of the product in the next few weeks.
Innovation can’t be forced but it can be fostered
A contributing factor to our ability to innovate has been the successful development and deployment of our new data processing platform, which gives us unprecedented insight into the data we collect. We now have the ability to slice and dice, mix ‘n match, and find the secret herbs and spices in our data. With an innovation-orientated mindset, consider how you can best expose your assets so your people can go wild with it!
The takeaway from our experience of creating an innovation culture at Apteligent is that it can’t be forced. You can’t make people innovate. What you can do is make sure that everyone from executives to staff set the right environment for it to proliferate: data access, time to experiment, a safe place where innovation is encouraged.
Innovate. Something amazing will emerge.