Liferay Analytics Cloud is planning a new feature to go beyond basic, out-of-the-box reports and provide for the specific reporting needs of our customers. This post is about the discovery phase — benchmarking against competitors, testing on potential users, and gathering customer feedback before a single line of code is committed.
In this post I will share a few of the things we solved prior to gathering customer feedback, to demonstrate the value that even basic, internal user-testing provides to product teams.
When we were first designing our product MVP, we recognized the need to track things beyond vanity metrics like views and bounce rate. While it's an important feature, it wasn't critical for the viability or our product — so we didn't launch with it. However, as our customer base continues to grow, this need has become increasingly more relevant. Because our customers are using Liferay DXP to provide “endless solutions”, Analytics Cloud needs to keep up and reflect the performance of these solutions in our reporting.
Earlier this year, we saw the need to add another level of customization for customers like:
This is the level of understanding we believe our customers require to deliver quality experiences for their users — and we can provide this to them with Analytics Cloud.
Close collaboration and good communication are critical for the designer / product manager relationship. One of the artifacts that we use to communicate is our product roadmap. Because we work on this together — it keeps all parties informed about what's coming down the pipeline. This allows us to work on long term features, while concurrently meeting immediate customer needs.
As this is a new feature, we want to ensure it provides value for our users and be as resource conscious as possible, by building the right thing. We approached this validation in 3 stages:
This feature is not a new idea and has been widely adopted across the industry. Large enterprise companies like Google and Adobe have their own products and features to support it while more focused analytics companies like Mixpanel and Amplitude built their entire platform around this feature. As a late entrant, we have benefited by being able to studying these implementations to jump start our ideation process.
We’ve had 2 rounds of user feedback cycles so far.
The first was a quick iteration cycle where I presented a low fidelity wireframe to one of our resident data analysts (outside of our product team) to get qualitative feedback based on his experience. He identified a few critical functions that could help define our MVP. Some things learned were:
Once those changes were made, we tapped the design research team to conduct our first official user testing session.
We conducted this test while still in the discovery phase — no code was written, no grooming was done — just a Figma prototype. At a macro level, we were trying to understand whether the design was learnable and matched the mental model of our potential user. At a more micro level, we were trying to document areas of friction and low levels of understanding.
Every user was able to complete all 5 tasks, much success! But, to stay on theme here, vanity metrics can be misleading.
During each test, our researcher captured comments made by our test participants and reported her personal observations. By removing myself and my biases from the testing process, we were able to discover some gaps that could impact our user’s understanding and overall experience (AKA qualitative gold). A few key areas we discovered users struggling in:
After taking the necessary steps to fill the gaps in our prototypes, we're confident that further testing with customers will provide even more value. Instead of having them uncover low-level issues, we're hoping they'll be able to provide high quality insights and feedback that we can use to further refine not only the designs, but also sales and marketing content.
This has all been done before a single line of code was ever committed. It's difficult to measure the impact this has on product quality (another post for another time) — but it's easy to see that had we launched this feature without any testing, we would have spent a significant amount of time and effort to fix the issues that we uncovered with fairly minimal cost.
Making quality products is a team effort — in this case people from product management, customer success, marketing, and others contributed. Huge thanks to @Miriam for the work in user testing.
In a future post, I'll share our process and results from customer feedback sessions — so check back!