I soaked up a beautiful summer day while waiting for a ride to the airport. I felt buoyant. Just moments ago, I finished a multi-day tour of a design innovation lab run by a well-respected Fortune 500 firm. Its mission: Improve employee and customer experiences around the world by evaluating and redesigning digital products that passed through the lab.
I actually got to witness, hands-on, one project go through the lab’s process. Incredibly well-run and thought out, I was filled with excitement about what the lab was able to create in such a short time. Although evidence was qualitative and anecdotal, it seemed obvious the lab uncovered relevant improvements to the stakeholder’s employees’ experience. As a designer, I was basking in the afterglow of this “feel-good” process.
But once I got on the plane, I had a sobering thought. What happens once the project leaves the lab? What if the project’s team can’t execute those improvements to the employee experience? How much support will the project have from designers in the months ahead?
In that moment, I had an epiphany: The excitement of going through any design lab needs to be tempered by an unbiased measure of the outcomes directly attributable to the lab itself.
Outcomes over output. Proof over potential.
What design innovation labs (DILs) need—beyond great process, people and governance—is a framework for measuring outcomes.
After all, what good is an DIL if it can’t deliver results in the real world?
Note: While this article focuses on DILs (funded by one or more business units and responsible for internal and/or customer-focused projects), some of these success metrics may be relevant to ad-hoc design labs as well.
A Wide-Ranging Landscape
Unless you’ve designed for/at a very large tech or engineering company, chances are you’ve probably never interacted with an DIL.
The DIL is a close cousin to a traditional innovation lab, with the purpose of uncovering newer, better ideas and processes and evangelizing them to the rest of the company. However, unlike an innovation lab, an DIL often ties its hitch to company operations. Yes, an DIL explores the innovation space, but it’s also often tasked to help business units apply better design to existing digital products and services.
At some companies, the DIL is owned and/or operated by the global UX leadership. While they may not be tasked with innovation by creating brand new technology, they are often charged with applying that innovative technology to enhance customer or employee experiences.
In other words, while an innovation lab often “lives in the clouds,” an DIL usually has to operate with both boots on the ground and eyes to the sky.
Some DILs are primarily designed to explore and push the edges of innovation. It’s merely enough to hop on the latest tech and prototype the “art of possible” to notch up wins. Other DILs run more like factories: A selected group of projects run through the lab because they are expected to go (eventually) into production.
That divergence makes it harder to measure success in a “one size fits all” way.
Furthermore, no two companies are alike, so DILs come in many shapes and sizes. Some are centralized, others distributed. Some manage projects end to end, others act as a drop-in design studio. Levels of touch also vary: some DILs can only consult and advise, while others can enforce a design system.
Qualities to Consider
That said, the common-thread mandate for most DILs is that for any digital project that passes through it, the lab delivers a process that:
- Enhances the designed user experience
- Drives measurable business value
- Ensures the product is technically feasible
If the projects that run through the DIL relate to marketing and sales initiatives, a fourth bullet pertains:
- Is on brand and strategically aligned
For DILs that have to defend spend, a critically important fifth bullet:
- Justifies the cost of operating the design lab
I’d love to hear from other DIL managers about any other proof points that should be added to this list. That said, I feel this rubric sets a solid foundation for success metrics.
Success Metrics: Pluses and Pitfalls
At the end of the day, there won’t be a single set of metrics that will universally apply to all companies, but here are some worth considering.
To measure enhancement of the user experience, it’ll be important to strip away bias. The natural bias of a participant going through any DIL will be to assume that the DIL positively affected the output. This makes NPS-type measurements of the lab problematic; they may validate the lab’s process but not the project’s outcome.
Instead, an DIL should consider baselining the current experience of the digital product or service before going into the lab, then do an identical measurement of the design output post-lab, and compare the results. A solid DIL process should see statistically significant improvements.
To measure business value, much will depend on the parent company’s business metrics (and more importantly, the business units sponsoring the DIL), as well as the perceived value of innovation vs. production within the company.
Companies that highly value innovation can compare their project outputs with competitors: Are we keeping up? (In some industries, the mere existence of an DIL can be a competitive differentiator regardless of its actual impact.)
In blended DILs, one can measure number of innovations actually implemented in production, then subsequently measure impact on operations, whether that’s time savings, efficiency improvement, cost reduction and/or revenue lift.
In production-focused DILs, one could measure number of projects (as a percentage of all company projects) that go through the DIL vs. do not. While it might be politically sensitive to draw contrasts from comparing the two streams, it would be relevant to count the number of employees and/or customers who benefit each year from projects that run through the DIL. (For DILs that have to defend a budget, measuring reach within the company is a useful metric.) If the company embraces design, one could even measure less vital but “feel good” storytelling metrics such as hours of collaboration between business units or number of happy quotes from participants.
If the DIL is responsible for managing a portfolio of innovations and/or projects, some traditional innovation measures can be considered:
- Number of projects in pipeline; value of pipeline
- Percentage of innovations that make it into production
- Speed to market (if speed is a relevant metric)
To measure technical feasiblity, the key metric should be the number of consulting engineers, keenly knowledgeable in your company’s IT and ops infrastructure, that are embedded in your DIL process. These engineers will be a critical “voice of reason” to bridge the gap between art of possible and real-world usage, and help the DIL prioritize which innovation-driven design projects are worth pursuing. The greatest innovation is meaningless if it can’t be built.
This may sound counter-intuitive, but one key metric to consider is the number of projects killed by the DIL, especially if it leans toward ops. If the DIL can filter out technically unfeasible projects, it can quantify the amount of money saved by stopping projects before they become budget black holes. Alternatively, an DIL can measure success by uncovering less expensive alternatives that still meet original project design goals, and take credit for those cost savings through better, more practical design.
To measure being on brand and strategic alignment, simple perception and awareness surveys should do the trick. If the DIL is perceived as a marketing and/or sales asset, then treat it just like any other product. Give it a defined marketing budget (both internal to employees and external to customers), a brand strategy and—if relevant—a timeline of events and activities that promote the benefits of the DIL and its team members.
Even if the DIL is more operationally based, it’ll be important to manage perception internally with good messaging to reinforce value. This is especially true if your DIL is a voluntary drop-in service and/or demand within the organization is low. Tangentially, one can measure the DIL’s participation (e.g. hosted meetups, thought leadership articles, speaking engagements, social media, sponsorships) as a vehicle for brand promotion.
Of course, at the end of the day most if not all DILs have to justify their existence to the budget gods. Hopefully, if you’re tracking one or more of the metrics above, it’ll help protect your DIL from the budget axe.
That said, if you have to defend spend, it’ll be critical to tie the outcomes from the DIL to relevant business value, and frame that value in terms that multiple business leaders can embrace. Finance will see value in a cost-neutral DIL. Business units that care about employees will see value in a lift to the work experience. Business units measured on cost or revenue will need to see the DIL effects on those metrics. Business units that touch the customer will need to see improvements in the customer experience as well as operational improvements to their own workflows.
In other words, if an DIL’s existence depends on buy-in from multiple business units, measure not just breadth and reach of the DIL, but also the number of defenders the DIL has in its corner vs. objectors.
It’ll be critical for any DIL to build a network of champions throughout an organization to protect it from changes in leadership and/or the economy. Your VP of Design or CX might defend the DIL today… but the next VP might not.
Sadly, the design lab story I mentioned at the start of the article didn’t have a happy ending. While the DIL proved its benefit by designing a better employee experience, it didn’t give project stakeholders enough juice to build a strong case for business value. Although the DIL process was amazing, it couldn’t pull the project over the finish line.
If you run an DIL, I’d love to interview you (either on the record or anonymously) — especially if you disagree with any of the ideas in this article. Alternatively, feel free to comment with a question or suggestion to improve this article. At the end of the day, I want all DILs to succeed so that design remains a vital, integral part of every large company. Hopefully, this article helps your DIL think through success metrics and identify those that matter most.