Overview
We were the first product in the construction space to offer an AI estimation generation capability leveraging Large Language Models. The feature received high exclamation and was adopted by over 50% of our US cohort. It further opened the doors for exclusive partnerships with The Home Depot in the US.
My Role
Lead Designer
Duration
June 2024 - Sept 2024
Team
1 Product Manger
4 Engineers
The Opportunity
Estimating is a manual and labour intensive process. Can AI help expedite estimate creation?
Builders typically take about 3-5 days to create an estimate, but often they end up not winning the job. Many factors determine whether a builders wins a job or not, and speed is one of them. The faster a builder can send out an estimate, the higher their likelihood of winning the job.
At Buildxact, we wanted to explore if AI, can help our builder customers create an estimate faster.
Key Criteria: Usable Generation
We recognised right from the start that that the quality of the output generation and Buildxact's integration with suppliers are tehe key to driving feature adoption among users. To achieve this we took the following approach:
Explore generation capabilities via the native model interfaces for various models.
Validate accuracy through our in-house estimating experts
Mocking a prototype to get early feedback on utility and usage
Researching architectural models to improve accuracy
User testing the beta release
Framework for generation quality
I worked extensively with the engineering team to define the principles of how we'd want the generation to work. My own research in this fields informed some of the the directives.
The cutting edge nature of this field mean't that the entire team owned the research and development. Through knowledge sharing, we arrived at what we considered the best approach for implementing this feature.
Concept exploration and purple UI
I decided to depart from the brand colour to adopt a UI colour scheme more apt for an AI interface. Not sure why products do that, but one of the arguments was to make the feature standout. The rest of the UI was fairly conventional of an AI generation product, so I quickly prototyped and tested those with a small cohort of customers for usability.
We didn't encounter any major usability issue, and participants offered overwhelmingly positive inclinations for such a tool, but were clearly concerned about the output quality. Since as a team we were highly confident on the output quality, we decided to go ahead with the development.
1
Compounding with power features
We leveraged Takeoffs and Calculator which were existing power features resulting in compounded value of the feature.
2
Building trust through partners
Ensuring prices, and measurements are accurate by leveraging our partner ecosystem for generations
3
Giving control and flexbility
Allowing the user to delete generations, add them directly to an estimate or build an assembly to save for later.
For a new customer, the AI estimator was avaialble only post connection with The Home Depot to ensure accurate pricing and materials
Release & Impact
Our US MQL to Subscription conversion shot up from 9% to 28% for the month following the release, a 211% increase
Based on average cost of item bundles added ($1.18k), $200K+ in THD item value had been added to Estimate Costings via the AI tool in the first week post full release.
Adoption was over 60% for the THD connected customers
The release broke some numbers. Marketing claimed that the release email had the best open ratio of any emails they've sent out to date and saw some of the highest adoption of any prior features released.
68% of generations were added to quote indicating high quality
We measured the quality of the generations based on the number of actual usages of generated items. The top user added 32 bundles of items to their costings. The next highest added 29 bundles of items. 167 bundles of items were added to Estimate Costings. (+26% week on week increase)
Key Learnings
AI can be useful, but many business are failing to implement real quality of life enhancement for their customers. Here are my top learnings from the successful rollout of this project.
Be relevant, push back on gimmicks
We actively avoided solving for a non-existent pain point, which was hard given the business was keen on shipping AI features.
Stability mandates staying current
Rapid changes in the model and technology impact the stability of the build. The whole team needs to be vigilant and and current on updates.
Involve cross functional partners early
There is an explosion in generic AI generation, relevancy needs verticalized expertise that involve cross functional partners. By partnering into The Home Depot and industry experts we have built a moat against any competition.
Quality of LLM generation is the key for User adoption
One of the feedback we received was, "I use AI for roughly 50% of my estimates now. I've previously used CHatGPT but, the results are nowhere as good".
Conclusion
The high US conversion, indicated strong demand for AI features, setting the roadmap for the feature launch in Australia. We further signed exclusive partnership agreements with The Home Depot in the US ( The details of which will hopefully be made public soon) .
