How Do You Measure the Success of New Technology Implementations?
CIO Grid
How Do You Measure the Success of New Technology Implementations?
In the rapidly evolving world of tech, measuring the success of new implementations is crucial. We've gathered insights from Chief Information Officers and other tech leaders, distilling their wisdom into six key strategies. From gauging success across multiple metrics to resolving client issues with new technology, discover how these experts quantify tech triumphs.
- Gauge Success Across Multiple Metrics
- Track Key Metrics and Define Goals
- Assess the 'Noise Factor'
- Consider Holistic Business Impact
- Evaluate with Comprehensive Business Alignment
- Resolve Client Issues with New Technology
Gauge Success Across Multiple Metrics
The success of a new technology implementation can be measured in terms of Performance, Financials, User Adoption & Satisfaction, Productivity, and Business impacts. It all depends on the goals of the implementation. Generally speaking, User Adoption & Satisfaction and Business Impact are the true measures as they reflect the "net" outcome of an initiative.
These can be measured in various ways depending on the initiative:
User Adoption & Satisfaction: Qualified through these metrics:
- Adoption Rate & Frequency of Use
- Onboarding Experience
- Issues/Errors Reported
- Net Promoter Score (NPS)
Business Impact: Qualified through these metrics:
- Productivity Gains & Operational Efficiency Increase
- Financial Performance
- Compliance & Risk Degree
- Innovation and Competitive Advantage
One key technology initiative undertaken at Acro was the Microsoft Great Plains ERP System implementation. This involved moving from a decades-old system to a modern system with add-on modules designed for our staffing industry. As with most ERP implementations, this initiative involved substantial people, process, and technology changes across Acro.
The primary goals were:
- Replace the old system
- Digitally transform the Accounts Payable function
- Improve efficiency of functions like Time Tracking, Payroll, AR, GL, Billing, etc.
- Reduce manual and offline processes
This implementation took us a lot longer than originally planned! But the implementation was still considered a success by most people.
So, what were the measures of success for this technology implementation? We can evaluate it in terms of the aspects listed earlier:
Performance: System response/speed, and uptime were great, with little to no errors.
Financials: We were able to operate with fewer people and support complex clients. True ROI could not be measured until much later, though.
User Adoption & Satisfaction: Within four weeks, all users were fully functional in the new system and comfortable with the processes.
Productivity and Business impacts: Over the next two years, we supported clients with $150+ million in revenue, and profits increased at a higher rate than revenue.
All in all, if user satisfaction with the new system is high within a few months of implementation, the project should be considered a success. In the medium to long term, financial results would reflect productivity increases, positive business impact, and the ability to handle more complex processes and customer requirements without proportional cost increases.
Track Key Metrics and Define Goals
Measuring the success of a new technology implementation is all about tracking key metrics. Also, clear goals need to be set beforehand, and what 'success' looks like must be defined, whether it's improving efficiency, increasing sales, or enhancing user experience. We implemented a new project management tool to streamline our workflows. Before rolling it out, we identified key performance indicators (KPIs) like project completion times, team productivity, and overall satisfaction. We tracked these metrics before and after the implementation. We saw a 20% reduction in project completion times and a noticeable boost in team collaboration. The tool's integration features made it easier for different departments to communicate, which was a game-changer for us.
Assess the 'Noise Factor'
There are dozens of different success criteria for a new technology implementation that can't all be crammed into a paragraph. One that perhaps is the clearest representation of all those criteria is what I'll call the 'noise factor.' Think of a 9-box grid with 'pre-implementation, during, and post-implementation' along the left-hand side. Then along the top, you have 'negative, neutral, and positive.' Imagine filling out that 9-box with evidential noise that was generated in each of the phases, and the tone of that noise. For example, in the during/negative box, you might find something that says, 'The team voiced displeasure about working several weekends in a row, including the July 4th holiday.'
When I think about the most successful implementations I've been a part of, for example, a new HR, Payroll, and Benefits system for a large nautical organization, there would be largely positive commentary throughout the various phases, and negative noise would be managed and minor. Again, the overall noise factor would tie directly back to overall success criteria, such as duration, effort, cost savings, error reduction, etc.
Consider Holistic Business Impact
The introduction of a new technology to the business is often difficult to quantify in terms of its ultimate impact and usefulness. Rarely does the implementation of a new technology have no unintended consequences, both good and bad. To accurately measure the success of a new technology, you must look at it holistically in terms of the business—did the technology ultimately result in a net addition to business value?
A good example of this is the application of generative AI. Yes, there are some obvious positive results, such as time savings. However, there are unintended negative results as well, such as the risk of exposing intellectual property to the public, and in some cases, inaccurate information. Only a holistic view considering all the positive and negative results can measure the success of new technology implementations.
Evaluate with Comprehensive Business Alignment
The optimal method for evaluating the success of a technology strategy incorporates a comprehensive approach that is in harmony with business goals and objectives. Critical metrics encompass return on investment (ROI), user satisfaction and adoption rates, and operational efficiency.
Resolve Client Issues with New Technology
We measure the success of new technology implementation by its ability to resolve client issues effectively.
A local company approached us about a problem it was having. They were having issues with voids/cavities appearing in a sealant product they manufactured. At first, it wasn’t clear what type of test would help us determine whether voids were appearing. We quickly realized that we were going to have to do some experimentation to determine whether we could find an appropriate test.
We inspected the bags of sealant using digital 'Real-Time' Radiography. This new technology would enable us to capture radiographic images of the sealant and give us a benchmark from which to work. We then used a syringe to simulate voids within the sealant, using the real-time feed to prove that detection was possible. The Real-Time Radiography did indeed pick up the cavities in the bags. This proved that this form of radiographic testing is a viable option for this client. The client will now be able to apply this technology to their manufacturing process to manage and hopefully reduce the number of cavities appearing in their products.
