These are edited excerpts from Brent Loewen’s presentation at the ITFMA Financial World of Information Technology Conference July 2011
One of the best practices for implementing new financial management systems, such as chargeback and others, is to run a pilot before full implementation. This article is about why, when and how to pilot.
Reasons to Pilot
Why would we want to pilot our chosen solution? There are various reasons, and some or all of them may apply for a given situation:
- Identifies issues prior to full-scale implementation
- Reduces disruption to the normal work environment
- Reduces risk of problems and failure
- Provides for smoother final implementation
- Uncovers unidentified issues
- Assists in obtaining buy-in
- Confirms expected results and relationships
- Experiments with alternative solutions
When should we consider piloting? There are a number of factors that drive the timing of piloting a solution, and these are some of the major situations:
- When change implementation costs are high
- When it is essential to reduce failure risk
- When the solution needs to be verified prior to implementation
- When the scope of the change is large
- When reversing the change would be impossible or impractical
Selecting a Piloting Strategy
Depending on time, resource and cost restrictions, there are a number of piloting strategies available. If there is a limited time window to complete the project, a fixed length of time should be set for the pilot. In cases of limited scope or resources, you can pilot at certain sites only or limit it to certain teams or projects. If the project is only a partial solution to a larger project, implement parts of the proposed solution independently. If a real-life simulation is required, perform the pilot in an "off-line” environment to simulate the real world. If a computer simulation is being considered, do this when you have predictable processes and product inputs that can be programmed into complex models.
Planning the Pilot
You will need to plan your pilot test with the same rigor as a full implementation. The only differences are scale and complexity. Be sure to document who is doing what by when. The following are typical pilot test plan components:
- Timelines and responsibilities
- New procedures, updated process maps, work instructions
- Communication requires identifying who is affected and letting them know
- Measurement plan in order to know the pilot has been successful
- Back-ups and contingency for problem prevention purposes
- Evaluation of both failure and success
During the Pilot
Consider the process operation during the pilot and the work environment. Is the pilot process running as expected? Note performance issues, which requires being there and watching it in operation. Is the output correct and as expected? If not, what defects are being produced? Is the area a suitable working environment?
Evaluating the Pilot
After the pilot test, you need to evaluate the methods and results against the project’s objective. Review your results by using appropriate statistical tools and tests. Ask the questions, "Did you follow the methods you defined? Did you get the results you wanted?” You may need to return to your solution’s ideas and pick alternative methods or you may need to make some changes based on what you learned. If the team followed the plans and the results meet the project objective, you’re ready for full-scale implementation.
Before implementing the solution, think about the things that could go wrong. What could go wrong with this new step? What are the causes? How can we prevent the causes? Think about how you would overcome resistance to your proposal. Document your problem prevention measures in your solution/plan.
Once you have successfully completed the pilot, you will be ready to begin full-scale Implementation. The tools for implementing your proven solution are the same as those used for the pilot. What is different is the scale and complexity. There are more variables, activities and people involved. Your key task here is to get everything up and running under real-world conditions and to fix any problems. Detailed and robust planning is essential!