Phoenix Falling or Phoenix Rising?
News on the Government of Canada central payroll system using PeopleSoft ERP, increasingly ironically called the “Phoenix Pay System“, is discouraging. The CBC reports that 228,000 cases remained at the end of July. Thousands of new cases were added in the month resulting in net reduction of 18,000. The government had committed in June last year to have problems resolved by the end of October. My twitter feed shows significant frustration among public servants. Many wonder why the problem can’t be fixed.
This ERP implementation has become highly politicized. Communications from the government is vague. The political press is unaware of technology and lack the ability to ask the right questions. The public doesn’t know what the problem is, or problems are.
I think that it’s high time for the government to explain the underlying technology problem or problems. Otherwise, it leads to endless speculation. The lack of transparency leads us to assume that there was no competence whatsoever in the managing of the Phoenix project by the Government of Canada. I’ve met many Canadian public servants in information technology who have excellent project management skill, and technology knowledge. ERP projects and complicated with discouraging success rates in government. There are patterns that have emerged from other government ERP failures: late delivery, over budget, few anticipated benefits – some projects abandoned. It’s wishful thinking that ministerial oversight will raise phoenix magically from the ashes.
So, I’m going to speculate on what could be the problem based on patterns in similar circumstances, covering these aspects:
- Incentives
- Quality
- Project Management
- Best practices
- Training
- ERP Technology
- Replacement
- Scalability
I will conclude by contemplating why anyone should care about this.
1. When Incentives Conspire for GovTech Failure
Government project team were given bonuses on completion of milestones. This is a toxic output incentive. It encourages gaming the output by public servants on one hand, and creates a pressure mechanism for sign off by providers. It creates an environment where the definition of success can be compromised. This can lead implementation teams to delivery scope and quality they they define as successful, but users do not.
Providers can leverage the mandate of on-time delivery to deliver with lower quality or less scope. Very large vendors know how to leverage contracts. They have legal departments to comb through contract details. It is my experience that there are always gaps between Request for Proposal requirements and government needs. Perhaps scope in functions that have been generating problems was not well-articulated. There can be a significant gap between the legal contractual target set for the provider and the real need.
I suspect that the incentives given to the government implementation team implies that management understood that the project was risky. It appears to me that the procurement approach was to create failure deniability by sole-sourcing PeopleSoft, the leading HR ERP package, and ensuring that only large systems integration firms could meet vendor qualifications. ‘No one ever got fired buying…’ way of thinking.
2. Is Phoenix Tormented with Bugs?
It’s unclear how 89,000 cases were handled in June. My sense is that these were handled by workarounds in the system and by financial transactions outside the system. Are there bugs in the system that is generating problems? If so, is it from incorrect business rules, or is it in certain functions of the system? The addition of 71,000 new cases in a month implies that any rule or validations bugs that are at fault have not been fixed.
Many systems integration firms blame customers for incomplete communications during requirements gathering. That seems inconceivable in this case. Government of Canada pay rules aren’t a mystery. The rules are complicated but well-understood. Many people employed by government know these rules – as do many in the private sector. For example, many government departments and agencies have used FreeBalance software for salary budgeting, and quality check on the legacy payroll system.
3. Projects Under Stress become Greek Tragedies
There is a typical management response to projects under stress – more oversight. Management oversight introduces more meetings, and more reports. It disrupts the project flow. It increases the likelihood of failure, like a Greek tragedy, by attempting to avoid failure. It’s inevitable that Oedipus will kill his father, and his ERP project will not deliver on time.
Project management under stress often results in following the judgment of least functional and technical qualified on the team: the most senior person. The voice of technically and functionally knowledgeable staff is often drowned in oversight, paperwork, and committees.
The tragedy of committees is that they report to more committees. Decisions are made increasingly by those who lack knowledge and context.
There is a typical response by committees: add more people, work longer hours. More people are added too late in the project. This staff never gets up to speed, constantly interrupting productivity. Meanwhile, working more hours compounds quality issues, introduces new bugs, cripples the project.
4. When “Best Practices” are the Worst
Government procurement is often predicated on the notion of predictable timelines aligned with the budget cycle. Typical government contracts for Enterprise Software have defined milestones based on previous projects that used different technologies. Payments occur at these defined milestones. Any project where the milestones are not strictly defined is seen as risky. That means that agile methodologies are avoided in complex IT procurement by most governments. Gaps between requirements and real needs go through change orders. Change orders cost money. Extra costs are avoided. This results in far higher costs in the long run because of misalignment with needs.
Predictability is also associated with documentation and oversight. Systems integration firms spend significant amount of time writing gap analysis, specifications, test cases, acceptance plans to where success is often measured by the number of trees destroyed and the number of meetings attended. This compromises project success. This is not the say documentation should be avoided. It’s that documentation as ceremony often inhibits success. For example, the documentation process often uses the legacy system as an anchor rather than the replacement. So, teams spend more time to determine and document how the new software will function like the software that is being replaced. The more optimal approach is to identify the problems that the legacy software solved and determining whether the replacement software solves these problems. It often turns out that the new software has more elegant ways to solve problems.
Government often specify the makeup of the system integration project team. This includes the number of years of experience with the chosen product, or in the HR domain. This may not have included understanding payroll in the Government of Canada, or governments in general. And, governments often specify the number of people required to be provided by the integration firm. These project teams can become unwieldy, requiring operating in silos with frequent context meeting among groups. More is rarely better in software implementation.
5. The Training Conundrum
Government spokespeople have indicated that a lack of training was one of the root causes of the problem. The government took over training from the systems integrator, IBM. This is not an unreasonable approach when government staff have more hands-on experience with payroll. But, many projects suffer when this tactic is used to cut costs. Training costs can appear to be horribly expensive in a large proposal. It’s a budget line item that can stick out like a sore thumb – easy to cut to reduce costs.
Sufficient training is required to achieve maximum utility from any enterprise software. Staff from the systems integrator often know the new product much better, and training by government staff can be sub par. “Train the trainer” schemes can result in the blind leading the blind. And, enterprise software training also assumes a minimum of domain knowledge that might not exist (even though clients insist that this knowledge exists). It’s often difficult for public servants to abstract generic training from a business context into lessons for government.
Is the lack of training a root cause? How does the lack of training result in invalid calculations including some public servants receiving no pay? There should be validation on data entry and exception reports that indicate that people are not getting paid. Budget reports should also show any gap between expected pay and actual. It appears that the lack of training is a contributor to hundreds of thousands of cases but not the root cause.
6. When Legacy Technology is Replaced by Legacy ERP
There is a common misunderstanding about the Phoenix project. Most observers believe that the system was “developed” by IBM. The Government of Canada sole-sourced PeopleSoft. It’s an official Treasury Board pronouncement. IBM became the systems integration firm after a competitive process. The problem is that the government is attempting to eliminate technical debt caused by legacy software by acquiring legacy ERP. This means that falling into “technical bankruptcy” has been delayed. But the doomsday IT clock is ticking.
It does little good to replace ancient systems with what the Gartner Group calls “legacy ERP“, that requires more customization than more modern enterprise applications. Governments often require more customizations because of unique legislative regulations. These customizations introduce new code. The more changes in code, the higher the possibility of quality problems. More bugs. Loss of data integrity.
The quality problem with highly customized ERP systems compound over time. Changes to legislation introduces more customization. Customization makes product upgrades difficult because the orphan code needs to be examined in detail to see what needs to change. If customizations are a root cause here, the problems are only going to get worse.
7. Replacement and Portfolio Myth
Improved integration and reduced total costs are typical value propositions for many large scale ERP projects. Yet, many government projects designed to replace multiple systems with a single system results in significant cost overruns. There could be change management issues in these circumstances. My sense is that IT decision-makers often fail to understand why different systems were added in the first place. There can be significant needed functionality in these systems that increases the customization footprint.
It makes sense that centralized IT functions should achieve positive economies of scale than many individual systems. It is reasonable to expect that the training and maintenance of a single integrated solution ought to be less than that of multiple solutions operating in different ways with different technologies. This is not always the case in practice. First, ERP systems tend to be more complex. It is our experience that more people may be required to operate and maintain a large ERP system than a collection of custom-built and COTS applications. Second, many ERP solutions are not unified. This comes from many acquisitions made by ERP vendors so that there are many technologies and code basis extant in these solutions. ERP vendors deploy monolithic versions of modules that use the same technology and code base. This requires metadata management and integration strategies when the same software is used because the definition of something in one module differs from others. The long-term total cost can be higher with a single legacy ERP than a portfolio of applications.
There is also something fundamentally different about government-wide ERP systems. There are negative returns that occur in many government organization when smaller departments and agencies grapple with highly complex standardized processes designed for much larger departments. It can also introduce more workload for departments and agencies with legitimately unique processes. For example, a government entity dealing with national security issues has very complex hiring criteria.
8. When Scaling Doesn’t Scale
Large payroll systems have scaling concerns. That’s because any payroll run requires running through thousands of rules. There were over 258,000 federal public servants in the Government of Canada in April 2016. This does not include the number of retired public servants who are receiving pensions. Therefore, it is somewhat important to ensure system scaling for peak payroll activities. PeopleSoft is built on proprietary technology. Scaling PeopleSoft likely requires some proprietary method of load balancing for batch processes. This method might not be elegant compared to some of the techniques available today. But, it ought to scale unless, like with some Shared Services Canada implementations, there is insufficient computing resources provided.
Why is this Important?
Phoenix is one of many large Government of Canada IT projects that have failed to meet expectations, along with the single web content management project, the Shared Services Canada e-mail project, and others This is public money that has been wasted. It appears these implementations will cost the government more than if the status quo was maintained. This does not bode well for any future large IT projects in the Government of Canada.
Some observers think public servants are over-entitled. One person responded to me it twitter that there ought to be far fewer employed in Canada. His point seemed to be that public servants losing homes, unable to pay for medicines or education had it coming to them. There seems to be a lack of empathy outside the National Capital Region. This notion of unproductive nameless bureaucrats is a myth. It is true that Canadian public servants have good pension plans, but many make less than private sector counterparts.
My observation is that public servants have gone through downsizing, rightsizing, shared services and outsourcing waves to where it would be hard to distinguish the quality and efficiency of work between the public and private sectors. Sure, there are officious bureaucrats in government, but many in the private sector as well. Businesses like Blackberry and Blockbuster made critical blunders.
There is a far higher ratio of mission-driven staff in government than the private sector. Most of the public servants that I’ve met are there to make a positive different. That’s something rarely reported. It should be.