Back to TopBack to Top

Fixing a Government ERP Fail: When Politics and Technology Clash


August 8, 2017

Phoenix Falling or Phoenix Rising?

News on the Government of Canada central payroll system using PeopleSoft ERP, increasingly ironically called the “Phoenix Pay System“, is discouraging.  The CBC reports that 228,000 cases remained at the end of July. Thousands of new cases were added in the month resulting in net reduction of 18,000. The government had committed in June last year to have problems resolved by the end of October. My twitter feed shows significant frustration among public servants. Many wonder why the problem can’t be fixed.

This ERP implementation has become highly politicized. Communications from the government is vague. The political press is unaware of technology and lack the ability to ask the right questions. The public doesn’t know what the problem is, or problems are.

I think that it’s high time for the government to explain the underlying technology problem or problems. Otherwise, it leads to endless speculation. The lack of transparency leads us to assume that there was no competence whatsoever in the managing of the Phoenix project by the Government of Canada. I’ve met many Canadian public servants in information technology who have excellent project management skill, and technology knowledge. ERP projects and complicated with discouraging success rates in government. There are patterns that have emerged from other government ERP failures: late delivery, over budget, few anticipated benefits – some projects abandoned. It’s wishful thinking that ministerial oversight will raise phoenix magically from the ashes.

So, I’m going to speculate on what could be the problem based on patterns in similar circumstances, covering these aspects:

  1. Incentives
  2. Quality
  3. Project Management
  4. Best practices
  5. Training
  6. ERP Technology
  7. Replacement
  8. Scalability

I will conclude by contemplating why anyone should care about this.

1. When Incentives Conspire for GovTech Failure

Government project team were given bonuses on completion of milestones. This is a toxic output incentive. It encourages gaming the output by public servants on one hand, and creates a pressure mechanism for sign off by providers. It creates an environment where the definition of success can be compromised. This can lead implementation teams to delivery scope and quality they they define as successful, but users do not.

Providers can leverage the mandate of on-time delivery to deliver with lower quality or less scope. Very large vendors know how to leverage contracts. They have legal departments to comb through contract details. It is my experience that there are always gaps between Request for Proposal requirements and government needs. Perhaps scope in functions that have been generating problems was not well-articulated. There can be a significant gap between the legal contractual target set for the provider and the real need.

I suspect that the incentives given to the government implementation team implies that management understood that the project was risky. It appears to me that the procurement approach was to create failure deniability by sole-sourcing PeopleSoft, the leading HR ERP package, and ensuring that only large systems integration firms could meet vendor qualifications. ‘No one ever got fired buying…’ way of thinking.

2. Is Phoenix Tormented with Bugs?

It’s unclear how 89,000 cases were handled in June. My sense is that these were handled by workarounds in the system and by financial transactions outside the system. Are there bugs in the system that is generating problems? If so, is it from incorrect business rules, or is it in certain functions of the system? The addition of 71,000 new cases in a month implies that any rule or validations bugs that are at fault have not been fixed.

Many systems integration firms blame customers for incomplete communications during requirements gathering. That seems inconceivable in this case. Government of Canada pay rules aren’t a mystery. The rules are complicated but well-understood. Many people employed by government know these rules – as do many in the private sector. For example, many government departments and agencies have used FreeBalance software for salary budgeting, and quality check on the legacy payroll system.

3. Projects Under Stress become Greek Tragedies

There is a typical management response to projects under stress – more oversight. Management oversight introduces more meetings, and more reports. It disrupts the project flow. It increases the likelihood of failure, like a Greek tragedy, by attempting to avoid failure. It’s inevitable that Oedipus will kill his father, and his ERP project will not deliver on time.

Project management under stress often results in following the judgment of least functional and technical qualified on the team: the most senior person. The voice of technically and functionally knowledgeable staff is often drowned in oversight, paperwork, and committees.

The tragedy of committees is that they report to more committees. Decisions are made increasingly by those who lack knowledge and context.

There is a typical response by committees: add more people, work longer hours. More people are added too late in the project. This staff never gets up to speed, constantly interrupting productivity. Meanwhile, working more hours compounds quality issues, introduces new bugs, cripples the project.

4. When “Best Practices” are the Worst

Government procurement is often predicated on the notion of predictable timelines aligned with the budget cycle. Typical government contracts for Enterprise Software have defined milestones based on previous projects that used different technologies. Payments occur at these defined milestones. Any project where the milestones are not strictly defined is seen as risky. That means that agile methodologies are avoided in complex IT procurement by most governments. Gaps between requirements and real needs go through change orders. Change orders cost money. Extra costs are avoided. This results in far higher costs in the long run because of misalignment with needs.

Predictability is also associated with documentation and oversight. Systems integration firms spend significant amount of time writing gap analysis, specifications, test cases, acceptance plans to where success is often measured by the number of trees destroyed and the number of meetings attended. This compromises project success. This is not the say documentation should be avoided. It’s that documentation as ceremony often inhibits success. For example, the documentation process often uses the legacy system as an anchor rather than the replacement. So, teams spend more time to determine and document how the new software will function like the software that is being replaced. The more optimal approach is to identify the problems that the legacy software solved and determining whether the replacement software solves these problems. It often turns out that the new software has more elegant ways to solve problems.

Government often specify the makeup of the system integration project team. This includes the number of years of experience with the chosen product, or in the HR domain. This may not have included understanding payroll in the Government of Canada, or governments in general. And, governments often specify the number of people required to be provided by the integration firm. These project teams can become unwieldy, requiring operating in silos with frequent context meeting among groups. More is rarely better in software implementation.

5. The Training Conundrum

Government spokespeople have indicated that a lack of training was one of the root causes of the problem. The government took over training from the systems integrator, IBM. This is not an unreasonable approach when government staff have more hands-on experience with payroll. But, many projects suffer when this tactic is used to cut costs. Training costs can appear to be horribly expensive in a large proposal. It’s a budget line item that can stick out like a sore thumb – easy to cut to reduce costs.

Sufficient training is required to achieve maximum utility from any enterprise software. Staff from the systems integrator often know the new product much better, and training by government staff can be sub par. “Train the trainer” schemes can result in the blind leading the blind. And, enterprise software training also assumes a minimum of domain knowledge that might not exist (even though clients insist that this knowledge exists). It’s often difficult for public servants to abstract generic training from a business context into lessons for government.

Is the lack of training a root cause? How does the lack of training result in invalid calculations including some public servants receiving no pay? There should be validation on data entry and exception reports that indicate that people are not getting paid. Budget reports should also show any gap between expected pay and actual. It appears that the lack of training is a contributor to hundreds of thousands of cases but not the root cause.

6. When Legacy Technology is Replaced by Legacy ERP

There is a common misunderstanding about the Phoenix project. Most observers believe that the system was “developed” by IBM. The Government of Canada sole-sourced PeopleSoft. It’s an official Treasury Board pronouncement. IBM became the systems integration firm after a competitive process. The problem is that the government is attempting to eliminate technical debt caused by legacy software by acquiring legacy ERP. This means that falling into “technical bankruptcy” has been delayed. But the doomsday IT clock is ticking.

It does little good to replace ancient systems with what the Gartner Group calls “legacy ERP“, that requires more customization than more modern enterprise applications. Governments often require more customizations because of unique legislative regulations. These customizations introduce new code. The more changes in code, the higher the possibility of quality problems. More bugs. Loss of data integrity.

The quality problem with highly customized ERP systems compound over time. Changes to legislation introduces more customization. Customization makes product upgrades difficult because the orphan code needs to be examined in detail to see what needs to change. If customizations are a root cause here, the problems are only going to get worse.

7. Replacement and Portfolio Myth

Improved integration and  reduced total costs are typical value propositions for many large scale ERP projects. Yet, many government projects designed to replace multiple systems with a single system results in significant cost overruns. There could be change management issues in these circumstances. My sense is that IT decision-makers often fail to understand why different systems were added in the first place. There can be significant needed functionality in these systems that increases the customization footprint.

It makes sense that centralized IT functions should achieve positive economies of scale than many individual systems. It is reasonable to expect that the training and maintenance of a single integrated solution ought to be less than that of multiple solutions operating in different ways with different technologies. This is not always the case in practice. First, ERP systems tend to be more complex. It is our experience that more people may be required to operate and maintain a large ERP system than a collection of custom-built and COTS applications. Second, many ERP solutions are not unified. This comes from many acquisitions made by ERP vendors so that there are many technologies and code basis extant in these solutions. ERP vendors deploy monolithic versions of modules that use the same technology and code base. This requires metadata management and integration strategies when the same software is used because the definition of something in one module differs from others. The long-term total cost can be higher with a single legacy ERP than a portfolio of applications.

There is also something fundamentally different about government-wide ERP systems. There are negative returns that occur in many government organization when smaller departments and agencies grapple with highly complex standardized processes designed for much larger departments. It can also introduce more workload for departments and agencies with legitimately unique processes. For example, a government entity dealing with national security issues has very complex hiring criteria.

8. When Scaling Doesn’t Scale

Large payroll systems have scaling concerns. That’s because any payroll run requires running through thousands of rules. There were over 258,000 federal public servants in the Government of Canada in April 2016. This does not include the number of retired public servants who are receiving pensions. Therefore, it is somewhat important to ensure system scaling for peak payroll activities. PeopleSoft is built on proprietary technology. Scaling PeopleSoft likely requires some proprietary method of load balancing for batch processes. This method might not be elegant compared to some of the techniques available today. But, it ought to scale unless, like with some Shared Services Canada implementations, there is insufficient computing resources provided.

Why is this Important?

Phoenix is one of many large Government of Canada IT projects that have failed to meet expectations, along with the single web content management project, the Shared Services Canada e-mail project, and others  This is public money that has been wasted. It appears these implementations will cost the government more than if the status quo was maintained. This does not bode well for any future large IT projects in the Government of Canada.

Some observers think public servants are over-entitled. One person responded to me it twitter that there ought to be far fewer employed in Canada. His point seemed to be that public servants losing homes, unable to pay for medicines or education had it coming to them. There seems to be a lack of empathy outside the National Capital Region. This notion of unproductive nameless bureaucrats is a myth. It is true that Canadian public servants have good pension plans, but many make less than private sector counterparts.

My observation is that public servants have gone through downsizing, rightsizing, shared services and outsourcing waves to where it would be hard to distinguish the quality and efficiency of work between the public and private sectors. Sure, there are officious bureaucrats in government, but many in the private sector as well. Businesses like Blackberry and Blockbuster made critical blunders.

There is a far higher ratio of mission-driven staff in government than the private sector. Most of the public servants that I’ve met are there to make a positive different. That’s something rarely reported. It should be.


The following two tabs change content below.
Doug Hadden

Doug Hadden

Executive Vice President, Innovation at FreeBalance
Doug is responsible for identifying new global markets, new technologies and trends, and new and enhanced internal processes. Doug leads a cross-functional international team that is responsible for developing product prototypes and innovative go-to-market strategies.

6 Responses to “Fixing a Government ERP Fail: When Politics and Technology Clash”

  1. David P says:

    Many “cases” are in fact routine HR/Compensation transactions – for example, Helen gets promoted; Frank retires; or Josee is given acting pay for two weeks while her boss is on vacation. All of those events create a “case”. Many are easy to resolve, but still require manual intervention in the system.

    Unfortunately, the system appears to have been built to mimic the workflow of the old systems, rather than designed to automate it. Thus, many transactions still require extensive manual calculations, increasing the risk of error. For example, in some instances,compensation staff accidentally invent new pay scales that do not exist and begin paying people at those rates – or mistakenly cut people’s pay when they are promoted. (A large factor in this is the inexperience of the compensation staff). These generate more work.

    The government let go hundreds of compensation staff as part of the “savings” Phoenix was to bring – and those timelines were announced in advance. So many fled those jobs before the deadline, increasing the size of the backlog before transition to the new system. Then most of the jobs were put in Miramichi, New Brunswick. Not because of any appreciable pool of talented HR and compensation experts, but because the old federal employer in town closed up shop, and this was a way to channel federal jobs back into the area.

    • Doug Hadden Doug Hadden says:

      The odd thing is that the previous legacy system had similar problems. Government departments and agencies used “salary management” software from Canadian vendors, FreeBalance or GX. Part of the purpose of the software was to model and predict salary budgets. These systems re-forecast expected budget after every pay period. These applications also identified errors: promotions, retirements, transfers to other departments and would issue Journal Vouchers for adjusting entries. A part of the proposed cost savings for Phoenix, in addition to getting rid of departmental pay advisors, was to stop using the software from Canadian vendors and save maintenance fees.

  2. Patrice Boivin says:

    When you refer to “Peoplesoft”, what do you mean exactly? Peoplesoft is still running and hasn’t been replaced by Phoenix. Do you mean that the old product was Oracle Financials , or some other vendor’s product? These systems are complex (ref. ) and probably the Achilles Heel of these projects is coaxing organizations to evolve, like when eBusiness started prior to Y2K and existing businesses refused to restructure.

    If it is indeed an Oracle product which was implemented, why would someone ask IBM to implement Oracle software? (ref. )

    • Doug Hadden Doug Hadden says:

      Some explanation is required. First, many government departments were running PeopleSoft for HR or to help in the creation of payroll. Systems sent payroll information to the Standard Pay System, a custom-developed application. There were often issues with this system. (Many departments used a FreeBalance product to trap errors, but the notion was that the new system would not have these errors.) At any rate, the policy of the Government of Canada was to replace the custom-developed system with an ERP. It was decided (no competition) to select PeopleSoft as the payroll system of record. PeopleSoft was acquired by Oracle. Phoenix is basically PeopleSoft that has been customized to Government of Canada requirements by the winning systems integration bidder, IBM. IBM, and any other bidder had no choice but to implement PeopleSoft. It is very unusual for ERP manufacturers like Oracle to bid on complex RFPs like this.

  3. David P says:

    Phoenix is the Government of Canada implementation of PeopleSoft Payroll North America. It has been customized beyond all recognition, and there was no adequate testing done in advance of roll out.

    I am also hearing that “closed” means different things to those reporting on the status that it would to Joe Q Public.

    There are issues where they’re doing part of the work, declaring “We’ll get to the rest later”, but calling them “closed” in order to boost their numbers.

    The actual status of what’s going on is not being accurately presented to the public.

  4. David P says:

    One other headshaking moment: With the new collective agreements being implemented, lots of employees were logging in to Phoenix to see what their retroactive pay would be.

    This put so much demand on the IT infrastructure that the system became almost unusable – so the pay clerks (I refuse to call them compensation advisors) couldn’t do their job.

    The solution? Bless those bureaucrats. Apparently scaling the servers to handle the load was out of the question – so instead, the IT wallahs disabled employee access to pay information – in other words, employees can no longer use the new and expensive pay system to do things like, you know, look at their pay information.

    The layers of failure keep getting stacked one on top of the other.

Leave a Reply