Case | HBS Case Collection | September 1998 (Revised May 2002)
Cisco Systems, Inc.: Implementing ERP
by Robert D. Austin, Richard L. Nolan and Mark J. Cotteleer
Reviews Cisco System's approach to implementing Oracle's Enterprise Resource Planning (ERP) software product. This case chronologically reviews the diverse, critical success factors and obstacles facing Cisco during its implementation. Cisco faced the need for information systems replacement based on its significant growth potential and its reliance on failing legacy systems. The discussion focuses on where management was particularly savvy in contrast to where it was the beneficiary of good fortune.
Keywords: Information Technology; Software; Technology Adoption; Complexity; Information Management;
Don't fix it if it's not broken - especially if you are talking about an extensive enterprise software foundation, right? Not necessarily.
In the case of the Oracle software environment, Cisco management teams decided that it was time to upgrade even though the Oracle 10.7 software was still getting the job done. IT, operations, and executive teams all concurred that this vital component of the company's IT infrastructure needed to be modernized. While there would be clear benefits from an upgrade to Oracle 11i, the main decision drivers were the need to move away from an outdated platform that was becoming cost-prohibitive for supporting Cisco's growth and evolving business requirements. In particular, any future development of enterprise solutions would be compromised if developed on a less-than-current Oracle platform.
"Remaining competitive on a global scale often comes down to our ability to efficiently respond to changing operations, to work with new partners and suppliers, and to make sure that our employees have fast access to the latest productivity-enhancing technology and decision-making tools. Having an integrated, modern platform allows us to more rapidly respond to changing business requirements and to improve partner productivity and customer satisfaction," says David Murray, director for release management, Oracle Projects, Cisco IT. "We knew that an upgrade to Oracle 11i would be a massive undertaking, but we also knew we had to build out the foundation to enable future capabilities."
Oracle applications have supported Cisco operations since 1995 and are used by every functional area. The initial installation and a subsequent upgrade carried out for Y2K provided executive and IT teams with points of reference to size the magnitude of this project and the level of risk mitigation required. In particular, it was recognized that the company had grown significantly since the last upgrade, and the business processes were appreciably more complex. Thousands of employees relied on the Oracle platform and tools to perform their jobs. Hundreds of applications and boundary systems were dependent on this platform, adding complexity to the upgrade.
“After all the rehearsals and contingency planning, the actual upgrade went like a dream. It was like getting a corporate heart transplant and seeing the patient up and walking around right after it was done. The success of the project resulted in recognition from our CEO at the company all-hands meeting, and winning the company’s teamwork award.”
Director for Release Management, Oracle Projects, Cisco IT manager
One of the priorities for manufacturing was to support their move to a more outsourced model. The challenge for this functional area was to minimize the investment required for the upgrade and to fully engage all of the manufacturing business teams and supply chain partners. For the finance team, the upgrade to Oracle 11i had to be carefully defined to support all finance-related business initiatives such as the globalization and automation of processes, and the reduction of customizations by moving to off-the-shelf and standard functionality.
Other success factors for the upgrade were based on growing recognition of the needs for:
- Increased collaboration across Cisco functions.
- External collaboration with suppliers, distributors, partners, and customers.
- Support for the move to an electronic business model.
During the timeframe of the upgrade to Oracle 11i, a primary driver for Cisco—increasing company productivity—greatly influenced the planning efforts. A strategy to transition Cisco to a process-focused enterprise emerged, and modernizing the Oracle foundation was soon recognized as a key prerequisite and enabler for making the shift. This effectively raised the project from a cross-functional task to a companywide priority. The new Oracle 11i enterprise architecture would serve as a critical foundation for the overall business, introducing common technology and business process platforms, and providing enhanced capabilities across functional areas.
Companywide Dedication and Priority Status
Cisco’s Oracle 11i upgrade was classified as a company initiative, ensuring that all the functions prioritized this upgrade with the necessary resources and funding. Expertise was brought in as needed, and resource contentions were quickly resolved since project priorities were clearly defined in advance. The upgrade project was able to garner the required resources at the required times, sometimes moving people in and out of the project for short-term engagements.
Rigorous Program Management
An effective governance structure served as a vital component of program management. Given the multiple internal and external groups ultimately affected by the upgrade, IT established a cross-functional governance structure spanning from the executive level to the project management level for all affected business areas. In parallel, a more rigorous release management process was put in place for the Oracle 11i program. Four teams were established to address the critical components of the process:
- Steering committee - This multifunctional team met monthly to provide governance and guidance for the program management team, and to proactively address program risks and resource issues.
- Program management team - IT provided dedicated, full-time staff to manage the development and deployment work. This team also provided day-to-day management of the release management and business flow teams.
- Release management team - This team coordinated the process of planning and implementing rehearsals and "go-live" events.
- Business flow team - This team was made up of representatives from the affected groups in each of the critical Oracle-supported business flows, who voiced the business requirements for the move to Oracle 11i with an emphasis on the user perspectives.
A designated business lead from each business process area represented their users’ requirements and maintained communications between the business area and the PMO. Each business lead’s responsibilities included overall project governance for that functional team, as well as serving as executive liaison with their respective steering committee representative.
With teams in place, the PMO established processes for traditional project management tasks—analysis, development, and testing. Increased focus was given to managing the transition to the new system and its impact to Cisco's business, partners, and customers:
- Transition readiness track - This track focused planning, rehearsal, and risk mitigation of all business and IT activities related to performing the upgrade and getting the new system ready for business use.
- Business readiness track - This track focused on understanding the impact of the upgrade to the business not only in terms of the changes and processes, but in terms of the impact of the three-day downtime on Cisco's operations, partners, and customers. Business readiness was divided into sub-tracks that focused on organizational adoption, communications, training, and contingency planning.
"Teamwork was an important success factor," says Murray. "Everyone from management to individual contributors recognized that we succeeded or failed together."
Software Change Management
For almost 10 years, Cisco IT managed software changes using a model based on segregation of duties—the person writing code did not place it into production, but rather submitted a request to the production team. Segregation of duties and an automated workflow protected the staff and business from many malicious or unintentional problems. Very early during the Oracle 11i project, IT recognized that the existing software configuration management model would not suffice. Oracle 11i required an unprecedented amount of change for Cisco, and between 400 and 500 developers would be programming the changes. To meet these challenges, Cisco IT drove the adoption of a new model and solution for configuration change management. For complete details on this vital component of the Oracle 11i project, read the Cisco IT case study, "Cisco IT Balances Innovation and Risk with Change Management Process" at h//www.cisco.com/web/about/ciscoitatwork/business_of_it/software_configuration_management.html.
In order to implement the change, Cisco Manufacturing would have to coordinate with numerous subcontracted vendors in approximately 20 locations worldwide. A great deal of time was spent keeping the teams informed and making sure they understood and were prepared for all of the changes.
Ahead-of-time communications were not enough. The actual transition to the new environment was rehearsed multiple times. Three end-to-end tests were carried out, each performed over the days of the week that would be used for the actual cutover. The business impact necessitated that the test be run around the clock. This called for a full commitment from the IT, finance, and manufacturing teams, as well as subcontractors, to support a 7x24 schedule. Hardware and software vendors were involved early on, and participated in the test runs. “Before the first readiness test, we defined the thousands of steps required for the upgrade and put a system in place to track those steps and their dependencies. As a backup, we defined points of contact and escalation processes for each team,” says Murray. “The tests greatly reduced our risks and helped us optimize the process. As a result, we shortened the time required for the cutover.”
In addition to the readiness tests, the teams conducted walkthroughs, including simulations of potential problems. The walkthroughs involved predetermined roles and scripts for the operational teams to follow. By simulating problems, the teams were ready for many scenarios and became accustomed to working together in a crisis situation.
The New Architecture
The move to Oracle 11i introduced radical architectural changes throughout the Cisco infrastructure. The main changes included:
Three-tier architecture - Prior to Oracle 11i, users accessing applications on a data center server required client software loaded onto their desktops. In the new Oracle 11i environment, clients can access the applications using Web browsers. This thin-client, three-tier model provides users with a new, friendlier interface.
Storage area network (SAN) - Storage requirements increased radically with the upgrade, especially during development and rollout testing when other projects were looking for copies of applications and data sets. Combined storage for design, development, staging, testing, training, and production added up to hundreds of terabytes. To connect servers and storage, two Cisco MDS 9000 Series multilayer director switches were introduced to replace unwieldy, smaller switches. The scalable, director-class switches provide 640 ports and allow scalable, flexible SAN configurations (Figure 1). Key features of the storage architectures, before and after the upgrade, are summarized in Table 1.
Load balancing - The Cisco Content Switch Module (CSM) optimizes Oracle 11i performance relating to forms, reports, and Java applications. Traffic is redirected automatically and dynamically, balancing the load in real time among servers, firewalls, and other devices on the network. As processing demands increase, more servers can be installed without the need to replace the systems that are already in place. As a plug-in module for Cisco Catalyst® 6500 Series switches, the CSM combines Layer 2–4 services and Layer 4–7 functions within a single, high-performance platform (Table 2).
Disaster recovery - Redundancy is built in at many levels. The main corporate database engine is hosted in a San Jose, California data center, using two servers in an active-passive configuration. A second site, in North Carolina, acts as a disaster recovery site in the event of a site failure at the main data center.
Internet access for partners - Changes to router-based access control lists (ACLs) allow external partners and suppliers to be given access to various Oracle 11i applications. In the past, partners had access only through dedicated leased lines; now partners and suppliers have increased access to Oracle applications over the Internet. Security is maintained for corporate data assets while enhancing collaboration and e-commerce abilities for the company.
|Storage Architecture||Before Upgrade||After Upgrade|
|Production frame||Sym 8430, SCSI disk drives, 16Gb Cache, 96 drives||DMX 1000 Performance Model, fibre disk drives, 64 GB Cache, 144 drives, 4 to 5x performance gain|
|Redo logs||Reside on the production frame itself without any redundancy||Two DMX 800 frames for High-Availability. Oracle Multiplexing for redo logs|
|Backup frame||Symmetrix 8430, SCSI drives, 16-GB Cache||DMX 1000 with backup disks raid protected, 4x performance gain|
|Storage Switches||Cisco Andiamo Switches||Cisco Andiamo Switches|
|Fibre Channel||1 Gbps||2 Gbps|
|Data volume LUN Size||8.4 GB without striping||34 GB Meta volume with striping|
|Archive data||Separate database and Separate storage frame||Reside on production database itself. Archive data alone could be on a separate, less-expensive frame|
|Disk backup||Dual backup (AM and PM) but disks are Unprotected||Dual backup (AM and PM) with Raid 3+1 Protected disks|
|RAS Features||Before Upgrade||After Upgrade|
|Network||Redundant Switches||Redundant Switches|
|CSM||None||Redundant Cisco CSM’s|
|Application server||Single Node with failover||Multiple Nodes with Load Balancing. Scalable Architecture|
|Application load balancing||None||Apache, Forms, and Reports Load balanced using Cisco CSM|
- Oracle 8i Parallel Server
- Oracle 8i Parallel Server
|Third-party applications||Server Cluster for High-Availability||Server Cluster for High-Availability|
|OID (LDAP) for authentication||None||High-Availability test is in progress|
|Policy server||None||Multiple Nodes with Load Balancing|
Cisco successfully upgraded its entire finance and manufacturing environment to Oracle 11i over a three-day holiday weekend at the end of 2003.
This monumental milestone was not marked with performance problems, corrupt data, or frustrated partners, but with a smooth, uneventful implementation. “After all the rehearsals and contingency planning, the actual upgrade went like a dream,” says Murray. “It was like getting a corporate heart transplant and seeing the patient up and walking around right after it was done. The success of the project resulted in recognition from our CEO at the company all-hands meeting, and winning the company’s teamwork award.”
The project team received companywide praise, and the executives noted the phenomenal success of this high-priority company initiative. John Chambers, president and CEO of Cisco, remarked, “Well done! I realize how important this was to us and how risky. The first time we put in our ERP system, we were naive and didn’t understand what it could have done to us. The Oracle 11i installation was flawless. In fact, they told me I wouldn’t be able to get our finance numbers over the weekend—but I could get my numbers every day except one. It was just an unbelievably good coordination effort, and it speaks to how we can drive business process across the company through IT implementation and teamwork.”
In the end, multiple factors contributed to the success of the implementation. Teamwork, management support, release planning, proactive communications, and company priority all played key roles in the smooth transition. Given the challenges and risks of implementing new technology and new functions, the success was especially significant.
The team learned the importance of defining strong governance and metrics management for program health, overall readiness, and solution stability. Inclusion of multiple end-to-end test cycles with increasingly strict exit criteria ensured the appropriate level of “go-live” quality. Multiple reviews focused the teams on business and systems readiness from early on in the project. Regular executive and project reviews were conducted to manage risk and deliverable status.
Software configuration best practices management (change control) was applied to reduce the velocity of change as the go-live date approached. Murray explains, “The closer you get, the more you must mitigate risk—it’s better to go ahead with known issues rather than making too many last-minute changes that could introduce risks and unknowns.” Before going live, the executive and business teams approved known issues, business risks, and workarounds.
For information on the technical support move to and rollout of new Oracle-related tools, see the related Cisco IT case study “ERP Technical Support” at //www.cisco.com/web/about/ciscoitatwork/business_of_it/erp_technical_support.html
Figure 2. Cisco CSM Delivers High Performance, Availability, and Scalability to Web-based Oracle 11i Applications.
Click on image to enlarge
The Cisco MDS 9000 Series enabled the consolidation of switches for the main data center and for the disaster recovery center. Today, custom scripts are used to synchronize the data centers at the application level. In the future, IT would like to introduce synchronization at the storage level, and the team is investigating how to apply Cisco MDS features for this purpose.
Content caching is also under investigation as a method for improving responsiveness of Web-based ERP applications. By locally storing the most commonly used information—images and forms—remote offices and sites can gain performance improvements by minimizing WAN delays. A caching solution also minimizes the bandwidth requirements, since less information is transmitted over the network. The Cisco Application and Content Network System (ACNS) caching engine is being deployed globally to benefit the new Oracle 11i implementation. Recent testing indicates that 50 to 90 percent of Oracle 11i objects can be effectively cached.
The Linux operating environment allows cost-effective Intel servers to replace more expensive servers. Cisco has already migrated some applications to the Linux platform, and will continue this trend for manufacturing and technical support applications.