The URL can be used to link to this page

Home My WebLink About11/16/1999, 5 - YEAR 2000 PREPAREDNESS UPDATE J council M°,°� aGEnba REpoRt lem N..hc CITY OF SAN LUIS O B I S P O FROM: Bob Neumann,Fire Chief Bill Statler,Director of Finance Teri Maa, Information Systems Manager–*k SUBJECT: YEAR 2000 PREPAREDNESS UPDATE CAO RECOMMENDATION ■ Review and discuss the status of the City's preparedness for Year 2000 and the recently prepared MIS Disaster Recovery Plan. ■ Appropriate $15,000 from the unreserved General Fund balance for the purchase of a back-up file server. REPORT-IN-BRIEF Year 2000 (Y2k) Readiness. The City has been working diligently over the past several years to ensure that our community and our organization are "Y2k" ready. This effort has both an external focus in working with community groups; and an internal focus in ensuring that the City's technology systems are Y2k compliant. As we approach January 1, 2000, the purpose of this report is to provide the Council with an update on the status of our Y2k readiness efforts. Overall, we believe our efforts to educate and prepare the community for Y2k have been successful; and that the City's technology systems— especially our"mission-critical"ones—are Y2k ready. Disaster Planning and Recovery. As part of our Y2k planning efforts, we have completed a broader information systems (MIS) disaster recovery plan,which will have lasting value to us in planning and recovering from any disaster,not just Y2k. In general, all the resources we believe are reasonable in recovering our`mission-critical"technology systems are already in place. This is largely due to our past efforts in planning for data recovery. It is also due to the system standardization (in hardware as well as system and application software) that we have achieved over the past several years, which provides us with a high-level of back-up in our day-today systems and operations. In short, if one system or application should fail in one location, we have similar resources elsewhere that can be transferred from a lower to higher priority area. The only notable exception is our recommendation to purchase a back-up file server at an estimated cost of $15,000. We will install this in our training room, where it can be fully configured and"hot-swapped"with any other server in the City if needed. There are two other areas of potential vulnerability that we are currently evaluating where we do not have automated back-up systems in place in the event of hardware failure: 5-1 Council Agenda Report—Year 2000 Preparedness Update Page 2. B Public safety application file server (Hewlett Packard 9000), which supports records management and emergency dispatch for our police.and fire operations. ® 911 communications system. . While recovery plans are in place at this time for these two systems (manual operations in the case of failure by the public safety system file server; and transfer of 911 calls. to the County Emergency Operations Center in the case of failure in our 9.11 communications system), we are currently reviewing other options. Depending on the results of these evaluations, we may return to the Council in the:future for additional back=up and recovery resources for these two systems. DISCUSSION Background f The City's efforts to ensure that the community and the organization are "Y2k ready" span several years. While not specifically driven by Y2k concerns, adoption of our Information Technology Master Plan in 1995—which provided for the replacement or upgrade for virtually all of our "rni'ssion-critical" applications was our first step among many in becoming Y2k ready. Our formal Y2k efforts began in March 1998 when we formed an organization-wide "Y2k Task Force"with representatives from all City departments. Their first task was to prepare an initial inventory and assessment of our technology systems. This wasfollowed by contracting with Information Integration Innovation & Associates for expert advice on Y2k compliance strategies. With their assistance, we prepared a formal compliance assessment, which we presented to the Council in September 1998. This led topreparing.a formal Y2k master plan as well as a broader MIS disaster prevention and recovery plan. As noted at that time, we do not see preparing for Y2k as significantly different from planning for recovery from any other type of disaster. In short, while Y2k was the stimulus, we used this as an opportunity to develop recovery procedures and plans that will continue to serve us long after January 2000. In February of 1999; we presented the Council with a comprehensive update on our Y2k readiness efforts. Based on our broaderr look at disaster recovery needs, the Council approved funding in the amount of$207,000 for additional portable generators,pumps, radios, advertising and emergency supplies. The Council also approved the Y2k master plan at that time. Attached is a brief summary from our Web site.on our Y2k readiness strategy and the status of our efforts to-date. Status Report on Our Community Awareness Efforts Since February of 1999, staff has been very active in educating the general public on not only what we are doing internally, but how they should prepare both their businesses and homes for potential Y2k impacts. The following highlights the results of our efforts and"what's left to do." 5=2 Council Agenda Report—Year 2000 Preparedness Update Page 3 What Have We Done Since February 1999? ■ The Fire Chief has been a primary stakeholder in the San Luis Obispo County Y2k Action Alliance. This group met every Tuesday at our headquarters fire station until just recently. The Alliance is made up of public and private interests from all over San Luis Obispo County. Our accomplishments include developing a web page, an Y2k hotline, a pamphlet and a speaker's forum as well as coordinating a series of disaster preparedness "Expos" throughout the county. The Alliance also coordinated a media campaign that was funded through a cooperative agreement between the County and a number of cities (including San Luis Obispo). ■ Produced, specific to the City, an Y2k pamphlet that was distributed through the City's utility billing system. ■ Updated and augmented our existing disaster supply caches and added two new caches at the water treatment plant and the wastewater reclamation facility. All major City facilities are now adequately equipped with disaster caches. ■ Sponsored two community awareness fairs at the Veteran's Hall. Our focus at these events was two-fold. We hosted a speaker's forum and developed an area for vendors to display disaster preparedness supplies and equipment. In addition, we took this opportunity to showcase the Fire Department's CERT program. ■ Ordered new pumps and generators to service our sewer and water systems. The pump is in service and delivery of generators is expected in late November. While we had hoped for earlier delivery, it was delayed by Hurricane Floyd. ■ Established employee standby lists for the last week in December and the first two weeks in January to assure that key staff members are available to oversee critical business functions. ■ Augmented our fuel supplies and delivery systems and developed an agreement with a private contractor for emergency fuel supplies. ■ Staffed a speaker's forum with the Fire Chief, the Utilities Director and the Information Systems Manager, who have spoken to a wide variety of groups on Y2k and possible impacts. What Do We Have Left to Do? ■ As requested by the CAO, the Fire Chief and Information Systems Manager will be hosting a brown bag lunch to share with our employees where we, as an organization, now stand. 5-3 Council Agenda Report—Year 2000 Preparedness Update Page 4 ■ A media blitz is planned for mid-December. It will emphasize a common sense approach for general disaster preparedness. ■ As the year comes to a close, the Fire Chief, Police Chief, Utilities Director, Public Works Director and Information Systems Manager will host a meeting with key staff members to review all critical functions and systems to ensure that we are as prepared as possible from an emergency response perspective. Status Report on City's Technology Systems What Have We Done Since February 1999? Over the past eight months, we have been focused on implementing our "remediation plans" (primarily replacing or upgrading non-compliant hardware and software) and making sure that the"remediation"solutions actually work by testing them. Highlights of these efforts include: ■ Replaced five of our six wide area network(WAN) servers with Y2k ready equipment. ■ Installed Y2k compliant Novell operating system software. ■ Installed Y2k compliant backup software products. ■ Replaced Pacific Bell leased lines with City owned fiber, thus eliminating our dependence on a third party for network connectivity. ■ Converted Fire's record management system from a non-compliant FoxPro database application to the public safety Spillman system. ■ Upgraded our Radix water meter reading software to be Y2k compliant. ■ Upgraded the city's fleet maintenance program software to be Y2k compliant. ■ Installed Y2k compliant chip sets in our fire alarm system. ■ Implemented hardware and firmware upgrades to bring three of our four heating, cooling and ventilation (HVAC) control systems into Y2k compliance (the fourth system was already compliant). ■ Changed-out traffic management equipment and software in our parking garages to be Y2k compliant. ■ Replaced hand-held ticket generator hardware and software to be Y2k compliant. ■ Installed an Y2k compliant chip in the oxidation-reduction potential unit at the Water Reclamation Facility. ■ Completed conversion of the accounting, payroll, budget, cash receipts, accounts receivable and business tax modules to the Pentamation financial management system. E Upgraded GroupWise,the City's office automation software(such email and calendaring) to the latest Y2k compliant version. 5-4 Council Agenda Report—Year 2000.Preparedness Update Page 5 What Do We Have Left to Do? As summarized below, we have several focused work efforts remaining. However, these efforts are well underway and mostly represent "fine-tuning." As such, they will easily be completed before January 2000. ■ Install the latest "firmware" on our hubs and routers (components of the City's wide area network(WAN )that manage the network). ■ Install switches at City Hall, Police and Corporation Yard to replace non-compliant hardware. ■ Roll-out Microsoft Office 2000 to every desktop. ■ Apply Microsoft Windows 95 Y2k update patch on every desktop. ■ Replace selected workstations to ensure Y2k compliancy. ■ Complete conversion of our utility billing system. ■ Complete conversion of Sigma (Human Resources applicant tracking system) to the Pentamation financial management system. ■ Finalize remaining remediation efforts for the public safety system applications. ■ Apply Unix Y2k update patch to the Pentamation system. What Is the Status of Our Testing Plans? As noted above,just replacing hardware and software to versions the manufacturer says are Y2k compliant is not enough. We need to test them and make sure they actually work. This is especially important in the case of our"mission-critical"applications. Because of this, preparing and implementing test plans is an integral part of our Y2k master plan. In some cases, testing is not feasible as it would be too disruptive or is not technically possible to do. Our testing plans reflected this where applicable, and contingency planning has taken place in these cases. The following highlights the status and results of our testing plans: N Y2k testing of the financial management system was successfully completed in October 1999. All modules correctly rolled-over from December 31, 1999 to January 1, 2000. Leap year calculations generated the correct results. ■ Utilities carefully reviewed their equipment at the water reclamation facility, water treatment plant and Whale Rock reservoir for Y2k vulnerabilities. Equipment at these locations fell into three categories: non-critical to plant operation; verified by the manufacturer to have no date sensitive components; or upgrades were required. All required upgrades have been made and testing is completed. 5-5 Council Agenda Report—Year 2000 Preparedness Update Page 6 ■ Public Works has completed their testing which included traffic signals, HVAC systems, burglar and alarm systems,and management software programs. ■ Final testing of the public safety system is still underway. Overall Assessment City staff has extensively prepared for the Year 2000. Based upon manufacturer statements and our own product testing, we believe the City is "Year 2000 Ready." Technically, this means that our electronics are capable of accurately processing, providing, and/or receiving date data from, into and between the twentieth and the twenty-first centuries, and the years 1999 and 2000, including leap year calculations. MIS Disaster Recovery Plan As discussed above, virtually all the procedures and resources recommended in the plan are already in place. The only exception is the purchase of a back-up file server, which we are recommending at this time. It is important to stress that the plan takes a "reasonable" approach in striking a balance between the costs of preventing and responding to all possible disasters, and the likelihood that an adverse event will occur (and the consequences if it does). In short, implementing the plan will not provide absolute assurance that all technology systems will avoid disruption or failure in the event of a disaster, or that recovery will be simple, cheap and quick when disruptions do occur. However, it does set forth reasonable plans and procedures for back- up and recovery in the event our technology systems are adversely affected by natural or human disasters. While many of the recommended procedures and plans were already in place before preparing the plan,there were three major benefits resulting from this effort: ■ Clearly stating our data back-up and recovery procedures. ■ Identifying and prioritizing our "critical business processes" as they relate to our technology systems, and how we will allocate resources in the event of natural or human disasters. Attached is Appendix C of the plan that sets forth these priorities. ■ Preparing our MIS recovery plans in accordance with the City's overall strategy for disaster preparedness, and assuring that roles and responsibilities are clearly defined. CONCURRENCES The MIS Steering Committee concurs with the recommended actions. 5-6 Council Agenda Report—Year 2000 Preparedness Update Page 7_ __ FISCAL INO--ACT Back-Up File Server. We recommend appropriating $15,000 for a back-up file server from the unreserved General Fund balance. Based oil interim results for 1998-99, adequatesesources are available to fund therecommended purchase and retain fund balance at minimum policy levels. Future Costs.. As discussed above; we may return to the Council for additional recovery resources for our public.safety system file server and 911 communications system: However, until we complete our assessment of reasonable options, it not possible to estimate future costs (if any). ATTACHMENTS ® The Millennium Bug—What's the City Doing About It? ® Information Technology Response.Priorities (Appendix C of the MIS Disaster:Recovery Plan) AVAILABLE FOR REVIEW IN THE COUNCIL OFFICE ~' a MIS.Disaster Recovery Plan ■ Year 2000 MasterPlan G:FinanceMlS PoliciesNear 2000 Project/Council Agenda Report=November 16,1999 5. city of san lues oslspo � C x The Millennium Bug—What's the City Doing About It? There has been a lot of"buzz" lately in the media about the Additionally, the Y2k problem is potentially more than just major problems in-store for any organization that is not correctly calculating dates. The Year 2000 is a leap year, and working hard right now to ensure that their technology many computer programs —even if they consider themselves systems will be Year 2000 (Y2k) compliant This report Y2k compliant—may not correctly account for this because of reviews what the "Y2k" problem is, what the City has done an obscure rule that only comes into play once every 400 to-date to address this problem and what we are doing to years. assure that our technology-driven systems will be Y2k compliant so we don't catch the millennium bug! How Big Is the Problem? What Is the Problem? There are three kinds of problems facing the City: Until pretty recently, computer memory was very limited — • Business process applications and systems. This is what and because of this, also very expensive. To save space (and we most commonly think of when considering technology money), for many years computer programmers used only the systems. It includes software and hardware used for daily last two digits of the year in writing date-related programs. business operations, such as desktop workstations,file servers, printers, operating system software, office automation For example, "1965" was programmed and entered as "65." software (like Word and Excel), and specialized applications In determining the length of time since an event last happened like geographic information systems (GIS), public safety, ,uch as a birth date and today), the "19" part of the date is finance, pavement management, work orders and building .sumed, so it is easy to calculate that 33 years have passed permits. from 1965 to 1998(98 minus 65). • Process and control systems. These are often hard to So why is this a problem? Because when the year 2000 rolls identify,because they use"embedded"software and hardware around,computers with only"65"in them will not know what for facilities and equipment, such as heating, cooling and to do. A straightforward calculation should result in a ventilation systems, telecommunication systems (telephones, negative 65 years passing since "65" (00 minus 65). At this voice mail, data communications), security systems, traffic point, it is not clear what most systems will do when they signals, process control systems at our water treatment and encounter a negative date number — other than it is unlikely wastewater reclamation plants, vehicle fuel pumps, street that an accurate answer will result! (This is why testing is an lights,radio communications and telemetry systems. integral part of most Y2k compliance plans.) • Data exchanges with other organizations. We rely on Dates are used in an overwhelming number of applications in electronic information from other agencies,and other agencies our daily lives, from computing interest earnings on savings rely on electronic information from us. Examples include accounts to determining retirement benefits;from telling when investments, banking, retirement system reporting, payroll the next inspection is due on an elevator(and it will not man if deposits, wire transfers and law enforcement information. In it is past due)to determining the freshness of food products. short, we rely upon electronic data from other agencies for a number of"mission-critical" operations,and they rely upon it So the short answer is: this is a problem because date-related from us. The problem: even if our own systems are Y2k computer calculations are everywhere in our daily lives, and compliant,this will not mean much if others we rely upon for we can be assured that serious (and in many cases, electronic interfaces are not Y2k compliant as well. unforeseen) consequences will result from hardware and software that cannot correctly deal with the millennium Business process applications and systems probably pose the change. lowest risk to the City for two reasons: bis date-calculation problem can be found in two places: • Based on the 1995 information technology master plan, embedded in the hardware itself in the built-in operating we have started replacing or upgrading virtually all of our system (BIOS); and in application and operating system "mission-critical" applications like "desktop" hardware and software. software, local and wide area network systemss and public The Millennium Bug—What's the City Doing About It? safety, finance, GIS applications. All of these efforts are the clock rolls over to January I, 2000. Among its underway and will be completed well before the Year 2000. recommendations was preparing a formal Y2k master plat and a more general disaster prevention and recovery plan. • Replacing `hon-mission critical" systems should be relatively easy and inexpensive (assuming we begin to do so • Preparing and implementing a formal Y2k master plan. on a timely basis); and the consequences of not being Year The Council approved the Y2k master plan in February 1999, 2000 compliant will be less significant. and as discussed below, almost all of the action steps have been completed as of September 1999. We are much more likely to experience serious difficulties with the 'Process control" and"data exchange" systems since impor• Identifying "critical" processes What are the most they are less visible, and in many cases,beyond our control to 2000?ant things for us to assure work correctly on January 1, � 2000? Realistically,we probably cannot make absolutely sure that every City system is Y2k compliant by then, so our What Has the City Done So Far,9 efforts need to be focused on the areas of greatest impact To do this,we identified and assessed our"critical"services. We then used the results of this effort to develop a broader disaster We are taking this very seriously, and we have adopted and prevention and recovery plan for our technology systems. started implementing an aggressive Y2k compliance action This plan identifies how we can continue to operate our most plan. Accomplishments to-date include: important services and facilities in the event of any major • Replacing "mission-critical". application software as disaster—not just in preparing for"Y21c." part of the information technology master plan adopted in . Replacing and upgrading software and hardware. This April of 1995. In getting a jump on this problem, we are is an essential part of our`remediation plan." Except in a few fortunate that we have been upgrading or replacing our cases,this was completed by September 1999. "Mission-critical"systems since 1995. While our main goal at the time we adopted the master plan was not primarily to • Testing. In conjunction with implementing our address Y2k compliance issues,this has been the result "remediation strategies,"we developed extensive testing plans to ensure that these solutions actually work- These test plan, • Forming an organization-wide "Y2k task force" with are underway,and should be completed by October 1999. representatives from every department. This was an essential first step in assuring an organization-wide approach to this • Planning for Y2k failures by others. The simple fact is problem. that we conduct a lot of our business electronically with others. And many of these agencies and organizations will not • Developing awareness We are making special efforts to be Y2k compliant, even if we are. Because of this, we have let our employees and the community know that we are taking developed contingency plans for how we will conduct this problem seriously, and are taking reasonable steps to business if some of our most important "electronic partners" address it We have included information on this in our Parks (like federal, state and county agencies) and other important & Recreation program catalog and on our web site. We are service providers (like PG&E and Pacific Bell) experience working closely with a community-based group—the Y2k Y2k compliance problems. Action Alliance—to help our community in preparing for Y2k. And we plan to send out finther "community preparation" For example, early in our assessment of risks, we identified information in our utility bills later in the Fall of 1999. possible power failures as a major Y2k service delivery problem. For this reason, we purchased four additional • Performing a detailed, organization-wide inventory of portable generators and a mechanical pump. With these all our systems This included extensive vendor research for resources—which will stand us in good stead in preparing for both hardware and software to identify potential Y2k natural disasters long after January 2000 rolls around—we are compliance problems. In any case where we were not able to confident we can continue to provide essential services even secure vendor Y2k certification from the vendor, we assumed in the event of protracted power outages. Similarly, we have that the application was Y2k vulnerable. made provisions for emergency fuel supplies should this be • Adopting a Y2k procurement policy. This ensures that necessary. all new purchases (not just technology-related ones) are Y2k For More Information About the City's Y2k Plan compliant • Completing a formal compliance assessment As reflected above, we have taken a number of steps to assur Completed in August 1998,this report concluded that the City that we are "Y2K Ready." If you have any questions about was off to a good start in getting ready for Y2k; however, it our plans,please call Bob Neumann Fire Chief, at 781-7380, noted that a lot of work remained ahead of us in assuring that or Teri Maa,Information Systems Manager,at 7817 31. ❑ our most important applications would still be working when z � z o N N N 3 � Q cc,� g o k 4 0 o 4) y Q £ % i oil oil It a y is Q4 h d o � m a1 Q V 02 Q� O Oil Ol 2 r�i A .oi r 8 L h •�.' cN 3 3 tt H' o i p 0.� Grw �L Q' Q' ►`� as U i i U as d u - ri u z c c u eio X -4 i •� � � O V "' iy u' Q y O1 4.3e cc Ta o U ori m oil ° oil 'e we c°l oil n V v O�l o > > •`• 8 O FSI � `• Ol QI Vii Oa� • • Pn a0i C4 R V a 5-10 . ■ % ee k Ind W4lim Cd CS O ■ � f _ 2 j ti - E 2 $ � \ { k \ g k « i5 � k4 « § - ■ Q. > \ q _ . i 7 - ( % / $ � . ■ a § 2 :1. k k k 2 a ■ 3 :. � . � § ■ % � 2 � � \ . - � k � ® ra to Cc % K 2 § ¥ 22 I \ # 2 ? I ■� � � � tk C%L-ft 41 § ] �� ■ § _ ; \ kt ; t « t2 § G § + ¢« ® % ` # ` ■ gym § & ` _ 2 � - k- ; � 2 MIS - �ar & :2! � � ■ - m - § - ts , , , ■ = a % § e t ■ e K § ■ o 2 2 `> ■ Cs. § k ] ` ! t - t e 2 ■ ; ; ■ ° ) , ■ _ k k ; K . ■ U , K C44 U \ w . 06 � ' § d y y c. dO z = a y � m tl it Z O r C 4) >y O Q i 41 � r h Ol C ay+ 03 > p� y w O 92 t Z A. O •a > � V Ol o sS �. m. as � 3 � L � I�r a' w a: r g a> w y Z p •� y r v vy, 9N�' a� l vi ami y u q = Q Q h s .r.. ► Q ► .q Q rEo R y a D i O d cqzLZ e � .ot i � yIs y r a ys'� a � d � v � '� Q � � v � U ,.�''•.v. q 0. Q � C � � � � � \ i �. ti 3 �.. 4 p y .�: •ytlr h -0 .:� ti 'n Gi fl O 4i N .Or Ol tl v 1 V! Ol Ol •.y. �0 .y,. -0 42 V m y Q 4 .Z4. :1. C i .rp44 y .p C •.. •.. d• t A � A � � C v o r i C p -fps- v b g •S". z U V U V vroa � IN awe a 0. 0. ..O C ++• r O u � � a r as H v y en Vj ► � M A �` .49 Q V V Oa � Q = m V V 'w i, G m - � 14) IS 0�1 tel` Orl rs ts p V r d0 WQ 'a 41 i o h t y $c �'°p °�' so c r a e ° v tm st `1 v m m m o m m v .� ,L'� d $ 'r d r •� URI. _. ' `OQ j OlOVi � 1L. � _ rO '„ � Ol CI � 9 o f o 'a m o 0 0 �, a o = o y v a o s o $ s o y j a n 4 obi 4` u P, a L e v ►r o 0. r 4 4 4 L W V V 4 a V 4 `o !7 k - Q O 2A lid cl � � 22kk% } 2 \ . ■ �4 ■ % � � ƒ\ ■ . § § ■ § � $ k \ �� 2 � - & « ® A § o k § \\ / k � k 2 \ K $ § § $ \k ■ $ � a ■ k k � � § � \ $ 2 z ■ ` ■ 2 a § & a s ■ ? I \ � ƒ CN • � 4 2 § t k 2 ® � > IL � ■ % ; ■ a q � a . 22 a 2k . } % ` ; - Q � � � § � , ° ■ aIzk &BI & $ k & & 7 « § ■ � & « E & % & § k W ■ ■ a A K82 � W4 4 . � 5-l4