Loading...
HomeMy WebLinkAbout04/07/2009, B4 - SCADA SYSTEM UPGRADE ASSESSMENT REPORT council - - oa j agenda Report ��N -;SL CITY OF SAN LUIS OBISPO FROM: Carrie Mattingly, Utilities Director Prepared By: Gary W. Henderson, Water Division Manager -4`11�A SUBJECT: SCADA SYSTEM UPGRADE ASSESSMENT REPORT RECOMMENDATION Receive a presentation on the Supervisory Control and Data Acquisition(SCADA) System Upgrade Assessment Report for Utilities Department operations prepared by DLT&V Systems Engineering DISCUSSION The City of San Luis Obispo Utilities Department utilizes numerous remote monitoring systems for the overall operations of our water and wastewater systems. Similar systems are used by most agencies to monitor and control system operations, provide discrete alarms for system failures, identify unusual events and collect data for system reporting requirements and optimization improvements. These monitoring systems are most commonly referred to as Supervisory Control and Data Acquisition Systems (SCADA) but have also been referred to as "Telemetry Systems". The primary function of any SCADA system is to alert system operators of potential and existing system problems to prevent hazardous sewage spills and assure water delivery to citizens. In addition, the systems provide operations data necessary for evaluating operational efficiency improvements (energy usage, chemical usage, etc.) to meet ongoing water quality regulations for both water and wastewater systems. In December of 2007, the City entered into a contract with DLT&V Systems Engineering to evaluate the existing, aged SCADA systems utilized by the Utilities Department to insure reliable and efficient operations of the water and wastewater systems. Much of the City's existing Water Distribution and Wastewater SCADA systems are over twenty years old and are no longer supported by the manufacturers. The City of San Luis Obispo has five distinct SCADA systems in the following areas: Water Treatment Plant, Whale Rock Reservoir, Water Reclamation Facility, Wastewater Collection, and Water Distribution. The two treatment plant systems mainly utilize fiber or other fixed hardwire for communication purposes, the wastewater collection and water distribution systems utilize radio links and the Whale Rock system uses leased phone lines. Most of the systems utilize Bristol Babcock Programmable Logic Controls (PLC's) which are similar to small computers and are programmed to control the equipment and send information and alarms to system data concentrators and other controllers. There are a few sites that utilize other PLC manufacturers, while others use multiple different models of Bristol Babcock equipment. The following sections provide a brief overview of each of the Utilities Department systems and recommendations for future upgrades where appropriate. J \ f i SCADA System Upgrade Assessment Report Page 2 Water Treatment Plant System As part of the 2008 Water Treatment Plant Upgrade Project, the existing SCADA system at the plant was upgraded. This included modification of the onsite communication system between the PLC's, upgrades to the existing PLC's, update of the existing software programs, and programming of the computer control screens to include the new plant process system operations and oversight. Since this facility's system has been recently upgraded, DLT&V's report did not identify any high priority projects. There were two recommended upgrades that should be considered in a future plant upgrade project. One involves adding a level transducer and radio link to provide the operators with water level information for the "Forebay". The Forebay is an in-ground reservoir that is utilized to provide a small amount of onsite raw water storage for improved control of water deliveries from Whale Rock and Salinas Reservoirs which are then routed through the water treatment plant processes. The second improvement would be to provide a wireless communication system to allow water treatment plant operators to monitor plant operations and receive alarms when they are working in various areas throughout the plant site. These improvements are not being recommended at this time although they will be included in future upgrade projects(Forebay) or in future operations budgets (wireless communication). Whale Roek.System The Whale Rock SCADA system is about twenty years old and was installed and programmed by in-house staff. This system monitors and controls the two pump stations along the 17 mile pipeline from Whale Rock Reservoir to the City's Water Treatment Plant. The pipeline also has turnouts along the way that provide water deliveries to the Cayucos Water Treatment Plant (County), the Dairy Creek Golf Course, the California Men's Colony, and raw water for Cal Poly agricultural operations. The existing Whale Rock system is utilizing a single DOS based 386 MHz computer and communicates to the remote sites via dedicated leased telephone lines. While this system has performed well for many years, the system is very old and will not be able to be maintained in the future due to outdated hardware and software systems. Several SCADA failures have recently occurred at the main site as well as the remote pump stations. The leased lines are still recommended as the preferred communication method due to the unsuitable terrain for wireless communication, but are recommended to be upgraded from voice grade to DSL to get the best performance for the new system. The consultant recommends that all of the PLC's on this system be replaced and reprogrammed and the computer system be upgraded to a Windows based system. This system is recommended as a high priority for replacement. Since it is a small relatively simple system it is recommended for the first upgrade implementation to test the hardware and software prior to rolling out to larger systems. Water Reclamation Facility The Water Reclamation Facility (WRF) SCADA system was originally installed in the early 1990's during upgrade projects at the site. This system allows operators to monitor and control SCADA System Upgrade Assessment Report Page 3 plant operations and processes. There have been several additions to the system over the years including additional PLC's for the Water Reuse. Project. The communication system at the facility site primarily utilizes hardwire between the PLC's and the plant computer monitoring/control system. There are also a few radio link communication systems that were added with the Water Reuse Project. At this time the SCADA system is relatively reliable overall, but some modifications should be addressed in the near future for upgrades to the software system to improve data collection and alarm report generation. Currently, there are no discrete plant alarms generated by the monitoring system. All alarms, major and minor, are sent to the operators as"Plant Alarm". The operators must then access the control system to determine the type and importance of the alarm. After hours, the on-call operator has to travel to the plant to assess the problem. Discrete alarms would enable the operators to remotely assess the problem and respond if necessary, or simply clear the alarm as appropriate. There is also the desire to provide a wireless communication system to allow water reclamation facility operators to monitor plant operations and receive alarms when they are working throughout the plant site. These improvements are recommended for consideration in the future upgrade projects currently under evaluation and study for the Water Reclamation Facility. Wastewater Collection System The Wastewater Collection system was installed and programmed in-house in the late 80's and early 90's. This system provides for monitoring and alarm notification for all of the sewer lift stations throughout the city. The communication between the sewer lift stations and the computer monitoring system at the Corp Yard uses a licensed radio system that includes a repeater site at Cuesta Peak. The system only provides general alarm information which requires operations staff to drive to the site to determine the specific problem and determine if corrective measures are needed. This system is currently reliable and performing well, but it has limited alarming capabilities and the software and computer systems should be upgraded in the near future to maintain system reliability. In addition, when these upgrades are undertaken the consultant recommends that the communication system be replaced with unlicensed spread spectrum radios which would utilize a repeater on the City-owned site at South Hill. This will eliminate the licensing fee for the current radios and standardize the Wastewater Collection and Water Distribution systems on the same radio which will reduce the costs for spare parts, it will also eliminate the lease fee the City now pays to SLO County for equipment space at Cuesta Peak.. This system is not recommended for immediate replacement but will be included in a future Financial Plan when deemed necessary by staff. Water Distribution System The Water Distribution System was also installed and programmed in-house and, along with the Whale Rock Reservoir system, is outdated and reaching the end of its service life. This system has minimal operations oversight and data collection and no alarm capability. 84 -3 SCADA System Upgrade Assessment Report Page 4 This mission critical system currently monitors water storage tank levels and provides this information to the Corp Yard for water distribution operators use as well as to the Water Treatment Plant operators. There is no control capability or data tracking for the pump stations or critical control valves in the City water system. Most of the pump stations and the associated storage tanks operate based on pressure switches at the pump station or tank site: A few of the pump stations communicate via radios and the PLC's at the pump stations control the pump operations based on the water level in the tank. At the Corp Yard there is a single DOS based computer that displays some of the tank water levels. Not all of the tank sites are displayed on the system. With this little monitoring and data acquisition, and no control or alarming capability, it brings up many concerns that should be addressed sooner rather than later. It is recommended that all of the PLC's be updated with the latest technology and that the DOS based systems be upgraded to a Windows based system. In addition, it is recommended that the licensed radio systems be replaced with an unlicensed radio system and the Cuesta Peak repeater be replaced with one at South Hill. The use of the City owned site at South Hill will eliminate the ongoing lease costs associated with the Cuesta peak location and standardize the Water Distribution and Wastewater Collection on the same radios. The Water Distribution system should be the highest priorities for the City to address after the Whale Rock system is upgraded. Currently water distribution operators find out about problems from citizens reporting they have no water, or that a water storage tank is overflowing, an unacceptable situation. Summary The SCADA System Upgrade Assessment Report evaluated all five separate systems used for operations for the Water Treatment Plant, Whale Rock Reservoir, Water Reclamation Facility, Wastewater Collection System, and Water Distribution System. While the report identifies recommended improvements for all five systems, the most pressing needs were identified for the Whale Rock and Water Distribution systems. These systems are out-dated and are posing reliability questions due to the age and lack of available replacement parts in the event of a failure of a component. In addition, installing monitoring, data acquisition and alarm capabilities at critical control valves that provide water into the "downtown"pressure zone will minimize the potential for multiple water mains breaks that have been experienced.several times in recent years. It is recommended that these two systems receive the highest priority in the near future. CONCURRENCES The Finance and Information Technology Department has provided a significant role in the oversight and development of this report and concurs with the recommendations and phasing strategies. FISCAL IMPACT There is no immediate fiscal impact associated with this report. Preliminary budgetary cost estimates for implementation and programming (excluding design costs) for the recommended improvements in each of the five system areas are shown in Table 1 below. Staff will prepare a CIP request as part of the 2009/11 Financial Plan for the recommended improvements for Whale l7 �' � SCADA System Upgrade Assessment Report Page 5 Rock and Water Distribution system upgrades for Council consideration. Improvements in the other areas will be included in future Financial Plan CIP requests when deemed appropriate. Table ]: Preliminary Cost Estimates Section Construction& Programming. Whale Rock $350,000 Water Distribution $1,500,000 Wastewater Collection $600,000 Water Reclamation Facility 5450,000 Water Treatment Plant $11,000 Council Reading File: SCADA System Upgrade Assessment Report, November 2008,prepared by DLT&V Systems Engineering