November 2011, Vol. 238 No. 11
Features
Intelligent Control Room Management For Oil And Gas Pipeline Safety
Globally, mankind has more than 1,500,000 km of oil and gas pipelines – with more than 600,000 km in North America alone. These numbers are even higher when one adds pipelines transporting other hazardous gases and liquids. Though efficient and critical, these pipelines carry high costs for individual safety incidents. Around the world, pipeline safety and security are immediate concerns, both for industry and government.
In the United States, the Department of Transportation Pipeline and Hazardous Materials Safety Administration (PHMSA) published a final rule, RIN 2137–AE28: “Pipeline Safety: Control Room Management/Human Factors.” The final rule amends the federal pipeline safety regulations to address human factors and other aspects of control room management for certain pipelines where controllers use supervisory control and data acquisition (SCADA) systems – and seeks to reduce risk and improve safety during the transportation of hazardous gases and liquids. This ruling sets forth improvements to control room management that have value in the United States where mandated and also around the world as good business practices.
Utilizing electronic Standard Operating Procedures (eSOPs) to comply with regulatory mandates such as RIN 2137–AE28 Pipeline Safety 1) drives good business practices throughout the world and 2) captures and proliferates expert knowledge and proven procedures.
This article discusses how intelligent control room management, based on a foundation of electronic standard operating procedures (eSOPs), can address the regulations, document the necessary actions for compliance validation, and drive operational excellence. With intelligent control room management, operations controllers can respond correctly with the right actions at the right time, which helps to minimize risk. Additionally, managers are able to generate plans and reports to prove compliance – as well as analyze and improve processes and performance.
Pipeline Safety
Operations controllers and managers in oil and gas facilities are inundated with information – thousands of alarms, remote equipment sensor data, etc. Companies also challenged by high employee turnover and an aging workforce – new employees are handling situations critical to community and life safety. Unfortunately, many companies can only rely on their expert workers to pass knowledge down and do not have formal step-by-step standard operating procedures in place or a way to capture best practice knowledge.
The PHMSA rulings provide an opportunity to address these challenges and improve business practices globally. As background, under the final rule, pipeline operators – people or corporations who engage in the transportation of gas – must implement methods to reduce the risk associated with controller fatigue.
A controller is defined as a qualified individual who remotely monitors and controls the safety-related operations of a pipeline facility via a SCADA system from a control room, and who has operational authority and accountability for the remote operational functions of the pipeline facility.
In addition, pipeline operators must define the roles and responsibilities of controllers and provide controllers with the necessary information, training and processes to fulfill these responsibilities. Affected operators must also 1) manage alarms, 2) assure control room considerations are taken into account when changing pipeline equipment or configurations, and 3) review reportable incidents or accidents to determine whether control room actions contributed to the event.
The final rule went into effect Feb. 1, 2010, and pipeline operators must develop control room procedures by Aug. 1, 2011. Companies must implement the new plans and procedures by Feb. 1, 2012, to be compliant with the PHMSA rules.
The final rule does not enumerate specific responsibilities that must be defined, as did the proposed rule. Instead, the final rule leaves the scope of controller responsibilities to be defined by each pipeline operator, taking into consideration the characteristics of its pipeline and its methods of safely managing pipeline operation.
This final rule imposes requirements for control room management for all gas and hazardous liquid pipelines subject to 49 CFR Part 192 and Part 195, respectively, that use SCADA systems and have at least one controller and control room.
PHMSA has also revised the scope of the final rule for gas pipeline operators such that each control room whose operations are limited to either or both criteria—1) distribution with fewer than 250,000 customers or 2) gas transmission without compressor stations—must follow procedures with appropriate documentation that implement only the requirements for fatigue management, validation and compliance and deviations.
Mandated Procedural Guidelines
The PHMSA regulation includes the following categories:
1) Roles and responsibilities – This rule requires pipeline operators to have and follow written control room management procedures. The operators must define the roles and responsibilities of controllers in normal, abnormal and emergency operating situations.
2) Provide adequate information – Pipeline operators are required by this final rule to assure that new SCADA displays and displays for SCADA systems that are expanded or replaced meet the provisions of the consensus standard governing such displays, API RP 1165. Displays for gas pipelines are required to meet only some provisions of the standard. Operators are required to validate the accuracy of SCADA displays whenever field equipment is added or moved and when other changes that may affect pipeline safety are made to field equipment or SCADA displays. Pipeline operators are also required to test any backup SCADA systems and to test and verify a means to manually operate the pipeline (in the event of a SCADA failure) at least annually.
3) Fatigue mitigation – Pipeline operators must implement measures to prevent fatigue that could influence a controller’s ability to perform as needed. Operators will need to schedule their shifts in a manner that allows each controller enough off-duty time to achieve eight hours of continuous sleep. Operators must train controllers and their supervisors to recognize the effects of fatigue and implement fatigue mitigation strategies. Finally, each operator’s procedures must establish a maximum limit on the number of hours that a controller can work.
PHMSA recognizes that there may be infrequent emergencies during which an operator may find the need to deviate from the maximum limit it has established to ensure adequate coverage in the control room for emergency response. Accordingly, the regulation provides that an operator’s procedures may provide for the deviation from the maximum limit in the case of an emergency. Such a deviation would only be permitted if necessary for the safe operation of the pipeline facility. PHMSA or the head of the appropriate State agency, as the case may be, may review the reasonableness of any deviation from an operator’s maximum limit on hours of service when considering whether to take enforcement action.
4) Alarm management – SCADA alarms are a key tool for managing pipeline operations, but excessive numbers of alarms can overwhelm controllers. This final rule requires pipeline operators to develop written alarm management plans., which must include monthly reviews of data points that have been taken off scan or have had forced or manual values for extended periods. Operators will also need to verify correct alarm set points, eliminate erroneous alarms, and review their alarm management plans at least annually. They are also required to monitor the content and volume of activity being directed to their controllers (including alarms and actions directed to controllers from sources other than the SCADA system) at least annually.
5) Change management – Operators must consider the effects of future changes to the pipeline on control room operations. They must involve controllers, controller representatives, or their management in planning prior to implementing significant hydraulic or configuration changes that could affect control room operations. This participation must be accomplished with enough time prior to the implementation to allow adequate training, procedure development and review by the affected controllers. Operators must also assure good communications when field personnel are implementing physical changes to pipeline equipment or configuration.
6) Operating experience – Pipeline operators are required to review their operating experience to identify lessons that might improve control room management. Specifically, operators are required to review any reportable event and determine if control room actions contributed to the event. Operators must identify, from these reviews, aspects of the event that may reflect on controller fatigue, field equipment, operation of any relief device, procedures, SCADA system configuration, and/or SCADA system performance. Operators must include lessons learned in controller training programs.
7) Training – Pipeline operators are required to have formal training programs, including computer-based or non-computer (e.g., tabletop) simulations to train controllers to recognize and deal with abnormal events. The training must also provide controllers with a working knowledge of the pipeline system, particularly as it may affect the progression of abnormal events, and their communication responsibilities under the operator’s emergency response plans.
8) Compliance validation and deviation – Operators must, upon request of pipeline safety regulators, submit their completed control room management programs to the regulator for review. A request to review the plan will usually be in the course of a regulatory inspection where the adequacy of control room management plans and training will be reviewed, as will the operator’s compliance with each of the above-referenced requirements.
Intelligent Management Best Practices
Let us consider improving safety and addressing mandates with intelligent control room management. An intelligent control room management solution layers on top of an existing SCADA solution and guides controllers in the proper actions to take. By capturing and digitizing best practices, companies can make every controller an expert – and can build a foundation for meeting pipeline safety regulations.
An intelligent control room management solution enables pipeline operators to guide controllers through critical standard operating procedures (SOPs) and track responses for compliance records. With templates for SOPs and reporting, operators can quickly and easily add this solution to their existing SCADA system and guide operators through proper procedures and corrective action — increasing controller effectiveness, reducing costs, easing compliance and minimizing risk.
With electronic standard operating procedures, pipeline controllers can filter through complex situations and sets of circumstances, managing the data from many sources and understanding what the next steps should be. Controllers need to know the steps for proper operations, security and much more. In addition to day-to-day activities, an eSOP system can guide controllers through actions to take during uncommon events that require the correct emergency response to minimize dangerous situations and risk to pipeline operators and customers.
Implementing intelligent control room management begins with a proven methodology to identify gaps and determine next steps. The technology solution usually incorporates Work Process Management (including Alarm Response Management), Change Management and HMI/SCADA software – which together can address the recommendations of PHMSA as well as API RP 1165 (the recommended practices for SCADA displays). There is a significant opportunity to seamlessly layer such a solution into any existing pipeline operations technology infrastructure.
Meeting The Requirements
Specifically related to the PHMSA rulings, an intelligent control room management solution aids in faster implementation and development. The solution addresses the mandates in the following ways: 1) alarm management, 2) roles and responsibilities, 3) provide adequate information, 4) fatigue mitigation, 5) change management, 6) operating experience, 7) training and 8) compliance validation and deviation.
Alarm management – Each operator using a SCADA system must have a written alarm management plan to provide for effective controller response to alarms.
Operators must produce a written plan that specifies the controller response to alarms and prove that controllers follow this plan. By developing SOPs and then using an eSOP method to present the information to controllers in the Work Process Management application, the operator can track the day to day activity of controllers’ actions.
As an example, all high priority alarms can be filtered to the Work Process Management system and trigger an appropriate eSOP workflow to minimize the possibility that a controller performs an “Acknowledge All” if a flood of alarms comes in and inadvertently misses an important alarm.
The system can also generate work requests that can serve as a reminder to controllers that certain conditions exist – such as equipment that needs to be tagged and the associated data points need to be off scan, alarm inhibited or manual values entered in the SCADA system, etc. An eSOP to show all open work requests can be run at shift changes. Furthermore, an expected time of completion event can trigger a reminder to the controller to check on the status of the work being done; the ability to extend the time can be allowed with an explanation also being entered. All distribution system control actions can require a workflow to be defined.
By using eSOPs to facilitate alarm response management, the operator can determine if a controller is inadvertently causing alarms to occur because of flawed equipment manipulation. Every time a workflow is kicked off, the date, time and controller can be recorded, and pipeline operators can compare controller performance as well as the ratio of controller to alarm statistics.
Using the properties in the ISA-95 standard, each piece of equipment can have set points associated with it. The controller supervisor can be notified during the log in process when a report needs to be run to verify these values.
Additionally, the operator can monitor the content and volume of activity required of each controller, to make sure they have time to analyze and react to alarms. A task history report can be generated to show the number of tasks completed on each workstation and can include the length of time it took to perform the task. Again, the controller supervisor can be notified during the log in process when this report needs to be run.
Roles and responsibilities- Each operator must define the roles and responsibilities of a controller during normal, abnormal, and emergency operating conditions.
Using Work Process Management and the ISA-95 standard, each controller can be classified by experience. The work processes defined can take into account the level of experience of the controller on duty and his/her work-related experience. When a process is kicked off – because of an alarm or a defined action – the eSOP system can step through the process and present information according to the controller’s classification. Every task completed is time stamped and recorded for reporting purposes.
In the event of an emergency condition, a work process can be initiated, and the controller can be guided through a series of steps to determine if emergency services should be notified.
At the end of a shift, if a work process is in progress, that status should be apparent in the eSOP system. The steps completed, and yet to be accomplished, can be viewed and discussed by the controllers.
Provide adequate information- Each operator must provide its controllers with the information, tools, processes, and procedures necessary for the controllers to carry out the roles and responsibilities the operator has defined.
The operator must ensure that what the controller sees on a SCADA screen is an accurate depiction of what is happening out in the field. This requirement calls for uncluttered screens, point-to-point verification of equipment represented on the screen, testing backup plans, and passing information to the next controller.
Point-to-point verification of SCADA displays and field equipment can be accomplished with a work process. Any time a piece of equipment is taken off-line poses a safety issue, and must be tracked; each time equipment goes back online, the controller must verify that what is seen on the SCADA screen accurately represents what is in the field. Using Work Process Management and eSOPs, the verification process can be tracked and recorded.
In the event the primary SCADA server fails, the backup systems should be put into service. Since this rarely happens, switching to the backup system must be tested once a year. An SOP can be generated to guide a controller through the proper sequence to perform the switchover to the backup server.
This section again identifies the necessity to have accurate information passed between shifts by the controllers and is addressed in the Roles and Responsibilities section.
Fatigue mitigation- Each operator must implement the following methods to reduce the risk associated with controller fatigue that could inhibit a controller’s ability to carry out the roles and responsibilities the operator has defined.
The operator must establish appropriate shift lengths, education on fatigue mitigation, and decide when working over the maximum limit can cause a problem. Using the work process system, the operator can set an event to trigger if there is a period of controller inactivity. Once that trigger occurs, the controller can be prompted to respond to an eSOP to verify alertness. If the controller does not respond in a timely manner, the on-duty supervisor can receive notification.
Shift lengths and frequency can be recorded by the sign-in process of the work process system, with a supervisor notification when a controller exceeds the maximum allowed hours.
eSOPs can also have alerts for response time, so that operators can review the alerts to try to understand the effect of hours worked on a controller.
Change management – Each operator must assure that changes that could affect control room operations are coordinated with the control room personnel.
Controllers should be notified in advance when work will be performed out in the field that will affect the safe operation of the pipeline system. Any time a piece of equipment transitions to online, the controller should be notified, allowing the controller to understand the workload required and be aware that changes will occur.
When a controller receives notification of work to be done, an eSOP can be initiated and an expected execution time entered. This allows the controller to plan and be ready in case something goes wrong. At the end of the expired time, the controller can extend the time or cancel the workflow. This method also allows tracking of SCADA data points being taken off scan and point-to-point verification.
The opened work requests can be tracked, and the controller can be notified when too much field work is scheduled to be handled safely and also keep track of the events that happen elsewhere in the pipeline system.
Operating experience – Each operator must assure that lessons learned from its operating experience are incorporated, as appropriate, into its control room management procedures.
The Work Process Management system can afford the ability to continuously record and make eSOPs available to future incidents. In the event that an incident takes place that has no eSOP, the operator can create a new workflow to be used in the future and for training.
By utilizing eSOP to control actions and handle alarms, the operator can determine the cause of the incident and then produce a new workflow by reviewing the resultant reports, coupled with the actual alarms produced during the incident.
Training – Each operator must establish a controller training program and review the training program content to identify potential improvements at least once each calendar year, but at intervals not to exceed 15 months. An operator’s program must provide for training each controller to carry out the roles and responsibilities defined by the operator.
Workflows developed during normal operation can be used for training purposes. Utilizing a stand-alone computer, the current HMI/SCADA screens can be combined with workflows to generate multiple scenarios that a controller may encounter. Values can be forced into data points during the simulation to evaluate the response of a controller.
Compliance validation and deviation – Upon request, operators must submit their procedures to PHMSA or, in the case of an intrastate pipeline facility regulated by a State, to the appropriate State agency.
An intelligent control room management solution tracks actions and records responses – and provides a historical record to demonstrate compliance. Additionally, the system digitizes SOPs for easy submittal and review of procedures. Standard and custom reports offer a way to analyze overall compliance with procedures as well as evaluate individual controller performance and opportunities for improvement.
Conclusion
Pipeline safety is a critical issue for both pipeline operators and the communities they serve. Government rulings establish guidelines for improving pipeline safety – and are good business practices globally. In addition to addressing the mandates, intelligent control room management can help pipeline operators overcome many challenges facing industry today, such as an aging workforce with experts retiring, lack of step-by-step Standard Operating Procedures that guide workers, alarm “noise” in the control room environment with thousands of alarms hitting controllers, and emergency situations requiring the correct response to maintain life and property safety.
With eSOPs at the heart of an intelligent control room management solution, controllers receive guidance on the proper actions to take. Companies can minimize nuisance alarms, train controllers – and probably most importantly, capture best practices and standardize procedures. By tracking and recording events and responses, historical records can be available that help prove compliance and offer a foundation for analysis and improvement that provides a long-term sustainable advantage.
Authors
Paul Thoman is president of S3 Development Corp. He received his formal education during the mini-computer era while working in the defense industry. Over the past two decades, he has played a major role in almost every SCADA design and / or implementation of the natural gas industry in the northeast U.S. He is best noted for his advanced and persistent design approach to reducing Total Cost of Ownership for the customer.
Mayank Mehta is the global industry manager for oil and gas at GE Intelligent Platforms. He is an electronics and communications engineer, with 29 years of experience in the field of automation and controls, optimization solutions and security. Before joining GE, he served 15 years at Emerson Process Management.
Alicia Bowers is a solutions manager – Infrastructure Industries, at GE Intelligent Platforms. She has served the industry for 15 years, with experience ranging from automation and control to operations management. She has been recognized for her volunteer roles within Manufacturing Enterprise Solutions Association International, providing industry leadership for more than three years. Contact information: alicia.bowers@ge.com.
Comments