Disaster Preparedness and Recovery

The Importance of Good Intelligence



SitePro Disaster Preparedness and Recovery

Introduction

According to a study recently published by the GSMA1 the total number of mobile connections reached 6.8 billion or 93% global penetration in the fourth quarter of 2012, including subscriptions associated with machine-to-machine communications. Furthermore the GSMA predict that by 2017 the global mobile penetration rate will rise to 114%, with penetration rates in both developed and developing markets set to exceed 100%.

Although penetration rates continue to rise (driven by the integration of mobile communications capabilities in previously unconnected devices – the Internet of Things, the adoption of superfast mobile broadband services and the rapid rise in mobile adoption in developing markets), carrier revenues are either stagnating or are in decline2, with some carriers
expecting average revenues to fall by 12% to 13% in 20133.

In order to effectively service a rapidly growing subscriber base, that are also becoming increasingly dependent upon mobile communications, while experiencing significant pressures on revenue, carriers and tower operating companies must ensure high levels of service availability, even when confronted by the most adverse of conditions.

The incidence of natural disasters all over the world is on the increase. North America has experienced the most significant rise, with a five-fold increase in weather related natural disasters over the past three decades, according to Munich Re. “Nowhere in the world is the rising number of natural catastrophes more evident than in North America” – Peter Hoeppe, Head of Munich Re’s Geo
Risks Research unit. There was also a four-fold increase in disasters in Asia, while the number doubled in Europe4.

Figures available from the International Disaster Database highlight the financial impact of this rising occurrence of natural disasters. Asia and the Americas combined now suffer damage from storms, floods and earthquakes, estimated to cost $8,000,000,000 per year.

The growing dependence on mobile communications to support first responders, the general public and media organisations, is forcing carriers and tower operating companies to take steps to harden their networks against the impact of such catastrophic events.

The services that mobile operators, broadcast, communications and utility companies provide become vitally important during times of crisis or disaster, but this is exactly the time when infrastructure is prone to significant failures. For example, during 2012 a major American carrier suffered, what were described by the FCC as “unacceptable failures”, which caused 911 emergency-calling service to fail for more than 2 million people after storms in June 20125. Site or network outages can cause significant disruption in the aftermath of a disaster and can in some cases hamper recovery efforts.

Figure 1: Natural Disasters Reported, 1900 – 2011

EM-DAT: The OFDA/CRED International Disaster Database – www.emdat.be – Université catholique de Louvain, Brussels, Belgium.

Failure to prepare effectively and recovery quickly for disasters can have major business impacts long after the disaster is over. In many cases this manifests itself as government censure, the introduction of a tougher regulatory regime or through delays in recovery which ties up valuable resource. Outperforming rival networks at a time of crisis can generate significant goodwill and also allows a carrier to maximise revenues at a time of higher network usage.

A document recently published by OFCOM6 indicates that approximately 45% of all network outages are caused by a failure in the power systems at the remote site, with a further 38% of outages caused by a failure in other hardware on a site.

Disaster Preparedness Systems

Reliable, high-quality intelligence is vital when planning for network resilience. Remote Monitoring and Control Systems (RMCS) are a core part of any Disaster Preparedness strategy. The deployment of such systems has been proven to minimise network downtime during natural disasters.

In 2007 Digicel, a Jamaican network operator, applied significant effort to their hurricane preparedness. Part of which was the deployment of Telemisis SitePro – Site Manager, an advanced Remote Monitoring and Control System (previously known as GenTrak). By using SitePro – Site Manager, before, during and after hurricane Dean (a Category 4 hurricane with winds of up to 145mph) struck the island, Digicel were the only mobile network still operating in all 14 parishes of Jamaica following the passage of storm. “Our hurricane preparedness measures allowed us to maintain the stability of our core network across the island”, “Loss of power was never an issue this year” – David Hall, CEO, Digicel7.

Advanced RMCS have a role to play, before, during and after the event, in making sure networks are prepared and able to recover in the shortest possible time, should the worst happen.

BEFORE THE EVENT

Power is one of the most critical systems that comes under the greatest pressure during storms or other natural disasters. It is therefore vital to ensure back-up power systems are functioning correctly, thus minimizing the chance that they will fail when needed instantly.

RMCS provides day-to-day and historical information to ensure site systems are ready. More advanced RMCS support rapid back-up power testing, through the provision of a remote control capability. RMCS can provide a wide range of information and control including:

  • Providing a method of test the starting and loading of any back-up generators without the need for a costly site visit.
  • Providing information on the condition of back-up batteries, so that their ability to power a site for the duration specified in the preparedness plan can be assessed.
  • Providing information on any issues that may prevent any back-up generators from starting, including:
  • Low generator battery voltage
  • Generator not in the correct start mode
  • Historic and live information as to whether a generator has failed to start
  • Empty or low fuel warnings, with advanced systems providing an estimate of run-time based upon fuel remaining and historic consumption.
  • Trending of site parameters and alerting of performance degradation, such as battery performance enabling timely service or exchange before a critical events arise.
  • Detecting shelter water ingress, thus allowing repair or improvement work to be scheduled and implemented as preventative maintenance, thus hardening the site against potential flood risks.

During the event

In many countries it is common for the main grid electricity supply to be suspended just prior to an event. This is done to minimize damage from shorts in the high voltage supply. When this occurs, back-up power at the remote site should kick in. The performance of the back-up power system can be monitored through the RMCS. If issues occur they can either be dealt with immediately or scheduled as part of the recovery plan.

RMCS can provide information and control during the disaster event to enable response or recovery planning by:

  • Providing visibility of grid failure or shutdown, as it moves across the network.
  • Monitoring generator start-up and provide the facility to manually intervene and remote start generators if required.
  • Fuel level and consumption monitoring combined with a prediction of when fuel will be exhausted, enables just in time refueling. Maximizing efficiency particularly when resources are stretched and access may be limited.
  • Where a site has only battery back-up on site, support staff can monitor the batteries for charge left and deploy emergency generators as necessary.
  • Flood sensing provides valuable information on the condition of the enclosure and the danger of equipment damage by water and power shorting and focus the deployment of pumping equipment.
  • Temperature & humidity detection will flag air conditioner failure or other shelter damage, which is also likely to take the sites off-air and so can be used to schedule repairs and deploy correctly skilled personnel.
  • Integration with other site systems such as microwave links, rectifiers and aircraft warning lights all feed into the recovery planning to optimize use of resource at a time when they will be significantly stretched.

After the event

Reliable and up-to-date information allows site operators to focus recovery efforts and ensure the right resources are deployed to the right sites. Sites that can be easily brought back on stream can be prioritised. Sites that are in less urgent need of attention can also be prioritised accordingly.

Once the crisis has passed the information reported from the RMCS can be used to provide feedback as part of a continuous improvement process and enhancing protection for any subsequent events.

Source:

1, GSMA, October 2012 – GSMA announces Global research that highlights significant growth opportunities for the mobile industry

2. ETNO, November 2013 – Annual Economic Report 2012

3. Bloomberg, April 2013 – France Telecom to Focus on Costs as Mobile Phone Bill Shrinks

4. Bloomberg, October 2012 – North America Has Biggest Rise In Weather Disasters

5. Bloomberg, January 2013 – Verizon to Face Backup Power Requirement on 911 Calling

6. OFCOM, December 2012 – Infrastructure Report

7. Jamaican Gleaner, 22nd August 2007

Author
Tony Richardson - Managing Director, Telemisis

Tony Richardson
Managing Director, Telemisis

More about Tony

Share this article



Telemisis SitePro

SitePro has been deployed in some of the world’s harshest environments and has been proven to help communications infrastructure operators manage and recover their systems in the event of natural disasters.