[VOIPSEC] Network Capacity During Emergencies

[Philip Walenta]

Some of the large deployments I've done have had a fair amount of fault
tolerance built in.

Unfortunately not all.  Rest of my comments inline below. 

Philip Walenta
262.951.1941
pwalenta at wi.rr.com

> -----Original Message-----
> From: Voipsec-bounces at voipsa.org 
> [mailto:Voipsec-bounces at voipsa.org] On Behalf Of Jonathan Zar
> Sent: Wednesday, May 10, 2006 9:09 AM
> To: voipsec at voipsa.org
> Subject: [VOIPSEC] Network Capacity During Emergencies
> 
>  
> I'd like to invite comment on the VOIPSEC list on what 
> happens to VOIP traffic during significant emergencies.
>  
> Are you seeing VOIP added to business continuity planning ?

I've had two clients that as part of the system install/evaluation go
through a complete plan and even went through significant disaster testing
where just about every type of single and multiple failure occurred (single
core outage, multiple data center outage).
>  
> In multinational operations with significant VOIP trunking, 
> what happens if there is a sudden spike in the VOIP traffic ?  
>  
Overflow trunking was built in, so that if all VoIP allowed bandwith was
used, it overflowed to PSTN based services.

> What would you recommend to your customers in the event data 
> remains up when other forms of communication fail ?

I've actually had this particular scneario happen only once, and that was
due to the voice traffic not being routed properly in the first place
(client had several networks consisting of frame, VPN and MPLS, and voice
traffic wasn't being routed properly, so when a disaster hit, the voice
traffic was misrouted virtually everywhere).  But what I've seen most
clients do is ensure there are published cell phone lists, and a few analog
lines spread among the buildings.  Not necessarily the most practical thing,
but it's at least better than nothing.

>  
> Finally, do you see overflow planning to allow employees to 
> use alternative VOIP systems in the event they are stranded 
> and unable to get to approved communication ?

Had only one client go down the path this far, only because they've kept
near-perfect records of disasters and phone system usage.  Essentially, they
had several "crash carts" waiting to be deployed in the event of this
significant of a failure because they knew the exact size of any given
system, and had complete backups and drives which could be restored on a
moments notice.

>  
> An emergency might include a major natural disaster, pandemic 
> or selective infrastructure failure.
>  
> Your voice of experience either from practice, planning or 
> statistical analysis and modeling is invited on this thread.

As far as general advice - most of the time it isn't the voip system itself
causing or having a problem - it's the network.  I cannot stress enough to
anyone I design or deploy a system for - build the network right, QoS,
security, routing, DHCP/DNS, load-balancing.  I would say at least 90% of
the time there's been some issue during an outage, it's a network
configuration causing the issue.  My biggest thing is eliminating
spanning-tree wherever possible.  Even with the advent of 802.1w (rapid
spanning-tree), I find that a routing table/protocol that already has active
multiple paths converges faster (when properly configured) than 802.1w.  In
essence, layer 3 is still faster and more reliable than layer two in my
experience.  I actually know of one company that is almost 100% layer 3,
where each closet switch (no matter the size of the closet) is L3 connected
to at least two different upstream devices - and they've not had a single
issue no matter what has gone on.

The one other thing I recommend is know your traffic.  Both network based,
and PSTN trunk based.  Capacity planning is another thing I've seen
seriously lacking in many facilities.  The network is built right, but not
monitored correctly - usually a major outage or two which might have been
forseeable cures this.  But I *hate* eagle-eye hindsight.

Hope this helps.

>  
> Best Regards,
>  
> Jonathan Zar
> Tel:    +1 (510) 275-1480
> Fax:   +1 (510) 275-1481
> Cell:   +1 (408) 209-0199
> _______________________________________________
> Voipsec mailing list
> Voipsec at voipsa.org
> http://voipsa.org/mailman/listinfo/voipsec_voipsa.org