Hurricane-proof data center built inside 770,000 gallon water tank
to Florida to start his Altamonte Springs IT job, the city’s network engineer walked up to him and said, “We’ve just lost everything. All the Novell servers, the Novell clusters, the backup, the SAN. Everything’s gone,” he recalls.
The city’s only backup consisted of a server running Veritas Backup Exec to a Spectra Logic AIT-3 tape library. “Tapes are unreliable,” DiGioia says. “Disaster recovery was nonexistent. It consisted of backup tapes in a box.”
Over a two-and-a-half week period, DiGioia says, he was able to recover most of the city’s data off the backup tape, but a significant amount was lost.
Things did not get better after that. In 2004 Altamonte Springs was hit by Category 3 and 4 hurricanes Charley, Frances and Jeanne. City Hall and the data center within it were constructed to resist Category 1 hurricanes. To safeguard the IT equipment, everything had to be packed for each storm and placed into storage units until the storm passed.
“We literally had to dismantle everything. It was a horrible experience. The emergency operation center was shut down also because there wasn’t infrastructure in place to support Internet access during a storm,” DiGioia says.
He decided it was time to build a better infrastructure, one that would include a separate disaster recovery facility, keep critical backup data online, and support recovery point and recovery time objectives.
The water tank just seemed like a logical site, he says. It had been decommissioned a few years back and was built like a fortress.
First, DiGioia got buy-in from Altamonte Springs’ political leaders. The city then commissioned the construction of two buildings on either side of the water tank; one wing now houses the networking equipment and the other has administrative offices.
In the old data center in City Hall, the networking infrastructure consisted of point-to-point T1 lines over copper wire to sixteen facilities, such as the police department and public works offices. For the new data center, the city ran dark fiber (unused optical fiber that can be tapped into if necessary) to the same facilities; it also leased dark fiber from surrounding county governments, greatly increasing bandwidth and distance for disaster recovery.
Mearian writes that DiGioia and his team also rolled out VMware ESX server software and reduced the physical server count from 80 boxes to 12 Dell quad-cores running 30 virtual machines. The servers were configured to boot from the SAN.
In the main data center, the city uses
