More on the phones...

I was forwarded this from Councilman Joe Shogan. It is a response to our questions about the phones from Garv Brakel, head of the city's IMS Division. A public thanks to the mayor, to our city councilmembers and especially to Mr. Brakel for answering Local 29's questions regarding the history behind the new phone system. It sounds like they've discovered the problems and the phones should now be working - good news for all!

The memo:

I was first notified of the problems with telephones at the fire stations on Monday, the 6th of June 2005, and focused the appropriate resources of my department for a quick resolution. Clearly, there has been a problem out there for several months. One of the issues has been that the problem was intermittent, and as a consequence has not been reported and escalated to management in affected departments and MIS. During this time, however, the 911 system was not impacted, nor, because of redundant systems, was there ever a situation when a fire station was without emergency communications capability. We isolated the main problem this week, however, we will continue to monitor performance. The fundamental problem was a programming error in some network components at remote sites.

Let me address the specific questions posed my Mr. Griffith:

1. The total cost of the new telephone system was $1,597,591.00. With tax, the total increased to $1,726,995.87. Included in this number is hardware, software, and installation services. It is true that each telephone instrument costs around $500, however, these telephones are actually computers on our citywide network that have many features not available on the old system. An example of this is the directory capability or unified messaging which makes voicemail available through the email system. Over time, we will be fielding additional enterprise applications to our customers via the telephone system. This number compares favorably with the $300 to $400 we were paying for rebuilt telephones for the old system. After inflation, the new telephones were actually cheaper than the old ones in 1989.
2. While the old system was not broken, it was 15 years old. To keep it operating following the year 2000 we bought parts from organizations that had already replaced their systems. We have been totally in the used parts arena for five years. At any time, a hardware failure could have caused an outage that would have lasted until we found the appropriate part. We have been on borrowed time for years.
3. The City began its search for a new system in 2000 with initial planning based upon our difficult experience in making the old system Y2K compliant. In 2001, we engaged the services of a telecommunications consultant to assist MIS staff in identifying each department’s requirements for a new system. We extended an invitation to all interested vendors to install demonstration systems for our evaluation. Five vendors accepted our offer and actually installed their systems in City Hall for use by selected users. City staff visited three other vendor sites for demonstrations. These requirements and experiences were reduced to a 120 page RFP that was released in early September 2003. On the due date of October 6, 2003, the City received 12 proposals for evaluation. The evaluation team consisted of City Council President Rob Higgins, Administrative Services Director Dorothy Webster, Mike Charter from the fire department, Dave Ingle from the police department, George Bailey from the solid waste department, and Ron Nicodemus the director of the utility billing department. I sat as a non-voting member of this committee. The final selection was based upon functionality and lowest cost over a 5 year period. The decision was approved by both City Council and Mayor West before the contract was signed.
4. The Voice over Internet Protocol (VoIP) technology is a mature product at this point. In fact, I delayed the whole process to the point that the technology was not only mature, but also at the point where prices dropped as they do with all new electronics. The exact same system selected is in use by several other cities and counties in Washington, Microsoft, Boeing, and over 60% of Fortune 500 companies. Mr. Griffith is probably confusing our system with some commercial offerings of VoIP where the Internet is the means of transport instead of normal public switched telephone system circuits. There are quality of service issues with these services. Ironically, the programming errors that we found cause our system to have these same problems at remote sites.
5. Once the installation has been completed, acceptance testing shall be performed upon the system following its cutover. This test and verification period, if successful, shall consist of thirty consecutive days of normal traffic load with no major component failures and no major alarms, defined as: 1) CPU failure; 2) main power supply failure; 3) a failure of the remote electronics at one of the City’s sites; 4) 25% of incoming/outgoing trunks inoperable; 5) 10% or more stations inoperable; or 6) 10% of the voice mail system inoperable. The City of Spokane shall accept the installed system after a signed letter of official system certification with successful acceptance test results, accompanied by two sets of as-built documentation provided by the Vendor, is received, reviewed with the Vendor, and accepted in writing by the City of Spokane. Because installation is not complete for this system, final acceptance testing has not begun.
6. It would be very difficult and prohibitively expensive to either go back to the old system or select a different technology at this point. One of the decision points for this system is that it rides the same network that we use for computers. With the old system, we had to maintain a separate physical network just for telephones. The savings over time will be significant. Telephonic customer service will improve this week. Funding for this system was via the City Capital Improvement Program and MIS equipment replacement reserves which are both one time funding resources for capital programs only and are not viable sources for continuing operations.