LITCHFIELD — A series of computer failures in early July handcuffed town staff – and residents – as officials worked to restore weeks of backup data.
Staff spent many hours fixing the problem, interrupting the ability of residents to complete basic tasks such as registering cars and paying taxes in the middle of the day.
Going forward, taxpayers also may be asked to pay more for computer investments, given the shortcomings highlighted by the crash.
“It was a perfect storm,” said Steve Ochmanski, Litchfield’s code enforcement officer, who helps manage the town’s computers and has been working with various companies to address the problems.
The issues arose on a Friday in early July, when the computers in the town office stopped working as an auditor was trying to deliver a presentation. But the problems had been germinating since spring, Ochmanski said.
The town installed a new, $2,150 server about six months ago. The server is hardware that sits in the basement of the town office and stores all digital records staff create on their computers.
Then, in May, the town had software problems related to automatic updates for the Windows 10 operating system, according to Ochmanski.
Those problems weren’t resolved until late June, but because of them, he said, the town was distracted from a more dire threat. A piece of hardware called the motherboard that controls many system functions was failing, Ochmanski said. That’s what caused the computers to crash on July 6.
Eventually, the town discovered that much of its software was corrupted and that a system for backing up its data to the web also had broken down.
“It was when we got it back up online that we realized the backup hadn’t been running,” Ochmanski said. “So then we went back and figured out the backup stopped on June 8 and everything after June 8, we didn’t have.”
Since then, staff have been working with an information-technology contractor and others to diagnose the problems, replace the broken hardware, recover the lost data and manually re-enter information from paper records back into the computers.
Those records included receipts from financial transactions such as vehicle registrations, building permits and transfer station fees.
Staff have nearly finished replacing the data, said Town Manager Trudy Lamoreau. To do so, workers have come in on Fridays, when the town office is normally closed. From Aug. 20-23, the town also closed the office to the public for five hours in the middle of each day.
The town sent out a letter on Aug. 16 notifying residents of the change, which still provided time in the morning and evening for the public to do business. Regular hours resumed last Monday.
During that time, Lamoreau said she was not aware of residents becoming frustrated with the arrangement.
“Thanks for their patience,” she said. “They were very supportive. They were very empathetic.”
Lamoreau also praised town employees for their efforts to re-enter the data.
The town hasn’t yet assessed the total costs of the computer crash, but Lamoreau and Ochmanski listed several obvious expenses that totaled nearly $5,000.
Lamoreau estimated that five employees each worked more than 44 hours to complete the data re-entry. Based on hourly wages, that represents a cost of about $2,800, Lamoreau said. Ochmanski, who is salaried, said he’s spent at least 30 hours on the problem.
The town’s server was under warranty from the company Dell, so there were no costs for the replacement hardware. But Lamoreau said the town will pay about $990 to the town’s contracted information worker; $280 for the town-wide mailing; and $700 to a Brunswick firm that helped recover the missing data.
The town also has improved the way it backs up data. In addition to saving that information to the web, it now stores it on two other hard drives, Ochmanski said. As an added layer of protection, Ochmanski said he now receives daily emails when the system has been backed up to the internet.
However, going forward, Ochmanski warned that the annual budget for information technology may have to be raised if the town hopes to adequately protect its computers and information. Those annual costs are now around $19,900, Lamoreau said.
“Until something like this happens, people don’t think it’s that important,” Ochmanski said.