After almost a week of sporadic e-mail service, technology officials brought recent and new e-mails to user inboxes late Wednesday night. Students should be able to view e-mail received Tuesday and Wednesday and send and receive new e-mail Thursday, said Bret Jones, director of technology engineering.
Information Systems and Services staff have taken all saved e-mail off the system to re-index and re-send, which should appear in student inboxes over the next 48 hours, Jones said.
Experts from Sun Microsystems, the company that manufactures GW’s e-mail storage hardware, still have not identified the cause of Friday’s severe hardware failure of the controller that allows students to view e-mail. Jones said that while the Webmail site most students use to access e-mail experiences occasional slowdowns and failures, GW’s system of e-mail storage has not failed in the two years since it was last upgraded.
Students should not lose any e-mail because of the downed hardware, Jones said, unless their inbox reaches its 20-megabyte quota.
Mail sent to students was returned with a message notifying the sender it was undeliverable, and attempts to re-send itself for about four days. If an inbox reaches its quota, it returns all mail to the sender with a message that the inbox is full.
Students named inconveniences from missed assignments and graduate school correspondence to health concerns as a result of the downed e-mail.
“I rely on Webmail for my medicines,” said Andrew Tabler, a freshman, whose political science and film studies classes also rely on the e-mail system. “I send the information to the company online and need the e-mail to work.”
“No one in the business world could run without an e-mail system,” said Jim Rector, a sophomore, whose English professor had sent e-mails to his class only to find that the system failed to deliver them to the class. “(The outage) is unbelievable.”
Charlie Spann, who runs the ISS help desk, said the e-mail server processes 150,000 messages in an average day and the backlog of e-mails sent to student inboxes over the weekend is approaching one million.
Eagerly anticipating the repair of GW’s e-mail system, students were let down once again after being able to use it for only part of the day Wednesday for the first time since Friday afternoon. Officials had to bring the server down again at about 4 p.m. after Tuesday’s re-indexing of thousands of old e-mails “did not go as well as we had anticipated,” Jones said.
When ISS restored the e-mail system early Wednesday morning, it ran slowly, some users were unable to log in and others received different users’ mail in their inboxes.
Jones said Sun is flying in experts from California and Michigan to identify and fix the cause of the controller failure that caused the outage at about 12:30 Friday afternoon.
ISS and Sun crews have worked around the clock since then to try to identify the problem, which affected the controller that allows students to view their Webmail inboxes. Jones said one staff member spent as long as 31 hours in the basement of the Academic Center, where ISS technology is stored, over the weekend.
“I’ll probably be spending the night here again,” Jones said Wednesday.
On Monday, system administrators replaced failed disks, but stored student mail had to be re-indexed and tested before it could be accessed. This test began at about 12:30 p.m. Tuesday, bringing all mail storage down temporarily. The test revealed that some folders were not indexed when messages were copied from the old storage device to a new one – some ISS employees on the downed server were not able to see their e-mail history when they logged in.
Spann said the way the mail application views the data in the messages may have been corrupted when the files were transferred, which officials thought may have been solved by the initial re-indexing of new messages. Officials then had to run a folder check.
ISS performs routine folder checks about once a week because deleting messages and receiving new mail can corrupt folders. The re-indexing of new messages was not as comprehensive as this folder check program.
“We did not think we would have to run the folder check – we planned to run the folder check as a last resort,” Spann said, adding that the re-indexing brought back some Webmail users’ old mail folders. “It was either, have people call us if they have a problem, or just do all of them.”
All long-term storage disks have remained intact, allowing ISS to queue all mail coming into the system.
“It’s like a library – you’ve got all these books but you lose the index,” explained Ron Bonig, executive director of technology operations. “While the books are still there, they haven’t gotten lost, but you can’t get to them fast.”
GW is currently upgrading its e-mail storage system to store all e-mails in one cabinet that would hold up to 16 terabytes of data. A terabyte is 1,000 gigabytes; the current student controller holds 1.4 terabytes, while the faculty and staff server holds one. The new system will be faster and more reliable.
Officials also hope to keep a real-time duplicate of this hardware at the Loudon County, Va., campus for safety.
“This is part of our ongoing disaster recovery plan,” Jones said. “When the World Trade Center incident happened, the companies that came back up within the first few days were running this system.”
GW is also upgrading the Web client students, staff and others will use to check e-mail. The University will switch from Webmail to iPlanet, the same back-end software that Internet service providers like America Online use for e-mail.
Kerry Washburn, ISS director of administrative applications, will help users adjust to the upgrade.
ISS conducted a user survey earlier this year to identify the GW community’s e-mail priorities. Washburn said a lot of the features those surveyed asked for are already available with clients like Webmail.
“For example, ‘I want to be able to automatically assign messages to different folders as they come in’ – well, with something like Netscape or Outlook you can apply what are known as filtering rules and that will happen,” she said.
The software upgrade is expected to be complete by late spring or early summer.
