UNIX crisis at SJSU?

First signs of trouble
The two main Email systems at SJSU are the UNIX system and Lotus Notes. UNIX is also used to authenticate to and provide access to the university's Web servers. Mid-day Thursday we saw the first signs of trouble with UNIX. Mid-afternoon we started getting calls at the Help Desk requesting password resets. When I went to log onto the UNIX system to reset these accounts my own password did not work. Normally I am the person who resets passwords, but now I needed my own password reset. More and more people were calling in reporting they could not log onto UNIX to either get email or update their web pages. I called the Computer Center to report there was a problem with the UNIX system. At first glance it appeared the authentication server might be down or having problems. I sent an Email to the SJSU techs Email list saying there appeared to be a problem with UNIX. I received a response that there was nothing wrong. In effect, I was told, this was a planned process to cull a handful of dormant and duplicate UNIX accounts.

Then, one tech from a university college called and reported to me that all the department Email accounts in that college appeared disabled and that mail to them appeared to be bouncing. That evening I sent an Email to all the students who work at the Help Desk to start marking all the tickets we receive with boiler plate language that would enable them to be traced. This went out Thursday after we closed:

Please copy this text and paste it into each GWI ticket you send to (Name Deleted) regarding the inability to log into client's email:

UNIX Account - user reports inability to log onto UNIX account. Please check to see if user's account has been locked and unlock it if it has.

~Steve

A busy Friday?
Normally Friday is a quiet day and I was not in the office that day. I called in several times to the students on Help Desk and they said it was the busiest day they had since the hectic start of the school year in September, normally the busiest days of the year. They estimated they turned in about 30 tickets using the boiler plate language. This volume of Help Desk calls is not good on a Friday.

Monday, is that a wave on the horizon?
There were thirty messages on the phone lines when I came in and the phone was ringing like crazy. Accounts that were locked included the library reference desk and HR accounts. At the rate we were going we would never be able to get ahead. On top of that administrators were calling wanting to know what was going on. Answering those questions took more time. I Emailed every staffer we have at the Help Desk asking for any staff who could come in to come in. We have three student work stations and one is down on maintenance. So counting me, we had three on computers and a couple out dealing with walk ins. By around three a search in the ticketing system using the boiler plate language turned up about 72 tickets. Those were just the tickets we created Friday and Monday and did not include Thursday tickets or tickets that may have come in through other portals. It is my guess there were 90-100 trouble tickets issued, or handled verbally, on this UNIX account issue by close of business Monday. According to one administrator, I am told, the total number of UNIX accounts marked to be locked was around 300 and about half of those were resolved in December. If that is true we have most of those reset already. But, another SJSU IT professional told me reportedly did a search of the database and found about 1,400 accounts that had been locked. If that is true, what we have seen so far may be the tip of the iceberg.

Should we be expecting a perfect storm?
Tuesday is the first day of the semester and Wednesday classes are starting. We were expecting the normal storm of folks coming to the Help Desk for routine stuff. We expected to add to this the storm of new users trying to figure out how to use the new wireless network. Now, we may be adding to this a lot of angry folks who are going to be coming back to SJSU just to find their UNIX Email accounts locked. We have a physical limitation in the help desk. We can only have three students on terminals answering phones and entering tickets at a time (when all the terminals are working.) We can maybe have a fourth moving around dealing with walk ins.

So, has this blown over? Will it be smooth sailing tomorrow? Or, will this be one of those cases where three smaller storms converge to form one, big, perfect storm?

Technorati Tags: San Jose, SJSU, CSU, California State University, ssloansjca