Chickenminnie’s Weblog

I’m settling into my new routine and it’s going fairly well in spite of some pretty sizable adjustments that I hadn’t anticipated.  The new hours are ok and I’m getting used to the busier, noisier office setting.  What has been harder to get used to is having to rely on other people for everyday things.

At the district, if I needed access to something I had it simply by virtue of being the Network Administrator.  At the RIC it’s a totally different scenario because of the fact that it’s a data center with numerous tiers of authority.  I was happily adding users the other day and discovered that I had permission to one container but not another, necessitating a request for a change in permissions.   If a file needs to be restored I have to open a support ticket and wait until the backup guy can restore it.  My requests for extra permissions or an errant file are quickly granted, but I always feel like the resident pest.

This scenario is typical of how large organizations work and understandable from the standpoint of security, but it has seriously hampered my ‘get it done’ approach.  Having spent nearly ten years running my networks alone, it has been an interesting exercise for me to suddenly have to work in a team environment; fortunately for me I don’t bring my ego to work with me!

There are other things about the new job that are not as easy to understand, such as why I still do not have voicemail.  When I queried my manager about it, he said something to effect that I needed justification for it.  This is French for “we are too cheap to pay for licenses for everyone”, but I was too stunned by that response to even formulate an answer.  How about the justification that I’m an IT professional who routinely gets calls from customers who probably wonder what sort of lame organization I’m working for that can’t even provide me with voice mail? Geez….

For now I’m choosing my battles, but if the stupidity layer gets too thick, I may have to pick up my toys and play elsewhere.

Life is a funny thing.  Faced with the regular irritants of everyday life, one develops little mechanisms to deal with them much like an oyster emits nacre to protect itself against a grain of sand it cannot expel.  In the case of an oyster a pearl is produced, but with humans it is often something else entirely.

When the irritant balance becomes too unwieldy I have a mechanism I call the Two Minute Warning; it’s the device that makes me realize that a lifestyle decision must be made, and it must be made now.  I have had several Two Minute Warnings in my life; the moment I realized that I could no longer be married to the man who is now my ex, the instant I realized that a college education was essential, and most recently when it became clear to me that a job change was in order.

At my district we have Tech Specialists who handle the day to day desktop stuff, leaving me to handle the network issues.  In their defense these are not well paid people, but the problem is that they tend to be computer enthusiasts and not people who understand networking. Because of that, they often do things unwittingly that cause me huge amounts of grief.

On Monday, like the proverbial straw that broke the camel’s back, a couple of things happened that made me realize I needed a change of scenery.  One of my Tech Specialists saved an 8 GB ghost image to the staff volume of my main server running it completely out of space, despite the fact that we have a server designated especially for such things.  It’s not a big deal to find the offending file and move it, but it disrupts people and causes problems with folks who auto-archive their email to their network space.  Shortly after fixing that issue, the chair of the Science department came to me because the Science Department folder containing years of shared documents was missing.  I took a look, and sure enough it was gone.

The great thing about my Novell network is that if a user deletes something it goes to a salvage area; deleted items stay there until the volume needs to reclaim the space.  When I did not see the missing folder in salvage I knew immediately that the system had purged the file to make room for the huge file my staffer had saved.  Undaunted, I assured the department chair that I could simply get the file from backup.

I created a restore job and after hitting “okay”, I got a testy message about the tape library being unavailable.  Huh?  I pinged the tape library and got no response.  Since it is not in my building, I drove over to the site where it is located and discovered that it was powered off!  Then I remembered the week before when another Tech Specialist accidentally unlugged one of the big heavy duty power strips fed by a UPS killing all the phones at the Middle School as well as the server.  Somehow in the fracas the tape library was never powered back on.

Argh!  I knew that even though the tape library was powered back on, the server was not going to “see” it until I rebooted, which was not going to happen until evening.  I had the rather unpalatable task of telling the department chair that I would not be able to restore the folder until the following day.  He was cool about it, but I could tell he was wondering if he was ever going to see his files again.

Later that evening I rebooted from home and was able to restore the folder for the staff member, but I knew that I had to make a change.  Running a large network is stressful enough on its own; I don’t need my tech staff making my job more difficult.  This sort of thing happens on a regular basis and repeated attempts to get my boss to hire more competent staff have failed.  He’s one of those nice people who never wants to rock the boat, so it looks like I will have to jump ship to save myself.

There is a white board in my cubicle upon which I keep a list of major projects that need doing; one of those items on the list is “Upgrade Websense” with a little sad face drawn next to it. The reason for this designation is that it is an upgrade that just never goes well. Ever. It always results in several long phone calls to San Diego where they are almost as mystified as I am.

You would not think this would be so difficult; I always dutifully print out the upgrade instructions and follow them to the letter, making certain that the event viewer is not open and that the Enterprise Manager is not running. The install manages to stop all the services properly, the file copy goes well and then bang! An Error:128 appears and I know I’m done; stick a fork in me, it’s time to call tech support.

Websense tech support is variable; sometimes you get someone phenomenal, and sometimes you get a dud. The tech I got was okay, but you would think the error messages I was kicking out would have been enough to go on, or documented somewhere. Instead he had me trying all kinds of things like reinstalling the old version and then upgrading to an incremental build, etc.  My gut tells me that the root of the evil is a config.xml file that has been through too many upgrades and ip address changes.

I suppose I could just do a clean install of the latest version, but the problem with that plan is that I would lose all the policies I have created along with my massive white list. When asked if it was possible to send them my config.xml file and have them convert it, they told me that they don’t do that; the only time they will convert a config file is if a customer migrates from Windows to Linux (this is very tempting since I have always felt that having “workhorse” services running on Windoze is a really dumb idea).

Anyway, after hours of being glued to the phone, the tech asked me if I had a spare server laying around so that I could set up a secondary policy server and migrate the xml file to it. The idea was that the secondary server would do the conversion, I could update the original and plunk the converted config file back in. I have been down that road before and got licensing issues and other mayhem, so I decided that for all the support money we pay those nice folks each year they could do it for me.

A couple of hours later the updated config file arrived and it worked fairly well after a bit of search-and-replace; all that pain for an .xml file that’s been around the block a few too many times!

I still wish I had gone with Linux.

Whenever someone finds out I work for a school district they invariably ask me if I have the summer off; people just naturally think that anyone who works in a school has a nice long lazy summer. But anyone who has darkened the door of a school building in July or August will find out that it is abuzz with activity.

While most people are kicking back and enjoying the slower pace, school techs are in a frenzy of upgrading, deploying and planning for the year ahead. If you are lucky enough to work for a district that is not undergoing some sort of construction project, life is good.  If you are like me, and working through the fourth year of a 92 million dollar construction project, life can get pretty crazy. Nothing adds interest to a network administrator’s job quite like asbestos abatement in your primary data facility. Or having a backhoe slice through the main fiber feed that services every building in the district…that’s fun. And there’s nothing like checking your email in the evening to discover that an entire data rack (the one housing the phone system) has lost power because an electrician hit the wrong breaker before leaving for the night. During construction every day is an adventure.

Compared to the corporate sector I suppose we are pretty lucky to get such a big chunk of time to work our magic. The down side to this, of course, is that the staff always returns in the fall refreshed and full of demands, while the tech staff has just emerged from eight weeks of exhausting work. If we seem little cranky try to remember that not everyone gets the summer off.

One of my colleagues gave me a newspaper clipping a few years ago which I have pinned up in my cubicle. It’s a photograph of a woman attempting to hold back a rather large group of military types; my guess is that seconds after the photo was taken she was trampled by this angry mob. He cut out the picture and gave it to me because he said it reminded him of what my job must be like.

The biggest battle on this job is, without a doubt, the whole issue of content filtering which is required by law for a school district. You can almost hear the screams about censorship from there, can’t you? I get angry emails daily from teachers who insist that I am hampering their freedom and inhibiting learning. The irony in all of this is that my only criteria for whether or not I “whitelist” a website is how much impact it will have on the network.

About 18 months ago I had to turn off the streaming audio and video category on our filter. We were plagued by phone calls from people whining about how slow the internet was, so I contacted the wide area guys at BOCES who provided me with a nifty little breakdown of where my bandwidth was going. I was astounded to see how much of the pipe was being used by “junk” traffic like streaming radio, so I made the decision to turn it off. My reasoning was that adding sites to an allow list is way easier than tracking down the offending user, and trying to explain the concept of finite bandwidth to a teacher is like nailing jello to a tree.

The internet slowdowns stopped, but I got a lot of hate mail from teachers who claim I destroyed their “teachable moments” (for those of you who do not work in education, a “teachable moment” is French for “I’m too lazy to do a lesson plan”). My rebuttal is that I am happy to add a site to the allow list…all they need to do is provide me with a URL. Most of them send along a link and I accommodate them by giving them same day service. Interestingly enough, the ones who scream the loudest don’t bother sending me anything.

Like most technology issues it is still a work in progress. We have kicked around the idea of using a packet shaper, but I hate throwing hardware at a people problem and at this point I have a sizable white list in place. Packet shapers tend to cause their own little set of issues, and lowering the priority of streaming is still going to dent people who are using it for instructional purposes…which is contrary to what I wanted to accomplish in the first place.

Many years ago when I was hired by BOCES I was told that some day I would need to take a Civil Service test for my position; I had already worked for another BOCES in a Civil Service capacity, so it came as no surprise to me. Years went past and my colleagues and I waited knowing that eventually we would be classified by the Civil Service gods; when the day finally came it was an experience none of us will forget.

Classifying people like us is tough because we do a little of everything. That, coupled with the fact that the folks who staff the Albany County Civil Service Department bring a new meaning to the term incompetent, and you have a recipe for some serious stress. Some of my colleagues have worked for BOCES for over twenty years, but rather than grandfathering them in, they had to take a test in order to keep their jobs. If someone off the street scored higher than they did, they would be out of a job!

There were a lot of other anxiety-producing rules and directives that went along with this classification process. The people reviewing the applications were apparently masochists; one of the programmers’ application for Civil Service status was denied because she had a degree in Mathematics, not Programming. It did not matter to them that she had been programming for us for seven years! Seniority for everyone was set at zero; we were able to keep our years in the retirement system, but as far as Civil Service was concerned our start date is the day we were given permanent status. And after reaching permanent status we had to be on probation for a year.

You would think our administration would be mighty concerned at the prospect of potentially losing a lot of employees, but instead we were greeted by the gallic shrug of management. Most of them are high enough on the food chain to be exempt from tests. Some people did actually get hired by their respective districts or took different positions to avoid taking the chance of losing their jobs.

I am happy to report that I finally reached permanent status as a Network Administrator, but it doesn’t end there. A few weeks ago I found out that my district was planning on posting my job at a higher rate, effectively giving me a raise. Nifty, huh? Except I can’t apply for my own job because the Senior Network Administrator exam I took and passed does not count!

Isn’t Civil Service great?