A couple of months ago, I was interviewed in my capacity as Systems DBA at Milton Keynes College about some of the challenges that we face as an organisation. You can read the resulting article, Defence Mechanisms, here.

After a nasty attack of Conficker, one of our Windows 2003 boxes refused to start the DHCP service, giving the following message:

Event Type: Error
Event Source: Service Control Manager
Event Category: None
Event ID: 7023
Date: Date
Time: Time
User: N/A
Computer: ServerName
Description: The DHCP Client service terminated with the following error: Access is denied.

The problem appears to be with permissions on a couple of registry keys, namely:

HKLM\SYSTEM\CurrentControlSet\Services\Dhcp
and
HKLM\SYSTEM\CurrentControlSet\Services\Tcpip

To fix the error (comes from Service Control Manager with a code of 7023), set the permissions for the Network Service account on the two keys above to Full Control. Make sure that the permissions are cascaded down to sub-keys by going to Advanced Permissions and ticking the box which states "Replace permission entries on all child objects with entries shown here that apply to child objects".

This should fix the issue

Sometime last week, I wrote that I'd be testing some virtualisation options for our new deployment and reporting back to you with the results... Well, in the interim, we've had an Active Directory death. This is not just a failed server, it's a full on meltdown. We noticed something was wrong when clients started dropping off of the domain. After taking some Wireshark traces, we noticed that there were a lot of DHCP requests floating about. Immediately alarm bells started flashing, as our DHCP box also runs DNS, as well as being the Global Catalog server, the PDC emulator and the FMSO master.

The server was dead - properly FUBAR'd. We're still not 100% sure what caused the failure, but I'm pretty sure it's a person rather than technology - this is what happens when you're forced to give 8000 students admin rights. Take note. If the dead server hadn't been the Global Catalog server then we could have cleared the remnants out of AD and removed it. However, as always happens, it was, which meant that Exchange went screwy too. Cue a restore from tape, which seems to have fixed most of the problems, although the first restore left the server with no Netlogon or Sysvol share! Apart from losing the DHCP reservations database, we appear to be OK, for the moment. Now I just need to prove who did it!

Since the restore, we've had problems with one of our finance apps - the clients keep reporting a SQL Connection Reset and then hundreds of VB Runtime errors occur. Long story short, it's network... If you're a DBA and you see this in your apps, get on to your server and network guys and shout at them - the only way a SQL app should generate this is if something kills the SPID, and of course that wouldn't happen on your server, would it!

I'll be back blogging again regularly soon - the virtualisation project is still live, and I have a review of the lovely Advent One 10" laptop to give you...

TTFN!

How bloody difficult is it to get a patch to install via Windows Update? The latest failure is KB955428, an update that closes a security hole in Works 8. It's one that you want to have... Luckily, the MS support site contains the solution.

Go to the KB955428 support page

Download the patch that corresponds to your version - the language code is on the end of the file name, so mine is EN-US

Extract the installer from the cabinet file using your extractor of choice (gotta be 7-Zip!)

Double click the installer, then think about why, after ten years of trying, Microsoft still can't get this bloody technology working right. Surely installing a security patch correctly is not too much to ask for!

Having come back to my desktop after 3 weeks, as you can imagine, there were a few patches to download. Most installed fine, with the exception of KB947562, which steadfastly refused to play nicely (7 goes at installing and no luck). If you search for this KB on the Microsoft Support Site, you'll see that it's an application compatibility update, which is a nice thing to have! If you're having similar issues with installing it there is an easy solution that gets it installed with a minimum of fuss:

Go to the stand alone installation site (don't worry, it's a Microsoft page).

Validate your copy of Vista.

Download the update (it's about 3.5mb).

Double click it and follow the prompts on screen.

Simple as that!

As a rule of thumb, you can usually find stand alone installers for any of the Windows updates by putting the KB number in the search box on the Microsoft support site - it can save a hell of a lot of hair tearing (trust me!)

As a DBA, I hate Microsoft Access with a passion. It is one of those products that started off with an admirable intent but really should have been retired somewhere around the late 1990s. The main problem is that users create simple little databases for their workgroup to use which grow silently until they become mission critical to entire organisations (trust me, I've seen it happen and it's not pretty). I repeatedly offer to design my users nice distributed databases in SQL Server, and am constantly rebuffed.

Another major headache is when the databases are created with passwords, "for security". They're then not used for a year, which means that the passwords end up forgotten. Luckily, I have this little utility that lets me recover the password with ease. All you need to do is download the zip file by clicking here, then drag the mdb file on to the executable - the password is revealed by magic!

Now all you need to do is get your users to stop using the bloody program in the first place

As you know, I hate anonymous proxies. In my opinion, they are the biggest single threat to network security at the present time. It was therefore with interest that I came across Peacfire.org, a site that is devoted to the circumvention of web filtering software - they even go so far as to provide a circumventor mailing list. Looking through the site, much of the content appears to be sub-standard adolescent whining. What really gets my goat however is the page entitled "Why we do this". Please click on the link and have a read.

As a sysadmin, I'd like to respond to some of the points raised in the article:

I am a libertarian. I believe in freedom of speech and all of that good stuff. However... My network is my responsibility. If it doesn't work, I get shouted at. Therefore, I have to make a decision as to how to protect it. I choose to block categories of sites because they are more likely to contain code that can harm my network. I do this not because I don't want you to look at Facebook. I really don't care. All I care about is the quality of service to my users. Filtering software is a blunt instrument, and it sometimes mis-classifies sites. It is however better than not filtering at all.

I have a 100MB connection to manage. This is probably 50 times faster than your connection at home. However, we have 8000 people using it at once. This is why I stop you using services that aren't work related. I don't care if you're watching videos on Youtube. I do care that by doing so, you're breaking our connection and people will shout at me. That's why I block streaming media.

There are lots of free MP3's on the internet. Feel free to download them at home. I don't care. I have no love myself for the RIAA, the MPAA or any of their ilk. However, if you download copyrighted material on my network, I am responsible for it. If they find out, I'll get a big fine and lose my job. I like my job. This is why I block P2P sites.

I don't care if you're straight, gay, bisexual or any or all of the above. As far as I'm concerned, you can look at what ever you want on the internet. However, other people may see it, and they may be offended by it. They may then sue us, which means I'll probably get fired for not stopping them seeing what you've accessed. As I've said, I like my job. That's why I block porn.

Can you see a pattern here? I don't care what you do. If you want to download thousands of MP3's and watch hardcore porn, go ahead. Do it at home. I'll support your right to do it. You know why? Because that's YOUR computer and YOUR internet connection, and YOU can decide what happens with it. When you're here, you're on MY network and I make the rules...

The most important thing about deploying updates to your live network is thoroughly testing them on your test network. Sounds obvious, doesn't it? You'd be amazed at how many sysadmins I talk to who don't have a test network. In years gone by, a test bed used to be a luxury that only the biggest of companies could afford. With decent virtualisation now available for free, it doesn't have to be a wallet busting exercise.

I started with an old SQL box, whose contents had been moved to our blade centre. The box is an HP rackmount with 4GB RAM, a dual core Xeon, and 500GB of storage. It's a decent spec, but to be honest, you can get away with a desktop PC with a similar spec - Dell will do you one with enough grunt for about £400.

The next step is to install the host OS. At this point in time, the sensible option is Windows 2003. I don't know of anyone with a live Windows 2008 deployment, so it's best to stick with what you know. It's also wise to stick the machine in its own Workgroup - you don't want any tentacles from your test network extending in to your live one...

The configuration of the box is down to you - from my own experience, putting the virtual hard drives on a RAID5 array is just common sense, so I've got a 400GB partition for this, with the rest for the host OS.

It's then up to you to choose your virtual host provider - VMWare or VirtualPC. Either are perfectly acceptable alternatives, but again on a personal level, I prefer VirtualPC. Once you've installed it (you can download it here) you can start installing your clients.

Obviously, if you're running a test environment you'll want to mirror your live network as perfectly as possible. The easiest way to do this is to take a backup of your DC's and application servers and restore them as virtual machines. You'll also want to put at least one of every client that you support in there too. It's important to remember though that you don't need 500 clients in your test network if you have 500 in your live one - traffic patterns can be extrapolated from a few machines. All you're looking at is testing proof of concept ideas and making sure that changes to your schema or similar don't have unforseen consequences.

The final big thing to check is your VM's network settings. ALL of their network adapters should be set to Local Only.

This stops your test machines interacting with your live setup - if you need to give them internet access, do it via a virtualised instance of ISA Server or similar, to ensure that there is no communication from test to live. Trust me - it's difficult enough dealing with your users problems without having to deal with ones that propagate over from your testbed!

And that's all there is to it! Once you've set this up (you can do it in a day if you concentrate) you'll wonder how you worked without it. It's invaluable for staff training - if they make a mistake, restore the VM. I use ours for showing our 2nd line guys the ins and outs of Group Policies, and it's always fun watching malware propagate around your network when you know you can restore it with the flick of a switch! Again, this is why you keep it seperate from the real world - you don't want any nasties escaping from it... It gives you the chance to throw some heavyweight tools around, safe in the knowledge that you can always reset. You can also try out some bizarre configurations too - try getting Windows for Workgroups talking to Ubuntu Server

I can't stress how important this is - sell it to whomever you have to, just make sure they pay for it, and soon!

As you're probably aware, I lie awake most nights worrying about threats to my network. The life of a sysadmin is a constant arms race with people who want to break your kit. It's therefore pleaseing to find a site such as ntsecurity.nu. Before clicking on the link, let me caution you that if you are working in a monitored environment, it may be flagged as a hacking site. It's not - it's a computer research site, which is completely different. You'll not have any nasties installed on your machine by visiting it, and all downloads are labelled with exactly what they do. Some of the downloads do nasty things, and will probably get flagged by your antivirus or antispyware programs. Be warned!

The site its self is run by Arne Vidstrom, a researcher at the Swedish Defence Research Agency. He has a huge list of publications, and has coded all of the tools on the site himself. It's worth looking through the tools section especially, as this will give you an idea of some of the things that my be used against your network. I'd suggest getting intimately acquainted with everything on this page! By using these tools on your TEST NETWORK, you can get a real feel for the way they may be used against you...

If you've been wondering where we've been for the last couple of days, the answer is quite simple. I broke the server.

As you may know, I've been on a Sharepoint developers course for the last week. Staying in a hotel away from home is boring as hell, so I decided to do some housekeeping and general tidying of my web server. I'm currently with Jumpline, who provide a complete virtualised box that included its own version of Apache, mySQL and so on. Connecting to it via FTP, I merrily started deleting files. I probably should have checked what I was actually deleting, because by the time I realised my mistake, half of the /etc path had been deleted... Things then went from bad to worse as the FTP connection dropped and (obviously) wouldn't allow me to reconnect. In desperation, I restarted the server, which then died completely.

I've spent most of today getting my box back to where it should be - we restored the VM from the day before and everything seems to have come back OK. I'm now $50 lighter and a lot more forgiving of users who fail to back up!

What did impress me though was the ease with which I could bring back a Joomla installation, even when the file system is corrupt. One of the sites I host was properly broken by the crash and was having problems after the restore (it uses a lot of externally hosted code). To get it back literally took 10 minutes - I installed a new instance of Joomla in another directory, repointed that at the old database, uploaded the site template, and away we go!