I have received some correspondence about a massive snow job at Waikato DHB. On Thursday NZPA reported that a major computer virus outbreak had essentially shutdown all of the DHB’s 3000 plus computers. On Friday it was reported that the DHB was now restarting all their PC’s after the incident, so far I haven’t heard how it all went.
My correspondent is a Senior IT person and is livid at what has gone on. The explanations given by spokes people at the DHB are laughable in the face of even basic knowledge of major IT systems. Anyway here is their take on issue, I couldn’t have written it better myself and I think a few judicious OIA requests may go in after the dust settles.
The problem with this incident in the Waikato is that it’s an identical incident to that of the MOH last year and must be indicative of a truly bizarre display of barrel scraping IT Management skills. Despite what is said in the last part of the first article this is in no way caused by a lack of complex passwords nor will implementing them actually help prevent further outbreaks of similar worm or malware type attacks.
The point of infection must have been introduced by some numpty on an infected USB key and given that it was during some “systems upgrade” in the early hours of the morning, that person was in IT. However the root cause which they’ll be trying hard to hide is that some utter utter fool allowed 3000+ PC’s to not have installed two very simple antivirus and Microsoft operating system patches that have been available for over 15 months.
“Retard” in the Paul Henry sense does not even begin to describe the managers responsible for letting this happen. As if the MOH cluster f*ck this time last year wasn’t enough of a paint-by-numbers example of what not to do they’ve reproduced it perfectly at WDHB. Those responsible at MOH got given the arse as soon as it was tidied up and I will expect the same again here.
But the loss to the public and the greater Waikato region is that grandma’s hip op is going to have to wait another month or so to happen and Uncle Jimmy’s hernia op may get cancelled due to the cost of the cleanup of this coming from somewhere. MOH’s episode got buried in PR spin and legal threats to staff to never speak of it again. I’d rather this one get a bit more visibility.
Also don’t read much into the “Microsoft called into help” bit, they’re not to blame, they’re there to help as part of the unique opportunity for them to negotiate new licensing and support terms for the next decade at exorbitant prices. WDHB will be paying whatever MS ask for just so they can try and pass the blame by throwing the word Microsoft together with worm or security exploit in the media for the public to join the dots to for a diversion.
Like the MOH debacle I’d say they’re in lockdown facing disciplinary action if they comment on anything. They must host their own website (as it’s utterly dead) so at this stage all we can go on is what is in the initial news report unless their comms team starts answering media enquiries.
From this you can assume that A) they have an XP desktop environment which had conficker patches release by MS in Oct 08 and the major AV companies in Nov 08, that B) the benefits of their ”Connected Health Network for Waikato project” haven’t quite played out as the small satellite health offices would have expected and that C) Alan Grainer will be having quite a different Christmas holiday than planned.
Maybe worth finding out how the average health clinic in Huntly is getting on with their PC’s if they’re part of the DHB WAN. They’ll be waaaay down the priority list yet probably able to do nothing today if the infection was not stopped at the local DHB HQ subnets.
The CIO seems to have previously been a Programme Manager at Unisys and a variety of other roles so theoretically should know his arse from his elbow. Although there’s a coincidentally
unfortunate linking of him to Alan Hesketh CIO at the MOH here on page 12\13 of this. I’d say they’ll certainly be on the phone together sharing tips on things other than leadership today, perhaps virus cleaning, PR strategy and CV updating.
Anyway my rage is aimed at criminal incompetence in letting their site be so vulnerable, the process of infection and how it is actually translated to the public via the media. Picking out statements from the Stuff and Herald articles so far:
“Waikato District Health Board has been crippled by a computer worm which has seen every PC in the organisation shut down”
Ok so over 3000+ PC’s are either infected or at risk of infection. This means that identical to the MOH debacle in Dec last year, they have for some reason, either through oversight or genius leadership choice, not deployed a basic Windows update that was available in October 2008. This patch was one that was flagged by MS and all major vendors at the time as a must have due to possible exploit, not to mention the first 6 months of this year where we had Conficker paranoia frenzy in the media. Even my grandmother was asking if her phone needing patching to prevent “these conficky worms”.
This patch is easily centrally deployable and took us all of 15 mins through automated policy at windows startup to get on every PC in a large PC environment with hundreds of PCs. With a few weeks of network scans double checking every PC in the place to ensure they successfully received and installed it. It’s a regular standard part of IT, this is not unusual, this is not hard, this does not cost additional taxpayer money, it is core operational work.
“Ms Gill said DHB technicians were working on a computer upgrade overnight when things started to go awry.”
Go awry??? Upgrading your HRIS system and then noticing your clever use of an infected USB key has spread a malicious worm throughout your 3000 PC network is not what I’d term “going awry”. Thats what I’d term as brown trouser material and I’d have serious thoughts about a sudden new life of sustainable living in the Urewera’s…
“We brought in Microsoft and have been working with them through the night.”
Really! Bringing them in is of no real use for fixing this. Any vendor like Gen-i, Datacom or Axon could assist just as well and likely better given that there isn’t a huge MS presence on the ground and each of those have sizeable presences in the Tron. All Microsoft are involved for is as a PR stunt. Being able to hint at the ” Don’t worry we’ve called in the pro’s”, “its a microsoft security hole so we called them to sort it etc etc” type connotations.
All MS are going to do is sign them up to a long term and expensive support agreement as a pre-requisite to take part in this shambles. Some form of OIA to MOH should show up the various MS deals that were signed there during and after that shambles as they did the exact same thing. It’s not MS’s fault and they’ll accept no blame but they’ll put on a helpful display for it and make out like a bandit at the WHDB for a long time to come for this.
Conficka has been identified as the culprit.
Well yes and no. A stingray was the culprit but Steve Irwin giving it a hug was more the reason.
“It reconstitutes itself as fast as you can fix it. It’s particularly virulent,” Ms Gill said.
It’s malicious SW, that’s what it does, that’s not new, that’s not unexpected. You don’t clean a virus and then not expect it to return. You have to patch the hole it came through then you clean it. And the second problem with all this is the
“small pamphlet explaining the problem had been printed off-site “and people are running around distributing them” at the hospitals, Ms Gill said.”
Fantastic result if you are unable to clean and connect a small set of PC’s and a printer to manage this task then the liklihood of things being operational by days end is lets just say optimistic….
“Ms Gill said the shutdown would probably result in an “even more robust password system” being introduced.”
Huh?? Brain explosion here. If I try and think this through they’re suggesting that the virus itself is cracking their authentication so they should increase the complexity of the passwords to make that task more difficult for the virus. Not, that perhaps they should ensure the equipment is patched to prevent the hole the virus used to propagate in the first place or address the no brainer requirement that the Antivirus SW should be up to date so as to be able to clean the virus from any location it has spread to…..
“It has millions of computers now under it control in more than 200 countries, according to the New York Times”
I’m assuming the Herald etc added this as that’s somewhat ridiculous. I would not be surprised if my porn surfing cousins PC became infected by conficker, I would however be concerned if my own home PC did and I am in a state of bubbling rage that any large government organisation could be in the situation that the WDHB finds itself.
I have been an IT Manager for quite a while and lurk in the public sector currently. There’s good and there’s bad, but there’s always several key things that you know will get you fired (and so they should!). Not mitigating risk by not patching your gear, be it pc’s, servers whatever, is one of these. It requires several levels of decision making and process failure and I am very sick of it being made out by the organisations concerned as “just one of those things” and an accepted part of IT. Heads should roll and sector wide audits need to happen. Audit NZ gets involved in the financial aspects of our organisations, why isn’t DIA (which seems to house the all-of-govt IT initiatives now) tasked with preventing these events and pinpointing the failures when they do so that accountability is upheld.
If it goes like the MOH there’ll be a further series of bollocks press releases creatively interpreting the situation. Like the MOH debacle I’d say they’re in lockdown facing disciplinary action if they comment on anything.
Regards,
Agent “X” in the public sector
Global sales of PCs fell 14% in the first three months of the year, the biggest fall since research firm IDC started tracking the industry in 1994.
Once again Labour Policy not properly costed ?