How to Manage the IT Support Desk

Last week I went on a 2-day UCISA workshop / course led by Noel Bruton looking at various topics associated with managing an IT support desk. The event had been recommended by colleagues and relates very closely to the work we are doing in IT Services to establish a consolidated service desk.

Reorganisation:- We were a group of 16 delegates, representing 14 HEIs including St Andrew’s, Huddersfield, Queens College Dublin, St John’s York, and Queen Mary’s London. Interestingly the majority of us had recently been though or were in the middle of some sort of reorganisation, either across IT as a whole or just for the service desk. One of those most radical transformations was the pending breakdown of a team of 33 field engineers to put 12 of them on the 1L help desk and lose 10 of them to save costs while preserving overall service levels. If only we had a similar army to work with here!

Help Desk FTE:- By way of introduction we each provided approximate figures for the number of 1L and 2L (desktop support) staff, and the size of our user base (staff/students counted separately). Ratios of support staff to users varied considerably, but Oxford clearly has one of the largest supported user bases combined with a rather unclear idea of help desk FTE given the devolved nature of IT support. The University of the Arts (London) has 8FTE 1L analysts plus a manager and 33 FTE field (2L) engineers serving 25,000 students. They noted that as they are split over 6 colleges and 16 sites these numbers need to be higher than elsewhere, however they plan to reduce the 2L contingent to 12FTE by expanding the 1L team and training them up.

ITIL isn’t IT Service Management:- Somehow we got talking about ITIL. Noel has a particular view on this. There is very little in ITIL that can’t be applied to other service disciplines, so it’s not really specific to IT. Service is the smile you get when provided with a cup of coffee in a hotel – you wanted the product but you got something intangible along with it; ITIL doesn’t talk about “service” in this sense, but rather about “services”. Finally, management is about the effective combination of people, process, and technology, but ITIL only talks about process so it can’t be management in any meaningful sense.

Very few people seem to have quite such strong anti-ITIL feelings – I suspect that in his consultancy role Noel has come across people who have engaged him to fix their service desk…which is fully ITIL compatible but clueless when it comes to dealing with actual IT users and issues.

A Brief Summary of Everything Help Desk:- Next we took a whirlwind tour of the help desk, directed largely by following up on particular problems or experiences within our own areas.

Some basic statistics: users typically make 11 – 50 calls to the help desk each year with a typical volume for a university being 16 calls a year; an average 1L analyst can handle 30-40 calls a day; a typical 2L resolver will resolve 5 – 8 issues per day.

It is absolutely critical that all calls are logged. This provides not only for measuring and reporting on current call volumes and performance, but also for identifying potential improvements such as call types that could be processed faster if an automated diagnostic tool was available.

Groups vs Teams:- It’s obvious that a team is more than just a group of individuals, but what is the crux of that difference? Each member of a team has a specific skillset and follows an agreed protocol to receive a workload, execute their part of the process, and then transfer the workload on to the next person in the team via an agreed protocol in order to deliver the outcome. Each team player adds value and is responsible for delivery of their activities, even though they actually perform the activities alone.

Mazlow, Demming, McGregor:- No management course would be complete without name dropping and a few potted theories of how people work. One discussion ran off into an unusual corner though – whether the top layer of Mazlow’s hierarchy of needs (“self-actualisation”) is truly applicable to everyone. The common definition for this highest level of motivation is realising one’s full potential, self-fulfillment, and peak experience. However, some of us felt that this is a somewhat self-centred pinnacle, and that an alternative exists at this level – an outward purpose whereby the individual is motivated to lose their self in order to benefit a community or other “whole”. We were probably going off on a tangent owing to misunderstanding Mazlow though, as explained in Chad’s sideways thoughts.

Don’t Ask for Permission:- There was a teacher called Lesley who was good at teaching. Lesley set up a business selling teaching. Things went well and there was too much work so Lesley hired a couple of people: Sam to handle the administration and buildings, and Vic to teach into a growing specialist market. Over time the business continued to grow and Sam had to take on a facilities manager (Alex) and an IT person (Chris). In time of course, Chris needed a Windows sysadmin and a network admin. And so on.

The point is that Lesley was a generalist who recruited two specialists who were better at specific parts of the business than she was. In turn, each manager took on specialists to help get their job done better as things grew. Each manager wrote job descriptions that must have been incomplete because they were for a more specialised position that their own.

On this basis (a) don’t ask for permission – your manager hired you because you have the specialist skills that they don’t have, and they expect to trust you and rely on you to come up with the answers (if they could do it then you wouldn’t have been taken on), and (b) hierarchy is an illusion – the relationship between a manager and their staff is one of generalisation vs specialisation.

Demand and Supply in the Support Desk:- We looked at some simple but effective ways of tabulating data to assess support demand (how many calls for which services at various times of day – done using tallies in a table of services vs time of day) and supply (skills matrix – levels of competence in each service area for each member of help desk staff).

As a stand alone item, the skills matrix enables a rapid assessment of which services are well supported, which are short of backup or capacity, which staff are your “gurus”, and which staff are yet to realise their potential through additional training.

By combining these charts it is possible to derive optimal staff rotas for the help desk, ensuring that sufficient qualified staff are on duty just in time for the demand peaks.

First line, second line, end of line:- At this stage we had talked quite a bit about our teams, how things work, and where problems are perceived. We had referred to first line support (1L), second line support (2L), and third line support (3L). Noel stopped us to think about this for a minute.

Support, incident resolution, help desk is all about getting the user back to work within the current operational parameters of the systems involved. Any more than this and you’d raise a change request (ITIL was back in fashion by this point).

So, a call comes in to the help desk. It is picked up and logged. At this stage the help desk analyst either fixes it there and then (First Contact Resolution), or escalates it to a team better placed to resolve it. An escalated call must be picked up by another team who either fix it and contact the user, or re-escalate it to a team better placed to resolve it. This next team picks up an escalated call and either fixes it or re-escalates to another team, and so on.

We have two functions here. One function receives a call from the user and either fixes or escalates – this is the 1L function. The other function receives an escalated call and either fixes or re-escalates – this is the 2L function. There is no other function – there is no such thing as third line support. Of course, re-escalation in the 2L function may involve progressively more “technical” teams operating at successively lower levels in the infrastructure, but they are all carrying out the same function, and any implied hierarchy here is mere technocracy.

Noel suggested one modification to this – a triage team who work alongside 1L to take recognised and well understood problems that would tie up a 1L person for too long. This means that your 1L staff can provide “hot” transfer of calls that they know can be resolved without involving 2L and get to the next incoming calls.

Who Speaks to the User?:- When a call has been escalated to 2L, and they have found a fix, who gets back to the user with the good news? Or alternatively, suppose that 2L have taken the call but need more diagnostics – who contacts the user to request further details?

There are only two options here, and both have their supporters.

Option 1 (call-back) says that it’s the 1L staff who are trained in customer service skills and are therefore best placed to speak to the user. 2L must therefore send their information or requests back to 1L who will then pass it on to the user. The trouble with this is that it incurs delays in the response as it moves between support lines, and can suffer from “chinese whispers”-style losses on the way.

Option 2 (call-through) says that the most efficient thing is for 2L to get straight back to the user and speak directly. This certainly cuts out various delays and risks, but requires 2L teams throughout the organisation to be willing and able to use “customer service skills”, especially those around customer language and empathy. A strong focus on getting users back to work can help this to work.

Service Desk Economics:- Finally we explored the financial impact of a service desk. A simple scenario was developed, based on an (observed) average of 16calls/user/year in an organisation of 1500 users, paying £18k to a 1L analyst and £25k to a 2L analyst.

In the first run we adopted a “classic” model of having a basic 1L capability who could deal with 25% of calls without escalation, and a busy 2L team who average 5 calls resolved each day. 16*1500=24,000calls/year, or around 100calls/day, requiring 3 * 1L analysts to take the calls (based on typical 1L processing rates), of which 75 will be escalated to 2L where 15 analysts are required to process the day’s workload. The total annual cost works out at £429k and the average resolution time is 71mins.

In the second model we imagined training our 1L staff up to resolve 50% of calls without resolution, and focussing our 2L staff on incident resolution so that each of them resolves and average of 8.3 calls per day. This time we still require 3 * 1L analyst, but only 50 calls are put through to 2L where we now only need 6 analysts to get through a day’s work. The total annual cost works out at £204k and the average resolution time is now just 38mins.

So we saved £225k/year and improved resolution times by nearly 50%. Now we can afford to pay our 1L staff £25k and keep training them…

(Much of the content from this workshop is contained in a set of slides published by Noel from an earlier rendition entitled The Future of the HelpDesk).

Advertisements

Easier Software Updates for Windows

It’s one of those fun weekly chores – going through all the software installed on your Windows PC and checking online for updates, downloading them to a USB stick (the bigger ones at least), and then going round each PC in turn to install them. We’ve got 4 PCs to update, and that can’t be uncommon in a modern household.

In the Linux world things are much easier – on Debian you’ll normally get away with running “apt-get update && apt-get upgrade”. If you’re really lazy you’ll configure unattended-upgrades and be done with it.

Windows is a different beast though. You’re on your own. Sure you’ve got Windows / Microsoft Update, Adobe Updaters, Java Auto-update, Apple Software Update, and lots of other tools to help, but how do you remember which updaters you’ve run on which systems? Also, when you (re-)install a machine how do you get all the right packages installed up-front?

Enter WPKG (wpkg.org). This handy tool allows you to wrap any software installer with some XML-based metadata that specifies which PCs on your network need which packages, how to install / upgrade / remove software from a PC using the distributed installer (MSI, EXE, whatever), and version information to enable auto-updating.

To get started you can just download the WPKG server package and unzip it to a network drive (on your home server, right). This sets up a folder structure into which you can add your software installers and the XML data. You’ll need to write three XML files to get going:

  • hosts.xml specifies which hosts should get which software bundles (“profiles”)
  • profiles.xml specifies the software packages comprising each profile
  • packages.xml – or more typically packages/*.xml – specifies how each software package should be managed, with install / upgrade / remove commands, the installer file location, current version number, conditions for installation, architecture/system dependent aspects, and pre- and post- installation commands

Now you can run the wpkg.js script to update your system. There is a handy client package that will install a service to update your system at each login, and you can configure periodic checks using Scheduled Tasks.

Combine that with subscribing to the announce maillists for your main software packages and life is much easier – in most cases you just download the new installers, update the version numbers in packages/*.xml and make the coffee. All your machines will update themselves next time they get used.