Not getting it right

Recent research from Bloor, which provides independent analysis for the IT sector, is worrying. It shows that the current success rate for the data migration portion of projects (that is, those that were delivered on time and to budget) is just 16 per cent. In total, 64 per cent of data migration projects are delivered late, and 37 per cent are over budget.

“One of the challenges with data migration and the reason why they often run late is that the challenges are not well perceived until you’re in the middle of the migration,” says Paul Hollingsworth, director of product marketing at Celona Technologies. “People don’t apply the right skills and resources to it and underestimate the time and cost involved.”

Although most charities operate a centralised database of contacts, there may be numerous duplicate or alternative versions of the data on individual computers. Before beginning a migration project, staff should be encouraged to reveal any personal data sets they are working from. Otherwise a false or incomplete version of the charity’s records will be transferred to the new system, undermining any improved functionality.

Care must also be taken not to merge documents containing permanent and temporary data: determining what is important from what is not could prove difficult otherwise.

“Often when the charity starts the project, they will say to the supplier that’s running it, ‘this is our data’ without realising that they’ve only actually supplied one snapshot of it,” says John Bird, managing director of ESiT. “The project goes ahead, then when they look at the data at the end they think there’s been a problem with the migration.”

When the correct data has been gathered together, it is mapped to the new database field by field; a process that is normally undertaken by the software supplier. This provides a design for data extraction and data loading. The design relates old data formats to the new system’s formats and requirements.

Over time errors will creep into even the best maintained databases, surviving from migration to migration like imperfections in a photocopy, until a thorough clean-up of the data is undertaken. Getting rid of inaccurate addresses or people who have cancelled their donations, for example, will mean you get the most from your new system.

Vendor support for database systems is often limited to their own suite of products. As a result, inexperienced IT managers could end up having to deal with two incompatible systems with which they have little or no experience. The source data must be in the correct format for use on the new system.

“You may transfer a whole load of data across and suddenly discover a country or region code which the recipient system has to have, but that you haven’t got in the old database,” says John Tate, IT advisor to the Charity Finance Directors Group.

How the software presents the information may also differ. For example, you may have some kind of donation summary for how much a donor has donated in any given year; one piece of software might include Gift Aid and one might not. “The data behind the scenes is right, but the charity looking at it will think these two pieces of information are different and assume that the data transfer has gone wrong,” says ESiT’s Bird.

After loading into the new system, a process of data verification aims to make sure all information has been accurately translated, is complete, and supports processes in the new system.

Top

Case in point

WWF UK moved from a bespoke AS400-based database to CS Group’s CARE database in February 2004. The old database had been tailored to the charity’s processes over a 14 year period and stored 30 years’ worth of fundraising information.

“It was critical to us that we successfully converted over this wealth of historic data as it underpins our ability to carry out detailed marketing analysis,” says Andrew Lockett, head of data analysis and insight at WWF UK.

“The conversion was a long and painstaking process. It involved a detailed documentation of the existing AS400 database whose evolved data structures were quite labyrinthine. We then needed to map these data structures over to the new CARE database’s relational structure. Finally we were in a position to make extracts from the AS400 database and write scripts to import them into CARE.”

Lockett believes the full involvement of all parties was key to the success of the project. He also says a good relationship with the existing database provider was instrumental in getting the required documentation and identifying nuances in the historic data. A core team of WWF UK staff made sure the data made sense once converted; and CS Group technical consultants ensured the scripts to move the data were robust.

“Needless to say, the migration process was not entirely smooth and involved many trial runs and management of errors,” says Lockett. “With a project of this complexity we were aware this would likely be the case and hence planned sufficient time to manage these trial runs and carefully document required modifications.”

A team of staff worked round the clock on the final live load of data onto the new system. The high level of preparation up to that point meant it was completed over a long weekend.

“There were some teething problems and internal process changes required to fully implement the new database. However, I can personally attest to the value of having made the painstaking effort to ensure full conversion of historic data,” says Lockett. “We now have a fantastic wealth of fundraising information on which to base our analysis and draw out insights into the long-term trends in supporter behaviour.”

In late 2006, the Foundation and Friends of the Royal Botanic Gardens made the decision to replace their existing database software with ESiT’s thankQ system. Nearly 350,000 contacts were migrated during the process.

“We didn’t actually look at it as a data migration project,” says Jean Waller, database manager at the Foundation. “We were just looking at what the best product was going to be at the end of the day, and the data migration was as a result of that. We had reservations about the migration because we knew that the Visual Alms data structure was quite complex, which proved founded really, but we never thought it was going to be an easy job.”

The Foundation’s legacy appeal data was all migrated with relative ease, but the membership data proved to be trickier.

“We have several different categories of membership, so people had moved around in the past,” says Waller. “With donations you can just bring the data across, it’s fairly straightforward, but with memberships it’s different. If somebody hasn’t renewed their membership immediately in the past, but three months later instead, there’s a gap there which had to be reflected in the migration.”

She advises other charities undertaking a migration project to build a good working relationship with the supplier of their new system. “Make sure it’s very iterative and the communication lines are open,” she says. “You need to work together as a team.”

Charities should also seek to involve all staff who work with the database in the project, particularly those with detailed knowledge of the data as their ability to spot inconsistencies can prove invaluable.

“You really need someone who can oversee the whole project,” says Tate. “That’s a challenge because a data migration can affect many different departments. You have to put a project team together and you have to be very clear on the ownership of the different sets of data.”

Running some scenarios before going live with the new system is well worthwhile. As Tate says: “Check, check and check again at every stage of the process. If you’re going to send out your first donor campaign, don’t send it out to everyone, send it out to the first ten donors and go through the results thoroughly.”

Top

To return to the April 08 features list click here