Converting

Converting data to your new system

• What happens
• Data that will assist the vendor
• Data Clean Up

Introduction

Data conversion can be one of the most difficult of all tasks and is frequently the reason for project cost overruns and schedule slippages.

When choosing new collections management software we recommend that you allow 2-3 months for your new system and your old system to operate in parallel. Using this approach allows for both the old and new systems to be operated simultaneously for some time. Running systems simultaneously will give time for data and system checking before the old system is shut off. It will ensure that all major problems in the new system have been solved before the old system is no longer available. Obviously, this strategy minimizes the risk of major flaws in the new system causing irreparable harm to your institution.

An abrupt transition to a new system creates many risks because there may still be problems that won’t be uncovered until the system has been in operation for at least 2-3 months. Also, there will be no verification process to ensure that data was transferred to your new system successfully.

What happens during system conversion

During the process of converting to a new system, a number of steps must be undertaken. They include:

1. Mapping data

Work with the vendor to develop a procedure for mapping data from your old system to your new system. Each of the vendors whose software was evaluated during the Collections Management Software Review process was given copies of the CHIN data dictionaries. They are, therefore, aware of the fields available through CHIN.

During the mapping process old fields are matched with fields in the new software. You must, therefore, know what fields are available in the new software. This information can be obtained from the vendor through their data dictionary. If you choose a development application for your new software be aware that they do not usually come with a data dictionary. This can create a more lengthy process as the data dictionary for development applications is defined according to your individual needs.

Data dictionaries are usually broken down into entities and elements (or fields). For example:

Entity:

• Title

Element (field):

• Title of Work
• Original Title
• Title Variation etc.

If the software has a relational structure, the entities would be known as tables and the elements (fields) would occur within these tables.

Taking into account the fields that are available within your new software, the old fields are then mapped to or matched with the appropriate new fields. During the mapping process, the vendor and the user must work together to identify new fields, indexing, data structure and field lengths that were not in the old system. Below is a sample chart for mapping.

• Request a copy of the mapping process from your vendor to ensure that all old fields are mapped to the new software.
• Check the field lengths on the mapping document to ensure that data is not truncated.
• Ask the vendor for audit trails of the importing procedure including record counts, fields and the number of characters (bytes) imported into your new system.
• Check random records within the new software to compare old fields to the new data fields and ensure data integrity.

2. Run the system in a test environment using your data

Produce a sample of test data from your old system and work with the vendor to load the data into your new software.

During this process, look at how well the product suits your needs and ensure that all criteria identified in your criteria checklist are met. During this process you should look for errors and omissions when comparing the system you received to the one outlined in your criteria checklist and Request For Proposal (RFP) specifications.

3. Test system performance and response time using all of the data from your old system

Test the system to verify that it works well with a normal workload. If the system does not perform well when working with all your data and under normal circumstances, then some programs may have to be rewritten to improve efficiency, or processing hardware may have to be replaced or upgraded.

4. Test backup and recovery procedures

At this point you will have full-sized computer files and databases with real data. It is crucial to test your backup and recovery procedures. Don’t wait until the first disaster to find an error in these procedures.

• Make sure your database is backed up, then simulate the loss of some data by deleting a test file.
• Test the time required to recover the file.
• Do a before and after comparison of the data to ensure that data was properly recovered.

5. Audit testing

Audit testing certifies that the system is free of errors and is ready to be placed into operation. It is important to monitor audit trails on the movement of data from your old system to your new system in order to ensure consistent record counts and file sizes. Have the people who will use the software on a daily basis do the testing.

When converting data from your old system to your new system:

• Keep records on the number of records and the number of characters that have been downloaded.
• Record the number of records and characters transmitted during this process.
• Compare the figures on records and characters downloaded to those in the new system to ensure data integrity.
• Take a snapshot of data from your current system just before you move to the new system and archive that information to safeguard against problems with the new system.
• Check random records in the new software to compare old fields to the new data fields and ensure data integrity.

Data that will assist the vendor

During CHIN member museums’ move to in-house collections management systems, CHIN provided data from our system in order to ensure a smooth transition to the new system. You will also need to supply your vendor with similar information. This will allow vendors to determine space requirements, mapping requirements, etc.

Be aware that this process will require time and other resources. Depending on your current system you may require the assistance of a database administrator or someone with intimate knowledge of the system to produce the type of statistics required.

Even when moving from a manual to an automated system, similar types of data will be required. You will be required to estimate field lengths and numbers of records that will be required in your new automated system. Taking averages and creating estimates can cause some risks in terms of accuracy but at least give the vendor some guidelines.

Database Study

A database study is very useful to help you and the vendor identify a number of important factors when mapping your data to a new system.

A database study will provide the following information:

• Field mnemonic or short form (i.e. AR)
• Field label (i.e. Artist/Maker)
• Field number (i.e. 70)
• The number of times that the field contains data (i.e. there are 4,300 records in this database that contain the field AR).
• The percentage of time that the field contains data (i.e. the field AR is present in 12.98% of the total 33,128 records).
• The minimum number of characters that are entered in the field. (i.e. 1)
• The maximum number of characters that are entered in the field. (i.e. 109)
• The total number of characters entered in the field. (i.e. 66,313)
• The average number of characters in the field based on the total characters in the field. (i.e. 15.42 characters)
• Percent of DB - The percentage of the database represented by the content of that field. (i.e. .27%) 

You may also want to include a summary of the entire database that identifies:

• The total number of records in the database. (i.e. 33,128)
• The total number of records selected for the database report (this is usually the entire database, however database studies can be run on department or a unique field within the database). (i.e. 33,128)
• The percentage of records selected for the report.
• The total number of characters in the database selected for the study not including field mnemonics. (i.e 24,622,580)
• The average selected record length in characters. (i.e. 743.26)

Activated Fields Report

In order for you and the vendor to properly map data to your new system you will need to outline the fields in your current database and how they are indexed. This information is extremely valuable to you and your vendor for the purpose of comparing indexing in your old system with the type of indexing in the new software.

Frequency Lists

Another example of data that may be useful to vendors is found in a frequency list. This provides a one column list of terms entered in a specified field. For example, a frequency list on Artist Province would provide a list of all the data in this field and the number of times that data occurred in the database.

Frequency

Frequency lists are also useful for data clean-up. When you request a frequency list on a field such as Artist Province you will be able to review all the Artist Provinces in your database including various spellings, typos, etc.

Data Clean Up

Cleaning up data for conversion is a very important step in the process. There are essentially three opportunities for data clean up.

• When you produce frequency lists and database studies you will find where you have duplicate records, typos, etc. which can be cleaned up by you before the conversion process.
• Between the process of downloading your data from your old system and uploading into your new system many vendors will assist you with a clean up process particularly if moving from a flat file system into a relational system.
• After loading to your new system you will want to continue the clean up process. Since these new systems will likely be more user friendly, the process of cleaning data should be easier.

Go to the exercise for this module.