Resource Discovery and Management: Record Loading

Record loading in Sierra is controlled by a command file called a load profile.  A load profile consists of commands lines for each MARC tag to be loaded.  A command line can apply to one tag or a range of tags, but if a MARC tag doesn't have a corresponding command in the load profile, it won't be loaded.  Most of these command lines are generic and contain no special loading instructions.  It is possible to create a command line that loads certain MARC tags differently.  One change that can be accomplished through a load command is to load an incoming MARC tag into a different MARC tag (loading 260 tags as 264 tags, for example.)  There is also a type of command line that can create a MARC note field, inserting text into the note, without reading a corresponding note field from the incoming record.

In addition, the load profile contains commands that pertain to the entire record set.  These include commands to create item and/or order records attached to the incoming bib record, designating record templates to set default values for created records, and how to apply holdings symbols.  The most important of these global loading commands, though, is whether to create a new bib record, overlay an existing bib record, or reject the record entirely, based on whether or not the record already exists in the database.  There is a complicated set of commands for controlling record overlay.  These include whether or not to overlay records, what fields to protect from overlay if they should not be changed by the loader, and what field or fields to use as a match point when determining whether a record should be overlaid or not.  One major limitation to the overlay functionality is that only indexed fields can be used as overlay points.  For example, if a vendor-specific identifier is moved from the 001 (usually indexed) to an 035 or 9xx field (usually not indexed), that vendor identifier can no longer be used as a match point for record overlay.

The load profile also contains a unique command to handle the 001 MARC tag.  This command line is designed to work with OCLC numbers as the unique identifiers in records.  OCLC numbers are prefixed with a set of letters: "ocn", "ocm" etc.  The 001 command line strips these prefixes off the OCLC number, only loading the numerical part.  This is significant, because vendor records may also contain a unique identifier in the 001, consisting of numbers with an alphabetic prefix, and if the prefix is stripped the loader may incorrectly match that record to another record with an OCLC number, and overlay the mismatched record.  Handling vendor identifiers in the 001 is a common reason for creating a custom load profile for vendor record sets.

Loading in Sierra is all or nothing; there is no way to load records conditionally.  A load profile has no capability for selective loading based on the contents of a record or the existence or nonexistence of particular fields.  A load profile assumes that the records in a record set are consistent and should be loaded the same way.

Record sets are provided by different vendors, either on a regular basis or when they make a new resource available.  For example, Serials Solutions creates record sets monthly; Marcive provides shipping list records weekly, full records monthly, and ERIC records on an irregular schedule.  These sets can vary considerably in quality, but unless there are issues that follow a pattern (and so can be corrected using a bulk update process, either in MARCEdit or a custom load profile) we accept them as-is.

Often when we enter into a contract for record sets with a new vendor, a custom load profile is required that matches the unique characteristics of these record sets.  Using Marcive as an example again, the weekly shipping list records have a unique identifier that must be accounted for in the load profile.  The monthly full records overlay shipping list records using a particular set of fields to identify a match, and that matching mechanism must also be accounted for in the load profile.  (There are two different load profiles involved, one for the shipping list records and a different one for the full records.)

Vendors usually notify Tech Services via email when a new record set is available.  Sometimes the vendor provides a direct link to the record set file in the email, but usually we have to download the record sets using an FTP client.  Several good FTP clients exist, among them WS_FTP and the Firefox extension FireFTP.  They all work similarly: the client connects to the vendor's FTP site, and the user selects which files to download and where to put them.  The vendor's FTP location and login information has to be configured before the connection can be made and files downloaded.  The vendor provides this information as part of the initial setup for record retrieval.  Campus IT can help install and configure an FTP client if necessary.

We receive record sets from quite a few vendors, so it is easiest to create a local folder to hold record sets that have not yet been loaded.  Once the record set has been loaded, it can be moved out of this staging folder to L:\Technical Services\Load profiling, where it is kept for a few months in case questions or issues arise, then deleted.

Select "Data Exchange" from the Sierra "Function" dropdown.  You will see a button bar, a "Select Process" dropdown, and a list of uploaded files.

In the "Select Process" dropdown, choose the type of load you want to do.  There are entries for bib record loads, authority record loads, patron loads, and a few profiles for pulling records out of Sierra.  Our custom load profiles are accessed by selecting "Load records via locally-created load profiles (local)".

In the button bar under the "Function" dropdown, a button is active only if it is possible to perform its function in the current context.  When no file is selected, the only active buttons are "Get FTS" and "Get PC".  Click "Get PC" to upload a record set to Sierra. Navigate to the folder where the record set file resides, and select it.  Click the "Upload" button, and click "OK" in the next popup that appears.  (This popup appends the suffix ".lfts" to the file as it is uploaded; this is the only suffix Data Exchange recognizes as valid files for the "Prep" function.)  The file, with the ".lfts" suffix appended, will appear in the file list.

Select the ".lfts" file; the "Prep" button will activate.  Click the "Prep" button to create a ".lmarc" file with the same name as the file you just uploaded.  As with ".lfts", ".lmarc" is the only file type Data Exchange recognizes as a valid loadable file.

Select the ".lmarc" file, and the "Load" button will activate.  Click the "Load" button.  If you are loading using a local profile, you wll be presented with a list of local loaders; click on the loader you want to use, and a load screen will appear.  If you are using another loader, not a local profile, you will be taken directly to the load screen.  The load screen shows the name of the loader and the input file, the start and stop block, and the maximum number of records to load.  The maximum is automatically set to 1,000,000, which should be sufficient to load all the records in the file.  In some cases, you may want to load only a few records from a file (when testing a new load profile, for example) and this can be controlled by decreasing the number of records to load.  There is also a check box labeled "Use Review Files"; checking this box will create review files of records inserted and records overlaid (two separate files) during the load.  These files can be retrieved in the "Create Lists" function for further review or processing.

Once all the options are set to your liking, click the "Load" button at the bottom of this screen.  The load will begin, and you will see a popup with a status bar which fills as the load proceeds.  A series of notification messages, one for each record loaded, also scrolls by in the background behind the popup.

When the load completes, a set of statistics displays in the load screen.  This shows how many records of each type were created or overlaid.  These statistics are also written to a log file, which can be accessed from the file list; it will have the same name as the file loaded, with the suffix ".errlog".  Click the "Close" button at the top of the load screen to close it and return to the file list.

Other buttons that appear on the button bar are "View", "Copy", and "Delete".  "View" can be used to check the ".errlog" file, or to look at the records in either the ".ltfs" or ".lmarc" files.  It includes a useful search function and a field count function.  "Delete" should be used to delete files after they have been loaded and any testing or post-load changes are done.  "Copy" copies an uploaded file to a specified location.

Due to the limitations of the Sierra loading technology and load profiles, it is sometimes necessary to modify records prior to loading them, after they are loaded, or both.  Pre-load changes are made using MARCEdit; post-load changes are done using Rapid Update, Global Update, or record-by-record manual updates within Sierra.  The specific changes to be made, and whether to make them before loading or within Sierra after loading, need to be determined as part of the planning process for receiving and loading a new record set. 

The Alliance Demand-Driven Acquisitions (DDA) project is an example where pre-load changes, post-load changes, and a custom load profile are all needed.  The project requires two record sets: one of discovery records from two DDA vendors, and another of fully cataloged records for purchased titles, cataloged by one of the participating libraries.  A custom load profile for the discovery records performs special handling of the vendor identifier in the 001, inserts several notes required by the project, and creates an item record based on a project-specific template.  The purchase records are loaded using a generic profile, but several local customizations are applied before loading, and the location and material type are updated after loading.  In addition, the purchase records can't be overlaid onto their corresponding discovery records automatically, so a manual process is required to mark the now obsolete discovery records for deletion.