ITS Serials Solutions procedures

• -deletes
• -updates
• -and new records

NOTE: the update feature of catalogload was NOT used since LOCAL tags like 856 would not be overlaid but would build up with each overlay. The idea of a separate format with NO local tags, just for serials solutions records was rejected. Even with overlay feature, one would have to remove the OLD copy and create a NEW copy for the record since the 949 call number etc. may be updated.

Emory gets a replacement file each time from serials solutions, this may be an option for us to explore if the incremental files become too problematic and complex in time.

Timing of the Load (adutext) / Deletes (adutext)

• LOAD on Friday via crontab at later hour-like 22:00--duplicate records will be in system for the night and all day Saturday which will cause RELAIS to get "001 not unique" error messages for those duplicate records

• DELETE the old matched records via crontab on Saturday night- RELAIS will get error message for "Unable to display record" for the removed records until adutext runs since the old removed record turns up first for their searches. Users may also see the "Unable to display record" message for the removed duplicate cases

• The LOAD and DELETE can BOTH be done on the same night via crontab if the number of records being dealt with can go through adutext without causing it to crook-which then would take a rebuildtext to fix. I have successfully ran an adutext in 1 hour 24 minutes after loading in 7,793 new records and deleting 7,572 old records giving adutext a total of 15,365 ckeys to deal with. (This testing was done on our test system of course.) If you were to hit the limit (KUNI.tbl) while running an adutext report, the report would error out with a message such as Inverted file size limit exceeded and at that point Sirsi would have to increase the size values in KUNI.tbl and run rebuildtext. See incident 64084 for details.

1. FTP, in binary mode, the marc file of records from the serials solutions site ftp.serialssolutions.com (see SS folder for login/password)

2. They should have provided ONE file with all the changed records, deleted records and new records. This is the file to work with. If they provided 3 separate files (one for changed records, delete records and new records IGNORE the brief record file for now per CMB), create a single file via:

   cat fileof_changed.mrc fileof_new.mrc fileof_deleted.mrc |\
   flatskip -im -om -aMARC > onefileall.mrc
   

3. Create a sample of 10 flat records for Joyce Fahlman to review in case of obvious problems. Use the sampleSS.ksh program or:
   • cat the_ftp_MARC_file_of_the_month | flatskip -aSERIAL -dN -im -of -w1 > jul04.sample
   • get a second random record via: cat the_ftp_MARC_file_of_the_month | flatskip -aSERIAL -dN -im -of -w1 -s100 >> jul04.sample
   • continue to use the s parameter to skip X number of records and the w parameter to write out a single record until you have a file of 10 or so for Carol.

4. Setup the crontab to run findmatchv2.ksh as in:

   30 11 2 2 * /s/sirsi/Unicorn/Work/NDL/Serialsolutions/findmatchv2.ksh /s/sirsi/Unicorn/Work/NDL/Serialsolutions/All_Calgary_MARC_2004-01-20.mrc 2>/s/sirsi/Unicorn/Work/ss.err2
   
   

   • The old manual program was: findmatch.ksh in Unicorn/Work/NDL/Serialsolutions as in: $findmatch.ksh UnivCalgary_MARC_2003-01.mrc janmatch.out

   • this program takes the marc input file (new from Serials Solutions) and outputs ckeys of catalogue records which are in our database with matching 001's to 001's in the new Serial solution file. (The program also outputs the matched 001s.)

   • Of interest ... since Aug 2002 the 001's are all ssj##...

   • The findmatchv2.ksh will check that the ckeys generated from findmatchv2.ksh are really Serials Solutions records which have been loaded in previously
     • cat ckeys from findmatch | sort > ckeyssort.matched

     • cat file of old loaded ckeys from previous months | sort > ckeysold.sort

     • comm -23 ckeyssort.matched ckeysold.sort > problems

       	Hopefully "problems" file is empty.  If ANY ckeys are output, this 
               means that incoming records have 001's that match records in our 
        	catalogue which are NOT serial solution records!  Investigate and 
       	make these catalogue records in our database non-serial solutions
       	001s and ensure that adutext runs after they are fixed.
       

   • The program does a cat the new Serial Solution marc file | flatskip -aSERIAL -dN -im -of -e"path/filename" > YYYYMM.flat

             Note that -dN parameter of flatskip does what it should, skips
             the output of records marked in the leader for "delete", although
             the log words it incorrectly. Sirsi is fixing this for v.2003.
     

   • Files created by this program in Work/NDL/Serialsolutions:
     • YYYYMM.flat which are all the records to load into the database
     • flat.err any errors from the cat of the marc SS file through flatskip
     • sstotalrecords which is a file with the number of marc records in the Serial Solution marc file for use in updating the serialsolutions.html webpage stats
     • stamp which is a file with the YYYYMM in it so it can be read and used by loadNEWcronv2.ksh
     • matchedckeys.matched001
     • matchedckeys.sort
     • ssfindmatch.err

   • Log is emailed out.

5. Setup the sirsi crontab to run the loadNEWcronv2.ksh program as in:

   30 12 3 2 * /s/sirsi/Unicorn/Work/NDL/Serialsolutions/loadNEWcronv2.ksh 2>/s/sirsi/Unicorn/Work/NDL/Serialsolutions/ssload.err
   
   

   • This program will read the stamp file to get the YYYYMM which it then uses to find the YYYYMM.flat file to load into the database with a date catalogued of 20010101 and create only catalogload parameter. The old date catalogued is used so that these records are NOT sent off with record exports to NLC!

   • This program will send it's log via email and a short report to my cellphone

   • Files created or removed from Work/NDL/Serialsolutions:
     • leftckeys which is a file of the ckeys which will remain in the database from past loads after the matchedckeys.sort ckeys are removed by the next program.
     • ALLckeys.sort.bak which is the old sorted file of all the serial solutions ckeys in the database
     • ALLckeys.sort which is a new sorted file of all the leftckeys plus new ckeys from this current programs loaded records
     • YYYYMM.flat.log.YYYYMMDY the log which is emailed is also in the directory
     • YYYYMM.flat.ckeys.YYYYMMDY list of ckeys for the new loaded SS records
     • ssloaded a file with just the number of SS record loaded
     • ssload.err a file of any errors which the crontab running of loadNEWcronv2.ksh has, should always be zero bytes of data
     • program does a rm -f of the stamp file since it is done loading records and nothing else needs to use the stamp file

   • older program was run loadNEW.ksh ssfilename.flat (loadNEW.ksh is set to use catalogload program with create only parameter). Use 20010101 as the date catalogued so that these records are NOT sent off with record exports to NLC!
     • check the logs
     • check the error file

6. Setup the sirsi crontab to run the removeoldcronv2.ksh program as in:

   43 14 3 2 * /s/sirsi/Unicorn/Work/NDL/Serialsolutions/removeoldcronv2.ksh 2>/s/sirsi/Unicorn/Work/NDL/Serialsolutions/ssremove.err
   
   

   • This program automatically uses the matchedckeys.sort from the findmatchv2.ksh program to remove the updated or deleted SS records.
   • A backup of the records to be removed is put into /Work/NDL/Serialsolutions/Backup. See the README file there if you ever have to reload the backedup and removed records.
   • Sends a complete log to email and a brief message of the number of records deleted to my cell phone.
   • Creates a few files in /Work/NDL/Serialsolutions:
     • matchedckeys.sort.log which is the same as the email version of the log
     • ssremove.err for any errors reported while running the program
     • moves the ssloaded file created by loadNEWcronv2.ksh to ssloaded.bak
     • moves the sstotalrecords file created by findmatchv2.ksh to sstotalrecords.bak
     • moves the matchedkceys.sort created by findmatchv2.ksh to matchedckeys.sort.bak
7. let adutext run to clean out the ckeys

8. update this webpage with stats which were in the emailed log from removeoldcronv2.ksh

* Incoming Serials Solutions file includes updates, new records and deleted records

** The number of records loaded are just the updated records and the new records

*** The number of deleted records are the delete cases and the updated cases

Backing Out of a Serials Solutions Load-YIKES!

• If the need arises to back out of a complete serials solutions load you would need to do the following steps:
• Find the ckeys of the records which were just loaded by doing: comm -23 ALLckeys.sort leftckeys > ckeysofjustloaded (Ensure the number of ckeys you get equals the stats for the last load for number loaded!)
• Use the removeold.ksh program to get rid of these records --This use the version that isn't hunting for all the normal monthly file names so you can input the proper ckey file for removal!

  % removeold.ksh ckeysofjustloaded 
  

• Now you can add back in the records that were deleted by the last load (updates and removed records) by going to the /Work/NDL/Serialsolutions/Backup directory and finding the last set of backed up records to reload. These are flat records already, so use the program in this directory to load them. (20010101 is the date catalogued that we always use for SS records.):

  % restorerecords.ksh removedrecords.20040621.bak.flat 20010101
  

• Now create the new ALLckeys.sort file in the /NDL/Serialsolutions directory for next month's load via:

  $ cat /s/sirsi/Unicorn/Work/NDL/Serialsolutions/leftckeys /s/sirsi/Unicorn/Work/NDL/Serialsolutions/Backup/removedrecords.20040621.bak.flat.ckeys | sort > ALLckeys.sort
  
  

• After all of this, adutext will reindex these records.