ITS Serials Solutions procedures as of Sept. 2010

We will be getting marc records which will be updated every month with a new file which will contain:

• -deletes
• -updates
• -and new records

Timing of the Process

• Run the findmatch process any day of the week, this process will find matches and :
  • remove local tags from the matched records
  • remove copies from the matched records
  • use api transactions to change the matched record's call number to AUTO class scheme and XX call number
• Run the load process right after, which will take the full Serials Solutions file and use catalogload to load it in. It will update the matched records, delete the records marked for deletion and load the new records. Parameters of catalogload are:
  • catalogload -aSERIAL -bf -im -e"$INPUT.err" -fg -hs"949" -n"LC,949,a" -mb -j -4"junktag.sersolutions" -r -ucm -Dd >$INPUT.ckeys.$today 2>${temp}log

• The findmatch and load programs can BOTH be done on the same night via crontab. Be careful to split the incoming file if the number of records being dealt with is over 10,000 records so that it can go through adutext without causing it to crook-which then would take a rebuildtext to fix. If you were to hit the limit (KUNI.tbl) while running an adutext report, the report would error out with a message such as "Inverted file size limit exceeded" and at that point Sirsi would have to increase the size values in KUNI.tbl and run rebuildtext. See incident 64084 for details.

1. FTP, in binary mode, the marc file of records from the serials solutions site ftp.serialssolutions.com (see SS folder for login/password) login and password are now part of the /s/sirsi/.netrc file so you can get automatically logged in

2. They should have provided ONE file with all the changed records, deleted records and new records. This is the file to work with. If they provided 3 separate files (one for changed records, delete records and new records IGNORE the brief record file for now per CMB), create a single file via:

   cat fileof_changed.mrc fileof_new.mrc fileof_deleted.mrc |\
   flatskip -im -om -aMARC > onefileall.mrc
   

3. Create a sample of 10 flat records for Joyce Fahlman to review in case of obvious problems. Use the sampleSS.ksh program or:
   • cat the_ftp_MARC_file_of_the_month | flatskip -aSERIAL -dN -im -of -w1 > jul04.sample
   • get a second random record via: cat the_ftp_MARC_file_of_the_month | flatskip -aSERIAL -dN -im -of -w1 -s100 >> jul04.sample
   • continue to use the s parameter to skip X number of records and the w parameter to write out a single record until you have a file of 10 or so for Carol.

4. Setup the crontab to run findmatchv3.ksh as in:

   30 11 2 2 * /s/sirsi/Unicorn/Work/NDL/Serialsolutions/findmatchv3.ksh /s/sirsi/Unicorn/Work/NDL/Serialsolutions/sept14all.mrc 2>/s/sirsi/Unicorn/Work/ss.err3
   
   

   • The old manual program was: findmatch.ksh in Unicorn/Work/NDL/Serialsolutions as in: $findmatch.ksh UnivCalgary_MARC_2003-01.mrc janmatch.out

   • this program takes the marc input file (new from Serials Solutions) and outputs ckeys of catalogue records which are in our database with matching 001's to 001's in the new Serial solution file. (The program also outputs the matched 001s.)

   • Of interest ... since Aug 2002 the 001's are all ssj##...

   • The findmatchv3.ksh will check that the ckeys generated from findmatchv3.ksh are really Serials Solutions records which have been loaded in previously
     • cat ckeys from findmatch | sort > ckeyssort.matched

     • cat file of old loaded ckeys from previous months | sort > ckeysold.sort

     • comm -23 ckeyssort.matched ckeysold.sort > problems

       	Hopefully "problems" file is empty.  If ANY ckeys are output, this 
               means that incoming records have 001's that match records in our 
        	catalogue which are NOT serial solution records!  Investigate and 
       	make these catalogue records in our database non-serial solutions
       	001s and ensure that adutext runs after they are fixed.
       

   • The program removes all LOCAL fields from the matched old SS records which are about to be overlaid, using editmarc
   • The program writes out api transactions using makeapifile.pl to change the matched records call number back to an XX auto generated call number style
   • The program deletes the copy on the old matched SS records via remitem since a new copy will be created when the new file of records is loaded into the system via loadNEWcronv3.ksh
   • The program does a cat the new Serial Solution marc file | flatskip -aSERIAL -im -om -e"path/filename" > YYYYMM.mrc

             Note that -dN parameter of flatskip IS NO LONGER USED in this version.  We want ALL records out even the deletes!

   • Files created by this program in Work/NDL/Serialsolutions:
     • YYYYMM.mrc which are all the records to load into the database
     • flat.err any errors from the cat of the marc SS file through flatskip
     • sstotalrecords which is a file with the number of marc records in the Serial Solution marc file for use in updating the serialsolutions.html webpage stats
     • stamp which is a file with the YYYYMM in it so it can be read and used by loadNEWcronv3.ksh
     • matchedckeys.matched001
     • matchedckeys.sort
     • ssfindmatchv3.err

   • Log is emailed out.

5. Setup the sirsi crontab to run the loadNEWcronv3.ksh program as in:

   30 12 3 2 * /s/sirsi/Unicorn/Work/NDL/Serialsolutions/loadNEWcronv3.ksh 2>/s/sirsi/Unicorn/Work/NDL/Serialsolutions/ssload.err
   
   

   • This program will read the stamp file to get the YYYYMM which it then uses to find the YYYYMM.mrc file to load into the database with a date catalogued of 20010101 and create/update catalogload parameter. The catalogload parameter of -dD will delete the records marked for deletions. The old date catalogued is used so that these records are NOT sent off with record exports to NLC or included in new book lists!

   • This program will send it's log via email

   • Files created or removed from Work/NDL/Serialsolutions:
     • leftckeys which is a file of the ckeys which will remain in the database from past loads after the matchedckeys.sort ckeys are removed by the next program.
     • ALLckeys.sort.bak which is the old sorted file of all the serial solutions ckeys in the database
     • ALLckeys.sort which is a new sorted file of all the leftckeys plus new ckeys from this current programs loaded records
     • YYYYMM.mrc.log.YYYYMMDY the log which is emailed is also in the directory
     • YYYYMM.mrc.ckeys.YYYYMMDY list of ckeys for the new loaded SS records
     • ssloaded a file with just the number of SS record loaded
     • ssload.err a file of any errors which the crontab running of loadNEWcronv2.ksh has, should always be zero bytes of data
     • program does a rm -f of the stamp file since it is done loading records and nothing else needs to use the stamp file
     • moves the ssloaded file created by loadNEWcronv2.ksh to ssloaded.bak
     • moves the sstotalrecords file created by findmatchv2.ksh to sstotalrecords.b ak
     • moves the matchedckeys.sort created by findmatchv2.ksh to matchedckeys.sort. bak
     • check the logs
     • check the error file

6. let adutext run to clean out the ckeys

7. update this webpage with stats which were in the emailed log from loadNEWcronv3.ksh

* Incoming Serials Solutions file includes updates, new records and deleted records

** The number of deletions is how many records were sent from SS for deletion

*** The number of replaced records is the number SS sent for update

**** The number of records loaded are just the NEW records