Full and Incremental Extracts from Symphony

This page is to document the process for Full and Incremental extracts processed from Symphony and how they are loaded to BlackCat Server.

What is included in a Symphony extract for BlackCat?

Everything from Sirsi,

• Excluding shadowed records but not temporary records. (Most of the temporary records are shadowed.)

• Review records will be excluded by default since they don't have any items associated with them (extraction begins at the item level)

• Excluding item types:
  CARRELKEY, EBOOKREADR, EQUIP14DAY, EQUIP24FEE, EQUIP24HR, EQUIP3DAY, EQUIP4HR, EQUIP5DAY, EQUIP7DAY, ILL, LAPTOP, PALCI

• Excluding Junk tags: Generated a custom junk tag list for Blackcat (dumpjunktag_blackcat on Symphony):

  001, 029, 090, 593, 594, 595, 596, 597, 598, 599, 900, 901, 920, 921, 926, 927, 930, 948, 949, 960, 961, 977, 980, 990, 994, 998, 999
  

• Bound-with extract:
  selects all the call numbers which are a bound-with child with their parent's holdings information, title and catkey of that bound-with child. The parent's title and catkey is appended and exported as a 591 field and the parent's holding information as a 949 to the child's extracted record.

• Holdings extract:
  The 949 field is used for current holdings.

• Marc Holdings extract -Summary holdings will be included in the extract. The fields 852 866 867 868 are extracted

• Records that exceeded the maximum export size to be catalog dumped

• Records that exceeded the maximum number of holdings to be catalog dumped: Split these catalog record into smaller bib records

Full Extracts

1. Full extract can start anytime after adutext finishes it's daily run of clearing out the Database Edit directory.

2. On blackcat QA server

   Where <extract date> equals a zero padded date stamp like YYYYMMDD

   $ sudo mkdir /data/symphony_data/full_extracts/<extract date>
   $ sudo chown ansible_deploy_bot:sirsi /data/symphony_data/full_extracts/<extract date>
   $ sudo chmod 775 /data/symphony_data/full_extracts/<extract date>
   

3. See https://psu.app.box.com/file/488353096744 for instructions on how to do a full extract of all the Symphony MARC data and secure copy it to /data/symphony_data/full_extracts/<extract date> directory on the Blackcat server. The full extract creates the following files:

   blackcat_<date>_<#>.mrc
   blackcat_<date>_bound_<#>.mrc
   blackcat_<date>_max_exceed.mrc
   blackcat_<date>_holdings_exceed.mrc
   

Incrementals

The incrementals are the add/updates and delete extracts from Symphony. The process is similar to full extract. The incrementals are done once a day. When a bib record is modified, created or deleted their catkey goes in the Database Edit directory. The daily extract picks up the previous day's add/updates/delete catkeys from this dir which have been saved off (prior to adutext) in a file. The catkeys are processed similar to the full extract (see above). Any record that is included becomes adds and updates. Records that are not included are the deletes. Any catalog dump errors are emailed to Sirsi developer to be reviewed. The add/updates file contains the marc records. The deletes file contains a list of catkeys to be deleted. They are scp'ed to blackcat01qa.