WIP: Enable duplicate detection via bag manifests

[ross-spencer]

Compare an accruals location to an AIP store

This commit introduces an accruals->aips comparison capability.

Digital objects in an accruals folder can now be compared to the
contents of an AIP store.

Where filepaths and checksums and dates match, the object is
considered to be identical (a true duplicate). Where they don't,
users can use modulo (%) to identify where the object isn't in fact
identical.

Much of the benefit of this work is derived from the nature of the
AIP structure imposed on a digital transfer.

Once the comparison is complete, three reports are output in CSV
format:

• True-duplicates.
• Near-duplicates (checksums match, but other components might not).
• Non-duplicates.

Additionally a summary report output in JSON.

Connected to archivematica/Issues#448

Configuration

API configuration, and transfer source location is done via this configuration file. Note the '"accruals_transfer_source"' parameter describes a transfer source in the storage service with the Description 'accruals'. But could equally be any other value more appropriate to your institution.

The primary script will also accept a value for this transfer source on the command line, e.g.

• python3 -m duplicates.accruals <my_transfer_source_description>

With everything configured correctly the successful output on the command line may look as follows:

$ python3 -m duplicates.accruals
INFO      2019-07-08 17:11:16 duplicates.py:171  No result for algorithm: md5
INFO      2019-07-08 17:11:16 duplicates.py:171  No result for algorithm: sha1
INFO      2019-07-08 17:11:17 duplicates.py:86   Filtering: data/METS.8a8e1cc5-82ec-491a-8bda-cc7d0223553f.xml
INFO      2019-07-08 17:11:17 duplicates.py:86   Filtering: data/README.html
INFO      2019-07-08 17:11:17 duplicates.py:86   Filtering: data/logs/fileFormatIdentification.log
INFO      2019-07-08 17:11:17 duplicates.py:86   Filtering: data/logs/filenameCleanup.log
INFO      2019-07-08 17:11:17 duplicates.py:82   Filtering: data/logs/transfers/1-465cca06-e0e2-4ee9-b2b6-ecc7feeed73a/logs/fileFormatIdentification.log
INFO      2019-07-08 17:11:17 duplicates.py:82   Filtering: data/logs/transfers/1-465cca06-e0e2-4ee9-b2b6-ecc7feeed73a/logs/filenameCleanup.log
INFO      2019-07-08 17:11:17 duplicates.py:82   Filtering: data/objects/metadata/transfers/1-465cca06-e0e2-4ee9-b2b6-ecc7feeed73a/directory_tree.txt
INFO      2019-07-08 17:11:17 duplicates.py:82   Filtering: data/objects/submissionDocumentation/transfer-1-465cca06-e0e2-4ee9-b2b6-ecc7feeed73a/METS.xml
INFO      2019-07-08 17:11:17 duplicates.py:171  No result for algorithm: sha512
INFO      2019-07-08 17:11:17 serialize_to_csv.py:41   Number of files in '1' AIPs in the AIP store: 4
INFO      2019-07-08 17:11:17 serialize_to_csv.py:44   Number of transfers: 3
INFO      2019-07-08 17:11:17 serialize_to_csv.py:47   Number of items in transfer 1: 4
INFO      2019-07-08 17:11:17 serialize_to_csv.py:47   Number of items in transfer 2: 5
INFO      2019-07-08 17:11:17 serialize_to_csv.py:47   Number of items in transfer 3: 2
{
    "count_of_files_across_aips": 4,
    "files_in_transfer-1": 4,
    "files_in_transfer-2": 5,
    "files_in_transfer-3": 2,
    "number_of_aips": 1,
    "numer_of_transfers": 3
}
ERROR     2019-07-08 17:11:17 serialize_to_csv.py:84   Outputting report to: true_duplicates_comparison.csv
ERROR     2019-07-08 17:11:17 serialize_to_csv.py:118  Outputting report to: near_matches_comparison.csv
ERROR     2019-07-08 17:11:17 serialize_to_csv.py:141  Outputting report to: non_matches_list.csv

The CSV files output as a result can then be used to compile a list of files specifically selected to be transferred into Archivematica.