How to update recognition training images?

[RensBloom]

Though I am not disappointed at all with Audiveris, I have the feel that I could contribute to improve the user experience. Regarding object recognition, I found that some symbols are not often recognized correctly. For example, the thrill looks slightly different in many scores, giving some troubles with recognition.

After digging a little bit through the code, I am still lost where I could add specific images such that Audiveris can handle them well. Should I be somewhere in omr-dataset-tools or in MuseScore? From the omr-dataset-tools wiki it is not clear for me where I should start trying to improve the OMR. I would be happy if someone can shortly explain how I could add images, retrain the model and test in Audiveris.

[hbitteur]

Thanks @RensBloom for pointing this out.

The way Audiveris engine recognizes objects depends on what object shapes it is looking for.
Roughly:

1. Some shapes, generally the ones that can vary in size, use ad hoc methods. Examples are: staff lines, bar-lines, stems, beams, slurs, etc
2. Heads use template matching (nota: a soon-to-be-released development will allow the use of several musical fonts)
3. Text recognition is delegated to Tesseract external library
4. All the other shapes use a glyph classifier, implemented as a neural network, which must be trained on a wide variety of representative samples.

Your trill example, not "thrill" as you wrote :-), belongs to the 4th category. And indeed, for some shapes like this trill shape, we still have too few samples in the training set.
How could we improve the situation? This is where open source shines.

If you want to do it yourself, the process is documented in the Advanced sections of Audiveris handbook. Please refer to:

1. Sample repository about the generation and management of samples.
2. Trainer interface about training the neural network.

An easier approach for the non-techy users, is to share with us representative sheet examples (just .omr files), so that I can retrieve their key samples and add them to the global sample repository.

In a next post, I will provide a list of these shapes for which we need additional samples.
Stay tuned.
/Hervé

[hbitteur]

Here is a print out of sample repository, with all the shapes handled by the glyph classifier.
Edited: List is now sorted alphabetically

1. Count <= 3 generally indicate artificial font-based samples.
2. We should aim at having at least 15 samples per shape, and as different-looking as possible.

Shape	Count
ACCENT	120
ARPEGGIATO	18
BREATH_MARK	5
BREVE_REST	5
CAESURA	3
CLUTTER	83
CODA	3
COMMON_TIME	560
CUT_TIME	96
C_CLEF	1431
DAL_SEGNO	2
DA_CAPO	2
DIGIT_0	7
DIGIT_1	7
DIGIT_2	6
DIGIT_3	6
DIGIT_4	7
DIGIT_5	3
DOT_set	1278
DOUBLE_FLAT	3
DOUBLE_SHARP	3
DYNAMICS_F	92
DYNAMICS_FF	16
DYNAMICS_FP	11
DYNAMICS_MF	23
DYNAMICS_MP	19
DYNAMICS_P	113
DYNAMICS_PP	21
DYNAMICS_SF	86
DYNAMICS_SFZ	2
EIGHTH_REST	758
FERMATA_ARC	11
FERMATA_ARC_BELOW	6
FLAG_1	446
FLAG_1_UP	499
FLAG_2	31
FLAG_2_UP	53
FLAG_3	1
FLAG_3_UP	1
FLAG_4	1
FLAG_4_UP	1
FLAG_5	1
FLAG_5_UP	1
FLAT	2483
F_CLEF	2184
F_CLEF_8VA	2
F_CLEF_8VB	2
F_CLEF_SMALL	29
G_CLEF	2172
G_CLEF_8VA	3
G_CLEF_8VB	14
G_CLEF_SMALL	41
HW_REST_set	280
LONG_REST	3
MORDENT	12
MORDENT_INVERTED	6
NATURAL	516
ONE_128TH_REST	3
ONE_16TH_REST	139
ONE_32ND_REST	10
ONE_64TH_REST	3
OTTAVA	12
PEDAL_MARK	151
PEDAL_UP_MARK	118
PERCUSSION_CLEF	20
PLUCK_A	3
PLUCK_I	4
PLUCK_M	4
PLUCK_P	7
QUARTER_REST	518
QUINDICESIMA	3
REPEAT_FOUR_BARS	3
REPEAT_ONE_BAR	166
REPEAT_TWO_BARS	10
ROMAN_I	3
ROMAN_II	3
ROMAN_III	3
ROMAN_IV	3
ROMAN_IX	3
ROMAN_V	3
ROMAN_VI	3
ROMAN_VII	3
ROMAN_VIII	3
ROMAN_X	3
ROMAN_XI	3
ROMAN_XII	3
SEGNO	3
SHARP	2710
SMALL_FLAG	7
SMALL_FLAG_SLASH	5
STACCATISSIMO	4
STRONG_ACCENT	20
TENUTO	7
TIME_EIGHT	409
TIME_FIVE	15
TIME_FIVE_FOUR	20
TIME_FOUR	1255
TIME_FOUR_FOUR	19
TIME_NINE	42
TIME_ONE	5
TIME_SEVEN	13
TIME_SIX	271
TIME_SIXTEEN	3
TIME_SIX_EIGHT	249
TIME_SIX_FOUR	4
TIME_THREE	693
TIME_THREE_EIGHT	22
TIME_THREE_FOUR	437
TIME_TWELVE	3
TIME_TWELVE_EIGHT	3
TIME_TWO	337
TIME_TWO_FOUR	318
TIME_TWO_TWO	10
TIME_ZERO	3
TR	6
TUPLET_SIX	3
TUPLET_THREE	38
TURN	17
TURN_INVERTED	1
TURN_SLASH	1
TURN_UP	1
VENTIDUESIMA	3

[RensBloom]

Thank you for your quick answer with the links. I'll give it a first try with examples from the score I want to process this week and see if I can already save myself from some manual actions. I have enough scores at home to collect samples from different publishers. That could be a good project for the Christmas holidays.

[hbitteur]

I noticed that the link to Google drive (that appears in the Sample repository section) is obsolete for 2 reasons:

• It points to 5.1 files. 5.2 files are in a separate folder visible at https://drive.google.com/drive/folders/1NaCZGzUjoz0jr7Jpsk94SLk6GEqFAEPg?usp=share_link
• I'm right now updating the files for 5.3 because of the coming multifont feature. I should publish them within one week.

But meanwhile, you can indeed read the documentation to get a good idea of how this works.
And you still have to gather relevant samples on your side...

[hbitteur]

@RensBloom
FYI, omr-dataset-tools project, still within Audiveris organization, is a work in progress. It deals with deep learning applied to the detection of musical symbols in a printed score.

Current Audiveris OMR engine uses several methods as described above, including a simple neural network for the glyph classifier. It works rather well, provided we can get the precise underlying glyph, that is detecting which pixels are part of the musical glyph and which pixels are not. This task, known as "segmentation", is an endless fight.

In omr-data-set tools instead, we develop an approach which considers all the pixels and thus avoids the segmentation problem. Last summer I got good results with a "patch classifier" which works as follows: you point to a given location and the patch classifier tells you the musical symbol, if any, centered on this location. This was the continuation of 6.0 protptype.
Accuracy of this MobileNet V2 implementation was about 94% IIRC. However, if I had to apply the patch classifier on each and every pixel of a score image, this would take several days on my PC. So the sliding window approach is not the correct one.
I have read literature about one-shot detection that should avoid the sliding window approach, but I still have to investigate this approach and come up with a working implementation.

Right now, I'm back to 5.x to develop a few missing features, such as the multi-font support. But someday, I'll turn back to 6.x again...