fix incorrect similar words

[KarlieZhao]

dhowe/rita#178
instead of 'toing', should be 'toeing'.

dhowe/rita#177
updated dictionary to fix the incorrect similar words

[dhowe]

Are there other cases you can think of for this? 'toe' is very uncommon as a verb -- I think its better to remove the verb form in the dictionary.

But also, why do I see "20,849 additions, 22,007 deletions" for the dict file ?

[KarlieZhao]

true, I'll remove the verb form of 'toe'. (But also I just found the word 'hoe''s presentpart form is returned as 'hoing', and there might be more similar cases -- I'll run more tests and modify the regex.)

[KarlieZhao]

But also, why do I see "20,849 additions, 22,007 deletions" for the dict file ?

there are a lot of changes in the dictionary because I was trying to remove those 'vbn' words that can be computed from their base forms (there were ~1000 such words so I didn't do it manually). Most lines in the new dict file remain the same but since I replaced the entire dictionary with a new one, they were recognized as additions & delections, I believe...not sure if there's a better way to fix this?

[dhowe]

there are a lot of changes in the dictionary because I was trying to remove those 'vbn' words that can be computed from their base forms (there were ~1000 such words so I didn't do it manually). Most lines in the new dict file remain the same but since I replaced the entire dictionary with a new one, they were recognized as additions & delections, I believe...not sure if there's a better way to fix this?

this is a very big decision/task... we should discuss more before trying it
for example we still need to be able to find those words when that pos is specified in a search or soundsLike, etc...

[KarlieZhao]

for example we still need to be able to find those words when that pos is specified in a search or soundsLike, etc...

I can think of two ways to solve this:
a. when looking for similar words (with soundsLike, etc), we not only iterate through the dictionary but also look at all other forms of each word, and check whether they rhyme with/sound/spell like the target word. So then we should be able to keep the base form of each word and remove the other forms in dict.
b. or, for words like 'discriminated', simply remove the 'vb' tag and keep 'vbn'.