Export queue rewrite and refactored code to use 'uri' #90
Conversation
mikewalch
force-pushed
the
export-rewrite
branch
from
556fd78 to
64f4564
Compare
| return IndexClient.genUriMutations((UriUpdate) export.getValue(), export.getSequence()); | ||
| } | ||
|
|
||
| log.error("Unknown IndexUpdate object placed on index export queue"); |
There was a problem hiding this comment.
This should probably throw an exception and include the class name
| public static Collection<Mutation> genDomainMutations(DomainUpdate update, long seq) { | ||
| Map<RowColumn, Bytes> oldData = genDomainData(update.getDomain(), update.getOldPageCount()); | ||
| Map<RowColumn, Bytes> newData = genDomainData(update.getDomain(), update.getNewPageCount()); | ||
| return AccumuloExportQueue.generateMutations(seq, oldData, newData); |
There was a problem hiding this comment.
I did not realize this while looking at the recipes PR, but looking at this PR made me realize that generateMutations is a really generic name.
There was a problem hiding this comment.
I agree. Do you want to create an issue?
There was a problem hiding this comment.
opened apache/fluo-recipes#108
| /** | ||
| * Used by URI collision free map | ||
| */ | ||
| public class UriInfo implements Serializable { |
There was a problem hiding this comment.
was this just copied out of UriMap? just wondering if there are any changes to review
There was a problem hiding this comment.
yes it was just copied out
| @@ -38,12 +39,12 @@ public Kryo create() { | |||
| // same order it would be ok) and ran into issue where Spark and Fluo code were using different | |||
| // ids for some reason. | |||
| kryo.register(UriInfo.class, 9); | |||
| kryo.register(DomainExport.class, 10); | |||
| kryo.register(UriCountExport.class, 11); | |||
| kryo.register(PageExport.class, 12); | |||
There was a problem hiding this comment.
need to update docs/code-guide.md for these changes
There was a problem hiding this comment.
updated code guide in f0cbafb
| import webindex.core.models.Link; | ||
| import webindex.core.models.Page; | ||
| import webindex.core.models.UriInfo; | ||
| import webindex.core.models.update.IndexUpdate; |
There was a problem hiding this comment.
is everything in models.update pkg related to exporting? If so I would prefer calling it models.export... but thats just my personal preference. Update also make sense, I am just thinking export is more specific.
There was a problem hiding this comment.
I wanted to group them but I am not stuck on 'update'. I can make the package 'export'
There was a problem hiding this comment.
change package name to 'export' in f0cbafb
|
+1 |
* Updates due to changes in apache/fluo-recipes#102 * All objects placed on export queue now implement IndexUpdate interface and only represent data that is being exported * Moved code that creates Accumulo mutations from export objects to IndexClient
mikewalch
force-pushed
the
export-rewrite
branch
from
f0cbafb to
22ebe46
Compare
No description provided.