ai.grakn.engine.postprocessing

Class ResourceDeduplicationTask.Job

java.lang.Object

ai.grakn.engine.postprocessing.ResourceDeduplicationTask.Job

All Implemented Interfaces:

Cloneable, org.apache.tinkerpop.gremlin.process.computer.MapReduce<String,ConceptId,String,Long,Long>

Enclosing class:

ResourceDeduplicationTask

public static class ResourceDeduplicationTask.Job
extends Object
implements org.apache.tinkerpop.gremlin.process.computer.MapReduce<String,ConceptId,String,Long,Long>

The map-reduce job submitted to the GraknGraphComputer that scan the whole set of resources in the graph and reduces by eliminating duplicates. The key in the mapping phase is formed by a resource's value and it's resource type.

Author:

borislav

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.tinkerpop.gremlin.process.computer.MapReduce

org.apache.tinkerpop.gremlin.process.computer.MapReduce.MapEmitter<K,V>, org.apache.tinkerpop.gremlin.process.computer.MapReduce.NullObject, org.apache.tinkerpop.gremlin.process.computer.MapReduce.ReduceEmitter<OK,OV>, org.apache.tinkerpop.gremlin.process.computer.MapReduce.Stage

Field Summary

Fields inherited from interface org.apache.tinkerpop.gremlin.process.computer.MapReduce

MAP_REDUCE

Constructor Summary

Constructors

Constructor and Description
Job()

Method Summary

Modifier and Type	Method and Description
org.apache.tinkerpop.gremlin.process.computer.MapReduce<String,ConceptId,String,Long,Long>	clone()
boolean	deleteUnattached() Return true if this job will delete unattached resources and false otherwise.
ResourceDeduplicationTask.Job	deleteUnattached(boolean deleteUnattached) Specify whether resources that are not associated with any entity or relationship should be deleted from the database.
boolean	doStage(org.apache.tinkerpop.gremlin.process.computer.MapReduce.Stage stage) We skip the combine stage and do only MAP and REDUCE.
Long	generateFinalResult(Iterator<org.apache.tinkerpop.gremlin.process.computer.KeyValue<String,Long>> keyValues)
String	getMemoryKey()
ResourceDeduplicationTask.Job	keyspace(String keyspace) Specify the keyspace to use for the deduplication job.
void	map(org.apache.tinkerpop.gremlin.structure.Vertex vertex, org.apache.tinkerpop.gremlin.process.computer.MapReduce.MapEmitter<String,ConceptId> emitter) Emit the resoucre index (value + type) mapped to the concept ID of the resource instance.
void	reduce(String key, Iterator<ConceptId> values, org.apache.tinkerpop.gremlin.process.computer.MapReduce.ReduceEmitter<String,Long> emitter) Here we simply collect all concepts for a key and ask our concept fixer to do its thing.

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.tinkerpop.gremlin.process.computer.MapReduce

addResultToMemory, combine, createMapReduce, getMapKeySort, getReduceKeySort, loadState, storeState, workerEnd, workerStart

Constructor Detail

Job

public Job()

Method Detail

keyspace

public ResourceDeduplicationTask.Job keyspace(String keyspace)

Specify the keyspace to use for the deduplication job.

deleteUnattached

public ResourceDeduplicationTask.Job deleteUnattached(boolean deleteUnattached)

Specify whether resources that are not associated with any entity or relationship should be deleted from the database.

deleteUnattached

public boolean deleteUnattached()

Return true if this job will delete unattached resources and false otherwise.

map

public final void map(org.apache.tinkerpop.gremlin.structure.Vertex vertex,
                      org.apache.tinkerpop.gremlin.process.computer.MapReduce.MapEmitter<String,ConceptId> emitter)

Emit the resoucre index (value + type) mapped to the concept ID of the resource instance.

Specified by:

map in interface org.apache.tinkerpop.gremlin.process.computer.MapReduce<String,ConceptId,String,Long,Long>

doStage

public boolean doStage(org.apache.tinkerpop.gremlin.process.computer.MapReduce.Stage stage)

We skip the combine stage and do only MAP and REDUCE. There is no optimization worth doing in a combine. The number of duplicates is not expected to be unmanageably large.

Specified by:

doStage in interface org.apache.tinkerpop.gremlin.process.computer.MapReduce<String,ConceptId,String,Long,Long>

reduce

public void reduce(String key,
                   Iterator<ConceptId> values,
                   org.apache.tinkerpop.gremlin.process.computer.MapReduce.ReduceEmitter<String,Long> emitter)

Here we simply collect all concepts for a key and ask our concept fixer to do its thing.

Specified by:

reduce in interface org.apache.tinkerpop.gremlin.process.computer.MapReduce<String,ConceptId,String,Long,Long>

getMemoryKey

public String getMemoryKey()

Specified by:

getMemoryKey in interface org.apache.tinkerpop.gremlin.process.computer.MapReduce<String,ConceptId,String,Long,Long>

clone

public org.apache.tinkerpop.gremlin.process.computer.MapReduce<String,ConceptId,String,Long,Long> clone()

Specified by:

clone in interface org.apache.tinkerpop.gremlin.process.computer.MapReduce<String,ConceptId,String,Long,Long>

Overrides:

clone in class Object

generateFinalResult

public Long generateFinalResult(Iterator<org.apache.tinkerpop.gremlin.process.computer.KeyValue<String,Long>> keyValues)

Specified by:

generateFinalResult in interface org.apache.tinkerpop.gremlin.process.computer.MapReduce<String,ConceptId,String,Long,Long>