public class KMeansWorker extends ml.shifu.guagua.worker.AbstractWorkerComputable<KMeansMasterParams,KMeansWorkerParams,GuaguaWritableAdapter<org.apache.hadoop.io.LongWritable>,GuaguaWritableAdapter<org.apache.hadoop.io.Text>>
KMeansWorker re-computes each record tagged with new category.
To calculate new k centers in master, KMeansWorker also help to accumulate worker info for new k centers by
using sum list and count list.
| Constructor and Description |
|---|
KMeansWorker() |
| Modifier and Type | Method and Description |
|---|---|
KMeansWorkerParams |
doCompute(ml.shifu.guagua.worker.WorkerContext<KMeansMasterParams,KMeansWorkerParams> context)
Using the new k centers to tag each record with index denoting the record belongs to which category.
|
void |
init(ml.shifu.guagua.worker.WorkerContext<KMeansMasterParams,KMeansWorkerParams> context) |
void |
initRecordReader(ml.shifu.guagua.io.GuaguaFileSplit fileSplit)
Reading input line by line
|
void |
load(GuaguaWritableAdapter<org.apache.hadoop.io.LongWritable> currentKey,
GuaguaWritableAdapter<org.apache.hadoop.io.Text> currentValue,
ml.shifu.guagua.worker.WorkerContext<KMeansMasterParams,KMeansWorkerParams> workerContext)
Loading data into memory.
|
protected void |
postLoad(ml.shifu.guagua.worker.WorkerContext<KMeansMasterParams,KMeansWorkerParams> context) |
public void initRecordReader(ml.shifu.guagua.io.GuaguaFileSplit fileSplit)
throws IOException
initRecordReader in class ml.shifu.guagua.worker.AbstractWorkerComputable<KMeansMasterParams,KMeansWorkerParams,GuaguaWritableAdapter<org.apache.hadoop.io.LongWritable>,GuaguaWritableAdapter<org.apache.hadoop.io.Text>>IOExceptionpublic void init(ml.shifu.guagua.worker.WorkerContext<KMeansMasterParams,KMeansWorkerParams> context)
init in class ml.shifu.guagua.worker.AbstractWorkerComputable<KMeansMasterParams,KMeansWorkerParams,GuaguaWritableAdapter<org.apache.hadoop.io.LongWritable>,GuaguaWritableAdapter<org.apache.hadoop.io.Text>>public KMeansWorkerParams doCompute(ml.shifu.guagua.worker.WorkerContext<KMeansMasterParams,KMeansWorkerParams> context)
doCompute in class ml.shifu.guagua.worker.AbstractWorkerComputable<KMeansMasterParams,KMeansWorkerParams,GuaguaWritableAdapter<org.apache.hadoop.io.LongWritable>,GuaguaWritableAdapter<org.apache.hadoop.io.Text>>protected void postLoad(ml.shifu.guagua.worker.WorkerContext<KMeansMasterParams,KMeansWorkerParams> context)
postLoad in class ml.shifu.guagua.worker.AbstractWorkerComputable<KMeansMasterParams,KMeansWorkerParams,GuaguaWritableAdapter<org.apache.hadoop.io.LongWritable>,GuaguaWritableAdapter<org.apache.hadoop.io.Text>>public void load(GuaguaWritableAdapter<org.apache.hadoop.io.LongWritable> currentKey, GuaguaWritableAdapter<org.apache.hadoop.io.Text> currentValue, ml.shifu.guagua.worker.WorkerContext<KMeansMasterParams,KMeansWorkerParams> workerContext)
load in class ml.shifu.guagua.worker.AbstractWorkerComputable<KMeansMasterParams,KMeansWorkerParams,GuaguaWritableAdapter<org.apache.hadoop.io.LongWritable>,GuaguaWritableAdapter<org.apache.hadoop.io.Text>>Copyright © 2015. All Rights Reserved.