Package ciir.umass.edu.eval

Class Evaluator

java.lang.Object

ciir.umass.edu.eval.Evaluator

public class Evaluator
extends java.lang.Object

Author:

vdang This class is meant to provide the interface to run and compare different ranking algorithms. It lets users specify general parameters (e.g. what algorithm to run, training/testing/validating data, etc.) as well as algorithm-specific parameters. Type "java -jar bin/RankLib.jar" at the command-line to see all the options.

Field Summary

Fields

Modifier and Type	Field	Description
static boolean	keepOrigFeatures
protected MetricScorerFactory	mFact
static java.lang.String	modelFile
static boolean	mustHaveRelDoc
static java.lang.String	newFeatureFile
static Normalizer	nml
static boolean	normalize
static java.lang.String	qrelFile
protected RankerFactory	rFact
protected MetricScorer	testScorer
static int	topNew
protected MetricScorer	trainScorer
protected RANKER_TYPE	type
static boolean	useSparseRepresentation

Constructor Summary

Constructors

Constructor	Description
Evaluator(RANKER_TYPE rType, METRIC metric, int k)
Evaluator(RANKER_TYPE rType, METRIC trainMetric, int trainK, METRIC testMetric, int testK)
Evaluator(RANKER_TYPE rType, METRIC trainMetric, METRIC testMetric)
Evaluator(RANKER_TYPE rType, METRIC trainMetric, METRIC testMetric, int k)
Evaluator(RANKER_TYPE rType, java.lang.String trainMetric, java.lang.String testMetric)

Method Summary

Modifier and Type	Method	Description
double	evaluate(Ranker ranker, java.util.List<RankList> rl)
void	evaluate(java.lang.String trainFile, double percentTrain, java.lang.String testFile, java.lang.String featureDefFile)	Evaluate the currently selected ranking algorithm using percenTrain% of the training samples for training the rest as validation data.
void	evaluate(java.lang.String sampleFile, java.lang.String featureDefFile, int nFold, float tvs, java.lang.String modelDir, java.lang.String modelFile)	Evaluate the currently selected ranking algorithm using with k-fold cross validation.
void	evaluate(java.lang.String sampleFile, java.lang.String featureDefFile, int nFold, java.lang.String modelDir, java.lang.String modelFile)	Evaluate the currently selected ranking algorithm using with k-fold cross validation.
void	evaluate(java.lang.String sampleFile, java.lang.String validationFile, java.lang.String featureDefFile, double percentTrain)	Evaluate the currently selected ranking algorithm using percenTrain% of the samples for training the rest for testing.
void	evaluate(java.lang.String trainFile, java.lang.String validationFile, java.lang.String testFile, java.lang.String featureDefFile)	Evaluate the currently selected ranking algorithm using .
static void	main(java.lang.String[] args)
void	normalize(java.util.List<RankList> samples)
void	normalize(java.util.List<RankList> samples, int[] fids)
void	normalizeAll(java.util.List<java.util.List<RankList>> samples, int[] fids)
void	rank(java.lang.String testFile, java.lang.String indriRanking)	Generate a ranking in Indri's format from the input ranking
void	rank(java.lang.String modelFile, java.lang.String testFile, java.lang.String indriRanking)	Use a pre-trained model to re-rank the test rankings.
void	rank(java.util.List<java.lang.String> modelFiles, java.lang.String testFile, java.lang.String indriRanking)	Use k pre-trained models to re-rank the test rankings.
void	rank(java.util.List<java.lang.String> modelFiles, java.util.List<java.lang.String> testFiles, java.lang.String indriRanking)	Similar to the above, except data has already been splitted.
int[]	readFeature(java.lang.String featureDefFile)
java.util.List<RankList>	readInput(java.lang.String inputFile)
void	savePerRankListPerformanceFile(java.util.List<java.lang.String> ids, java.util.List<java.lang.Double> scores, java.lang.String prpFile)	Save systems' performance to file
void	score(java.lang.String modelFile, java.lang.String testFile, java.lang.String outputFile)	Write the model's score for each of the documents in a test rankings.
void	score(java.util.List<java.lang.String> modelFiles, java.lang.String testFile, java.lang.String outputFile)	Write the models' score for each of the documents in a test rankings.
void	score(java.util.List<java.lang.String> modelFiles, java.util.List<java.lang.String> testFiles, java.lang.String outputFile)	Similar to the above, except data has already been split.
void	test(java.lang.String testFile)	Evaluate the performance (in -metric2T) of the input rankings
void	test(java.lang.String testFile, java.lang.String prpFile)
void	test(java.lang.String modelFile, java.lang.String testFile, java.lang.String prpFile)	Evaluate the performance (in -metric2T) of a pre-trained model.
void	test(java.util.List<java.lang.String> modelFiles, java.lang.String testFile, java.lang.String prpFile)	Evaluate the performance (in -metric2T) of k pre-trained models.
void	test(java.util.List<java.lang.String> modelFiles, java.util.List<java.lang.String> testFiles, java.lang.String prpFile)	Similar to the above, except data has already been splitted.
void	testWithScoreFile(java.lang.String testFile, java.lang.String scoreFile)	Re-order the input rankings and measure their effectiveness (in -metric2T)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

mustHaveRelDoc

public static boolean mustHaveRelDoc

useSparseRepresentation

public static boolean useSparseRepresentation

normalize

public static boolean normalize

nml

public static Normalizer nml

modelFile

public static java.lang.String modelFile

qrelFile

public static java.lang.String qrelFile

newFeatureFile

public static java.lang.String newFeatureFile

keepOrigFeatures

public static boolean keepOrigFeatures

topNew

public static int topNew

rFact

protected RankerFactory rFact

mFact

protected MetricScorerFactory mFact

trainScorer

protected MetricScorer trainScorer

testScorer

protected MetricScorer testScorer

type

protected RANKER_TYPE type

Constructor Detail

Evaluator

public Evaluator(RANKER_TYPE rType,
                 METRIC trainMetric,
                 METRIC testMetric)

Evaluator

public Evaluator(RANKER_TYPE rType,
                 METRIC trainMetric,
                 int trainK,
                 METRIC testMetric,
                 int testK)

Evaluator

public Evaluator(RANKER_TYPE rType,
                 METRIC trainMetric,
                 METRIC testMetric,
                 int k)

Evaluator

public Evaluator(RANKER_TYPE rType,
                 METRIC metric,
                 int k)

Evaluator

public Evaluator(RANKER_TYPE rType,
                 java.lang.String trainMetric,
                 java.lang.String testMetric)

Method Detail

main

public static void main(java.lang.String[] args)

Parameters:

args -

readInput

public java.util.List<RankList> readInput(java.lang.String inputFile)

normalize

public void normalize(java.util.List<RankList> samples)

normalize

public void normalize(java.util.List<RankList> samples,
                      int[] fids)

normalizeAll

public void normalizeAll(java.util.List<java.util.List<RankList>> samples,
                         int[] fids)

readFeature

public int[] readFeature(java.lang.String featureDefFile)

evaluate

public double evaluate(Ranker ranker,
                       java.util.List<RankList> rl)

evaluate

public void evaluate(java.lang.String trainFile,
                     java.lang.String validationFile,
                     java.lang.String testFile,
                     java.lang.String featureDefFile)

Evaluate the currently selected ranking algorithm using .

Parameters:

trainFile -

validationFile -

testFile -

featureDefFile -

evaluate

public void evaluate(java.lang.String sampleFile,
                     java.lang.String validationFile,
                     java.lang.String featureDefFile,
                     double percentTrain)

Evaluate the currently selected ranking algorithm using percenTrain% of the samples for training the rest for testing.

Parameters:

sampleFile -

validationFile - Empty string for "no validation data"

featureDefFile -

percentTrain -

evaluate

public void evaluate(java.lang.String trainFile,
                     double percentTrain,
                     java.lang.String testFile,
                     java.lang.String featureDefFile)

Evaluate the currently selected ranking algorithm using percenTrain% of the training samples for training the rest as validation data. Test data is specified separately.

Parameters:

trainFile -

percentTrain -

testFile - Empty string for "no test data"

featureDefFile -

evaluate

public void evaluate(java.lang.String sampleFile,
                     java.lang.String featureDefFile,
                     int nFold,
                     java.lang.String modelDir,
                     java.lang.String modelFile)

Evaluate the currently selected ranking algorithm using with k-fold cross validation.

Parameters:

sampleFile -

featureDefFile -

nFold -

modelDir -

modelFile -

evaluate

public void evaluate(java.lang.String sampleFile,
                     java.lang.String featureDefFile,
                     int nFold,
                     float tvs,
                     java.lang.String modelDir,
                     java.lang.String modelFile)

Evaluate the currently selected ranking algorithm using with k-fold cross validation.

Parameters:

sampleFile -

featureDefFile -

nFold -

tvs - Train-validation split ratio.

modelDir -

modelFile -

test

public void test(java.lang.String testFile)

Evaluate the performance (in -metric2T) of the input rankings

Parameters:

testFile - Input rankings

test

public void test(java.lang.String testFile,
                 java.lang.String prpFile)

test

public void test(java.lang.String modelFile,
                 java.lang.String testFile,
                 java.lang.String prpFile)

Evaluate the performance (in -metric2T) of a pre-trained model. Save its performance on each of the ranked list if this is specified.

Parameters:

modelFile - Pre-trained model

testFile - Test data

prpFile - Per-ranked list performance file: Model's performance on each of the ranked list. These won't be saved if prpFile="".

test

public void test(java.util.List<java.lang.String> modelFiles,
                 java.lang.String testFile,
                 java.lang.String prpFile)

Evaluate the performance (in -metric2T) of k pre-trained models. Data in the test file will be splitted into k fold, where k=|models|. Each model will be evaluated on the data from the corresponding fold.

Parameters:

modelFiles - Pre-trained models

testFile - Test data

prpFile - Per-ranked list performance file: Model's performance on each of the ranked list. These won't be saved if prpFile="".

test

public void test(java.util.List<java.lang.String> modelFiles,
                 java.util.List<java.lang.String> testFiles,
                 java.lang.String prpFile)

Similar to the above, except data has already been splitted. The k-th model will be applied on the k-th test file.

Parameters:

modelFiles -

testFiles -

prpFile -

testWithScoreFile

public void testWithScoreFile(java.lang.String testFile,
                              java.lang.String scoreFile)

Re-order the input rankings and measure their effectiveness (in -metric2T)

Parameters:

testFile - Input rankings

scoreFile - The model score file on each of the documents

score

public void score(java.lang.String modelFile,
                  java.lang.String testFile,
                  java.lang.String outputFile)

Write the model's score for each of the documents in a test rankings.

Parameters:

modelFile - Pre-trained model

testFile - Test data

outputFile - Output file

score

public void score(java.util.List<java.lang.String> modelFiles,
                  java.lang.String testFile,
                  java.lang.String outputFile)

Write the models' score for each of the documents in a test rankings. These test rankings are splitted into k chunks where k=|models|. Each model is applied on the data from the corresponding fold.

Parameters:

modelFiles -

testFile -

outputFile -

score

public void score(java.util.List<java.lang.String> modelFiles,
                  java.util.List<java.lang.String> testFiles,
                  java.lang.String outputFile)

Similar to the above, except data has already been split. The k-th model will be applied on the k-th test file.

Parameters:

modelFiles -

testFiles -

outputFile -

rank

public void rank(java.lang.String modelFile,
                 java.lang.String testFile,
                 java.lang.String indriRanking)

Use a pre-trained model to re-rank the test rankings. Save the output ranking in indri's run format

Parameters:

modelFile -

testFile -

indriRanking -

rank

public void rank(java.lang.String testFile,
                 java.lang.String indriRanking)

Generate a ranking in Indri's format from the input ranking

Parameters:

testFile -

indriRanking -

rank

public void rank(java.util.List<java.lang.String> modelFiles,
                 java.lang.String testFile,
                 java.lang.String indriRanking)

Use k pre-trained models to re-rank the test rankings. Test rankings will be splitted into k fold, where k=|models|. Each model will be used to rank the data from the corresponding fold. Save the output ranking in indri's run format.

Parameters:

modelFiles -

testFile -

indriRanking -

rank

public void rank(java.util.List<java.lang.String> modelFiles,
                 java.util.List<java.lang.String> testFiles,
                 java.lang.String indriRanking)

Similar to the above, except data has already been splitted. The k-th model will be applied on the k-th test file.

Parameters:

modelFiles -

testFiles -

indriRanking -

savePerRankListPerformanceFile

public void savePerRankListPerformanceFile(java.util.List<java.lang.String> ids,
                                           java.util.List<java.lang.Double> scores,
                                           java.lang.String prpFile)

Save systems' performance to file

Parameters:

ids - Ranked list IDs.

scores - Evaluation score (in whatever measure specified/calculated upstream such as NDCG@k, ERR@k, etc.)

prpFile - Output filename.