ml.shifu.guagua.example.nn

Class MemoryDiskMLDataSet

java.lang.Object

ml.shifu.guagua.example.nn.MemoryDiskMLDataSet

All Implemented Interfaces:

Iterable<org.encog.ml.data.MLDataPair>, org.encog.ml.data.MLDataSet

public class MemoryDiskMLDataSet
extends Object
implements org.encog.ml.data.MLDataSet

A hybrid data set combining BasicMLDataSet and BufferedMLDataSet together.

With this data set, element is added firstly in memory, if over maxByteSize then element will be added into disk.

This data set provide a very important feature to make in memory computing more stable. Even for some cases no enough memory, memory and disk will be leveraged together to accelerate computing.

Example almost same as BufferedMLDataSet:

 MemoryDiskMLDataSet dataSet = new MemoryDiskMLDataSet(400, "a.txt");
 dataSet.beginLoad(10, 1);
 dataSet.add(pair);
 dataSet.endLoad();
 
 Iterator iterator = dataSet.iterator();
 while(iterator.hasNext()) {
     MLDataPair next = iterator.next();
     ...
 }
 
 dataSet.close();
 

Author:

Zhang David (pengzhang@paypal.com)

Constructor Summary

Constructors

Constructor and Description
MemoryDiskMLDataSet(long maxByteSize, String fileName) Constructor with maxByteSize and fileName
MemoryDiskMLDataSet(long maxByteSize, String fileName, int inputCount, int outputCount) Constructor with maxByteSize, fileName, inputCount and outputCount.
MemoryDiskMLDataSet(String fileName, int inputCount, int outputCount) Constructor with fileName, inputCount and outputCount

Method Summary

Modifier and Type	Method and Description
void	add(org.encog.ml.data.MLData data)
void	add(org.encog.ml.data.MLData inputData, org.encog.ml.data.MLData idealData)
void	add(org.encog.ml.data.MLDataPair inputData)
void	beginLoad(int inputSize, int idealSize) Setting input variable size and output target size.
void	close()
void	endLoad() This method should be called once all the data has been loaded.
long	getDiskCount()
int	getIdealSize()
int	getInputSize()
long	getMemoryCount()
void	getRecord(long index, org.encog.ml.data.MLDataPair pair)
long	getRecordCount()
boolean	isSupervised()
Iterator<org.encog.ml.data.MLDataPair>	iterator()
static void	main(String[] args)
org.encog.ml.data.MLDataSet	openAdditional()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.lang.Iterable

forEach, spliterator

Constructor Detail

MemoryDiskMLDataSet

public MemoryDiskMLDataSet(String fileName,
                           int inputCount,
                           int outputCount)

Constructor with fileName, inputCount and outputCount

MemoryDiskMLDataSet

public MemoryDiskMLDataSet(long maxByteSize,
                           String fileName)

Constructor with maxByteSize and fileName

MemoryDiskMLDataSet

public MemoryDiskMLDataSet(long maxByteSize,
                           String fileName,
                           int inputCount,
                           int outputCount)

Constructor with maxByteSize, fileName, inputCount and outputCount.

Method Detail

beginLoad

public final void beginLoad(int inputSize,
                            int idealSize)

Setting input variable size and output target size.

Parameters:

inputSize - input variable size

idealSize - output target size

endLoad

public final void endLoad()

This method should be called once all the data has been loaded. The underlying file will be closed. The binary fill will then be opened for reading.

iterator

public Iterator<org.encog.ml.data.MLDataPair> iterator()

Specified by:

iterator in interface Iterable<org.encog.ml.data.MLDataPair>

getIdealSize

public int getIdealSize()

Specified by:

getIdealSize in interface org.encog.ml.data.MLDataSet

getInputSize

public int getInputSize()

Specified by:

getInputSize in interface org.encog.ml.data.MLDataSet

isSupervised

public boolean isSupervised()

Specified by:

isSupervised in interface org.encog.ml.data.MLDataSet

getRecordCount

public long getRecordCount()

Specified by:

getRecordCount in interface org.encog.ml.data.MLDataSet

getRecord

public void getRecord(long index,
                      org.encog.ml.data.MLDataPair pair)

Specified by:

getRecord in interface org.encog.ml.data.MLDataSet

openAdditional

public org.encog.ml.data.MLDataSet openAdditional()

Specified by:

openAdditional in interface org.encog.ml.data.MLDataSet

add

public void add(org.encog.ml.data.MLData data)

Specified by:

add in interface org.encog.ml.data.MLDataSet

add

public void add(org.encog.ml.data.MLData inputData,
                org.encog.ml.data.MLData idealData)

Specified by:

add in interface org.encog.ml.data.MLDataSet

add

public void add(org.encog.ml.data.MLDataPair inputData)

Specified by:

add in interface org.encog.ml.data.MLDataSet

close

public void close()

Specified by:

close in interface org.encog.ml.data.MLDataSet

getMemoryCount

public long getMemoryCount()

Returns:

the memoryCount

getDiskCount

public long getDiskCount()

Returns:

the diskCount

main

public static void main(String[] args)