org.encog.neural.data.buffer
Class BufferedNeuralDataSet

java.lang.Object
  org.encog.persist.BasicPersistedObject
      org.encog.neural.data.buffer.BufferedNeuralDataSet

All Implemented Interfaces:

Serializable, Iterable<NeuralDataPair>, EngineDataSet, EngineIndexableSet, Indexable, NeuralDataSet, EncogPersistedObject

public class BufferedNeuralDataSet
extends BasicPersistedObject
implements NeuralDataSet, Indexable, Serializable

This class is not memory based, so very long files can be used, without running out of memory. This dataset uses a Encog binary training file as a buffer. When used with a slower access dataset, such as CSV, XML or SQL, where parsing must occur, this dataset can be used to load from the slower dataset and train at much higher speeds. This class makes use of Java file channels for maximum file access performance. If you are going to create a binary file, by using the add methods, you must call beginLoad to cause Encog to open an output file. Once the data has been loaded, call endLoad. You can also use the BinaryDataLoader class, with a CODEC, to load many other popular external formats. The binary files produced by this class are in the Encog binary training format, and can be used with any Encog platform. Encog binary files are stored using "little endian" numbers.

See Also:

Serialized Form

Field Summary

static String	ERROR_ADD Error message for ADD.
static String	ERROR_REMOVE Error message for REMOVE.

Constructor Summary

BufferedNeuralDataSet(File binaryFile)
Construct the dataset using the specified binary file.

Method Summary

void	add(NeuralData data1) Add only input data, for an unsupervised dataset.
void	add(NeuralData inputData, NeuralData idealData) Add both the input and ideal data.
void	add(NeuralDataPair pair) Add a data pair of both input and ideal data.
void	beginLoad(int inputSize, int idealSize) Begin loading to the binary file.
void	close() Close the dataset.
Persistor	createPersistor() Create a persistor for this object.
void	endLoad() This method should be called once all the data has been loaded.
EncogEGBFile	getEGB()
File	getFile()
int	getIdealSize()
int	getInputSize()
BufferedNeuralDataSet	getOwner()
void	getRecord(long index, EngineData pair) Read an individual record.
long	getRecordCount() Determine the total number of records in the set.
boolean	isSupervised()
Iterator<NeuralDataPair>	iterator()
void	load(NeuralDataSet training) Load the specified training set.
NeuralDataSet	loadToMemory() Load the binary dataset to memory.
void	open() Open the binary file for reading.
BufferedNeuralDataSet	openAdditional() Opens an additional instance of this dataset.
void	removeAdditional(BufferedNeuralDataSet child) Remove an additional dataset that was created.
void	setOwner(BufferedNeuralDataSet owner) Set the owner of this dataset.

Methods inherited from class org.encog.persist.BasicPersistedObject

getCollection, getDescription, getName, setCollection, setDescription, setName

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

ERROR_ADD

public static final String ERROR_ADD

Error message for ADD.

See Also:

Constant Field Values

ERROR_REMOVE

public static final String ERROR_REMOVE

Error message for REMOVE.

See Also:

Constant Field Values

Constructor Detail

BufferedNeuralDataSet

public BufferedNeuralDataSet(File binaryFile)

Construct the dataset using the specified binary file.

Parameters:

binaryFile - The file to use.

Method Detail

open

public void open()

Open the binary file for reading.

iterator

public Iterator<NeuralDataPair> iterator()

Specified by:

iterator in interface Iterable<NeuralDataPair>

Returns:

An iterator.

getRecordCount

public long getRecordCount()

Description copied from interface: EngineIndexableSet

Determine the total number of records in the set.

Specified by:

getRecordCount in interface EngineIndexableSet

Returns:

The record count.

getRecord

public void getRecord(long index,
                      EngineData pair)

Read an individual record.

Specified by:

getRecord in interface EngineIndexableSet

Parameters:

index - The zero-based index. Specify 0 for the first record, 1 for the second, and so on.

pair - THe data to read.

openAdditional

public BufferedNeuralDataSet openAdditional()

Description copied from interface: EngineIndexableSet

Opens an additional instance of this dataset.

Specified by:

openAdditional in interface EngineIndexableSet

Returns:

An additional training set.

add

public void add(NeuralData data1)

Add only input data, for an unsupervised dataset.

Specified by:

add in interface NeuralDataSet

Parameters:

data1 - The data to be added.

add

public void add(NeuralData inputData,
                NeuralData idealData)

Add both the input and ideal data.

Specified by:

add in interface NeuralDataSet

Parameters:

inputData - The input data.

idealData - The ideal data.

add

public void add(NeuralDataPair pair)

Add a data pair of both input and ideal data.

Specified by:

add in interface NeuralDataSet

Parameters:

pair - The pair to add.

close

public void close()

Close the dataset.

Specified by:

close in interface NeuralDataSet

getIdealSize

public int getIdealSize()

Specified by:

getIdealSize in interface EngineDataSet

Specified by:

getIdealSize in interface NeuralDataSet

Returns:

The ideal data size.

getInputSize

public int getInputSize()

Specified by:

getInputSize in interface EngineDataSet

Specified by:

getInputSize in interface NeuralDataSet

Returns:

The input data size.

isSupervised

public boolean isSupervised()

Specified by:

isSupervised in interface EngineDataSet

Specified by:

isSupervised in interface NeuralDataSet

Returns:

True if this dataset is supervised.

getOwner

public BufferedNeuralDataSet getOwner()

Returns:

If this dataset was created by openAdditional, the set that created this object is the owner. Return the owner.

setOwner

public void setOwner(BufferedNeuralDataSet owner)

Set the owner of this dataset.

Parameters:

owner - The owner.

removeAdditional

public void removeAdditional(BufferedNeuralDataSet child)

Remove an additional dataset that was created.

Parameters:

child - The additional dataset to remove.

beginLoad

public void beginLoad(int inputSize,
                      int idealSize)

Begin loading to the binary file. After calling this method the add methods may be called.

Parameters:

inputSize - The input size.

idealSize - The ideal size.

endLoad

public void endLoad()

This method should be called once all the data has been loaded. The underlying file will be closed. The binary fill will then be opened for reading.

createPersistor

public Persistor createPersistor()

Description copied from class: BasicPersistedObject

Create a persistor for this object.

Specified by:

createPersistor in interface EncogPersistedObject

Overrides:

createPersistor in class BasicPersistedObject

Returns:

An Encog persistor for this object.

getFile

public File getFile()

Returns:

The binary file used.

getEGB

public EncogEGBFile getEGB()

Returns:

The EGB file to use.

loadToMemory

public NeuralDataSet loadToMemory()

Load the binary dataset to memory. Memory access is faster.

Returns:

A memory dataset.

load

public void load(NeuralDataSet training)

Load the specified training set.

Parameters:

training - The training set to load.