de.gsi.dataset.utils

Class DataSetUtils

java.lang.Object

de.gsi.dataset.utils.DataSetUtilsHelper

de.gsi.dataset.utils.DataSetUtils

public class DataSetUtils
extends DataSetUtilsHelper

Author:

braeun, rstein, akrimm

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	DataSetUtils.Compression Supported Compression Types
static class	DataSetUtils.ErrType Error type short handle

Field Summary

Fields

Modifier and Type	Field and Description
protected static boolean	exportMetaDataByDefault
protected static boolean	useFloat32BinaryStandard

Fields inherited from class de.gsi.dataset.utils.DataSetUtilsHelper

BYTE_ARRAY_CACHE_LOCK, byteArrayCache, STRING_BUFFER_CACHE_LOCK, stringBuilderCache

Method Summary

Modifier and Type	Method and Description
static AbstractDataSet<?>	copyDataSet(DataSet2D ds) perform and return a deep copy of the data set
protected static double[]	cropToLength(double[] in, int length) small helper routine to crop data array in case it's to long
static double	error(DataSet dataSet, DataSetUtils.ErrType eType, double x) convenience short-hand notation for getting error variable at an exact x value.
static double	error(DataSet dataSet, DataSetUtils.ErrType eType, int index) convenience short-hand notation for getting error variable at a specific data point index.
protected static double	error(DataSet dataSet, DataSetUtils.ErrType eType, int index, double x, boolean interpolate) convenience short-hand notation for getting error variables.
static double[]	errors(DataSet dataSet, DataSetUtils.ErrType eType) convenience short-hand notation for getting error variables (if defined for dataset)
static String	getFileName(DataSet dataSet, String fileName) Produce a filename from a dataSet and a String that can contain {datafield;type;format} style placeholders.
static String	getISODate(long timeMillis, String format) Get ISO date from milliseconds since Jan 01, 1970
protected static String	getKey(String line, String replace)
protected static String	getValue(String line)
static DataSet	readDataSetFromByteArray(byte[] byteArray) Read a Dataset from a byte array containing comma separated values. The data format is a custom extension of csv with an additional #-commented Metadata Header and a $-commented column header.
static DataSet	readDataSetFromFile(String fileName) Read a Dataset from a file containing comma separated values. Automatically determines compression from the file extension. The data format is a custom extension of csv with an additional #-commented Metadata Header and a $-commented column header.
static DataSet	readDataSetFromFile(String fileName, DataSetUtils.Compression compression) Read a Dataset from a file containing comma separated values. The data format is a custom extension of csv with an additional #-commented Metadata Header and a $-commented column header.
static DataSet	readDataSetFromStream(de.gsi.dataset.utils.DataSetUtils.SplitCharByteInputStream inputStream) Read a Dataset from a stream containing comma separated values. The data format is a custom extension of csv with an additional #-commented Metadata Header and a $-commented column header.
protected static DataSet	readNumericDataFromFile(BufferedReader inputFile, String dataSetName, boolean is3D, int nSamplesGuessed)
static void	setExportMetaDataByDefault(boolean state)
static void	setUseFloat32BinaryStandard(boolean state)
static boolean	useExportMetaDataByDefault()
static boolean	useFloat32BinaryStandard()
static void	writeDataSetToByteArray(DataSet dataSet, ByteArrayOutputStream byteOutput, boolean binary, boolean asFloat) Write data set into byte buffer.
static String	writeDataSetToFile(DataSet dataSet, Path path, String fileName) Export the contents of the supplied dataSet to file as comma separated values with an additional comment header containing metaData if existent. The filename can contain placeholders of the form {metadatafield;type;format}, where metadatafield references a field in the metadata as specified by the metaDataDataSet interface.
static String	writeDataSetToFile(DataSet dataSet, Path path, String fileName, boolean binary) Export the contents of the supplied dataSet to file as comma separated values with an additional comment header containing metaData if existent. The filename can contain placeholders of the form {metadatafield;type;format}, where metadatafield references a field in the metadata as specified by the metaDataDataSet interface.
static String	writeDataSetToFile(DataSet dataSet, Path path, String fileName, DataSetUtils.Compression compression) Export the contents of the supplied dataSet to file as comma separated values with an additional comment header containing metaData if existent. The filename can contain placeholders of the form {metadatafield;type;format}, where metadatafield references a field in the metadata as specified by the metaDataDataSet interface.
static String	writeDataSetToFile(DataSet dataSet, Path path, String fileName, DataSetUtils.Compression compression, boolean binary) Export the contents of the supplied dataSet to file as comma separated values with an additional comment header containing metaData if existent. The filename can contain placeholders of the form {metadatafield;type;format}, where metadatafield references a field in the metadata as specified by the metaDataDataSet interface.
protected static void	writeHeaderDataToStream(OutputStream outputStream, DataSet dataSet)
protected static void	writeMetaDataToStream(OutputStream outputStream, DataSet dataSet)
protected static void	writeNumericDataToStream(OutputStream outputFile, DataSet dataSet)

Methods inherited from class de.gsi.dataset.utils.DataSetUtilsHelper

getCachedDoubleArray, getCachedStringBuilder, integralSimple, mean, readDoubleArrayFromBuffer, release, release, rootMeanSquare, toDoubleArray, toFloatArray, writeDoubleArrayAsFloatToByteBuffer, writeDoubleArrayToByteBuffer

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

useFloat32BinaryStandard

protected static boolean useFloat32BinaryStandard

exportMetaDataByDefault

protected static boolean exportMetaDataByDefault

Method Detail

copyDataSet

public static AbstractDataSet<?> copyDataSet(DataSet2D ds)

perform and return a deep copy of the data set

Parameters:

ds - data set to be copied

Returns:

deep copy of data set

cropToLength

protected static double[] cropToLength(double[] in,
                                       int length)

small helper routine to crop data array in case it's to long

Parameters:

in - input data array

length - length of output array

Returns:

cropped/zero-padded array of size length.

error

public static double error(DataSet dataSet,
                           DataSetUtils.ErrType eType,
                           double x)

convenience short-hand notation for getting error variable at an exact x value. Returns 0.0 if no errors are defined.

Parameters:

dataSet - the source data set

eType - the error type

x - the data set x-value for which the error should be interpolated

Returns:

the given interpolated error

error

public static double error(DataSet dataSet,
                           DataSetUtils.ErrType eType,
                           int index)

convenience short-hand notation for getting error variable at a specific data point index. Returns 0.0 if no errors are defined.

Parameters:

dataSet - the source data set

eType - the error type

index - the data set index

Returns:

the given error at data point "index"

error

protected static double error(DataSet dataSet,
                              DataSetUtils.ErrType eType,
                              int index,
                              double x,
                              boolean interpolate)

convenience short-hand notation for getting error variables. Returns 0.0 if no errors are defined.

Parameters:

dataSet - the source data set

eType - the error type

index - the data set index

x - the data set x-value for which the error should be interpolated

interpolate - determines if the value at index or the interpolated value at x should be returned

Returns:

the given error

errors

public static double[] errors(DataSet dataSet,
                              DataSetUtils.ErrType eType)

convenience short-hand notation for getting error variables (if defined for dataset)

Parameters:

dataSet - the source data set

eType - the error type

Returns:

the given error array (cropped to data set length if necessary)

getFileName

public static String getFileName(DataSet dataSet,
                                 String fileName)

Produce a filename from a dataSet and a String that can contain {datafield;type;format} style placeholders. Datafield references a field in the metadata as specified by the metaDataDataSet interface. The special field "systemTime" can be used to use the current system time. Also the standard DataSet fields can be used ("dataSetName", "xMin", "xMax", "yMin", "yMax"). The optional type field supports "string", "date", "int/long" and "float/double", where "string" is the default. The optional format field can be used to provide format strings. The default for date is SimpleDateFormat "yyyyMMdd_HHmmss", for int and float it is printf's "%d" and "%e".

Parameters:

dataSet - A dataSet containing all the data field referenced in the filename pattern

fileName - Filename (with "{metadatafield;type;format}" placeholders for variables)

Returns:

The filename with the placeholders replaced

getISODate

public static String getISODate(long timeMillis,
                                String format)

Get ISO date from milliseconds since Jan 01, 1970

Parameters:

timeMillis - time to be converted

format - time format string

Returns:

ISO formatted UTC date-time string

getKey

protected static String getKey(String line,
                               String replace)

getValue

protected static String getValue(String line)

readDataSetFromByteArray

public static DataSet readDataSetFromByteArray(byte[] byteArray)

Read a Dataset from a byte array containing comma separated values.
The data format is a custom extension of csv with an additional #-commented Metadata Header and a $-commented column header. Expects the following columns in this order to be present: index, x, y, eyn, eyp.

Parameters:

byteArray - byte array.

Returns:

DataSet with the data and metadata read from the file

readDataSetFromFile

public static DataSet readDataSetFromFile(String fileName)

Read a Dataset from a file containing comma separated values.
Automatically determines compression from the file extension.
The data format is a custom extension of csv with an additional #-commented Metadata Header and a $-commented column header. Expects the following columns in this order to be present: index, x, y, eyn, eyp.

Parameters:

fileName - Path and name of file containing csv data.

Returns:

DataSet with the data and metadata read from the file

readDataSetFromFile

public static DataSet readDataSetFromFile(String fileName,
                                          DataSetUtils.Compression compression)

Read a Dataset from a file containing comma separated values.
The data format is a custom extension of csv with an additional #-commented Metadata Header and a $-commented column header. Expects the following columns in this order to be present: index, x, y, eyn, eyp.

Parameters:

fileName - Path and name of file containing csv data.

compression - Compression of the file (GZIP, ZIP or NONE). Supply AUTO or omit this value to use file extension.

Returns:

DataSet with the data and metadata read from the file

readDataSetFromStream

public static DataSet readDataSetFromStream(de.gsi.dataset.utils.DataSetUtils.SplitCharByteInputStream inputStream)

Read a Dataset from a stream containing comma separated values.
The data format is a custom extension of csv with an additional #-commented Metadata Header and a $-commented column header. Expects the following columns in this order to be present: index, x, y, eyn, eyp.

Parameters:

inputStream - Path and name of file containing csv data.

Returns:

DataSet with the data and metadata read from the file

readNumericDataFromFile

protected static DataSet readNumericDataFromFile(BufferedReader inputFile,
                                                 String dataSetName,
                                                 boolean is3D,
                                                 int nSamplesGuessed)

setExportMetaDataByDefault

public static void setExportMetaDataByDefault(boolean state)

setUseFloat32BinaryStandard

public static void setUseFloat32BinaryStandard(boolean state)

useExportMetaDataByDefault

public static boolean useExportMetaDataByDefault()

useFloat32BinaryStandard

public static boolean useFloat32BinaryStandard()

writeDataSetToByteArray

public static void writeDataSetToByteArray(DataSet dataSet,
                                           ByteArrayOutputStream byteOutput,
                                           boolean binary,
                                           boolean asFloat)

Write data set into byte buffer.

Parameters:

dataSet - The DataSet to export

byteOutput - byte output stream (N.B. keep caching this object)

binary - true: encode data as binary (smaller size, performance), or false as string (human readable, easier debugging)

asFloat - true: encode data as binary floats (smaller size, performance), or false as double (better precision)

writeDataSetToFile

public static String writeDataSetToFile(DataSet dataSet,
                                        Path path,
                                        String fileName)

Export the contents of the supplied dataSet to file as comma separated values with an additional comment header containing metaData if existent.
The filename can contain placeholders of the form {metadatafield;type;format}, where metadatafield references a field in the metadata as specified by the metaDataDataSet interface. The special field systemTime can be used to use the current system time. Also the standard DataSet fields can be used ("dataSetName", "xMin", "xMax", "yMin", "yMax"). The optional type field supports "string", "date", "int/long" and "float/double", where "string" is the default. The optional format field can be used to provide format strings. The default for date is SimpleDateFormat "yyyyMMdd_HHmmss", for int and float it is printf's "%d" and "%e".
The compression method is automatically determined from the file extension.
The data format is a custom extension of csv with an additional #-commented Metadata Header and a $-commented column header. Expects the following columns in this order to be present: index, x, y, eyn, eyp.

Parameters:

dataSet - The DataSet to export

path - Path to the location of the file

fileName - Filename (with "{metadatafield;type;format}" placeholders for variables)

Returns:

actual name of the file that was written or none in case of errors

writeDataSetToFile

public static String writeDataSetToFile(DataSet dataSet,
                                        Path path,
                                        String fileName,
                                        boolean binary)

Export the contents of the supplied dataSet to file as comma separated values with an additional comment header containing metaData if existent.
The filename can contain placeholders of the form {metadatafield;type;format}, where metadatafield references a field in the metadata as specified by the metaDataDataSet interface. The special field systemTime can be used to use the current system time. Also the standard DataSet fields can be used ("dataSetName", "xMin", "xMax", "yMin", "yMax"). The optional type field supports "string", "date", "int/long" and "float/double", where "string" is the default. The optional format field can be used to provide format strings. The default for date is SimpleDateFormat "yyyyMMdd_HHmmss", for int and float it is printf's "%d" and "%e".
The compression method is automatically determined from the file extension.
The data format is a custom extension of csv with an additional #-commented Metadata Header and a $-commented column header. Expects the following columns in this order to be present: index, x, y, eyn, eyp.

Parameters:

dataSet - The DataSet to export

path - Path to the location of the file

fileName - Filename (with "{metadatafield;type;format}" placeholders for variables)

binary - true: whether to store data as binary or string

Returns:

actual name of the file that was written or none in case of errors

writeDataSetToFile

public static String writeDataSetToFile(DataSet dataSet,
                                        Path path,
                                        String fileName,
                                        DataSetUtils.Compression compression)

Export the contents of the supplied dataSet to file as comma separated values with an additional comment header containing metaData if existent.
The filename can contain placeholders of the form {metadatafield;type;format}, where metadatafield references a field in the metadata as specified by the metaDataDataSet interface. The special field systemTime can be used to use the current system time. Also the standard DataSet fields can be used ("dataSetName", "xMin", "xMax", "yMin", "yMax"). The optional type field supports "string", "date", "int/long" and "float/double", where "string" is the default. The optional format field can be used to provide format strings. The default for date is SimpleDateFormat "yyyyMMdd_HHmmss", for int and float it is printf's "%d" and "%e".
The compression method is automatically determined from the file extension.
The data format is a custom extension of csv with an additional #-commented Metadata Header and a $-commented column header. Expects the following columns in this order to be present: index, x, y, eyn, eyp.

Parameters:

dataSet - The DataSet to export

path - Path to the location of the file

fileName - Filename (with "{metadatafield;type;format}" placeholders for variables)

compression - compression type @see Compression

Returns:

actual name of the file that was written or none in case of errors

writeDataSetToFile

public static String writeDataSetToFile(DataSet dataSet,
                                        Path path,
                                        String fileName,
                                        DataSetUtils.Compression compression,
                                        boolean binary)

Export the contents of the supplied dataSet to file as comma separated values with an additional comment header containing metaData if existent.
The filename can contain placeholders of the form {metadatafield;type;format}, where metadatafield references a field in the metadata as specified by the metaDataDataSet interface. The special field systemTime can be used to use the current system time. Also the standard DataSet fields can be used ("dataSetName", "xMin", "xMax", "yMin", "yMax"). The optional type field supports "string", "date", "int/long" and "float/double", where "string" is the default. The optional format field can be used to provide format strings. The default for date is SimpleDateFormat "yyyyMMdd_HHmmss", for int and float it is printf's "%d" and "%e".
The data format is a custom extension of csv with an additional #-commented Metadata Header and a $-commented column header. Expects the following columns in this order to be present: index, x, y, eyn, eyp.

Parameters:

dataSet - The DataSet to export

path - Path to the location of the file

fileName - Filename (with "{metadatafield;type;format}" placeholders for variables)

compression - Compression of the file (GZIP, ZIP or NONE). Supply AUTO or omit this value to use file extension.

binary - true: whether to store data as binary or string

Returns:

actual name of the file that was written or none in case of errors

writeHeaderDataToStream

protected static void writeHeaderDataToStream(OutputStream outputStream,
                                              DataSet dataSet)

writeMetaDataToStream

protected static void writeMetaDataToStream(OutputStream outputStream,
                                            DataSet dataSet)

writeNumericDataToStream

protected static void writeNumericDataToStream(OutputStream outputFile,
                                               DataSet dataSet)