Package de.gsi.dataset.utils

Class DataSetUtils

    • Field Detail

      • useFloat32BinaryStandard

        protected static boolean useFloat32BinaryStandard

      • exportMetaDataByDefault

        protected static boolean exportMetaDataByDefault

    • Method Detail

      • copyDataSet

        public static AbstractDataSet<?> copyDataSet​(DataSet ds)
        perform and return a deep copy of the data set
        Parameters:
        ds - data set to be copied
        Returns:
        deep copy of data set

      • cropToLength

        protected static double[] cropToLength​(double[] in,
                                       int length)
        small helper routine to crop data array in case it's to long
        Parameters:
        in - input data array
        length - length of output array
        Returns:
        cropped/zero-padded array of size length.

      • error

        public static double error​(DataSet dataSet,
                           DataSetUtils.ErrType eType,
                           double x)
        convenience short-hand notation for getting error variable at an exact x value. Returns 0.0 if no errors are defined.
        Parameters:
        dataSet - the source data set
        eType - the error type
        x - the data set x-value for which the error should be interpolated
        Returns:
        the given interpolated error

      • error

        public static double error​(DataSet dataSet,
                           DataSetUtils.ErrType eType,
                           int index)
        convenience short-hand notation for getting error variable at a specific data point index. Returns 0.0 if no errors are defined.
        Parameters:
        dataSet - the source data set
        eType - the error type
        index - the data set index
        Returns:
        the given error at data point "index"

      • error

        protected static double error​(DataSet dataSet,
                              DataSetUtils.ErrType eType,
                              int index,
                              double x,
                              boolean interpolate)
        convenience short-hand notation for getting error variables. Returns 0.0 if no errors are defined.
        Parameters:
        dataSet - the source data set
        eType - the error type
        index - the data set index
        x - the data set x-value for which the error should be interpolated
        interpolate - determines if the value at index or the interpolated value at x should be returned
        Returns:
        the given error

      • errors

        public static double[] errors​(DataSet dataSet,
                              DataSetUtils.ErrType eType)
        convenience short-hand notation for getting error variables (if defined for dataset)
        Parameters:
        dataSet - the source data set
        eType - the error type
        Returns:
        the given error array (cropped to data set length if necessary)

      • evaluateAutoCompression

        protected static DataSetUtils.Compression evaluateAutoCompression​(java.lang.String fileName)
        Determine the compression method from the file extension.
        Parameters:
        fileName - to be opened
        Returns:
        Compression Enum for the extension of the supplied filename. Defaults to Compression.NONE

      • getFileName

        public static java.lang.String getFileName​(DataSet dataSet,
                                           java.lang.String fileName)
        Produce a filename from a dataSet and a String that can contain {datafield;type;format} style placeholders. Datafield references a field in the metadata as specified by the metaDataDataSet interface. The special field "systemTime" can be used to use the current system time. Also the standard DataSet fields can be used ("dataSetName", "xMin", "xMax", "yMin", "yMax"). The optional type field supports "string", "date", "int/long" and "float/double", where "string" is the default. The optional format field can be used to provide format strings. The default for date is SimpleDateFormat "yyyyMMdd_HHmmss", for int and float it is printf's "%d" and "%e".
        Parameters:
        dataSet - A dataSet containing all the data field referenced in the filename pattern
        fileName - Filename (with "{metadatafield;type;format}" placeholders for variables)
        Returns:
        The filename with the placeholders replaced

      • getISODate

        public static java.lang.String getISODate​(long timeMillis,
                                          java.lang.String format)
        Get ISO date from milliseconds since Jan 01, 1970
        Parameters:
        timeMillis - time to be converted
        format - time format string
        Returns:
        ISO formatted UTC date-time string

      • getKey

        protected static java.lang.String getKey​(java.lang.String line,
                                         java.lang.String replace)

      • getValue

        protected static java.lang.String getValue​(java.lang.String line)

      • readDataSetFromByteArray

        public static DataSet readDataSetFromByteArray​(byte[] byteArray)
        Read a Dataset from a byte array containing comma separated values.
        The data format is a custom extension of csv with an additional #-commented Metadata Header and a $-commented column header. Expects the following columns in this order to be present: index, x, y, eyn, eyp.
        Parameters:
        byteArray - byte array.
        Returns:
        DataSet with the data and metadata read from the file

      • readDataSetFromFile

        public static DataSet readDataSetFromFile​(java.lang.String fileName)
        Read a Dataset from a file containing comma separated values.
        Automatically determines compression from the file extension.
        The data format is a custom extension of csv with an additional #-commented Metadata Header and a $-commented column header. Expects the following columns in this order to be present: index, x, y, eyn, eyp.
        Parameters:
        fileName - Path and name of file containing csv data.
        Returns:
        DataSet with the data and metadata read from the file

      • readDataSetFromFile

        public static DataSet readDataSetFromFile​(java.lang.String fileName,
                                          DataSetUtils.Compression compression)
        Read a Dataset from a file containing comma separated values.
        The data format is a custom extension of csv with an additional #-commented Metadata Header and a $-commented column header. Expects the following columns in this order to be present: index, x, y, eyn, eyp.
        Parameters:
        fileName - Path and name of file containing csv data.
        compression - Compression of the file (GZIP, ZIP or NONE). Supply AUTO or omit this value to use file extension.
        Returns:
        DataSet with the data and metadata read from the file

      • readDataSetFromStream

        public static DataSet readDataSetFromStream​(DataSetUtils.SplitCharByteInputStream inputStream)
        Read a Dataset from a stream containing comma separated values.
        The data format is a custom extension of csv with an additional #-commented Metadata Header and a $-commented column header. Expects the following columns in this order to be present: index, x, y, eyn, eyp.
        Parameters:
        inputStream - Path and name of file containing csv data.
        Returns:
        DataSet with the data and metadata read from the file

      • readNumericDataFromFile

        protected static DataSet readNumericDataFromFile​(java.io.BufferedReader inputFile,
                                                 java.lang.String dataSetName,
                                                 boolean is3D,
                                                 int nSamplesGuessed)

      • setExportMetaDataByDefault

        public static void setExportMetaDataByDefault​(boolean state)

      • setUseFloat32BinaryStandard

        public static void setUseFloat32BinaryStandard​(boolean state)

      • useExportMetaDataByDefault

        public static boolean useExportMetaDataByDefault()

      • useFloat32BinaryStandard

        public static boolean useFloat32BinaryStandard()

      • writeDataSetToByteArray

        public static void writeDataSetToByteArray​(DataSet dataSet,
                                           java.io.ByteArrayOutputStream byteOutput,
                                           boolean binary,
                                           boolean asFloat)
        Write data set into byte buffer.
        Parameters:
        dataSet - The DataSet to export
        byteOutput - byte output stream (N.B. keep caching this object)
        binary - true: encode data as binary (smaller size, performance), or false as string (human readable, easier debugging)
        asFloat - true: encode data as binary floats (smaller size, performance), or false as double (better precision)

      • writeDataSetToFile

        public static java.lang.String writeDataSetToFile​(DataSet dataSet,
                                                  java.nio.file.Path path,
                                                  java.lang.String fileName)
        Export the contents of the supplied dataSet to file as comma separated values with an additional comment header containing metaData if existent.
        The filename can contain placeholders of the form {metadatafield;type;format}, where metadatafield references a field in the metadata as specified by the metaDataDataSet interface. The special field systemTime can be used to use the current system time. Also the standard DataSet fields can be used ("dataSetName", "xMin", "xMax", "yMin", "yMax"). The optional type field supports "string", "date", "int/long" and "float/double", where "string" is the default. The optional format field can be used to provide format strings. The default for date is SimpleDateFormat "yyyyMMdd_HHmmss", for int and float it is printf's "%d" and "%e".
        The compression method is automatically determined from the file extension.
        The data format is a custom extension of csv with an additional #-commented Metadata Header and a $-commented column header. Expects the following columns in this order to be present: index, x, y, eyn, eyp.
        Parameters:
        dataSet - The DataSet to export
        path - Path to the location of the file
        fileName - Filename (with "{metadatafield;type;format}" placeholders for variables)
        Returns:
        actual name of the file that was written or none in case of errors

      • writeDataSetToFile

        public static java.lang.String writeDataSetToFile​(DataSet dataSet,
                                                  java.nio.file.Path path,
                                                  java.lang.String fileName,
                                                  boolean binary)
        Export the contents of the supplied dataSet to file as comma separated values with an additional comment header containing metaData if existent.
        The filename can contain placeholders of the form {metadatafield;type;format}, where metadatafield references a field in the metadata as specified by the metaDataDataSet interface. The special field systemTime can be used to use the current system time. Also the standard DataSet fields can be used ("dataSetName", "xMin", "xMax", "yMin", "yMax"). The optional type field supports "string", "date", "int/long" and "float/double", where "string" is the default. The optional format field can be used to provide format strings. The default for date is SimpleDateFormat "yyyyMMdd_HHmmss", for int and float it is printf's "%d" and "%e".
        The compression method is automatically determined from the file extension.
        The data format is a custom extension of csv with an additional #-commented Metadata Header and a $-commented column header. Expects the following columns in this order to be present: index, x, y, eyn, eyp.
        Parameters:
        dataSet - The DataSet to export
        path - Path to the location of the file
        fileName - Filename (with "{metadatafield;type;format}" placeholders for variables)
        binary - true: whether to store data as binary or string
        Returns:
        actual name of the file that was written or none in case of errors

      • writeDataSetToFile

        public static java.lang.String writeDataSetToFile​(DataSet dataSet,
                                                  java.nio.file.Path path,
                                                  java.lang.String fileName,
                                                  DataSetUtils.Compression compression)
        Export the contents of the supplied dataSet to file as comma separated values with an additional comment header containing metaData if existent.
        The filename can contain placeholders of the form {metadatafield;type;format}, where metadatafield references a field in the metadata as specified by the metaDataDataSet interface. The special field systemTime can be used to use the current system time. Also the standard DataSet fields can be used ("dataSetName", "xMin", "xMax", "yMin", "yMax"). The optional type field supports "string", "date", "int/long" and "float/double", where "string" is the default. The optional format field can be used to provide format strings. The default for date is SimpleDateFormat "yyyyMMdd_HHmmss", for int and float it is printf's "%d" and "%e".
        The compression method is automatically determined from the file extension.
        The data format is a custom extension of csv with an additional #-commented Metadata Header and a $-commented column header. Expects the following columns in this order to be present: index, x, y, eyn, eyp.
        Parameters:
        dataSet - The DataSet to export
        path - Path to the location of the file
        fileName - Filename (with "{metadatafield;type;format}" placeholders for variables)
        compression - compression type @see Compression
        Returns:
        actual name of the file that was written or none in case of errors

      • writeDataSetToFile

        public static java.lang.String writeDataSetToFile​(DataSet dataSet,
                                                  java.nio.file.Path path,
                                                  java.lang.String fileName,
                                                  DataSetUtils.Compression compression,
                                                  boolean binary)
        Export the contents of the supplied dataSet to file as comma separated values with an additional comment header containing metaData if existent.
        The filename can contain placeholders of the form {metadatafield;type;format}, where metadatafield references a field in the metadata as specified by the metaDataDataSet interface. The special field systemTime can be used to use the current system time. Also the standard DataSet fields can be used ("dataSetName", "xMin", "xMax", "yMin", "yMax"). The optional type field supports "string", "date", "int/long" and "float/double", where "string" is the default. The optional format field can be used to provide format strings. The default for date is SimpleDateFormat "yyyyMMdd_HHmmss", for int and float it is printf's "%d" and "%e".
        The data format is a custom extension of csv with an additional #-commented Metadata Header and a $-commented column header. Expects the following columns in this order to be present: index, x, y, eyn, eyp.
        Parameters:
        dataSet - The DataSet to export
        path - Path to the location of the file
        fileName - Filename (with "{metadatafield;type;format}" placeholders for variables)
        compression - Compression of the file (GZIP, ZIP or NONE). Supply AUTO or omit this value to use file extension.
        binary - true: whether to store data as binary or string
        Returns:
        actual name of the file that was written or none in case of errors

      • writeHeaderDataToStream

        protected static void writeHeaderDataToStream​(java.io.OutputStream outputStream,
                                              DataSet dataSet)

      • writeMetaDataToStream

        protected static void writeMetaDataToStream​(java.io.OutputStream outputStream,
                                            DataSet dataSet)

      • writeNumericDataToStream

        protected static void writeNumericDataToStream​(java.io.OutputStream outputFile,
                                               DataSet dataSet)