Package de.gsi.dataset.utils

Class DataSetUtils

    • Method Summary

      All Methods Static Methods Concrete Methods 
      Modifier and Type Method Description
      static AbstractDataSet<?> copyDataSet​(DataSet ds)
      perform and return a deep copy of the data set
      protected static double[] cropToLength​(double[] in, int length)
      small helper routine to crop data array in case it's to long
      static double error​(DataSet dataSet, DataSetUtils.ErrType eType, double x)
      convenience short-hand notation for getting error variable at an exact x value.
      static double error​(DataSet dataSet, DataSetUtils.ErrType eType, int index)
      convenience short-hand notation for getting error variable at a specific data point index.
      protected static double error​(DataSet dataSet, DataSetUtils.ErrType eType, int index, double x, boolean interpolate)
      convenience short-hand notation for getting error variables.
      static double[] errors​(DataSet dataSet, DataSetUtils.ErrType eType)
      convenience short-hand notation for getting error variables (if defined for dataset)
      static java.lang.String getFileName​(DataSet dataSet, java.lang.String fileName)
      Produce a filename from a dataSet and a String that can contain {datafield;type;format} style placeholders.
      static java.lang.String getISODate​(long timeMillis, java.lang.String format)
      Get ISO date from milliseconds since Jan 01, 1970
      protected static java.lang.String getKey​(java.lang.String line, java.lang.String replace)  
      protected static java.lang.String getValue​(java.lang.String line)  
      static DoubleErrorDataSet readDataSetFromByteArray​(byte[] byteArray)
      Read a Dataset from a byte array containing comma separated values.
      The data format is a custom extension of csv with an additional #-commented Metadata Header and a $-commented column header.
      static DataSet readDataSetFromFile​(java.lang.String fileName)
      Read a Dataset from a file containing comma separated values.
      Automatically determines compression from the file extension.
      The data format is a custom extension of csv with an additional #-commented Metadata Header and a $-commented column header.
      static DataSet readDataSetFromFile​(java.lang.String fileName, DataSetUtils.Compression compression)
      Read a Dataset from a file containing comma separated values.
      The data format is a custom extension of csv with an additional #-commented Metadata Header and a $-commented column header.
      static DoubleErrorDataSet readDataSetFromStream​(de.gsi.dataset.utils.DataSetUtils.SplitCharByteInputStream inputStream)
      Read a Dataset from a stream containing comma separated values.
      The data format is a custom extension of csv with an additional #-commented Metadata Header and a $-commented column header.
      protected static void readNumericDataFromFile​(java.io.BufferedReader inputFile, DoubleErrorDataSet dataSet)  
      static void writeDataSetToByteArray​(DataSet dataSet, java.io.ByteArrayOutputStream byteOutput, boolean binary, boolean asFloat)
      Write data set into byte buffer.
      static java.lang.String writeDataSetToFile​(DataSet dataSet, java.nio.file.Path path, java.lang.String fileName)
      Export the contents of the supplied dataSet to file as comma separated values with an additional comment header containing metaData if existent.
      The filename can contain placeholders of the form {metadatafield;type;format}, where metadatafield references a field in the metadata as specified by the metaDataDataSet interface.
      static java.lang.String writeDataSetToFile​(DataSet dataSet, java.nio.file.Path path, java.lang.String fileName, boolean binary)
      Export the contents of the supplied dataSet to file as comma separated values with an additional comment header containing metaData if existent.
      The filename can contain placeholders of the form {metadatafield;type;format}, where metadatafield references a field in the metadata as specified by the metaDataDataSet interface.
      static java.lang.String writeDataSetToFile​(DataSet dataSet, java.nio.file.Path path, java.lang.String fileName, DataSetUtils.Compression compression)
      Export the contents of the supplied dataSet to file as comma separated values with an additional comment header containing metaData if existent.
      The filename can contain placeholders of the form {metadatafield;type;format}, where metadatafield references a field in the metadata as specified by the metaDataDataSet interface.
      static java.lang.String writeDataSetToFile​(DataSet dataSet, java.nio.file.Path path, java.lang.String fileName, DataSetUtils.Compression compression, boolean binary)
      Export the contents of the supplied dataSet to file as comma separated values with an additional comment header containing metaData if existent.
      The filename can contain placeholders of the form {metadatafield;type;format}, where metadatafield references a field in the metadata as specified by the metaDataDataSet interface.
      protected static void writeHeaderDataToStream​(java.io.OutputStream outputStream, DataSet dataSet)  
      protected static void writeMetaDataToStream​(java.io.OutputStream outputStream, DataSet dataSet)  
      protected static void writeNumericDataToStream​(java.io.OutputStream outputFile, DataSet dataSet)  

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    • Field Detail

      • USE_FLOAT32_BINARY_STANDARD

        public static boolean USE_FLOAT32_BINARY_STANDARD

    • Method Detail

      • copyDataSet

        public static AbstractDataSet<?> copyDataSet​(DataSet ds)
        perform and return a deep copy of the data set
        Parameters:
        ds - data set to be copied
        Returns:
        deep copy of data set

      • error

        public static double error​(DataSet dataSet,
                           DataSetUtils.ErrType eType,
                           int index)
        convenience short-hand notation for getting error variable at a specific data point index. Returns 0.0 if no errors are defined.
        Parameters:
        dataSet - the source data set
        eType - the error type
        index - the data set index
        Returns:
        the given error at data point "index"

      • error

        public static double error​(DataSet dataSet,
                           DataSetUtils.ErrType eType,
                           double x)
        convenience short-hand notation for getting error variable at an exact x value. Returns 0.0 if no errors are defined.
        Parameters:
        dataSet - the source data set
        eType - the error type
        x - the data set x-value for which the error should be interpolated
        Returns:
        the given interpolated error

      • error

        protected static double error​(DataSet dataSet,
                              DataSetUtils.ErrType eType,
                              int index,
                              double x,
                              boolean interpolate)
        convenience short-hand notation for getting error variables. Returns 0.0 if no errors are defined.
        Parameters:
        dataSet - the source data set
        eType - the error type
        index - the data set index
        x - the data set x-value for which the error should be interpolated
        interpolate - determines if the value at index or the interpolated value at x should be returned
        Returns:
        the given error

      • cropToLength

        protected static double[] cropToLength​(double[] in,
                                       int length)
        small helper routine to crop data array in case it's to long
        Parameters:
        in - input data array
        length - length of output array
        Returns:
        cropped/zero-padded array of size length.

      • errors

        public static double[] errors​(DataSet dataSet,
                              DataSetUtils.ErrType eType)
        convenience short-hand notation for getting error variables (if defined for dataset)
        Parameters:
        dataSet - the source data set
        eType - the error type
        Returns:
        the given error array (cropped to data set length if necessary)

      • getISODate

        public static java.lang.String getISODate​(long timeMillis,
                                          java.lang.String format)
        Get ISO date from milliseconds since Jan 01, 1970
        Parameters:
        timeMillis - time to be converted
        format - time format string
        Returns:
        ISO formatted UTC datetime string

      • getFileName

        public static java.lang.String getFileName​(DataSet dataSet,
                                           java.lang.String fileName)
        Produce a filename from a dataSet and a String that can contain {datafield;type;format} style placeholders. Datafield references a field in the metadata as specified by the metaDataDataSet interface. The special field "systemTime" can be used to use the current system time. Also the standard DataSet fields can be used ("dataSetName", "xMin", "xMax", "yMin", "yMax"). The optional type field supports "string", "date", "int/long" and "float/double", where "string" is the default. The optional format field can be used to provide format strings. The default for date is SimpleDateFormat "yyyyMMdd_HHmmss", for int and float it is printf's "%d" and "%e".
        Parameters:
        dataSet - A dataSet containing all the data field referenced in the filename pattern
        fileName - Filename (with "{metadatafield;type;format}" placeholders for variables)
        Returns:
        The filename with the placeholders replaced

      • writeDataSetToFile

        public static java.lang.String writeDataSetToFile​(DataSet dataSet,
                                                  java.nio.file.Path path,
                                                  java.lang.String fileName,
                                                  boolean binary)
        Export the contents of the supplied dataSet to file as comma separated values with an additional comment header containing metaData if existent.
        The filename can contain placeholders of the form {metadatafield;type;format}, where metadatafield references a field in the metadata as specified by the metaDataDataSet interface. The special field systemTime can be used to use the current system time. Also the standard DataSet fields can be used ("dataSetName", "xMin", "xMax", "yMin", "yMax"). The optional type field supports "string", "date", "int/long" and "float/double", where "string" is the default. The optional format field can be used to provide format strings. The default for date is SimpleDateFormat "yyyyMMdd_HHmmss", for int and float it is printf's "%d" and "%e".
        The compression method is automatically determined from the file extension.
        The data format is a custom extension of csv with an additional #-commented Metadata Header and a $-commented column header. Expects the following columns in this order to be present: index, x, y, eyn, eyp.
        Parameters:
        dataSet - The DataSet to export
        path - Path to the location of the file
        fileName - Filename (with "{metadatafield;type;format}" placeholders for variables)
        binary - true: whether to store data as binary or string
        Returns:
        actual name of the file that was written or none in case of errors

      • writeDataSetToFile

        public static java.lang.String writeDataSetToFile​(DataSet dataSet,
                                                  java.nio.file.Path path,
                                                  java.lang.String fileName)
        Export the contents of the supplied dataSet to file as comma separated values with an additional comment header containing metaData if existent.
        The filename can contain placeholders of the form {metadatafield;type;format}, where metadatafield references a field in the metadata as specified by the metaDataDataSet interface. The special field systemTime can be used to use the current system time. Also the standard DataSet fields can be used ("dataSetName", "xMin", "xMax", "yMin", "yMax"). The optional type field supports "string", "date", "int/long" and "float/double", where "string" is the default. The optional format field can be used to provide format strings. The default for date is SimpleDateFormat "yyyyMMdd_HHmmss", for int and float it is printf's "%d" and "%e".
        The compression method is automatically determined from the file extension.
        The data format is a custom extension of csv with an additional #-commented Metadata Header and a $-commented column header. Expects the following columns in this order to be present: index, x, y, eyn, eyp.
        Parameters:
        dataSet - The DataSet to export
        path - Path to the location of the file
        fileName - Filename (with "{metadatafield;type;format}" placeholders for variables)
        Returns:
        actual name of the file that was written or none in case of errors

      • writeDataSetToFile

        public static java.lang.String writeDataSetToFile​(DataSet dataSet,
                                                  java.nio.file.Path path,
                                                  java.lang.String fileName,
                                                  DataSetUtils.Compression compression)
        Export the contents of the supplied dataSet to file as comma separated values with an additional comment header containing metaData if existent.
        The filename can contain placeholders of the form {metadatafield;type;format}, where metadatafield references a field in the metadata as specified by the metaDataDataSet interface. The special field systemTime can be used to use the current system time. Also the standard DataSet fields can be used ("dataSetName", "xMin", "xMax", "yMin", "yMax"). The optional type field supports "string", "date", "int/long" and "float/double", where "string" is the default. The optional format field can be used to provide format strings. The default for date is SimpleDateFormat "yyyyMMdd_HHmmss", for int and float it is printf's "%d" and "%e".
        The compression method is automatically determined from the file extension.
        The data format is a custom extension of csv with an additional #-commented Metadata Header and a $-commented column header. Expects the following columns in this order to be present: index, x, y, eyn, eyp.
        Parameters:
        dataSet - The DataSet to export
        path - Path to the location of the file
        fileName - Filename (with "{metadatafield;type;format}" placeholders for variables)
        compression - compression type @see Compression
        Returns:
        actual name of the file that was written or none in case of errors

      • writeDataSetToFile

        public static java.lang.String writeDataSetToFile​(DataSet dataSet,
                                                  java.nio.file.Path path,
                                                  java.lang.String fileName,
                                                  DataSetUtils.Compression compression,
                                                  boolean binary)
        Export the contents of the supplied dataSet to file as comma separated values with an additional comment header containing metaData if existent.
        The filename can contain placeholders of the form {metadatafield;type;format}, where metadatafield references a field in the metadata as specified by the metaDataDataSet interface. The special field systemTime can be used to use the current system time. Also the standard DataSet fields can be used ("dataSetName", "xMin", "xMax", "yMin", "yMax"). The optional type field supports "string", "date", "int/long" and "float/double", where "string" is the default. The optional format field can be used to provide format strings. The default for date is SimpleDateFormat "yyyyMMdd_HHmmss", for int and float it is printf's "%d" and "%e".
        The data format is a custom extension of csv with an additional #-commented Metadata Header and a $-commented column header. Expects the following columns in this order to be present: index, x, y, eyn, eyp.
        Parameters:
        dataSet - The DataSet to export
        path - Path to the location of the file
        fileName - Filename (with "{metadatafield;type;format}" placeholders for variables)
        compression - Compression of the file (GZIP, ZIP or NONE). Supply AUTO or omit this value to use file extension.
        binary - true: whether to store data as binary or string
        Returns:
        actual name of the file that was written or none in case of errors

      • writeDataSetToByteArray

        public static void writeDataSetToByteArray​(DataSet dataSet,
                                           java.io.ByteArrayOutputStream byteOutput,
                                           boolean binary,
                                           boolean asFloat)
        Write data set into byte buffer.
        Parameters:
        dataSet - The DataSet to export
        byteOutput - byte output stream (N.B. keep caching this object)
        binary - true: encode data as binary (smaller size, performance), or false as string (human readable, easier debugging)
        asFloat - true: encode data as binary floats (smaller size, performance), or false as double (better precision)

      • writeHeaderDataToStream

        protected static void writeHeaderDataToStream​(java.io.OutputStream outputStream,
                                              DataSet dataSet)

      • writeMetaDataToStream

        protected static void writeMetaDataToStream​(java.io.OutputStream outputStream,
                                            DataSet dataSet)

      • writeNumericDataToStream

        protected static void writeNumericDataToStream​(java.io.OutputStream outputFile,
                                               DataSet dataSet)

      • getKey

        protected static java.lang.String getKey​(java.lang.String line,
                                         java.lang.String replace)

      • getValue

        protected static java.lang.String getValue​(java.lang.String line)

      • readDataSetFromFile

        public static DataSet readDataSetFromFile​(java.lang.String fileName)
        Read a Dataset from a file containing comma separated values.
        Automatically determines compression from the file extension.
        The data format is a custom extension of csv with an additional #-commented Metadata Header and a $-commented column header. Expects the following columns in this order to be present: index, x, y, eyn, eyp.
        Parameters:
        fileName - Path and name of file containing csv data.
        Returns:
        DataSet with the data and metadata read from the file

      • readDataSetFromFile

        public static DataSet readDataSetFromFile​(java.lang.String fileName,
                                          DataSetUtils.Compression compression)
        Read a Dataset from a file containing comma separated values.
        The data format is a custom extension of csv with an additional #-commented Metadata Header and a $-commented column header. Expects the following columns in this order to be present: index, x, y, eyn, eyp.
        Parameters:
        fileName - Path and name of file containing csv data.
        compression - Compression of the file (GZIP, ZIP or NONE). Supply AUTO or omit this value to use file extension.
        Returns:
        DataSet with the data and metadata read from the file

      • readDataSetFromByteArray

        public static DoubleErrorDataSet readDataSetFromByteArray​(byte[] byteArray)
        Read a Dataset from a byte array containing comma separated values.
        The data format is a custom extension of csv with an additional #-commented Metadata Header and a $-commented column header. Expects the following columns in this order to be present: index, x, y, eyn, eyp.
        Parameters:
        byteArray - byte array.
        Returns:
        DataSet with the data and metadata read from the file

      • readDataSetFromStream

        public static DoubleErrorDataSet readDataSetFromStream​(de.gsi.dataset.utils.DataSetUtils.SplitCharByteInputStream inputStream)
        Read a Dataset from a stream containing comma separated values.
        The data format is a custom extension of csv with an additional #-commented Metadata Header and a $-commented column header. Expects the following columns in this order to be present: index, x, y, eyn, eyp.
        Parameters:
        inputStream - Path and name of file containing csv data.
        Returns:
        DataSet with the data and metadata read from the file

      • readNumericDataFromFile

        protected static void readNumericDataFromFile​(java.io.BufferedReader inputFile,
                                              DoubleErrorDataSet dataSet)