Package org.gorpipe.gor.manager

Class TableManager

java.lang.Object
org.gorpipe.gor.manager.TableManager
public class TableManager extends Object
Class to manage gor tables (dictionaries and tables).

Includes commands to:

load table insert entries into table delete entries from table select bucketize remove entries from buckets delete buckets

Created by gisli on 18/08/16.

TODO: Why not move bucketize methods into the DictionaryTable object. TODO: bucketzationInfo should be part of the table and stored in the table header (only passed in when table is created). Use sensible defaults otherwise.

  • Field Details

    • DEFAULT_LOCK_TIMEOUT

      public static final Duration DEFAULT_LOCK_TIMEOUT

    • DEFAULT_LOCK_TYPE

      public static final Class<? extends org.gorpipe.gor.table.lock.TableLock> DEFAULT_LOCK_TYPE

  • Constructor Details

    • TableManager

      public TableManager()
      Default constructor.

  • Method Details

    • newBuilder

      public static TableManager.Builder newBuilder()

    • setMinBucketSize

      public void setMinBucketSize(int minBucketSize)

    • getBucketSize

      public int getBucketSize()

    • setBucketSize

      public void setBucketSize(int bucketSize)
      Set the bucket size. Also updates the minBucketSize if it is larger than bucketSize.
      Parameters:
      bucketSize - the requested bucket size.

    • getfileReader

      public org.gorpipe.gor.model.FileReader getfileReader()

    • setfileReader

      public void setfileReader(org.gorpipe.gor.model.FileReader fileReader)

    • getLockType

      public Class<? extends org.gorpipe.gor.table.lock.TableLock> getLockType()

    • getLockTimeout

      public Duration getLockTimeout()

    • initTable

      public org.gorpipe.gor.table.dictionary.BaseDictionaryTable initTable(Path path)
      Initialize the table given by path.
      Parameters:
      path - path to the table.
      Returns:
      the table given by path.

    • insert

      public void insert(Path tableFile, BucketManager.BucketPackLevel packLevel, int workers, org.gorpipe.gor.table.dictionary.DictionaryEntry... entries)
      Insert new data file (partition) into the table. This might trigger bucketization.
      Parameters:
      tableFile - the path to the table file.
      packLevel - pack level to use (see BucketPackLevel).
      workers - number of workers to use for bucketization (if needed).
      entries - Files/lines to insert.

    • insert

      public void insert(org.gorpipe.gor.table.dictionary.BaseDictionaryTable table, BucketManager.BucketPackLevel packLevel, int workers, org.gorpipe.gor.table.dictionary.DictionaryEntry... entries)
      Insert new data file (partition) into the table. This might trigger bucketization.
      Parameters:
      table - the table to insert into.
      packLevel - pack level to use (see BucketPackLevel).
      workers - number of workers to use for bucketization (if needed).
      entries - Files/lines to insert.

    • save

      public void save(org.gorpipe.gor.table.dictionary.BaseDictionaryTable table, BucketManager.BucketPackLevel packLevel, int workers)
      Save the given table.
      Parameters:
      table - the table to save.
      packLevel - pack level to use (see BucketPackLevel).
      workers - number of workers to use for bucketization (if needed).

    • delete

      public void delete(Path tableFile, org.gorpipe.gor.table.dictionary.DictionaryEntry... entries)
      Delete entries from the table.
      Parameters:
      tableFile - path to the table file.
      entries - the entries to delete.

    • delete

      public void delete(Path tableFile, org.gorpipe.gor.table.dictionary.TableFilter entries)
      Delete entries from the table.
      Parameters:
      tableFile - path to the table file.
      entries - the entries to delete.

    • select

      public List<? extends org.gorpipe.gor.table.dictionary.DictionaryEntry> select(Path tableFile, String[] files, String[] aliases, String[] tags, String[] buckets, String chrRange, boolean includedDeleted)
      Select entries from the table.

      This function is useful to select entries as input into other methods of this class.

      Parameters:
      tableFile - path to the table file.
      files - list of files to include.
      aliases - list of aliases to include.
      tags - list of tags to include.
      buckets - list of buckets to include.
      chrRange - filter range, string in format [chr from]:[pos from]-[chr to]:[pos to]. Example: chr1:10000-chr1:200000
      includedDeleted - Should deleted files be included in the result.
      Returns:
      entries from the table, matching the given criteria.

    • selectAll

      public Collection<? extends org.gorpipe.gor.table.dictionary.DictionaryEntry> selectAll(Path tableFile)
      Select all entries from table.

      This function is useful to select entries as input into other methods of this class.

      Parameters:
      tableFile - path to the table file.
      Returns:
      all entries from table as a collection.

    • print

      public void print(org.gorpipe.gor.table.dictionary.TableFilter lines)

    • bucketize

      public void bucketize(Path tableFile, BucketManager.BucketPackLevel packLevel, int workers, int maxBucketCount, List<String> bucketDirs)
      Bucketize the given table.
      Parameters:
      tableFile - the path to the table file.
      packLevel - pack level to use (see BucketPackLevel).
      workers - number of workers to use for bucketization (if needed).
      maxBucketCount - Maximum number of buckets to generate on this call.
      bucketDirs - array of directories to bucketize to, ignored if null. The dirs are absolute or relative to the table dir.

    • deleteBuckets

      public void deleteBuckets(org.gorpipe.gor.table.dictionary.BaseDictionaryTable table, boolean force, String... buckets)
      Delete the given buckets.
      Parameters:
      table - table to update.
      force - force clean up (ignore grace periods).
      buckets - list of buckets to be deleted.

    • deleteBuckets

      public void deleteBuckets(Path tableFile, String... buckets)
      Delete the given buckets.
      Parameters:
      tableFile - the path to the table file.
      buckets - list of buckets to be deleted.