Package org.gorpipe.gor.manager

Class BucketManager<T extends org.gorpipe.gor.table.dictionary.DictionaryEntry>

java.lang.Object
org.gorpipe.gor.manager.BucketManager<T>
public class BucketManager<T extends org.gorpipe.gor.table.dictionary.DictionaryEntry> extends Object

  • Field Details

    • DEFAULT_MIN_BUCKET_SIZE

      public static final int DEFAULT_MIN_BUCKET_SIZE
      See Also:

    • DEFAULT_BUCKET_SIZE

      public static final int DEFAULT_BUCKET_SIZE
      See Also:

    • DEFAULT_MAX_BUCKET_COUNT

      public static final int DEFAULT_MAX_BUCKET_COUNT
      See Also:

    • DEFAULT_BUCKET_PACK_LEVEL

      public static final BucketManager.BucketPackLevel DEFAULT_BUCKET_PACK_LEVEL

    • DEFAULT_LOCK_TIMEOUT

      public static final Duration DEFAULT_LOCK_TIMEOUT

    • BUCKET_CLEANUP_INTERVAL

      public static final Duration BUCKET_CLEANUP_INTERVAL

    • DEFAULT_LOCK_TYPE

      public static final Class<? extends org.gorpipe.gor.table.lock.TableLock> DEFAULT_LOCK_TYPE

    • HEADER_MIN_BUCKET_SIZE_KEY

      public static final String HEADER_MIN_BUCKET_SIZE_KEY
      See Also:

    • HEADER_BUCKET_SIZE_KEY

      public static final String HEADER_BUCKET_SIZE_KEY
      See Also:

    • HEADER_BUCKET_DIRS_KEY

      public static final String HEADER_BUCKET_DIRS_KEY
      See Also:

    • HEADER_BUCKET_DIRS_LOCATION_KEY

      public static final String HEADER_BUCKET_DIRS_LOCATION_KEY
      See Also:

    • HEADER_BUCKET_MAX_BUCKETS

      public static final String HEADER_BUCKET_MAX_BUCKETS
      See Also:

    • gracePeriodForDeletingBuckets

      protected Duration gracePeriodForDeletingBuckets

  • Constructor Details

    • BucketManager

      public BucketManager(org.gorpipe.gor.table.dictionary.BaseDictionaryTable<T> table)
      Default constructor.

  • Method Details

    • newBuilder

      public static BucketManager.Builder newBuilder(org.gorpipe.gor.table.dictionary.BaseDictionaryTable table)

    • bucketize

      public void bucketize()

    • bucketize

      public int bucketize(BucketManager.BucketPackLevel packLevel, int maxBucketCount)

    • bucketize

      public int bucketize(BucketManager.BucketPackLevel packLevel, int maxBucketCount, List<String> bucketDirs, boolean forceClean)
      Bucketize the given table.
      Parameters:
      packLevel - pack level to use (see BucketPackLevel).
      maxBucketCount - Maximum number of buckets to generate on this call, 0 or less means not set in which case we read it from the table, if its not set on the table we use default.
      bucketDirs - array of directories to bucketize to, ignored if null. The dirs are absolute or relative to the table dir.
      forceClean - Should we force clean bucket files (if force clean we ignoring grace periods).
      Returns:
      buckets created.

    • deleteBuckets

      public void deleteBuckets(String... buckets)
      Delete the given buckets.
      Parameters:
      buckets - list of buckets to be deleted.

    • deleteBuckets

      public void deleteBuckets(boolean force, String... buckets)
      Delete the given buckets.
      Parameters:
      force - should grace period be ignored.
      buckets - list of buckets to be deleted.

    • getEffectiveMinBucketSize

      protected int getEffectiveMinBucketSize()
      Get effective min bucket size.
      Returns:
      effective min bucket size.

    • getBucketSize

      public int getBucketSize()

    • setBucketSize

      public void setBucketSize(int bucketSize)

    • getMinBucketSize

      public int getMinBucketSize()

    • setMinBucketSize

      public void setMinBucketSize(int minBucketSize)

    • getDefaultBucketDir

      protected String getDefaultBucketDir()

    • getLockTimeout

      public Duration getLockTimeout()

    • setBucketDirs

      public void setBucketDirs(List<String> newBucketDirs)
      Set the bucket dir list.
      Parameters:
      newBucketDirs - the new list of bucket dirs, absolute or relative to table dir.

    • pickBucketDir

      protected final String pickBucketDir()
      Pick a bucket dir from a list. There couple of strategies to choose from (using the gor.table.buckets.directory.strategy option.) 'random': pick a directory by random. 'least_used': pick the directory that is least used by the dictionary. Default.
      Returns:
      path to the chosen bucket dir, relative to path or absolute.

    • cleanOldBucketFiles

      protected void cleanOldBucketFiles(org.gorpipe.gor.table.lock.TableLock bucketizeLock, boolean force) throws IOException
      Cleans up bucketFiles that are not in use and have not been accessed for a given period of time.

      Notes: 1. We only manage bucketdirs that are listed in the table meta data and are in use. So if we stop using a folder and there are no buckets there listed in the table then we need to manage it manually.

      Parameters:
      bucketizeLock - the bucketize lock to use.
      force - force cleaning, ignoring grace periods.
      Throws:
      IOException