Package org.gorpipe.gor.manager

Class BucketManager<T extends org.gorpipe.gor.table.BucketableTableEntry>

  • public class BucketManager<T extends org.gorpipe.gor.table.BucketableTableEntry>
extends java.lang.Object

    • Field Detail

      • DEFAULT_MIN_BUCKET_SIZE

        public static final int DEFAULT_MIN_BUCKET_SIZE
        See Also:
        Constant Field Values

      • DEFAULT_MAX_BUCKET_COUNT

        public static final int DEFAULT_MAX_BUCKET_COUNT
        See Also:
        Constant Field Values

      • DEFAULT_LOCK_TIMEOUT

        public static final java.time.Duration DEFAULT_LOCK_TIMEOUT

      • DEFAULT_LOCK_TYPE

        public static final java.lang.Class<? extends org.gorpipe.gor.table.lock.TableLock> DEFAULT_LOCK_TYPE

      • HEADER_MIN_BUCKET_SIZE_KEY

        public static final java.lang.String HEADER_MIN_BUCKET_SIZE_KEY
        See Also:
        Constant Field Values

      • HEADER_BUCKET_SIZE_KEY

        public static final java.lang.String HEADER_BUCKET_SIZE_KEY
        See Also:
        Constant Field Values

      • HEADER_BUCKET_DIRS_KEY

        public static final java.lang.String HEADER_BUCKET_DIRS_KEY
        See Also:
        Constant Field Values

      • HEADER_BUCKET_MAX_BUCKETS

        public static final java.lang.String HEADER_BUCKET_MAX_BUCKETS
        See Also:
        Constant Field Values

      • gracePeriodForDeletingBuckets

        protected java.time.Duration gracePeriodForDeletingBuckets

    • Constructor Detail

      • BucketManager

        public BucketManager​(org.gorpipe.gor.table.BaseTable<T> table)
        Default constructor.

    • Method Detail

      • bucketize

        public void bucketize()

      • bucketize

        public int bucketize​(BucketManager.BucketPackLevel packLevel,
                     int maxBucketCount,
                     java.util.List<java.nio.file.Path> bucketDirs,
                     boolean forceClean)
        Bucketize the given table.
        Parameters:
        packLevel - pack level to use (see BucketPackLevel).
        maxBucketCount - Maximum number of buckets to generate on this call, 0 or less means not set in which case we read it from the table, if its not set on the table we use default.
        bucketDirs - array of directories to bucketize to, ignored if null. The dirs are absolute or relative to the table dir.
        forceClean - Should we force clean bucket files (if force clean we ignoring grace periods).
        Returns:
        buckets created.

      • deleteBuckets

        public void deleteBuckets​(java.nio.file.Path... buckets)
        Delete the given bucket.
        Parameters:
        buckets - list of buckets to be deleted.

      • getEffectiveMinBucketSize

        protected int getEffectiveMinBucketSize()
        Get effective min bucket size.
        Returns:
        effective min bucket size.

      • getBucketSize

        public int getBucketSize()

      • setBucketSize

        public void setBucketSize​(int bucketSize)

      • getMinBucketSize

        public int getMinBucketSize()

      • setMinBucketSize

        public void setMinBucketSize​(int minBucketSize)

      • getDefaultBucketDir

        protected java.nio.file.Path getDefaultBucketDir()

      • getLockTimeout

        public java.time.Duration getLockTimeout()

      • setBucketDirs

        public void setBucketDirs​(java.util.List<java.nio.file.Path> newBucketDirs)
        Set the bucket dir list.
        Parameters:
        newBucketDirs - the new list of bucket dirs, absolute or relative to table dir.

      • pickBucketDir

        protected final java.nio.file.Path pickBucketDir()
        Pick a bucket dir from a list. There couple of strategies to choose from (using the gor.table.buckets.directory.strategy option.) 'random': pick a directory by random. 'least_used': pick the directory that is least used by the dictionary. Default.
        Returns:
        path to the chosen bucket dir, relative to path or absolute.

      • cleanBucketFiles

        protected void cleanBucketFiles​(org.gorpipe.gor.table.lock.TableLock bucketizeLock,
                                boolean force)
                         throws java.io.IOException
        Cleans up bucketFiles that are not in use and have not been accessed for a given period of time.

        TODO: We are not handling changed bucket dirs very well, i.e. if we remove bucket dir from the list of bucket dirs because we don't want add more files to that directory it will not be included in the cleaning. Hard to changes this until we add the bucket dirs to the table header (then we can keep two lists). Note, not enough to check the directories in use in the file as we remove them from the gord file before actual delete the files (because of the delayed delete).

        Parameters:
        bucketizeLock - the bucketize lock to use.
        force - force cleaning, ignoring grace periods.
        Throws:
        java.io.IOException