Package org.projectnessie.gc.base

Class DistributedIdentifyContents

java.lang.Object

org.projectnessie.gc.base.DistributedIdentifyContents

public class DistributedIdentifyContents
extends Object

Identify the expired and live contents in a distributed way using the spark and bloom filter by walking all the references (both dead and live).

Constructor Summary

Constructors

Constructor	Description
DistributedIdentifyContents(org.apache.spark.sql.SparkSession session, GCParams gcParams)

Method Summary

Modifier and Type	Method	Description
IdentifiedResult	getIdentifiedResults(Map<String,ContentBloomFilter> liveContentsBloomFilterMap, List<org.projectnessie.model.Reference> references)	Gets the expired contents per content id by walking all the live and dead references in a distributed way using spark and checking the contents against the live bloom filter results.
Map<String,ContentBloomFilter>	getLiveContentsBloomFilters(List<org.projectnessie.model.Reference> references, long bloomFilterSize, Map<org.projectnessie.model.Reference,Instant> droppedRefTimeMap)	Compute the bloom filter per content id by walking all the live references in a distributed way using spark.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DistributedIdentifyContents

public DistributedIdentifyContents(org.apache.spark.sql.SparkSession session,
                                   GCParams gcParams)

Method Detail

getLiveContentsBloomFilters

public Map<String,ContentBloomFilter> getLiveContentsBloomFilters(List<org.projectnessie.model.Reference> references,
                                                                        long bloomFilterSize,
                                                                        Map<org.projectnessie.model.Reference,Instant> droppedRefTimeMap)

Compute the bloom filter per content id by walking all the live references in a distributed way using spark.

Parameters:

references - list of all the references

bloomFilterSize - size of bloom filter to be used

droppedRefTimeMap - map of dropped time for reference@hash

Returns:

map of ContentBloomFilter per content-id.

getIdentifiedResults

public IdentifiedResult getIdentifiedResults(Map<String,ContentBloomFilter> liveContentsBloomFilterMap,
                                             List<org.projectnessie.model.Reference> references)

Gets the expired contents per content id by walking all the live and dead references in a distributed way using spark and checking the contents against the live bloom filter results.

Parameters:

liveContentsBloomFilterMap - live contents bloom filter per content id.

references - list of all the references to walk (live and dead)

Returns:

IdentifiedResult object.