Class BaseElasticSearchIndexBuilder
- java.lang.Object
-
- org.sakaiproject.search.elasticsearch.BaseElasticSearchIndexBuilder
-
- All Implemented Interfaces:
org.sakaiproject.search.api.SearchIndexBuilder,org.sakaiproject.search.elasticsearch.ElasticSearchIndexBuilder
- Direct Known Subclasses:
SiteElasticSearchIndexBuilder
public abstract class BaseElasticSearchIndexBuilder extends Object implements org.sakaiproject.search.elasticsearch.ElasticSearchIndexBuilder
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description protected classBaseElasticSearchIndexBuilder.BulkContentIndexerTaskThis is the task that searches for any docs in the search index that do not have content yet, digests the content and loads it into the index.static classBaseElasticSearchIndexBuilder.IndexActionprotected static classBaseElasticSearchIndexBuilder.NoOpTaskprotected classBaseElasticSearchIndexBuilder.RebuildIndexTask
-
Field Summary
Fields Modifier and Type Field Description protected static StringADD_RESOURCE_VALIDATION_KEY_CONTENT_PRODUCERprotected static StringADD_RESOURCE_VALIDATION_KEY_ENTITY_IDprotected static StringADD_RESOURCE_VALIDATION_KEY_INDEX_ACTIONprotected static StringADD_RESOURCE_VALIDATION_KEY_RESOURCE_NAMEKey in the "validation map" built up byvalidateAddResourceEvent(Event)(specificallyvalidateResourceName(Event, Map).protected static org.sakaiproject.authz.api.SecurityAdvisorallowAllAdvisorprotected TimerbackgroundSchedulerindexing thread that performs loading the actual content into the index.protected intbulkRequestSizeNumber of actions to send in one elasticsearch bulk index call defaults to 10.protected org.elasticsearch.client.RestHighLevelClientclientES Client, with access to any indexes in the cluster.protected intcontentIndexBatchSizeNumber of documents to index at a time for each run of the context indexing task (defaults to 500).protected static StringDEFAULT_FACET_NAMEprotected static StringDEFAULT_SUGGESTION_MATCHING_FIELD_NAMEprotected StringdefaultIndexSettingsResourceprotected StringdefaultMappingResourceprotected intdelaynumber seconds of wait after startup before starting the BulkContentIndexerTask (defaults to 3 minutes)protected static StringDELETE_RESOURCE_KEY_DOCUMENT_IDprotected static StringDELETE_RESOURCE_KEY_ENTITY_REFERENCEprotected org.sakaiproject.search.elasticsearch.ElasticSearchIndexBuilderEventRegistrareventRegistrarprotected org.sakaiproject.event.api.EventTrackingServiceeventTrackingServiceprotected StringfacetNameprotected intfacetTermSizeN most frequent termsprotected org.sakaiproject.search.elasticsearch.filter.SearchItemFilterfilterprotected StringindexedDocumentTypeprotected StringindexNamethe ES indexnameprotected org.elasticsearch.common.settings.SettingsindexSettingsprotected StringindexSettingsConfigExpects a JSON string of ElasticSearch index settings.protected longlastLoadprotected StringmappingCombination ofmappingConfig, its fallback read fromdefaultMappingResource, and any overrides implemented ininitializeElasticSearchMapping(String).protected StringmappingConfigby default the mapping in configured in the mapping.json file.protected intmaxNumberOfSuggestionsmax number of suggestions to return when looking for suggestions (this populates the autocomplete drop down in the UI)protected StringnameLogical, well-known name, possibly distinct from the physical ESindexNameprotected intperiodhow often the BulkContentIndexerTask runs in seconds (defaults to 1 minute)protected Set<org.sakaiproject.search.api.EntityContentProducer>producersprotected booleanrebuildIndexOnStartupset to true to force an index rebuild at startup time, defaults to false.protected String[]searchResultFieldNamesprotected org.sakaiproject.authz.api.SecurityServicesecurityServiceprotected org.sakaiproject.component.api.ServerConfigurationServiceserverConfigurationServiceprotected longstartTimeprotected StringsuggestionMatchingFieldNameprotected String[]suggestionResultFieldNamesprotected booleantestModethis turns off the threads and does indexing inline.protected Set<String>triggerFunctionsprotected booleanuseFacettingprotected booleanuseSuggestions
-
Constructor Summary
Constructors Constructor Description BaseElasticSearchIndexBuilder()
-
Method Summary
All Methods Instance Methods Abstract Methods Concrete Methods Modifier and Type Method Description protected org.elasticsearch.xcontent.XContentBuilderaddContent(org.elasticsearch.xcontent.XContentBuilder contentSourceBuilder, String resourceName, org.sakaiproject.search.api.EntityContentProducer ecp, boolean includeContent)protected org.elasticsearch.xcontent.XContentBuilderaddCustomProperties(org.elasticsearch.xcontent.XContentBuilder contentSourceBuilder, String resourceName, org.sakaiproject.search.api.EntityContentProducer ecp, boolean includeContent)protected abstract org.elasticsearch.xcontent.XContentBuilderaddFields(org.elasticsearch.xcontent.XContentBuilder contentSourceBuilder, String resourceName, org.sakaiproject.search.api.EntityContentProducer ecp, boolean includeContent)protected org.elasticsearch.action.search.SearchRequestaddFindContentQueueRequestParams(org.elasticsearch.action.search.SearchRequest searchRequest)voidaddResource(org.sakaiproject.event.api.Notification notification, org.sakaiproject.event.api.Event event)protected voidaddSearchCoreParams(org.elasticsearch.action.search.SearchRequest searchRequest)protected voidaddSearchFacetting(org.elasticsearch.action.search.SearchRequest searchRequest)protected voidaddSearchPagination(org.elasticsearch.action.search.SearchRequest searchRequest, int start, int end)protected voidaddSearchQuery(org.elasticsearch.action.search.SearchRequest searchRequest, String searchTerms, List<String> references, List<String> siteIds)protected voidaddSearchReferences(org.elasticsearch.action.search.SearchRequest searchRequest, List<String> references)protected voidaddSearchResultFields(org.elasticsearch.action.search.SearchRequest searchRequest)protected abstract voidaddSearchSiteIds(org.elasticsearch.action.search.SearchRequest searchRequest, List<String> siteIds)protected voidaddSearchSuggestionResultFields(org.elasticsearch.action.search.SearchRequest searchRequest)protected voidaddSearchSuggestionsCoreParams(org.elasticsearch.action.search.SearchRequest searchRequest)protected voidaddSearchSuggestionsPagination(org.elasticsearch.action.search.SearchRequest searchRequest)protected voidaddSearchSuggestionsQuery(org.elasticsearch.action.search.SearchRequest searchRequest, String searchString, String currentSite, boolean allMySites)protected abstract voidaddSearchSuggestionsSites(org.elasticsearch.action.search.SearchRequest searchRequest, String currentSite, boolean allMySites)protected abstract voidaddSearchSuggestionsTerms(org.elasticsearch.action.search.SearchRequest searchRequest, String searchString)protected voidaddSearchTerms(org.elasticsearch.action.search.SearchRequest searchRequest, String searchTerms)protected voidassureIndex()creates a new index if one does not existprotected abstract voidbeforeBackgroundSchedulerInitialization()Called after all ES config has been processed but before the background scheduler has been set up and before any index startup ops have been invoked (initializeIndex().protected abstract voidbeforeElasticSearchConfigInitialization()Gives subclasses a chance to initialize configuration prior to reading/processing any ES configs.protected org.elasticsearch.xcontent.XContentBuilderbuildIndexRequestContentSource(String resourceName, org.sakaiproject.search.api.EntityContentProducer ecp, boolean includeContent)protected abstract voidcompleteAddResourceEventValidations(org.sakaiproject.event.api.Event event, Map<String,Object> validationContext)protected abstract org.elasticsearch.action.delete.DeleteRequestcompleteDeleteRequest(org.elasticsearch.action.delete.DeleteRequest deleteRequest, Map<String,Object> deleteParams)protected abstract org.elasticsearch.action.search.SearchRequestcompleteFindContentQueueRequest(org.elasticsearch.action.search.SearchRequest searchRequest)protected abstract org.elasticsearch.action.index.IndexRequestcompleteIndexRequest(org.elasticsearch.action.index.IndexRequest indexRequest, String resourceName, org.sakaiproject.search.api.EntityContentProducer ecp, boolean includeContent)protected org.elasticsearch.xcontent.XContentBuildercompleteIndexRequestContentSourceBuilder(org.elasticsearch.xcontent.XContentBuilder contentSourceBuilder, String resourceName, org.sakaiproject.search.api.EntityContentProducer ecp, boolean includeContent)protected abstract voidcompleteSearchRequestBuilders(org.elasticsearch.action.search.SearchRequest searchRequest, String searchTerms, List<String> references, List<String> siteIds)protected abstract voidcompleteSearchSuggestionsRequestBuilders(org.elasticsearch.action.search.SearchRequest searchRequest, String searchString, String currentSite, boolean allMySites)protected voidcreateIndex()creates a new index, does not check if the exist existsprotected voiddeleteDocument(org.elasticsearch.search.SearchHit searchHit)protected voiddeleteDocument(NoContentException noContentException)protected voiddeleteDocumentWithParams(Map<String,Object> deleteParams)protected org.elasticsearch.action.delete.DeleteResponsedeleteDocumentWithRequest(org.elasticsearch.action.delete.DeleteRequest deleteRequest)voiddestroy()protected voiddisableAzgSecurityAdvisor()Disable the security advisor.protected voiddispatchValidatedAddResource(Map<String,Object> validationContext)protected voidenableAzgSecurityAdvisor()Establish a security advisor to allow the "embedded" work to occur with no need for additional security permissions.protected voidexecuteBulkRequest(org.elasticsearch.action.bulk.BulkRequest bulkRequest)protected Map<String,Collection<String>>extractCustomProperties(String resourceName, org.sakaiproject.search.api.EntityContentProducer contentProducer)Extract properties from the givenEntityContentProducerprotected Map<String,Object>extractDeleteDocumentParams(Map<String,Object> validationContext)protected Map<String,Object>extractDeleteDocumentParams(org.elasticsearch.search.SearchHit searchHit)protected Map<String,Object>extractDeleteDocumentParams(NoContentException noContentException)protected org.elasticsearch.action.search.SearchResponsefindContentQueue()protected org.elasticsearch.action.search.SearchResponsefindContentQueueWithRequest(org.elasticsearch.action.search.SearchRequest searchRequest)List<org.sakaiproject.search.model.SearchBuilderItem>getAllSearchItems()Set<String>getContentFunctions()List<org.sakaiproject.search.api.EntityContentProducer>getContentProducers()get all the producers registered, as a clone to avoid concurrent modification exceptionsStringgetFacetName()StringgetFieldFromSearchHit(String field, org.elasticsearch.search.SearchHit hit)loads the field from the SearchHit.org.sakaiproject.search.elasticsearch.filter.SearchItemFiltergetFilter()List<org.sakaiproject.search.model.SearchBuilderItem>getGlobalMasterSearchItems()org.elasticsearch.cluster.health.ClusterIndexHealthgetIndexHealth()List<String>getIndices()protected abstract org.slf4j.LoggergetLog()StringgetName()longgetNDocs()intgetPendingDocuments()org.sakaiproject.search.api.SearchStatusgetSearchStatus()StringBuildergetStatus(StringBuilder into)Set<String>getTriggerFunctions()booleangetUseFacetting()protected voidindexAdd(String resourceName, org.sakaiproject.search.api.EntityContentProducer ecp)schedules content for indexing.voidinitialize(org.sakaiproject.search.elasticsearch.ElasticSearchIndexBuilderEventRegistrar eventRegistrar, org.elasticsearch.client.RestHighLevelClient client)protected TimerinitializeBackgroundScheduler()protected TimerTaskinitializeContentQueueProcessingTask()protected org.elasticsearch.common.settings.SettingsinitializeElasticSearchIndexSettings(String injectedConfig)protected StringinitializeElasticSearchMapping(String injectedConfig)protected voidinitializeIndex()booleanisBuildQueueEmpty()booleanisEnabled()protected booleanisSupportedIndexAction(BaseElasticSearchIndexBuilder.IndexAction action)protected Optional<org.sakaiproject.search.api.EntityContentProducer>matchEntityContentProducer(Predicate<org.sakaiproject.search.api.EntityContentProducer> matcher)protected TimerTasknewBulkContentIndexerTask()org.sakaiproject.search.api.EntityContentProducernewEntityContentProducer(String ref)Find aEntityContentProducercapable of handling the given entity reference, or null if no such producer has been registered.org.sakaiproject.search.api.EntityContentProducernewEntityContentProducer(org.sakaiproject.event.api.Event event)Find aEntityContentProducercapable of handling the givenEvent, or null if no such producer has been registered.protected org.elasticsearch.action.index.IndexRequestnewIndexRequest(String resourceName, org.sakaiproject.search.api.EntityContentProducer ecp, boolean includeContent)protected org.elasticsearch.xcontent.XContentBuildernewIndexRequestContentSourceBuilder(String resourceName, org.sakaiproject.search.api.EntityContentProducer ecp, boolean includeContent)protected TimerTasknewRebuildIndexTask()protected org.elasticsearch.action.search.SearchRequestnewSearchRequestAndQueryBuilders()protected org.elasticsearch.action.search.SearchRequestnewSearchSuggestionsRequestAndQueryBuilders(String searchString)protected abstract org.elasticsearch.xcontent.XContentBuildernoContentForIndexRequest(org.elasticsearch.xcontent.XContentBuilder contentSourceBuilder, String resourceName, org.sakaiproject.search.api.EntityContentProducer ecp, boolean includeContent)protected voidnoContentProducerForContentQueueEntry(org.elasticsearch.search.SearchHit hit, String reference)protected <L,R>
org.apache.commons.lang3.tuple.Pair<L,R>pairOf(L left, R right)protected org.elasticsearch.action.delete.DeleteRequestprepareDeleteDocument(Map<String,Object> deleteParams)protected org.elasticsearch.action.search.SearchRequestprepareFindContentQueue()protected org.elasticsearch.action.index.IndexRequestprepareIndex(String resourceName, org.sakaiproject.search.api.EntityContentProducer ecp, boolean includeContent)protected voidprepareIndexAdd(String resourceName, org.sakaiproject.search.api.EntityContentProducer ecp, boolean includeContent)protected org.elasticsearch.action.search.SearchRequestprepareSearchRequest(String searchTerms, List<String> references, List<String> siteIds, int start, int end)protected org.elasticsearch.action.search.SearchRequestprepareSearchSuggestionsRequest(String searchString, String currentSite, boolean allMySites)protected voidprocessContentQueue()Searches for any docs in the search index that have not been indexed yet, digests the content and loads it into the index.protected voidprocessContentQueueEntry(org.elasticsearch.search.SearchHit hit, org.elasticsearch.action.bulk.BulkRequest bulkRequest)voidrebuildIndex()Removes any existing index, creates a new index, and rebuilds the index from the entities own stored stateprotected abstract voidrebuildIndexImmediately()protected voidrecreateIndex()removes any existing index and creates a new onevoidrefreshIndex()refresh the index from the current stored statevoidregisterEntityContentProducer(org.sakaiproject.search.api.EntityContentProducer ecp)register an entity content producer to provide content to the search engineprotected voidrequireConfiguration()protected doubleroundTwoDecimals(double d)org.elasticsearch.action.search.SearchResponsesearch(String searchTerms, List<String> references, List<String> siteIds, int start, int end)org.elasticsearch.action.search.SearchResponsesearch(String searchTerms, List<String> references, List<String> siteIds, int start, int end, Map<String,String> additionalSearchInformation)String[]searchSuggestions(String searchString, String currentSite, boolean allMySites)voidsetBulkRequestSize(int bulkRequestSize)voidsetContentIndexBatchSize(int contentIndexBatchSize)voidsetDefaultIndexSettingsResource(String defaultIndexSettingsResource)voidsetDefaultMappingResource(String defaultMappingResource)voidsetDelay(int delay)voidsetEventTrackingService(org.sakaiproject.event.api.EventTrackingService eventTrackingService)voidsetFacetName(String facetName)voidsetFacetTermSize(int facetTermSize)voidsetFilter(org.sakaiproject.search.elasticsearch.filter.SearchItemFilter filter)voidsetIndexedDocumentType(String indexedDocumentType)voidsetIndexName(String indexName)voidsetIndexSettingsConfig(String indexSettingsConfig)voidsetMappingConfig(String mappingConfig)voidsetMaxNumberOfSuggestions(int maxNumberOfSuggestions)voidsetName(String name)voidsetPeriod(int period)voidsetRebuildIndexOnStartup(boolean rebuildIndexOnStartup)voidsetSearchResultFieldNames(String[] searchResultFieldNames)voidsetSecurityService(org.sakaiproject.authz.api.SecurityService securityService)voidsetServerConfigurationService(org.sakaiproject.component.api.ServerConfigurationService serverConfigurationService)voidsetSuggestionMatchingFieldName(String suggestionMatchingFieldName)voidsetSuggestionResultFieldNames(String[] suggestionResultFieldNames)voidsetTestMode(boolean testMode)voidsetTriggerFunctions(Collection<String> triggerFunctions)voidsetUseFacetting(boolean useFacetting)voidsetUseSuggestions(boolean useSuggestions)protected Map<String,Object>validateAddResourceEvent(org.sakaiproject.event.api.Event event)protected voidvalidateContentProducer(org.sakaiproject.event.api.Event event, Map<String,Object> validationContext)protected voidvalidateIndexable(org.sakaiproject.event.api.Event event, Map<String,Object> validationContext)protected voidvalidateIndexAction(org.sakaiproject.event.api.Event event, Map<String,Object> validationContext)protected voidvalidateResourceName(org.sakaiproject.event.api.Event event, Map<String,Object> validationContext)protected voidvalidateServiceEnabled(org.sakaiproject.event.api.Event event, Map<String,Object> validationContext)
-
-
-
Field Detail
-
DEFAULT_FACET_NAME
protected static final String DEFAULT_FACET_NAME
- See Also:
- Constant Field Values
-
DEFAULT_SUGGESTION_MATCHING_FIELD_NAME
protected static final String DEFAULT_SUGGESTION_MATCHING_FIELD_NAME
- See Also:
- Constant Field Values
-
ADD_RESOURCE_VALIDATION_KEY_RESOURCE_NAME
protected static final String ADD_RESOURCE_VALIDATION_KEY_RESOURCE_NAME
Key in the "validation map" built up byvalidateAddResourceEvent(Event)(specificallyvalidateResourceName(Event, Map).- See Also:
- Constant Field Values
-
ADD_RESOURCE_VALIDATION_KEY_CONTENT_PRODUCER
protected static final String ADD_RESOURCE_VALIDATION_KEY_CONTENT_PRODUCER
- See Also:
- Constant Field Values
-
ADD_RESOURCE_VALIDATION_KEY_INDEX_ACTION
protected static final String ADD_RESOURCE_VALIDATION_KEY_INDEX_ACTION
- See Also:
- Constant Field Values
-
ADD_RESOURCE_VALIDATION_KEY_ENTITY_ID
protected static final String ADD_RESOURCE_VALIDATION_KEY_ENTITY_ID
- See Also:
- Constant Field Values
-
DELETE_RESOURCE_KEY_DOCUMENT_ID
protected static final String DELETE_RESOURCE_KEY_DOCUMENT_ID
- See Also:
- Constant Field Values
-
DELETE_RESOURCE_KEY_ENTITY_REFERENCE
protected static final String DELETE_RESOURCE_KEY_ENTITY_REFERENCE
- See Also:
- Constant Field Values
-
allowAllAdvisor
protected static final org.sakaiproject.authz.api.SecurityAdvisor allowAllAdvisor
-
securityService
protected org.sakaiproject.authz.api.SecurityService securityService
-
serverConfigurationService
protected org.sakaiproject.component.api.ServerConfigurationService serverConfigurationService
-
eventTrackingService
protected org.sakaiproject.event.api.EventTrackingService eventTrackingService
-
client
protected org.elasticsearch.client.RestHighLevelClient client
ES Client, with access to any indexes in the cluster.
-
indexName
protected String indexName
the ES indexname
-
name
protected String name
Logical, well-known name, possibly distinct from the physical ESindexName
-
rebuildIndexOnStartup
protected boolean rebuildIndexOnStartup
set to true to force an index rebuild at startup time, defaults to false. This is probably something you never want to use, other than in development or testing
-
useSuggestions
protected boolean useSuggestions
-
maxNumberOfSuggestions
protected int maxNumberOfSuggestions
max number of suggestions to return when looking for suggestions (this populates the autocomplete drop down in the UI)
-
suggestionMatchingFieldName
protected String suggestionMatchingFieldName
-
suggestionResultFieldNames
protected String[] suggestionResultFieldNames
-
searchResultFieldNames
protected String[] searchResultFieldNames
-
filter
protected org.sakaiproject.search.elasticsearch.filter.SearchItemFilter filter
-
useFacetting
protected boolean useFacetting
-
facetName
protected String facetName
-
facetTermSize
protected int facetTermSize
N most frequent terms
-
contentIndexBatchSize
protected int contentIndexBatchSize
Number of documents to index at a time for each run of the context indexing task (defaults to 500). Setting this too low will slow things down, setting it to high won't allow all nodes in the cluster to share the load.
-
bulkRequestSize
protected int bulkRequestSize
Number of actions to send in one elasticsearch bulk index call defaults to 10. Setting this to too high a number will have memory implications as you'll be keeping more content in memory until the request is executed.
-
delay
protected int delay
number seconds of wait after startup before starting the BulkContentIndexerTask (defaults to 3 minutes)
-
period
protected int period
how often the BulkContentIndexerTask runs in seconds (defaults to 1 minute)
-
startTime
protected long startTime
-
lastLoad
protected long lastLoad
-
testMode
protected boolean testMode
this turns off the threads and does indexing inline. DO NOT enable this in prod. It is meant for testing, especially unit tests only.
-
defaultMappingResource
protected String defaultMappingResource
-
mappingConfig
protected String mappingConfig
by default the mapping in configured in the mapping.json file. This can be overridden by injecting json into this property. See {@link elasticsearch mapping reference } for more information on configuration that is available. For example, if you want to change the analyzer config for a particular field this is the place to do it.
-
mapping
protected String mapping
Combination ofmappingConfig, its fallback read fromdefaultMappingResource, and any overrides implemented ininitializeElasticSearchMapping(String). (Currently there are no such overrides in that method... just the fallback resource lookup. And historically the mapping config was stored exclusively inmappingConfig. This newmappingMergedfield was added for symmetry withindexSettings).
-
defaultIndexSettingsResource
protected String defaultIndexSettingsResource
-
indexSettingsConfig
protected String indexSettingsConfig
Expects a JSON string of ElasticSearch index settings. You can set this in your sakai.properties files and inject a value using the indexSettings@org.sakaiproject.search.api.SearchIndexBuilder property. By default this value is configured by the indexSettings.json files. See {@link elasticsearch index modules} for more information on configuration that is available.
-
indexSettings
protected org.elasticsearch.common.settings.Settings indexSettings
-
indexedDocumentType
protected String indexedDocumentType
-
backgroundScheduler
protected Timer backgroundScheduler
indexing thread that performs loading the actual content into the index.
-
producers
protected Set<org.sakaiproject.search.api.EntityContentProducer> producers
-
eventRegistrar
protected org.sakaiproject.search.elasticsearch.ElasticSearchIndexBuilderEventRegistrar eventRegistrar
-
-
Method Detail
-
isEnabled
public boolean isEnabled()
-
destroy
public void destroy()
- Specified by:
destroyin interfaceorg.sakaiproject.search.api.SearchIndexBuilder
-
initialize
public void initialize(org.sakaiproject.search.elasticsearch.ElasticSearchIndexBuilderEventRegistrar eventRegistrar, org.elasticsearch.client.RestHighLevelClient client)- Specified by:
initializein interfaceorg.sakaiproject.search.elasticsearch.ElasticSearchIndexBuilder
-
beforeElasticSearchConfigInitialization
protected abstract void beforeElasticSearchConfigInitialization()
Gives subclasses a chance to initialize configuration prior to reading/processing any ES configs. May be important for setting up defaults, for example, or for ensuring subclass-specific configs are in place before any background tasks are in place. (Though the latter would be better factored intobeforeBackgroundSchedulerInitialization()
-
requireConfiguration
protected void requireConfiguration()
-
beforeBackgroundSchedulerInitialization
protected abstract void beforeBackgroundSchedulerInitialization()
Called after all ES config has been processed but before the background scheduler has been set up and before any index startup ops have been invoked (initializeIndex(). I.e. this is a subclass's last chance to set up any configs on which background jobs and/or index maintenance in general might depend.
-
initializeElasticSearchMapping
protected String initializeElasticSearchMapping(String injectedConfig)
-
initializeElasticSearchIndexSettings
protected org.elasticsearch.common.settings.Settings initializeElasticSearchIndexSettings(String injectedConfig)
-
initializeBackgroundScheduler
protected Timer initializeBackgroundScheduler()
-
initializeContentQueueProcessingTask
protected TimerTask initializeContentQueueProcessingTask()
-
newBulkContentIndexerTask
protected TimerTask newBulkContentIndexerTask()
-
newRebuildIndexTask
protected TimerTask newRebuildIndexTask()
-
processContentQueue
protected void processContentQueue()
Searches for any docs in the search index that have not been indexed yet, digests the content and loads it into the index. Any docs with empty content will be removed from the index.
-
processContentQueueEntry
protected void processContentQueueEntry(org.elasticsearch.search.SearchHit hit, org.elasticsearch.action.bulk.BulkRequest bulkRequest) throws NoContentException- Throws:
NoContentException
-
executeBulkRequest
protected void executeBulkRequest(org.elasticsearch.action.bulk.BulkRequest bulkRequest)
-
noContentProducerForContentQueueEntry
protected void noContentProducerForContentQueueEntry(org.elasticsearch.search.SearchHit hit, String reference) throws NoContentException- Throws:
NoContentException
-
findContentQueue
protected org.elasticsearch.action.search.SearchResponse findContentQueue()
-
prepareFindContentQueue
protected org.elasticsearch.action.search.SearchRequest prepareFindContentQueue()
-
addFindContentQueueRequestParams
protected org.elasticsearch.action.search.SearchRequest addFindContentQueueRequestParams(org.elasticsearch.action.search.SearchRequest searchRequest)
-
completeFindContentQueueRequest
protected abstract org.elasticsearch.action.search.SearchRequest completeFindContentQueueRequest(org.elasticsearch.action.search.SearchRequest searchRequest)
-
findContentQueueWithRequest
protected org.elasticsearch.action.search.SearchResponse findContentQueueWithRequest(org.elasticsearch.action.search.SearchRequest searchRequest)
-
deleteDocument
protected void deleteDocument(org.elasticsearch.search.SearchHit searchHit)
-
deleteDocument
protected void deleteDocument(NoContentException noContentException)
-
prepareDeleteDocument
protected org.elasticsearch.action.delete.DeleteRequest prepareDeleteDocument(Map<String,Object> deleteParams)
-
completeDeleteRequest
protected abstract org.elasticsearch.action.delete.DeleteRequest completeDeleteRequest(org.elasticsearch.action.delete.DeleteRequest deleteRequest, Map<String,Object> deleteParams)
-
deleteDocumentWithRequest
protected org.elasticsearch.action.delete.DeleteResponse deleteDocumentWithRequest(org.elasticsearch.action.delete.DeleteRequest deleteRequest) throws IOException- Throws:
IOException
-
extractDeleteDocumentParams
protected Map<String,Object> extractDeleteDocumentParams(org.elasticsearch.search.SearchHit searchHit)
-
extractDeleteDocumentParams
protected Map<String,Object> extractDeleteDocumentParams(NoContentException noContentException)
-
initializeIndex
protected void initializeIndex()
-
assureIndex
protected void assureIndex()
creates a new index if one does not exist
-
createIndex
protected void createIndex()
creates a new index, does not check if the exist exists
-
recreateIndex
protected void recreateIndex()
removes any existing index and creates a new one
-
rebuildIndex
public void rebuildIndex()
Removes any existing index, creates a new index, and rebuilds the index from the entities own stored state- Specified by:
rebuildIndexin interfaceorg.sakaiproject.search.api.SearchIndexBuilder
-
rebuildIndexImmediately
protected abstract void rebuildIndexImmediately()
-
refreshIndex
public void refreshIndex()
refresh the index from the current stored state- Specified by:
refreshIndexin interfaceorg.sakaiproject.search.api.SearchIndexBuilder
-
prepareIndex
protected org.elasticsearch.action.index.IndexRequest prepareIndex(String resourceName, org.sakaiproject.search.api.EntityContentProducer ecp, boolean includeContent) throws IOException, NoContentException
- Parameters:
resourceName-ecp-- Returns:
- Throws:
IOExceptionNoContentException
-
newIndexRequest
protected org.elasticsearch.action.index.IndexRequest newIndexRequest(String resourceName, org.sakaiproject.search.api.EntityContentProducer ecp, boolean includeContent)
-
completeIndexRequest
protected abstract org.elasticsearch.action.index.IndexRequest completeIndexRequest(org.elasticsearch.action.index.IndexRequest indexRequest, String resourceName, org.sakaiproject.search.api.EntityContentProducer ecp, boolean includeContent)
-
buildIndexRequestContentSource
protected org.elasticsearch.xcontent.XContentBuilder buildIndexRequestContentSource(String resourceName, org.sakaiproject.search.api.EntityContentProducer ecp, boolean includeContent) throws NoContentException, IOException
- Throws:
NoContentExceptionIOException
-
newIndexRequestContentSourceBuilder
protected org.elasticsearch.xcontent.XContentBuilder newIndexRequestContentSourceBuilder(String resourceName, org.sakaiproject.search.api.EntityContentProducer ecp, boolean includeContent) throws IOException
- Throws:
IOException
-
addFields
protected abstract org.elasticsearch.xcontent.XContentBuilder addFields(org.elasticsearch.xcontent.XContentBuilder contentSourceBuilder, String resourceName, org.sakaiproject.search.api.EntityContentProducer ecp, boolean includeContent) throws IOException- Throws:
IOException
-
addCustomProperties
protected org.elasticsearch.xcontent.XContentBuilder addCustomProperties(org.elasticsearch.xcontent.XContentBuilder contentSourceBuilder, String resourceName, org.sakaiproject.search.api.EntityContentProducer ecp, boolean includeContent) throws IOException- Throws:
IOException
-
addContent
protected org.elasticsearch.xcontent.XContentBuilder addContent(org.elasticsearch.xcontent.XContentBuilder contentSourceBuilder, String resourceName, org.sakaiproject.search.api.EntityContentProducer ecp, boolean includeContent) throws NoContentException, IOException- Throws:
NoContentExceptionIOException
-
noContentForIndexRequest
protected abstract org.elasticsearch.xcontent.XContentBuilder noContentForIndexRequest(org.elasticsearch.xcontent.XContentBuilder contentSourceBuilder, String resourceName, org.sakaiproject.search.api.EntityContentProducer ecp, boolean includeContent) throws NoContentException- Throws:
NoContentException
-
completeIndexRequestContentSourceBuilder
protected org.elasticsearch.xcontent.XContentBuilder completeIndexRequestContentSourceBuilder(org.elasticsearch.xcontent.XContentBuilder contentSourceBuilder, String resourceName, org.sakaiproject.search.api.EntityContentProducer ecp, boolean includeContent) throws IOException- Throws:
IOException
-
prepareIndexAdd
protected void prepareIndexAdd(String resourceName, org.sakaiproject.search.api.EntityContentProducer ecp, boolean includeContent) throws NoContentException
- Parameters:
resourceName-ecp-- Throws:
NoContentException
-
indexAdd
protected void indexAdd(String resourceName, org.sakaiproject.search.api.EntityContentProducer ecp)
schedules content for indexing.- Parameters:
resourceName-ecp-
-
getPendingDocuments
public int getPendingDocuments()
- Specified by:
getPendingDocumentsin interfaceorg.sakaiproject.search.api.SearchIndexBuilder
-
isBuildQueueEmpty
public boolean isBuildQueueEmpty()
- Specified by:
isBuildQueueEmptyin interfaceorg.sakaiproject.search.api.SearchIndexBuilder
-
addResource
public void addResource(org.sakaiproject.event.api.Notification notification, org.sakaiproject.event.api.Event event)- Specified by:
addResourcein interfaceorg.sakaiproject.search.api.SearchIndexBuilder
-
validateAddResourceEvent
protected Map<String,Object> validateAddResourceEvent(org.sakaiproject.event.api.Event event) throws IllegalArgumentException, IllegalStateException
-
validateServiceEnabled
protected void validateServiceEnabled(org.sakaiproject.event.api.Event event, Map<String,Object> validationContext) throws IllegalArgumentException, IllegalStateException
-
validateResourceName
protected void validateResourceName(org.sakaiproject.event.api.Event event, Map<String,Object> validationContext) throws IllegalArgumentException, IllegalStateException
-
validateContentProducer
protected void validateContentProducer(org.sakaiproject.event.api.Event event, Map<String,Object> validationContext) throws IllegalArgumentException, IllegalStateException
-
validateIndexable
protected void validateIndexable(org.sakaiproject.event.api.Event event, Map<String,Object> validationContext) throws IllegalArgumentException, IllegalStateException
-
validateIndexAction
protected void validateIndexAction(org.sakaiproject.event.api.Event event, Map<String,Object> validationContext) throws IllegalArgumentException, IllegalStateException, UnsupportedOperationException
-
isSupportedIndexAction
protected boolean isSupportedIndexAction(BaseElasticSearchIndexBuilder.IndexAction action)
-
completeAddResourceEventValidations
protected abstract void completeAddResourceEventValidations(org.sakaiproject.event.api.Event event, Map<String,Object> validationContext) throws IllegalArgumentException, IllegalStateException
-
dispatchValidatedAddResource
protected void dispatchValidatedAddResource(Map<String,Object> validationContext)
-
extractDeleteDocumentParams
protected Map<String,Object> extractDeleteDocumentParams(Map<String,Object> validationContext)
-
extractCustomProperties
protected Map<String,Collection<String>> extractCustomProperties(String resourceName, org.sakaiproject.search.api.EntityContentProducer contentProducer)
Extract properties from the givenEntityContentProducerThe
EntityContentProducer.getCustomProperties(String)method returns a map of different kind of elements. To avoid casting and calls toinstanceof, extractCustomProperties does all the work and returns a formated map containing onlyCollection.- Parameters:
resourceName- affected resourcecontentProducer- producer providing properties for the given resource- Returns:
- a formated map of
Collection
-
search
public org.elasticsearch.action.search.SearchResponse search(String searchTerms, List<String> references, List<String> siteIds, int start, int end)
- Specified by:
searchin interfaceorg.sakaiproject.search.elasticsearch.ElasticSearchIndexBuilder
-
search
public org.elasticsearch.action.search.SearchResponse search(String searchTerms, List<String> references, List<String> siteIds, int start, int end, Map<String,String> additionalSearchInformation)
- Specified by:
searchin interfaceorg.sakaiproject.search.elasticsearch.ElasticSearchIndexBuilder
-
prepareSearchRequest
protected org.elasticsearch.action.search.SearchRequest prepareSearchRequest(String searchTerms, List<String> references, List<String> siteIds, int start, int end)
-
newSearchRequestAndQueryBuilders
protected org.elasticsearch.action.search.SearchRequest newSearchRequestAndQueryBuilders()
-
addSearchCoreParams
protected void addSearchCoreParams(org.elasticsearch.action.search.SearchRequest searchRequest)
-
addSearchQuery
protected void addSearchQuery(org.elasticsearch.action.search.SearchRequest searchRequest, String searchTerms, List<String> references, List<String> siteIds)
-
addSearchTerms
protected void addSearchTerms(org.elasticsearch.action.search.SearchRequest searchRequest, String searchTerms)
-
addSearchReferences
protected void addSearchReferences(org.elasticsearch.action.search.SearchRequest searchRequest, List<String> references)
-
addSearchSiteIds
protected abstract void addSearchSiteIds(org.elasticsearch.action.search.SearchRequest searchRequest, List<String> siteIds)
-
addSearchResultFields
protected void addSearchResultFields(org.elasticsearch.action.search.SearchRequest searchRequest)
-
addSearchPagination
protected void addSearchPagination(org.elasticsearch.action.search.SearchRequest searchRequest, int start, int end)
-
addSearchFacetting
protected void addSearchFacetting(org.elasticsearch.action.search.SearchRequest searchRequest)
-
completeSearchRequestBuilders
protected abstract void completeSearchRequestBuilders(org.elasticsearch.action.search.SearchRequest searchRequest, String searchTerms, List<String> references, List<String> siteIds)
-
searchSuggestions
public String[] searchSuggestions(String searchString, String currentSite, boolean allMySites)
- Specified by:
searchSuggestionsin interfaceorg.sakaiproject.search.elasticsearch.ElasticSearchIndexBuilder
-
prepareSearchSuggestionsRequest
protected org.elasticsearch.action.search.SearchRequest prepareSearchSuggestionsRequest(String searchString, String currentSite, boolean allMySites)
-
completeSearchSuggestionsRequestBuilders
protected abstract void completeSearchSuggestionsRequestBuilders(org.elasticsearch.action.search.SearchRequest searchRequest, String searchString, String currentSite, boolean allMySites)
-
newSearchSuggestionsRequestAndQueryBuilders
protected org.elasticsearch.action.search.SearchRequest newSearchSuggestionsRequestAndQueryBuilders(String searchString)
-
addSearchSuggestionsCoreParams
protected void addSearchSuggestionsCoreParams(org.elasticsearch.action.search.SearchRequest searchRequest)
-
addSearchSuggestionsQuery
protected void addSearchSuggestionsQuery(org.elasticsearch.action.search.SearchRequest searchRequest, String searchString, String currentSite, boolean allMySites)
-
addSearchSuggestionsTerms
protected abstract void addSearchSuggestionsTerms(org.elasticsearch.action.search.SearchRequest searchRequest, String searchString)
-
addSearchSuggestionsSites
protected abstract void addSearchSuggestionsSites(org.elasticsearch.action.search.SearchRequest searchRequest, String currentSite, boolean allMySites)
-
addSearchSuggestionResultFields
protected void addSearchSuggestionResultFields(org.elasticsearch.action.search.SearchRequest searchRequest)
-
addSearchSuggestionsPagination
protected void addSearchSuggestionsPagination(org.elasticsearch.action.search.SearchRequest searchRequest)
-
getIndexHealth
public org.elasticsearch.cluster.health.ClusterIndexHealth getIndexHealth()
-
getStatus
public StringBuilder getStatus(StringBuilder into)
- Specified by:
getStatusin interfaceorg.sakaiproject.search.elasticsearch.ElasticSearchIndexBuilder
-
roundTwoDecimals
protected double roundTwoDecimals(double d)
-
getNDocs
public long getNDocs()
- Specified by:
getNDocsin interfaceorg.sakaiproject.search.elasticsearch.ElasticSearchIndexBuilder
-
getSearchStatus
public org.sakaiproject.search.api.SearchStatus getSearchStatus()
- Specified by:
getSearchStatusin interfaceorg.sakaiproject.search.elasticsearch.ElasticSearchIndexBuilder
-
newEntityContentProducer
public org.sakaiproject.search.api.EntityContentProducer newEntityContentProducer(String ref)
Find aEntityContentProducercapable of handling the given entity reference, or null if no such producer has been registered.- Specified by:
newEntityContentProducerin interfaceorg.sakaiproject.search.api.SearchIndexBuilder- Parameters:
ref- the entity reference
-
newEntityContentProducer
public org.sakaiproject.search.api.EntityContentProducer newEntityContentProducer(org.sakaiproject.event.api.Event event)
Find aEntityContentProducercapable of handling the givenEvent, or null if no such producer has been registered.- Specified by:
newEntityContentProducerin interfaceorg.sakaiproject.search.api.SearchIndexBuilder- Parameters:
event-- Returns:
-
matchEntityContentProducer
protected Optional<org.sakaiproject.search.api.EntityContentProducer> matchEntityContentProducer(Predicate<org.sakaiproject.search.api.EntityContentProducer> matcher)
-
getContentProducers
public List<org.sakaiproject.search.api.EntityContentProducer> getContentProducers()
get all the producers registered, as a clone to avoid concurrent modification exceptions- Specified by:
getContentProducersin interfaceorg.sakaiproject.search.api.SearchIndexBuilder- Returns:
-
registerEntityContentProducer
public void registerEntityContentProducer(org.sakaiproject.search.api.EntityContentProducer ecp)
register an entity content producer to provide content to the search engine- Specified by:
registerEntityContentProducerin interfaceorg.sakaiproject.search.api.SearchIndexBuilder
-
getContentFunctions
public Set<String> getContentFunctions()
- Specified by:
getContentFunctionsin interfaceorg.sakaiproject.search.elasticsearch.ElasticSearchIndexBuilder
-
enableAzgSecurityAdvisor
protected void enableAzgSecurityAdvisor()
Establish a security advisor to allow the "embedded" work to occur with no need for additional security permissions.
-
disableAzgSecurityAdvisor
protected void disableAzgSecurityAdvisor()
Disable the security advisor.
-
getFieldFromSearchHit
public String getFieldFromSearchHit(String field, org.elasticsearch.search.SearchHit hit)
loads the field from the SearchHit. Loads from field not from source since we aren't storing the source.- Specified by:
getFieldFromSearchHitin interfaceorg.sakaiproject.search.elasticsearch.ElasticSearchIndexBuilder- Parameters:
field-hit-- Returns:
-
pairOf
protected <L,R> org.apache.commons.lang3.tuple.Pair<L,R> pairOf(L left, R right)
-
getAllSearchItems
public List<org.sakaiproject.search.model.SearchBuilderItem> getAllSearchItems()
- Specified by:
getAllSearchItemsin interfaceorg.sakaiproject.search.api.SearchIndexBuilder
-
setIndexedDocumentType
public void setIndexedDocumentType(String indexedDocumentType)
-
setTestMode
public void setTestMode(boolean testMode)
-
setRebuildIndexOnStartup
public void setRebuildIndexOnStartup(boolean rebuildIndexOnStartup)
-
getUseFacetting
public boolean getUseFacetting()
- Specified by:
getUseFacettingin interfaceorg.sakaiproject.search.elasticsearch.ElasticSearchIndexBuilder
-
setUseFacetting
public void setUseFacetting(boolean useFacetting)
-
setFacetName
public void setFacetName(String facetName)
-
setMaxNumberOfSuggestions
public void setMaxNumberOfSuggestions(int maxNumberOfSuggestions)
-
setUseSuggestions
public void setUseSuggestions(boolean useSuggestions)
-
setSuggestionResultFieldNames
public void setSuggestionResultFieldNames(String[] suggestionResultFieldNames)
-
setSuggestionMatchingFieldName
public void setSuggestionMatchingFieldName(String suggestionMatchingFieldName)
-
setSearchResultFieldNames
public void setSearchResultFieldNames(String[] searchResultFieldNames)
-
setFilter
public void setFilter(org.sakaiproject.search.elasticsearch.filter.SearchItemFilter filter)
-
getFilter
public org.sakaiproject.search.elasticsearch.filter.SearchItemFilter getFilter()
- Specified by:
getFilterin interfaceorg.sakaiproject.search.elasticsearch.ElasticSearchIndexBuilder
-
getFacetName
public String getFacetName()
- Specified by:
getFacetNamein interfaceorg.sakaiproject.search.elasticsearch.ElasticSearchIndexBuilder
-
setFacetTermSize
public void setFacetTermSize(int facetTermSize)
-
getName
public String getName()
- Specified by:
getNamein interfaceorg.sakaiproject.search.api.SearchIndexBuilder
-
setName
public void setName(String name)
-
setIndexName
public void setIndexName(String indexName)
-
setDefaultIndexSettingsResource
public void setDefaultIndexSettingsResource(String defaultIndexSettingsResource)
-
setMappingConfig
public void setMappingConfig(String mappingConfig)
-
setIndexSettingsConfig
public void setIndexSettingsConfig(String indexSettingsConfig)
-
setDefaultMappingResource
public void setDefaultMappingResource(String defaultMappingResource)
-
setDelay
public void setDelay(int delay)
-
setPeriod
public void setPeriod(int period)
-
setContentIndexBatchSize
public void setContentIndexBatchSize(int contentIndexBatchSize)
-
setBulkRequestSize
public void setBulkRequestSize(int bulkRequestSize)
-
setSecurityService
public void setSecurityService(org.sakaiproject.authz.api.SecurityService securityService)
-
setServerConfigurationService
public void setServerConfigurationService(org.sakaiproject.component.api.ServerConfigurationService serverConfigurationService)
-
setEventTrackingService
public void setEventTrackingService(org.sakaiproject.event.api.EventTrackingService eventTrackingService)
-
setTriggerFunctions
public void setTriggerFunctions(Collection<String> triggerFunctions)
-
getTriggerFunctions
public Set<String> getTriggerFunctions()
- Specified by:
getTriggerFunctionsin interfaceorg.sakaiproject.search.elasticsearch.ElasticSearchIndexBuilder
-
getGlobalMasterSearchItems
public List<org.sakaiproject.search.model.SearchBuilderItem> getGlobalMasterSearchItems()
- Specified by:
getGlobalMasterSearchItemsin interfaceorg.sakaiproject.search.api.SearchIndexBuilder
-
getLog
protected abstract org.slf4j.Logger getLog()
-
-