Package org.dspace.discovery

Class SolrServiceImpl

java.lang.Object

org.dspace.discovery.SolrServiceImpl

All Implemented Interfaces:

IndexingService, SearchService

@Service
public class SolrServiceImpl
extends Object
implements SearchService, IndexingService

SolrIndexer contains the methods that index Items and their metadata, collections, communities, etc. It is meant to either be invoked from the command line (see dspace/bin/index-all) or via the indexContent() methods within DSpace.

The Administrator can choose to run SolrIndexer in a cron that repeats regularly, a failed attempt to index from the UI will be "caught" up on in that cron. The SolrServiceImpl is registered as a Service in the ServiceManager via a Spring configuration file located under classpath://spring/spring-dspace-applicationContext.xml Its configuration is Autowired by the ApplicationContext

Author:

Kevin Van de Velde (kevin at atmire dot com), Mark Diggory (markd at atmire dot com), Ben Bosman (ben at atmire dot com)

Field Summary

Fields

Modifier and Type	Field	Description
protected ConfigurationService	configurationService
protected ContentServiceFactory	contentServiceFactory
protected GroupService	groupService
protected IndexObjectFactoryFactory	indexObjectServiceFactory
static final String	SOLR_FIELD_SUFFIX_FACET_PREFIXES
protected SolrSearchCore	solrSearchCore

Constructor Summary

Constructors

Modifier	Constructor	Description
protected	SolrServiceImpl()

Method Summary

Modifier and Type	Method	Description
void	atomicUpdate(Context context, String uniqueIndexId, String field, Map<String,Object> fieldModifier)	Atomically update the index of a single field for an object
void	buildSpellCheck()
String	calculateExtremeValue(Context context, String valueField, String sortField, DiscoverQuery.SORT_ORDER sortOrder)	This method returns us either the highest or lowest value for the field that we give to it depending on what sortOrder we give this method.
void	cleanIndex()	Iterates over all documents in the Lucene index and verifies they are in database, if not, they are removed.
void	commit()
void	createIndex(Context c)	create full index - wiping old index
String	createLocationQueryForAdministrableItems(Context context)	Method to create a Query that includes all communities and collections a user may administrate.
void	deleteIndex()	Removes all documents from the Lucene index
protected void	emailException(Exception exception)
String	escapeQueryChars(String query)	Utility method to escape any special characters in a user's query
protected IndexableObject	findIndexableObject(Context context, org.apache.solr.common.SolrDocument doc)	Find the indexable object by type and UUID
FacetYearRange	getFacetYearRange(Context context, IndexableObject scope, DiscoverySearchFilterFacet facet, List<String> filterQueries, DiscoverQuery parentQuery)
List<Item>	getRelatedItems(Context context, Item item, DiscoveryMoreLikeThisConfiguration mltConfig)
void	indexContent(Context context, IndexableObject dso)	If the handle for the "dso" already exists in the index, and the "dso" has a lastModified timestamp that is newer than the document in the index then it is updated, otherwise a new document is added.
void	indexContent(Context context, IndexableObject indexableObject, boolean force)	If the handle for the "dso" already exists in the index, and the "dso" has a lastModified timestamp that is newer than the document in the index then it is updated, otherwise a new document is added.
void	indexContent(Context context, IndexableObject dso, boolean force, boolean commit)
void	indexContent(Context context, IndexableObject indexableObject, boolean force, boolean commit, boolean preDb)	Index a given DSO
Iterator<Item>	iteratorSearch(Context context, IndexableObject dso, DiscoverQuery query)	Convenience method to call @see #search(Context, DSpaceObject, DiscoverQuery) and getting an iterator for the results
String	locationToName(Context context, String field, String value)
void	optimize()	Maintenance to keep a SOLR index efficient.
void	reIndexContent(Context context, IndexableObject dso)	reIndexContent removes something from the index, then re-indexes it
protected boolean	requiresIndexing(String uniqueId, Instant lastModified)	Is stale checks the lastModified time stamp in the database and the index to determine if the index is stale.
protected org.apache.solr.client.solrj.SolrQuery	resolveToSolrQuery(Context context, DiscoverQuery discoveryQuery)
protected DiscoverResult	retrieveResult(Context context, DiscoverQuery query)
List<IndexableObject>	search(Context context, String query, int offset, int max, String... filterquery)
List<IndexableObject>	search(Context context, String query, String orderfield, boolean ascending, int offset, int max, String... filterquery)
DiscoverResult	search(Context context, DiscoverQuery discoveryQuery)	Convenient method to call SearchService.search(Context, DSpaceObject, DiscoverQuery) with a null DSpace Object as scope (i.e. all the repository)
DiscoverResult	search(Context context, IndexableObject dso, DiscoverQuery discoveryQuery)	Convenient method to call @see #search(Context, DSpaceObject, DiscoverQuery, boolean) with includeWithdrawn=false
DiscoverFilterQuery	toFilterQuery(Context context, String field, String operator, String value, DiscoveryConfiguration config)	Transforms the given string field and value into a filter query
String	toSortFieldIndex(String metadataField, String type)	Transforms the metadata field of the given sort configuration into the indexed field which we can then use in our Solr queries.
protected String	transformAuthorityValue(Context context, String field, String value)
protected String	transformDisplayedValue(Context context, String field, String value)
protected String	transformFacetField(DiscoverFacetField facetFieldConfig, String field, boolean removePostfix)
protected String	transformPrefixFacetField(DiscoverFacetField facetFieldConfig, String field, boolean removePostfix)	Gets the solr field that contains the facet value split on each word break to the end, so can be searched on each word in the value, see #saveFacetPrefixParts(SolrInputDocument, DiscoverySearchFilter, String, String) Only applicable to facets of type DiscoveryConfigurationParameters.TYPE_TEXT, otherwise uses the regular facet filter field
protected String	transformSortValue(Context context, String field, String value)
void	unIndexContent(Context context, String searchUniqueID)	Unindex a Document in the Lucene index.
void	unIndexContent(Context context, String searchUniqueID, boolean commit)	Unindex a Document in the Lucene Index.
void	unIndexContent(Context context, IndexableObject dso)	unIndex removes an Item, Collection, or Community
void	unIndexContent(Context context, IndexableObject indexableObject, boolean commit)	unIndex removes an Item, Collection, or Community
protected void	update(Context context, IndexFactory indexableObjectService, IndexableObject indexableObject)
protected void	update(Context context, IndexFactory indexableObjectService, IndexableObject indexableObject, boolean preDB)	Update the given indexable object using a given service
void	updateIndex(Context context)	Iterates over all Items, Collections and Communities.
void	updateIndex(Context context, boolean force)	Iterates over all Items, Collections and Communities.
void	updateIndex(Context context, boolean force, String type)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

SOLR_FIELD_SUFFIX_FACET_PREFIXES

public static final String SOLR_FIELD_SUFFIX_FACET_PREFIXES

See Also:

• Constant Field Values

contentServiceFactory

@Autowired
protected ContentServiceFactory contentServiceFactory

groupService

@Autowired
protected GroupService groupService

indexObjectServiceFactory

@Autowired
protected IndexObjectFactoryFactory indexObjectServiceFactory

solrSearchCore

@Autowired
protected SolrSearchCore solrSearchCore

configurationService

@Autowired
protected ConfigurationService configurationService

Constructor Details

SolrServiceImpl

protected SolrServiceImpl()

Method Details

indexContent

public void indexContent(Context context,
 IndexableObject dso)
                  throws SQLException

If the handle for the "dso" already exists in the index, and the "dso" has a lastModified timestamp that is newer than the document in the index then it is updated, otherwise a new document is added.

Specified by:

indexContent in interface IndexingService

Parameters:

context - Users Context

dso - DSpace Object (Item, Collection or Community

Throws:

SQLException - if error

indexContent

public void indexContent(Context context,
 IndexableObject indexableObject,
 boolean force)

If the handle for the "dso" already exists in the index, and the "dso" has a lastModified timestamp that is newer than the document in the index then it is updated, otherwise a new document is added.

Specified by:

indexContent in interface IndexingService

Parameters:

context - Users Context

indexableObject - The object we want to index

force - Force update even if not stale.

update

protected void update(Context context,
 IndexFactory indexableObjectService,
 IndexableObject indexableObject)
               throws IOException,
SQLException,
org.apache.solr.client.solrj.SolrServerException

Throws:

IOException

SQLException

org.apache.solr.client.solrj.SolrServerException

update

protected void update(Context context,
 IndexFactory indexableObjectService,
 IndexableObject indexableObject,
 boolean preDB)
               throws IOException,
SQLException,
org.apache.solr.client.solrj.SolrServerException

Update the given indexable object using a given service

Parameters:

context - The DSpace Context

indexableObjectService - The service to index the object with

indexableObject - The object to index

preDB - Add a "preDB" status to the document

Throws:

IOException

SQLException

org.apache.solr.client.solrj.SolrServerException

unIndexContent

public void unIndexContent(Context context,
 IndexableObject dso)
                    throws SQLException,
IOException

unIndex removes an Item, Collection, or Community

Specified by:

unIndexContent in interface IndexingService

Parameters:

context - The relevant DSpace Context.

dso - DSpace Object, can be Community, Item, or Collection

Throws:

SQLException - if database error

IOException - if IO error

unIndexContent

public void unIndexContent(Context context,
 IndexableObject indexableObject,
 boolean commit)
                    throws SQLException,
IOException

unIndex removes an Item, Collection, or Community

Specified by:

unIndexContent in interface IndexingService

Parameters:

context - The relevant DSpace Context.

indexableObject - The object to be indexed

commit - if true force an immediate commit on SOLR

Throws:

SQLException - if database error

IOException - if IO error

unIndexContent

public void unIndexContent(Context context,
 String searchUniqueID)
                    throws IOException

Unindex a Document in the Lucene index.

Specified by:

unIndexContent in interface IndexingService

Parameters:

context - the dspace context

searchUniqueID - the search uniqueID of the document to be deleted

Throws:

IOException - if IO error

unIndexContent

public void unIndexContent(Context context,
 String searchUniqueID,
 boolean commit)
                    throws IOException

Unindex a Document in the Lucene Index.

Specified by:

unIndexContent in interface IndexingService

Parameters:

context - the dspace context

searchUniqueID - the search uniqueID of the document to be deleted

commit - commit the update immediately.

Throws:

IOException - if IO error

reIndexContent

public void reIndexContent(Context context,
 IndexableObject dso)
                    throws SQLException,
IOException

reIndexContent removes something from the index, then re-indexes it

Specified by:

reIndexContent in interface IndexingService

Parameters:

context - context object

dso - object to re-index

Throws:

SQLException - passed through.

IOException - passed through.

createIndex

public void createIndex(Context c)
                 throws SQLException,
IOException

create full index - wiping old index

Specified by:

createIndex in interface IndexingService

Parameters:

c - context to use

Throws:

SQLException - passed through.

IOException - passed through.

updateIndex

public void updateIndex(Context context)

Iterates over all Items, Collections and Communities. And updates them in the index. Uses decaching to control memory footprint. Uses indexContent and isStale to check state of item in index.

Specified by:

updateIndex in interface IndexingService

Parameters:

context - the dspace context

updateIndex

public void updateIndex(Context context,
 boolean force)

Iterates over all Items, Collections and Communities. And updates them in the index. Uses decaching to control memory footprint. Uses indexContent and isStale to check state of item in index.

At first it may appear counterintuitive to have an IndexWriter/Reader opened and closed on each DSO. But this allows the UI processes to step in and attain a lock and write to the index even if other processes/jvms are running a reindex.

Specified by:

updateIndex in interface IndexingService

Parameters:

context - the dspace context

force - whether or not to force the reindexing

updateIndex

public void updateIndex(Context context,
 boolean force,
 String type)

Specified by:

updateIndex in interface IndexingService

deleteIndex

public void deleteIndex()

Removes all documents from the Lucene index

Specified by:

deleteIndex in interface IndexingService

cleanIndex

public void cleanIndex()
                throws IOException,
SQLException,
SearchServiceException

Iterates over all documents in the Lucene index and verifies they are in database, if not, they are removed.

Specified by:

cleanIndex in interface IndexingService

Throws:

IOException - IO exception

SQLException - sql exception

SearchServiceException - occurs when something went wrong with querying the solr server

optimize

public void optimize()

Maintenance to keep a SOLR index efficient. Note: This might take a long time.

Specified by:

optimize in interface IndexingService

buildSpellCheck

public void buildSpellCheck()
                     throws SearchServiceException,
IOException

Specified by:

buildSpellCheck in interface IndexingService

Throws:

SearchServiceException

IOException

atomicUpdate

public void atomicUpdate(Context context,
 String uniqueIndexId,
 String field,
 Map<String,Object> fieldModifier)
                  throws org.apache.solr.client.solrj.SolrServerException,
IOException

Description copied from interface: IndexingService

Atomically update the index of a single field for an object

Specified by:

atomicUpdate in interface IndexingService

Parameters:

context - The DSpace context

uniqueIndexId - The unique index ID of the object to update the index for

field - The field to update

fieldModifier - The modifiers for the field to update. More information on how to atomically update a solr field using a field modifier can be found here: https://yonik.com/solr/atomic-updates/

Throws:

org.apache.solr.client.solrj.SolrServerException

IOException

emailException

protected void emailException(Exception exception)

requiresIndexing

protected boolean requiresIndexing(String uniqueId,
 Instant lastModified)
                            throws SQLException,
IOException,
SearchServiceException

Is stale checks the lastModified time stamp in the database and the index to determine if the index is stale.

Parameters:

uniqueId - the unique identifier of the object that we want to index

lastModified - the last modified date of the DSpace object

Returns:

a boolean indicating if the dso should be re indexed again

Throws:

SQLException - sql exception

IOException - io exception

SearchServiceException - if something went wrong with querying the solr server

createLocationQueryForAdministrableItems

public String createLocationQueryForAdministrableItems(Context context)
                                                throws SQLException

Description copied from interface: SearchService

Method to create a Query that includes all communities and collections a user may administrate. If a user has the appropriate rights to administrate communities and/or collections we want to look up all contents of those communities and/or collections, ignoring the read policies of the items (e.g. to list all private items of communities/collections the user administrates). This method returns a query to filter for items that belong to those communities/collections only.

Specified by:

createLocationQueryForAdministrableItems in interface SearchService

Parameters:

context - The relevant DSpace Context.

Returns:

query string specific to the user's rights

Throws:

SQLException - An exception that provides information on a database access error or other errors.

locationToName

public String locationToName(Context context,
 String field,
 String value)
                      throws SQLException

Throws:

SQLException

search

public DiscoverResult search(Context context,
 IndexableObject dso,
 DiscoverQuery discoveryQuery)
                      throws SearchServiceException

Description copied from interface: SearchService

Convenient method to call @see #search(Context, DSpaceObject, DiscoverQuery, boolean) with includeWithdrawn=false

Specified by:

search in interface SearchService

Parameters:

context - DSpace Context object

dso - a DSpace Object to use as scope of the search (only results within this object)

discoveryQuery - the discovery query object

Returns:

discovery search result object

Throws:

SearchServiceException - if search error

iteratorSearch

public Iterator<Item> iteratorSearch(Context context,
 IndexableObject dso,
 DiscoverQuery query)
                              throws SearchServiceException

Description copied from interface: SearchService

Convenience method to call @see #search(Context, DSpaceObject, DiscoverQuery) and getting an iterator for the results

Specified by:

iteratorSearch in interface SearchService

Parameters:

context - DSpace context object

dso - a DSpace object to use as a scope of the search

query - the discovery query object

Returns:

an iterator iterating over all results from the search

Throws:

SearchServiceException - if search error

search

public DiscoverResult search(Context context,
 DiscoverQuery discoveryQuery)
                      throws SearchServiceException

Description copied from interface: SearchService

Convenient method to call SearchService.search(Context, DSpaceObject, DiscoverQuery) with a null DSpace Object as scope (i.e. all the repository)

Specified by:

search in interface SearchService

Parameters:

context - DSpace Context object.

discoveryQuery - the discovery query object.

Returns:

discovery search result object

Throws:

SearchServiceException - if search error

resolveToSolrQuery

protected org.apache.solr.client.solrj.SolrQuery resolveToSolrQuery(Context context,
 DiscoverQuery discoveryQuery)
                                                             throws SearchServiceException

Throws:

SearchServiceException

retrieveResult

protected DiscoverResult retrieveResult(Context context,
 DiscoverQuery query)
                                 throws SQLException,
org.apache.solr.client.solrj.SolrServerException,
IOException,
SearchServiceException

Throws:

SQLException

org.apache.solr.client.solrj.SolrServerException

IOException

SearchServiceException

findIndexableObject

protected IndexableObject findIndexableObject(Context context,
 org.apache.solr.common.SolrDocument doc)
                                       throws SQLException

Find the indexable object by type and UUID

Parameters:

context - The relevant DSpace Context.

doc - the solr document, the following fields MUST be present RESOURCE_TYPE_FIELD, RESOURCE_ID_FIELD and HANDLE_FIELD

Returns:

an IndexableObject

Throws:

SQLException - An exception that provides information on a database access error or other errors.

search

public List<IndexableObject> search(Context context,
 String query,
 int offset,
 int max,
 String... filterquery)

search

public List<IndexableObject> search(Context context,
 String query,
 String orderfield,
 boolean ascending,
 int offset,
 int max,
 String... filterquery)

Specified by:

search in interface SearchService

toFilterQuery

public DiscoverFilterQuery toFilterQuery(Context context,
 String field,
 String operator,
 String value,
 DiscoveryConfiguration config)
                                  throws SQLException

Description copied from interface: SearchService

Transforms the given string field and value into a filter query

Specified by:

toFilterQuery in interface SearchService

Parameters:

context - The relevant DSpace Context.

field - the field of the filter query

operator - equals/notequals/notcontains/authority/notauthority

value - the filter query value

config - (nullable) the discovery configuration (if not null, field's corresponding facet.type checked to be standard so suffix is not added for equals operator)

Returns:

a filter query

Throws:

SQLException - if database error An exception that provides information on a database access error or other errors.

getRelatedItems

public List<Item> getRelatedItems(Context context,
 Item item,
 DiscoveryMoreLikeThisConfiguration mltConfig)

Specified by:

getRelatedItems in interface SearchService

toSortFieldIndex

public String toSortFieldIndex(String metadataField,
 String type)

Description copied from interface: SearchService

Transforms the metadata field of the given sort configuration into the indexed field which we can then use in our Solr queries.

Specified by:

toSortFieldIndex in interface SearchService

Parameters:

metadataField - the metadata field

type - see DiscoveryConfigurationParameters

Returns:

the indexed field

transformPrefixFacetField

protected String transformPrefixFacetField(DiscoverFacetField facetFieldConfig,
 String field,
 boolean removePostfix)

Gets the solr field that contains the facet value split on each word break to the end, so can be searched on each word in the value, see #saveFacetPrefixParts(SolrInputDocument, DiscoverySearchFilter, String, String) Only applicable to facets of type DiscoveryConfigurationParameters.TYPE_TEXT, otherwise uses the regular facet filter field

transformFacetField

protected String transformFacetField(DiscoverFacetField facetFieldConfig,
 String field,
 boolean removePostfix)

transformDisplayedValue

protected String transformDisplayedValue(Context context,
 String field,
 String value)
                                  throws SQLException

Throws:

SQLException

transformAuthorityValue

protected String transformAuthorityValue(Context context,
 String field,
 String value)
                                  throws SQLException

Throws:

SQLException

transformSortValue

protected String transformSortValue(Context context,
 String field,
 String value)
                             throws SQLException

Throws:

SQLException

indexContent

public void indexContent(Context context,
 IndexableObject dso,
 boolean force,
 boolean commit)
                  throws SearchServiceException,
SQLException

Specified by:

indexContent in interface IndexingService

Throws:

SearchServiceException

SQLException

indexContent

public void indexContent(Context context,
 IndexableObject indexableObject,
 boolean force,
 boolean commit,
 boolean preDb)
                  throws SearchServiceException,
SQLException

Description copied from interface: IndexingService

Index a given DSO

Specified by:

indexContent in interface IndexingService

Parameters:

context - The DSpace Context

indexableObject - The DSpace Object to index

force - Force update even if not stale

commit - Commit the changes

preDb - Add a "preDB" status to the index (only applicable to Items)

Throws:

SearchServiceException

SQLException

commit

public void commit()
            throws SearchServiceException

Specified by:

commit in interface IndexingService

Throws:

SearchServiceException

escapeQueryChars

public String escapeQueryChars(String query)

Description copied from interface: SearchService

Utility method to escape any special characters in a user's query

Specified by:

escapeQueryChars in interface SearchService

Parameters:

query - User's query to escape.

Returns:

query with any special characters escaped

getFacetYearRange

public FacetYearRange getFacetYearRange(Context context,
 IndexableObject scope,
 DiscoverySearchFilterFacet facet,
 List<String> filterQueries,
 DiscoverQuery parentQuery)
                                 throws SearchServiceException

Specified by:

getFacetYearRange in interface SearchService

Throws:

SearchServiceException

calculateExtremeValue

public String calculateExtremeValue(Context context,
 String valueField,
 String sortField,
 DiscoverQuery.SORT_ORDER sortOrder)
                             throws SearchServiceException

Description copied from interface: SearchService

This method returns us either the highest or lowest value for the field that we give to it depending on what sortOrder we give this method.

Specified by:

calculateExtremeValue in interface SearchService

Parameters:

context - The relevant DSpace context

valueField - The field in solr for which we'll calculate the extreme value

sortField - The field in solr for which we'll sort the calculated extreme value on This is typically the valueField appended with "_sort"

sortOrder - Entering ascending will return the minimum value Entering descending will return the maximum value

Returns:

Returns the min or max value based on the field in the parameters.

Throws:

SearchServiceException