Package org.dspace.statistics

Class SolrLoggerServiceImpl

java.lang.Object

org.dspace.statistics.SolrLoggerServiceImpl

All Implemented Interfaces:

SolrLoggerService, InitializingBean

public class SolrLoggerServiceImpl
extends Object
implements SolrLoggerService, InitializingBean

Static holder for a HttpSolrClient connection pool to issue usage logging events to Solr from DSpace libraries, and some static query composers.

Author:

ben at atmire.com, kevinvandevelde at atmire.com, mdiggory at atmire.com

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
class	SolrLoggerServiceImpl.ResultProcessor
static enum	SolrLoggerServiceImpl.StatisticsType	Possible values of the type field of a usage event document.

Field Summary

Fields

Modifier and Type	Field	Description
protected BitstreamService	bitstreamService
protected ClientInfoService	clientInfoService
protected ConfigurationService	configurationService
protected ContentServiceFactory	contentServiceFactory
static final String	DATE_FORMAT_8601
static final String	DATE_FORMAT_DCDATE
protected GeoIpService	geoIpService
protected com.maxmind.geoip2.DatabaseReader	locationService
protected org.apache.solr.client.solrj.SolrClient	solr
protected SolrStatisticsCore	solrStatisticsCore
protected boolean	useProxies

Constructor Summary

Constructors

Modifier	Constructor	Description
protected	SolrLoggerServiceImpl()

Method Summary

Modifier and Type	Method	Description
protected void	addAdditionalSolrYearCores(org.apache.solr.client.solrj.SolrQuery solrQuery)
protected void	addDocumentsToFile(Context context, org.apache.solr.common.SolrDocumentList docs, File exportOutput)
void	afterPropertiesSet()
Object	anonymizeIp(String ip)	Anonymize a given ip
void	commit()	Commit the solr core.
protected org.apache.solr.client.solrj.impl.HttpSolrClient	createCore(org.apache.solr.client.solrj.impl.HttpSolrClient solr, String coreName)
void	deleteRobots()	Delete all 'statistics' documents having 'isBot:true'.
void	exportHits()	Export all SOLR usage statistics for viewing/downloading content to a flat text file.
protected String	generateURL(String baseURL, Map<String,String> parameters)
protected org.apache.solr.common.SolrInputDocument	getCommonSolrDoc(DSpaceObject dspaceObject, jakarta.servlet.http.HttpServletRequest request, EPerson currentUser)	Returns a solr input document containing common information about the statistics regardless if we are logging a search or a view of a DSpace object
protected org.apache.solr.common.SolrInputDocument	getCommonSolrDoc(DSpaceObject dspaceObject, jakarta.servlet.http.HttpServletRequest request, EPerson currentUser, String referrer)	Returns a solr input document containing common information about the statistics regardless if we are logging a search or a view of a DSpace object
protected org.apache.solr.common.SolrInputDocument	getCommonSolrDoc(DSpaceObject dspaceObject, String ip, String userAgent, String xforwardedfor, EPerson currentUser, String referrer)
protected String	getDateView(String name, String type)
Set<String>	getMultivaluedFieldNames()	Retrieves a list of all the multi valued fields in the solr core.
protected void	initSolrYearCores()
boolean	isUseProxies()
void	markRobots()	Scan the entire 'statistics' collection for documents that should be marked 'isBot:true' according to SpiderDetector.isSpider(java.lang.String, java.lang.String, java.lang.String, java.lang.String).
void	post(DSpaceObject dspaceObject, jakarta.servlet.http.HttpServletRequest request, EPerson currentUser)	Old post method, use the new SolrLoggerService.postView(org.dspace.content.DSpaceObject, jakarta.servlet.http.HttpServletRequest, org.dspace.eperson.EPerson) method instead !
void	postSearch(DSpaceObject resultObject, jakarta.servlet.http.HttpServletRequest request, EPerson currentUser, List<String> queries, int rpp, String sortBy, String order, int page, DSpaceObject scope)
void	postView(DSpaceObject dspaceObject, jakarta.servlet.http.HttpServletRequest request, EPerson currentUser)	Store a usage event into Solr.
void	postView(DSpaceObject dspaceObject, jakarta.servlet.http.HttpServletRequest request, EPerson currentUser, String referrer)	Store a usage event into Solr.
void	postView(DSpaceObject dspaceObject, String ip, String userAgent, String xforwardedfor, EPerson currentUser)
void	postView(DSpaceObject dspaceObject, String ip, String userAgent, String xforwardedfor, EPerson currentUser, String referrer)
void	postWorkflow(UsageWorkflowEvent usageWorkflowEvent)
void	query(String query, int max, int facetMinCount)
org.apache.solr.client.solrj.response.QueryResponse	query(String query, String filterQuery, String facetField, int rows, int max, String dateType, String dateStart, String dateEnd, List<String> facetQueries, String sort, boolean ascending, int facetMinCount)	Perform a solr query.
org.apache.solr.client.solrj.response.QueryResponse	query(String query, String filterQuery, String facetField, int rows, int max, String dateType, String dateStart, String dateEnd, List<String> facetQueries, String sort, boolean ascending, int facetMinCount, boolean defaultFilterQueries)	Perform a solr query.
ObjectCount[]	queryFacetDate(String query, String filterQuery, int max, String dateType, String dateStart, String dateEnd, boolean showTotal, Context context, int facetMinCount)	Query used to get values grouped by the date.
ObjectCount[]	queryFacetField(String query, String filterQuery, String facetField, int max, boolean showTotal, List<String> facetQueries, int facetMinCount)	Query used to get values grouped by the given facet field.
Map<String,Integer>	queryFacetQuery(String query, String filterQuery, List<String> facetQueries, int facetMinCount)
Map<String,List<String>>	queryField(String query, List oldFieldVals, String field)
ObjectCount	queryTotal(String query, String filterQuery, int facetMinCount)
void	reindexBitstreamHits(boolean removeDeletedBitstreams)
void	removeIndex(String query)	Delete data from the index, as described by a query.
void	shardSolrIndex()
void	storeParents(org.apache.solr.common.SolrInputDocument doc1, DSpaceObject dso)	Method just used to log the parents.
void	update(String query, String action, List<String> fieldNames, List<List<Object>> fieldValuesList)
void	update(String query, String action, List<String> fieldNames, List<List<Object>> fieldValuesList, boolean commit)	Update the solr core.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

DATE_FORMAT_8601

public static final String DATE_FORMAT_8601

See Also:

• Constant Field Values

DATE_FORMAT_DCDATE

public static final String DATE_FORMAT_DCDATE

See Also:

• Constant Field Values

locationService

protected com.maxmind.geoip2.DatabaseReader locationService

useProxies

protected boolean useProxies

bitstreamService

@Autowired(required=true)
protected BitstreamService bitstreamService

contentServiceFactory

@Autowired(required=true)
protected ContentServiceFactory contentServiceFactory

configurationService

@Autowired(required=true)
protected ConfigurationService configurationService

clientInfoService

@Autowired(required=true)
protected ClientInfoService clientInfoService

solrStatisticsCore

@Autowired
protected SolrStatisticsCore solrStatisticsCore

geoIpService

@Autowired
protected GeoIpService geoIpService

solr

protected org.apache.solr.client.solrj.SolrClient solr

Constructor Details

SolrLoggerServiceImpl

protected SolrLoggerServiceImpl()

Method Details

afterPropertiesSet

public void afterPropertiesSet()
                        throws Exception

Specified by:

afterPropertiesSet in interface InitializingBean

Throws:

Exception

post

public void post(DSpaceObject dspaceObject,
 jakarta.servlet.http.HttpServletRequest request,
 EPerson currentUser)

Description copied from interface: SolrLoggerService

Old post method, use the new SolrLoggerService.postView(org.dspace.content.DSpaceObject, jakarta.servlet.http.HttpServletRequest, org.dspace.eperson.EPerson) method instead !

Specified by:

post in interface SolrLoggerService

Parameters:

dspaceObject - the object used.

request - the current request context.

currentUser - the current session's user.

postView

public void postView(DSpaceObject dspaceObject,
 jakarta.servlet.http.HttpServletRequest request,
 EPerson currentUser)

Description copied from interface: SolrLoggerService

Store a usage event into Solr.

Specified by:

postView in interface SolrLoggerService

Parameters:

dspaceObject - the object used.

request - the current request context.

currentUser - the current session's user.

postView

public void postView(DSpaceObject dspaceObject,
 jakarta.servlet.http.HttpServletRequest request,
 EPerson currentUser,
 String referrer)

Description copied from interface: SolrLoggerService

Store a usage event into Solr.

Specified by:

postView in interface SolrLoggerService

Parameters:

dspaceObject - the object used.

request - the current request context.

currentUser - the current session's user.

referrer - the optional referrer.

postView

public void postView(DSpaceObject dspaceObject,
 String ip,
 String userAgent,
 String xforwardedfor,
 EPerson currentUser)

Specified by:

postView in interface SolrLoggerService

postView

public void postView(DSpaceObject dspaceObject,
 String ip,
 String userAgent,
 String xforwardedfor,
 EPerson currentUser,
 String referrer)

Specified by:

postView in interface SolrLoggerService

getCommonSolrDoc

protected org.apache.solr.common.SolrInputDocument getCommonSolrDoc(DSpaceObject dspaceObject,
 jakarta.servlet.http.HttpServletRequest request,
 EPerson currentUser)
                                                             throws SQLException

Returns a solr input document containing common information about the statistics regardless if we are logging a search or a view of a DSpace object

Parameters:

dspaceObject - the object used.

request - the current request context.

currentUser - the current session's user.

Returns:

a solr input document

Throws:

SQLException - in case of a database exception

getCommonSolrDoc

protected org.apache.solr.common.SolrInputDocument getCommonSolrDoc(DSpaceObject dspaceObject,
 jakarta.servlet.http.HttpServletRequest request,
 EPerson currentUser,
 String referrer)
                                                             throws SQLException

Returns a solr input document containing common information about the statistics regardless if we are logging a search or a view of a DSpace object

Parameters:

dspaceObject - the object used.

request - the current request context.

currentUser - the current session's user.

referrer - the optional referrer.

Returns:

a solr input document

Throws:

SQLException - in case of a database exception

getCommonSolrDoc

protected org.apache.solr.common.SolrInputDocument getCommonSolrDoc(DSpaceObject dspaceObject,
 String ip,
 String userAgent,
 String xforwardedfor,
 EPerson currentUser,
 String referrer)
                                                             throws SQLException

Throws:

SQLException

postSearch

public void postSearch(DSpaceObject resultObject,
 jakarta.servlet.http.HttpServletRequest request,
 EPerson currentUser,
 List<String> queries,
 int rpp,
 String sortBy,
 String order,
 int page,
 DSpaceObject scope)

Specified by:

postSearch in interface SolrLoggerService

postWorkflow

public void postWorkflow(UsageWorkflowEvent usageWorkflowEvent)
                  throws SQLException

Specified by:

postWorkflow in interface SolrLoggerService

Throws:

SQLException

storeParents

public void storeParents(org.apache.solr.common.SolrInputDocument doc1,
 DSpaceObject dso)
                  throws SQLException

Description copied from interface: SolrLoggerService

Method just used to log the parents.

• Community log: owning comms.
• Collection log: owning comms and their comms.
• Item log: owning colls/comms.
• Bitstream log: owning item/colls/comms.

Specified by:

storeParents in interface SolrLoggerService

Parameters:

doc1 - the current SolrInputDocument

dso - the current dspace object we want to log

Throws:

SQLException - if database error ignore it

isUseProxies

public boolean isUseProxies()

Specified by:

isUseProxies in interface SolrLoggerService

removeIndex

public void removeIndex(String query)
                 throws IOException,
org.apache.solr.client.solrj.SolrServerException

Description copied from interface: SolrLoggerService

Delete data from the index, as described by a query.

Specified by:

removeIndex in interface SolrLoggerService

Parameters:

query - description of the records to be deleted.

Throws:

IOException - A general class of exceptions produced by failed or interrupted I/O operations.

org.apache.solr.client.solrj.SolrServerException - Exception from the Solr server to the solrj Java client.

queryField

public Map<String,List<String>> queryField(String query,
 List oldFieldVals,
 String field)
                                    throws IOException

Specified by:

queryField in interface SolrLoggerService

Throws:

IOException

markRobots

public void markRobots()

Description copied from interface: SolrLoggerService

Scan the entire 'statistics' collection for documents that should be marked 'isBot:true' according to SpiderDetector.isSpider(java.lang.String, java.lang.String, java.lang.String, java.lang.String).

Specified by:

markRobots in interface SolrLoggerService

deleteRobots

public void deleteRobots()

Description copied from interface: SolrLoggerService

Delete all 'statistics' documents having 'isBot:true'.

Specified by:

deleteRobots in interface SolrLoggerService

update

public void update(String query,
 String action,
 List<String> fieldNames,
 List<List<Object>> fieldValuesList)
            throws org.apache.solr.client.solrj.SolrServerException,
IOException

Specified by:

update in interface SolrLoggerService

Throws:

org.apache.solr.client.solrj.SolrServerException

IOException

update

public void update(String query,
 String action,
 List<String> fieldNames,
 List<List<Object>> fieldValuesList,
 boolean commit)
            throws org.apache.solr.client.solrj.SolrServerException,
IOException

Description copied from interface: SolrLoggerService

Update the solr core.

Specified by:

update in interface SolrLoggerService

Parameters:

query - query indicating which documents to update

action - the update action keyword

fieldNames - the fields to update

fieldValuesList - the values for the fields to update

commit - whether to commit the changes

Throws:

org.apache.solr.client.solrj.SolrServerException

IOException

query

public void query(String query,
 int max,
 int facetMinCount)
           throws org.apache.solr.client.solrj.SolrServerException,
IOException

Specified by:

query in interface SolrLoggerService

Throws:

org.apache.solr.client.solrj.SolrServerException

IOException

queryFacetField

public ObjectCount[] queryFacetField(String query,
 String filterQuery,
 String facetField,
 int max,
 boolean showTotal,
 List<String> facetQueries,
 int facetMinCount)
                              throws org.apache.solr.client.solrj.SolrServerException,
IOException

Description copied from interface: SolrLoggerService

Query used to get values grouped by the given facet field.

Specified by:

queryFacetField in interface SolrLoggerService

Parameters:

query - the query to be used

filterQuery - filter query

facetField - the facet field on which to group our values

max - the max number of values given back (in case of 10 the top 10 will be given)

showTotal - a boolean determining whether the total amount should be given back as the last element of the array

facetQueries - list of facet queries

facetMinCount - Minimum count of results facet must have to return a result

Returns:

an array containing our results

Throws:

org.apache.solr.client.solrj.SolrServerException - Exception from the Solr server to the solrj Java client.

IOException - passed through.

queryFacetDate

public ObjectCount[] queryFacetDate(String query,
 String filterQuery,
 int max,
 String dateType,
 String dateStart,
 String dateEnd,
 boolean showTotal,
 Context context,
 int facetMinCount)
                             throws org.apache.solr.client.solrj.SolrServerException,
IOException

Description copied from interface: SolrLoggerService

Query used to get values grouped by the date.

Specified by:

queryFacetDate in interface SolrLoggerService

Parameters:

query - the query to be used

filterQuery - filter query

max - the max number of values given back (in case of 10 the top 10 will be given)

dateType - the type to be used (example: DAY, MONTH, YEAR)

dateStart - the start date Format:(-3, -2, ..) the date is calculated relatively on today

dateEnd - the end date stop Format (-2, +1, ..) the date is calculated relatively on today

showTotal - a boolean determining whether the total amount should be given back as the last element of the array

context - The relevant DSpace Context.

facetMinCount - Minimum count of results facet must have to return a result

Returns:

and array containing our results

Throws:

org.apache.solr.client.solrj.SolrServerException - Exception from the Solr server to the solrj Java client.

IOException - passed through.

queryFacetQuery

public Map<String,Integer> queryFacetQuery(String query,
 String filterQuery,
 List<String> facetQueries,
 int facetMinCount)
                                    throws org.apache.solr.client.solrj.SolrServerException,
IOException

Specified by:

queryFacetQuery in interface SolrLoggerService

Throws:

org.apache.solr.client.solrj.SolrServerException

IOException

queryTotal

public ObjectCount queryTotal(String query,
 String filterQuery,
 int facetMinCount)
                       throws org.apache.solr.client.solrj.SolrServerException,
IOException

Specified by:

queryTotal in interface SolrLoggerService

Throws:

org.apache.solr.client.solrj.SolrServerException

IOException

getDateView

protected String getDateView(String name,
 String type)

query

public org.apache.solr.client.solrj.response.QueryResponse query(String query,
 String filterQuery,
 String facetField,
 int rows,
 int max,
 String dateType,
 String dateStart,
 String dateEnd,
 List<String> facetQueries,
 String sort,
 boolean ascending,
 int facetMinCount)
                                                          throws org.apache.solr.client.solrj.SolrServerException,
IOException

Description copied from interface: SolrLoggerService

Perform a solr query.

Specified by:

query in interface SolrLoggerService

Parameters:

query - the query to be used

filterQuery - filter query

facetField - field to facet the results by

rows - the max number of results to return

max - the max number of facets to return

dateType - the type to be used (example: DAY, MONTH, YEAR)

dateStart - the start date Format:(-3, -2, ..) the date is calculated relatively on today

dateEnd - the end date stop Format (-2, +1, ..) the date is calculated relatively on today

facetQueries - list of facet queries

sort - the sort field

ascending - the sort direction (true: ascending)

facetMinCount - Minimum count of results facet must have to return a result

Throws:

org.apache.solr.client.solrj.SolrServerException - Exception from the Solr server to the solrj Java client.

IOException - passed through.

query

public org.apache.solr.client.solrj.response.QueryResponse query(String query,
 String filterQuery,
 String facetField,
 int rows,
 int max,
 String dateType,
 String dateStart,
 String dateEnd,
 List<String> facetQueries,
 String sort,
 boolean ascending,
 int facetMinCount,
 boolean defaultFilterQueries)
                                                          throws org.apache.solr.client.solrj.SolrServerException,
IOException

Description copied from interface: SolrLoggerService

Perform a solr query.

Specified by:

query in interface SolrLoggerService

Parameters:

query - the query to be used

filterQuery - filter query

facetField - field to facet the results by

rows - the max number of results to return

max - the max number of facets to return

dateType - the type to be used (example: DAY, MONTH, YEAR)

dateStart - the start date Format:(-3, -2, ..) the date is calculated relatively on today

dateEnd - the end date stop Format (-2, +1, ..) the date is calculated relatively on today

facetQueries - list of facet queries

sort - the sort field

ascending - the sort direction (true: ascending)

facetMinCount - Minimum count of results facet must have to return a result

defaultFilterQueries - use the default filter queries

Throws:

org.apache.solr.client.solrj.SolrServerException - Exception from the Solr server to the solrj Java client.

IOException - passed through.

shardSolrIndex

public void shardSolrIndex()
                    throws IOException,
org.apache.solr.client.solrj.SolrServerException

Specified by:

shardSolrIndex in interface SolrLoggerService

Throws:

IOException

org.apache.solr.client.solrj.SolrServerException

createCore

protected org.apache.solr.client.solrj.impl.HttpSolrClient createCore(org.apache.solr.client.solrj.impl.HttpSolrClient solr,
 String coreName)
                                                               throws IOException,
org.apache.solr.client.solrj.SolrServerException

Throws:

IOException

org.apache.solr.client.solrj.SolrServerException

getMultivaluedFieldNames

public Set<String> getMultivaluedFieldNames()
                                     throws org.apache.solr.client.solrj.SolrServerException,
IOException

Retrieves a list of all the multi valued fields in the solr core.

Returns:

all fields tagged as multivalued

Throws:

org.apache.solr.client.solrj.SolrServerException - When getting the schema information from the SOLR core fails

IOException - When connection to the SOLR server fails

reindexBitstreamHits

public void reindexBitstreamHits(boolean removeDeletedBitstreams)
                          throws Exception

Specified by:

reindexBitstreamHits in interface SolrLoggerService

Throws:

Exception

exportHits

public void exportHits()
                throws Exception

Description copied from interface: SolrLoggerService

Export all SOLR usage statistics for viewing/downloading content to a flat text file. The file goes to a series

Specified by:

exportHits in interface SolrLoggerService

Throws:

Exception - if error

commit

public void commit()
            throws IOException,
org.apache.solr.client.solrj.SolrServerException

Description copied from interface: SolrLoggerService

Commit the solr core.

Specified by:

commit in interface SolrLoggerService

Throws:

IOException

org.apache.solr.client.solrj.SolrServerException

addDocumentsToFile

protected void addDocumentsToFile(Context context,
 org.apache.solr.common.SolrDocumentList docs,
 File exportOutput)
                           throws SQLException,
DateTimeParseException,
IOException

Throws:

SQLException

DateTimeParseException

IOException

generateURL

protected String generateURL(String baseURL,
 Map<String,String> parameters)
                      throws UnsupportedEncodingException

Throws:

UnsupportedEncodingException

addAdditionalSolrYearCores

protected void addAdditionalSolrYearCores(org.apache.solr.client.solrj.SolrQuery solrQuery)

initSolrYearCores

protected void initSolrYearCores()

anonymizeIp

public Object anonymizeIp(String ip)
                   throws UnknownHostException

Description copied from interface: SolrLoggerService

Anonymize a given ip

Specified by:

anonymizeIp in interface SolrLoggerService

Parameters:

ip - The ip to anonymize.

Throws:

UnknownHostException