com.googlecode.fascinator.harvester.csv

Class CSVHarvester

java.lang.Object

com.googlecode.fascinator.common.harvester.impl.GenericHarvester

com.googlecode.fascinator.harvester.csv.CSVHarvester

All Implemented Interfaces:

com.googlecode.fascinator.api.harvester.Harvester, com.googlecode.fascinator.api.Plugin

public class CSVHarvester
extends com.googlecode.fascinator.common.harvester.impl.GenericHarvester

Harvester for CSV files.

Configuration options:

• fileLocation: The location of the csv file (required)
• idColumn: the column holding the primary key. If not provided, the row number will be used.
• recordIDPrefix: Adds a prefix to the value found in the ID column. For example, setting this as "http://id.example.com/" with an ID value of "453" will result in http://id.example.com/453 as the ID.
• delimiter: The csv delimiter. Comma (,) is the default (optional)
• ignoredFields: An array of fields (columns) ignored by the harvest.
• includedFields: An array of fields (columns) included by the harvest.
• multiValueFields: An array of fields (columns) that contain several values (optional)
• multiValueFieldDelimiter: The delimiter for multi-value fields. Semi-colon (;) is the default (optional)
• payloadId: The payload identifier used to store the JSON data (defaults to "metadata.json")
• batchSize: The number of rows in the CSV file to process, before being harvested (defaults to 50)
• maxRows: The number of rows to process where -1 means harvest all (defaults to -1), note that if filters are in place the maxRows applies to all rows not just the ones that pass the filter
• filters: If a filter is provided it must match for a row to be imported. The filters are defined as an array of a map of the following
  • field: Name of the field (column), note that this field can be one that's being ignored
  • multi: Only applies to multi-value fields, can have values of ANY or ALL, ANY is the default (optional)
  • regex: The case sensitive regex that must match part of the value in the given column. To do an entire match surround with ^ and $

You can select to harvest all columns (fields) or be selective:

• If you don't set ignoredFields or includedFields: all fields are harvested.
• ignoredFields has precedence over includedFields: If you have the same field in both lists, it will get ignored.
• If you only provide fields in ignoredFields (and leave includedFields blank), all other fields will be harvested
• If you only provide fields in includedFields (and leave ignoredFields blank), only these fields will be harvested

Fields with repeated names result in only the first value being stored.

Based on Greg Pendlebury's CallistaHarvester.

Author:

Duncan Dickinson

2011 - 2012 - QCIF undertakes maintenance work.

2012 - innodev.com.au addition for filtering

, Greg Pendlebury, Duncan Dickinson

Constructor Summary

Constructors

Constructor and Description
CSVHarvester() Constructs the CSV harvester plugin.

Method Summary

Methods

Modifier and Type	Method and Description
Set<String>	getObjectIdList() Harvest the next batch of rows and return their object IDs.
boolean	hasMoreObjects() Check if there are more objects to harvest.
void	init() Initialise the CSV harvester plugin.
void	shutdown() Shutdown the plugin.

Methods inherited from class com.googlecode.fascinator.common.harvester.impl.GenericHarvester

getDeletedObjectIdList, getId, getJsonConfig, getName, getObjectId, getPluginDetails, getStorage, hasMoreDeletedObjects, init, init, setStorage

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

CSVHarvester

public CSVHarvester()

Constructs the CSV harvester plugin.

Method Detail

init

public void init()
          throws com.googlecode.fascinator.api.harvester.HarvesterException

Initialise the CSV harvester plugin.

Specified by:

init in class com.googlecode.fascinator.common.harvester.impl.GenericHarvester

Throws:

com.googlecode.fascinator.api.harvester.HarvesterException - if an error occurred

shutdown

public void shutdown()
              throws com.googlecode.fascinator.api.harvester.HarvesterException

Shutdown the plugin.

Specified by:

shutdown in interface com.googlecode.fascinator.api.Plugin

Overrides:

shutdown in class com.googlecode.fascinator.common.harvester.impl.GenericHarvester

Throws:

com.googlecode.fascinator.api.harvester.HarvesterException - if an error occurred

hasMoreObjects

public boolean hasMoreObjects()

Check if there are more objects to harvest.

Returns:

true if there are more, false otherwise

getObjectIdList

public Set<String> getObjectIdList()
                            throws com.googlecode.fascinator.api.harvester.HarvesterException

Harvest the next batch of rows and return their object IDs.

Returns:

the set of object IDs just harvested

Throws:

com.googlecode.fascinator.api.harvester.HarvesterException - if an error occurred