com.googlecode.fascinator.harvester.csv
Class CSVHarvester

java.lang.Object
  com.googlecode.fascinator.common.harvester.impl.GenericHarvester
      com.googlecode.fascinator.harvester.csv.CSVHarvester

All Implemented Interfaces:

com.googlecode.fascinator.api.harvester.Harvester, com.googlecode.fascinator.api.Plugin

public class CSVHarvester
extends com.googlecode.fascinator.common.harvester.impl.GenericHarvester

Harvester for CSV files.

Configuration options:

• fileLocation: The location of the csv file (required)
• idColumn: the column holding the primary key. If not provided, the row number will be used.
• recordIDPrefix: Adds a prefix to the value found in the ID column. For example, setting this as "http://id.example.com/" with an ID value of "453" will result in http://id.example.com/453 as the ID.
• delimiter: The csv delimiter. Comma (,) is the default
• ignoredFields: An array of fields (columns) ignored by the harvest.
• includedFields: An array of fields (columns) included by the harvest
• payloadId: The payload identifier used to store the JSON data (defaults to "metadata.json")
• batchSize: The number of rows in the CSV file to process, before being harvested (defaults to 50)
• maxRows: The number of rows to process where -1 means harvest all (defaults to -1)

You can select to harvest all columns (fields) or be selective:

• If you don't set ignoredFields or includedFields: all fields are harvested.
• ignoredFields has precedence over includedFields: If you have the same field in both lists, it will get ignored.
• If you only provide fields in ignoredFields (and leave includedFields blank), all other fields will be harvested
• If you only provide fields in includedFields (and leave ignoredFields blank), only these fields will be harvested

Fields with repeated names result in only the first value being stored.

Based on Greg Pendlebury's CallistaHarvester.

Author:

Duncan Dickinson

2011 - QCIF undertakes maintenance work.

, Greg Pendlebury

Constructor Summary

CSVHarvester()
Constructs the CSV harvester plugin.

Method Summary

Set<String>	getObjectIdList() Harvest the next batch of rows and return their object IDs.
boolean	hasMoreObjects() Check if there are more objects to harvest.
void	init() Initialise the CSV harvester plugin.
void	shutdown() Shutdown the plugin.

Methods inherited from class com.googlecode.fascinator.common.harvester.impl.GenericHarvester

getDeletedObjectIdList, getId, getJsonConfig, getName, getObjectId, getPluginDetails, getStorage, hasMoreDeletedObjects, init, init, setStorage

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

CSVHarvester

public CSVHarvester()

Constructs the CSV harvester plugin.

Method Detail

init

public void init()
          throws com.googlecode.fascinator.api.harvester.HarvesterException

Initialise the CSV harvester plugin.

Specified by:

init in class com.googlecode.fascinator.common.harvester.impl.GenericHarvester

Throws:

com.googlecode.fascinator.api.harvester.HarvesterException - if an error occurred

shutdown

public void shutdown()
              throws com.googlecode.fascinator.api.harvester.HarvesterException

Shutdown the plugin.

Specified by:

shutdown in interface com.googlecode.fascinator.api.Plugin

Overrides:

shutdown in class com.googlecode.fascinator.common.harvester.impl.GenericHarvester

Throws:

com.googlecode.fascinator.api.harvester.HarvesterException - if an error occurred

hasMoreObjects

public boolean hasMoreObjects()

Check if there are more objects to harvest.

Returns:

true if there are more, false otherwise

getObjectIdList

public Set<String> getObjectIdList()
                            throws com.googlecode.fascinator.api.harvester.HarvesterException

Harvest the next batch of rows and return their object IDs.

Returns:

the set of object IDs just harvested

Throws:

com.googlecode.fascinator.api.harvester.HarvesterException - if an error occurred