org.allenai.datastore

Datastore

Related Docs: object Datastore | package datastore

class Datastore extends Logging

Represents a datastore

This is a thin layer over an S3 bucket that stores the data. Data is identified by group ("org.allenai.something"), name ("WordNet"), and version (an integer). It supports files as well as directories.

Items are published to the datastore, and then referred to with the *path() methods. All data is cached, so access to all items should be very fast, except for the first time.

It might make more sense to get Datastore objects from the companion object, rather than creating them here.

Linear Supertypes
Logging, AnyRef, Any
Known Subclasses
Datastore, PrivateDatastore
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. Datastore
  2. Logging
  3. AnyRef
  4. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Instance Constructors

  1. new Datastore(name: String, s3: AmazonS3Client)

    name

    name of the datastore. Corresponds to the name of the bucket in S3. Currently we have "public" and "private".

    s3

    properly authenticated S3 client.

Type Members

  1. class AlreadyExistsException extends Exception

    Exception indicating that we tried to upload an item to the datastore that already exists.

    Exception indicating that we tried to upload an item to the datastore that already exists.

    Data in the datastore is (mostly) immutable. Replacing an item is possible, but you have to set a flag. If you don't set the flag, and you're replacing something, this exception gets thrown.

  2. class DoesNotExistException extends Exception

    Exception indicating that we tried to access an item in the datastore that wasn't there.

  3. case class Locator(group: String, name: String, version: Int, directory: Boolean) extends Product with Serializable

    Identifies a single version of a file or directory in the datastore

Value Members

  1. final def !=(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any

  2. final def ##(): Int

    Definition Classes
    AnyRef → Any

  3. final def ==(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any

  4. object Locator extends Serializable

  5. final def asInstanceOf[T0]: T0

    Definition Classes
    Any

  6. def bucketName: String

    Returns the name of the bucket backing this datastore

  7. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  8. def createBucketIfNotExists(): Unit

    Creates the bucket backing this datastore if necessary

    Creates the bucket backing this datastore if necessary

    You only need to call this if you're setting up a new datastore.

  9. def directoryExists(group: String, name: String, version: Int): Boolean

    Checks whether a directory exists in the datastore

    Checks whether a directory exists in the datastore

    group

    group of the directory in the datastore

    name

    name of the directory in the datastore

    version

    version of the directory in the datastore

    returns

    true if the directory exists, false otherwise

  10. def directoryPath(group: String, name: String, version: Int): Path

    Gets a local path for a directory in the datastore

    Gets a local path for a directory in the datastore

    Downloads the directory from S3 if necessary

    group

    the group of the directory

    name

    the name of the directory

    version

    the version of the directory

    returns

    path to the directory on the local filesystem

  11. def directoryUrl(group: String, name: String, version: Int): URL

    Gets a URL for a directory in the datastore

    Gets a URL for a directory in the datastore

    group

    group of the directory

    name

    name of the directory

    version

    version of the directory

    returns

    URL pointing to the directory. This URL will always point to a zip file containing the directory's contents.

  12. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  13. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any

  14. def exists(locator: Locator): Boolean

    Checks whether an item exists in the datastore

    Checks whether an item exists in the datastore

    locator

    locator of the item in the datastore

    returns

    true if the item exists, false otherwise

  15. def fileExists(group: String, name: String, version: Int): Boolean

    Checks whether a file exists in the datastore

    Checks whether a file exists in the datastore

    group

    group of the file in the datastore

    name

    name of the file in the datastore

    version

    version of the file in the datastore

    returns

    true if the file exists, false otherwise

  16. def filePath(group: String, name: String, version: Int): Path

    Gets a local path for a file in the datastore

    Gets a local path for a file in the datastore

    Downloads the file from S3 if necessary

    group

    the group of the file

    name

    the name of the file

    version

    the version of the file

    returns

    path to the file on the local filesystem

  17. def fileUrl(group: String, name: String, version: Int): URL

    Gets a URL for a file in the datastore

    Gets a URL for a file in the datastore

    group

    group of the file

    name

    name of the file

    version

    version of the file

    returns

    URL pointing to the file

  18. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )

  19. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any

  20. def hashCode(): Int

    Definition Classes
    AnyRef → Any

  21. val internalLogger: Logger

    Definition Classes
    Logging

  22. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any

  23. def listGroupContents(group: String): Set[Locator]

    Lists all items in a group

    Lists all items in a group

    group

    group to search over

    returns

    a set of locators, one for each item in the group. Multiple versions are multiple locators.

  24. def listGroups: Set[String]

    Lists all groups in the datastore

    Lists all groups in the datastore

    returns

    a set of all groups in the datastore

  25. object logger

    Definition Classes
    Logging

  26. val name: String

    name of the datastore.

    name of the datastore. Corresponds to the name of the bucket in S3. Currently we have "public" and "private".

  27. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  28. final def notify(): Unit

    Definition Classes
    AnyRef

  29. final def notifyAll(): Unit

    Definition Classes
    AnyRef

  30. def path(locator: Locator): Path

    Gets a local path for an item in the datastore

    Gets a local path for an item in the datastore

    Downloads the item from S3 if necessary

    locator

    locator for the item in the datastore

    returns

    path to the item on the local filesystem

  31. def publish(path: Path, locator: Locator, overwrite: Boolean): Unit

    Publishes an item to the datastore

    Publishes an item to the datastore

    path

    path to the item to be published

    locator

    locator to publish the item under

    overwrite

    if true, overwrites possible existing items in the datastore

  32. def publishDirectory(path: Path, group: String, name: String, version: Int, overwrite: Boolean): Unit

    Publishes a directory to the datastore

    Publishes a directory to the datastore

    path

    path to the directory to be published

    group

    group to publish the directory under

    name

    name to publish the directory under

    version

    version to publish the directory under

    overwrite

    if true, overwrites possible existing items in the datastore

  33. def publishDirectory(path: String, group: String, name: String, version: Int, overwrite: Boolean): Unit

    Publishes a directory to the datastore

    Publishes a directory to the datastore

    path

    name of the directory to be published

    group

    group to publish the directory under

    name

    name to publish the directory under

    version

    version to publish the directory under

    overwrite

    if true, overwrites possible existing items in the datastore

  34. def publishFile(file: Path, group: String, name: String, version: Int, overwrite: Boolean): Unit

    Publishes a file to the datastore

    Publishes a file to the datastore

    file

    path to the file to be published

    group

    group to publish the file under

    name

    name to publish the file under

    version

    version to publish the file under

    overwrite

    if true, overwrites possible existing items in the datastore

  35. def publishFile(file: String, group: String, name: String, version: Int, overwrite: Boolean): Unit

    Publishes a file to the datastore

    Publishes a file to the datastore

    file

    name of the file to be published

    group

    group to publish the file under

    name

    name to publish the file under

    version

    version to publish the file under

    overwrite

    if true, overwrites possible existing items in the datastore

  36. val s3: AmazonS3Client

    properly authenticated S3 client.

  37. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef

  38. def toString(): String

    Definition Classes
    AnyRef → Any

  39. def url(locator: Locator): URL

    Gets the URL for an item in the datastore

    Gets the URL for an item in the datastore

    locator

    locator of the item

    returns

    URL pointing to the locator

  40. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  41. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  42. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  43. def wipeCache(): Unit

    Wipes the cache for this datastore

Inherited from Logging

Inherited from AnyRef

Inherited from Any

Ungrouped