Resource Index Guide

2. What's New

With release 2.2, Fedora's Resource Index has improved in several significant ways:

In addition to Kowari, the RI can now be backed by MPTStore. MPTStore is a fast, highly scalable RDBMS-based triplestore, useful for applications that need fast, synchronous writes and the ability to work with the triplestore using familiar RDBMS utilities. Note, however, that MPTStore does not yet support a rich, native RDF query language like SPARQL or Kowari's iTQL. See the MPTStore Datastore Configuration section for more information on configuring MPTStore for use with the Resource Index.
The Resource Index module has been refactored for efficiency and stability. As noted in the release notes, several RI-related bugs have been eliminated for this release. In addition, the strategy for sending triple updates to the underlying triplestore has been streamlined to be much more efficient. This has resulted in a significant performance improvement over Fedora 2.1.1's Resource Index.
Automatic, synchronous writes are now supported. Using the new syncUpdates module parameter, Fedora can be configured to provide the guarantee that the triples associated with a given API-M operation have been flushed before the API-M call returns.

See the Fedora 2.2 API-M Performance Report for more details on how these changes and configuration options affect performance.

The Fedora object model can be abstractly viewed as a directed graph, consisting of internal arcs that relate digital object nodes to their dissemination nodes and external arcs between digital objects. The Resource Index is a Fedora service that allows storage and query of this graph. The Resource Index is automatically updated whenever an object is added or modified.

The Resource Index builds on the RDF primitives build within the semantic web community. Fedora supplies a base relationship ontology [RELS-EXT](defining a core set of internal and external relationships) that can co-exist with domain-specific relationship ontologies from other namespaces. Each digital object's external relationships to other digital objects are expressing in RDF/XML within a reserved datastream in the respective object. A relationship graph over the digital objects in the repository can then be derived by merging the internal relationships implied by the Fedora object model with the external relationships explicitly stated in their relationship datastreams.

A. System Relationships & Properties

The Fedora base ontology describes such relations and properties as the behavior definition implemented, behavior mechanism used, creation date, state, and mime-type.

In the figure below, the graph (abbreviated for clarity) represents three objects in the repository. demo:SmileyStuff uses the behavior mechanism demo:dualResImageCollection, which in turn implements the behavior definition demo:Collection.

Dublin Core statements, as shown in the figure below, are automatically extracted from an object's DC datastream and inserted into the Resource Index. datastream, as shown in the figure below:

B. User-defined Relationships

The Resource Index will automatically index object-to-object relationships defined in the RELS-EXT datastream. Please consult Fedora Metadata for Object-to-Object Relationships for more information.

4. Configuration

Please note that many configuration changes require a full rebuild of the Resource Index to ensure consistency. For example, turning the Resource Index Module off and on again will result in an inconsistent state, as the Resource Index will know nothing about the digital objects created or modified while the module was not loaded. Similarly, enabling full-text indexing [Full-Text] after the repository has already been populated will only add new objects to the full-text model. In general, the only safe configuration changes to make on a running repository are limited to the performance-related pool, buffer and flush parameters. In all cases, configuration changes require a restart of the Fedora server before taking effect.

The Resource Index is configured within two sections of fedora.fcfg, module and datastore.

A. Module Configuration

The Resource Index module is configured with fedora.fcfg

Here's an example of a Resource Index module configuration that uses Kowari with delayed updates:

<module role="fedora.server.resourceIndex.ResourceIndex" 
        class="fedora.server.resourceIndex.ResourceIndexModule">
  <param name="level" value="2"/>
  <param name="datastore" value="localKowariTriplestore"/>
  <param name="syncUpdates" value="false"/>
</module>

Here's another example, this time using MPTStore with immediate updates:

<module role="fedora.server.resourceIndex.ResourceIndex" 
        class="fedora.server.resourceIndex.ResourceIndexModule">
  <param name="level" value="2"/>
  <param name="datastore" value="localPostgresMPTTriplestore"/>
  <param name="syncUpdates" value="true"/>
</module>

An explanation of the parameters and their possible values:

level
Sets the operating level of the Resource Index.
- 0
  Off: the Resource Index will not load at server startup.
- 1
  On: the Resource Index will index system properties, inter & intra-object relationships, and user-defined relationships.
- 2
  Same as 1, but adds indexing of parameterized methods, where the parameters have a finite domain (e.g., the method getImage with parameter size, whose domain is {small, large}). Because calculating method parameters may result in a combinatorial explosion of statements in the Resource Index (depending on the design of a particular repository's Behavior Definition Objects), this level of indexing must be explicitly set.
datastore
The id of the datastore to use with the Resource Index. The referenced datastore must assert a connectorClassName parameter with a valid Trippi Connector class.
syncUpdates
Whether to flush the triple buffer before returning from object modification operations. This defaults to false. Specifying this as true will ensure that RI queries immediately reflect the latest triples. Specifying false will not provide this guarantee, but can reduce roundtrip time for API-M operations (especially when using Kowari).

B. Kowari Datastore Configuration

The example datastore configuration below (with the path parameter modified for the installation environment) would provide a local Kowari triplestore that buffers up to 20,000 triples in memory at a time or waits for 5 seconds of buffer inactivity before flushing them to disk. Because writing triples to disk is a relatively expensive operation, the buffer takes advantage of Kowari's bulk update handler to ingest a mass of triples at a time. The performance gain is significant during a bulk ingest of objects. The size or inactivity interval of the buffer may be adjusted according to performance needs and physical memory capacity.

<datastore id="localKowariTriplestore">
  <param name="connectorClassName" value="org.trippi.impl.kowari.KowariConnector"/>
  <param name="remote" value="false"/>
  <param name="path" value="/opt/fedora/store/resourceIndex"/>
  <param name="serverName" value="fedora"/>
  <param name="modelName" value="ri"/>
  <param name="poolInitialSize" value="3"/>
  <param name="poolMaxGrowth" value="-1"/>
  <param name="readOnly" value="false"/>
  <param name="autoCreate" value="true"/>
  <param name="autoTextIndex" value="false"/>
  <param name="memoryBuffer" value="true"/>
  <param name="autoFlushDormantSeconds" value="5"/>
  <param name="autoFlushBufferSize" value="20000"/>
  <param name="bufferSafeCapacity" value="40000"/>
  <param name="bufferFlushBatchSize" value="20000"/>
</datastore>

An explanation of the parameters and their possible values follows. Certain parameters require other parameters, as indicated in the hierarchy below. Optional parameters are also indicated below. As noted previously, many of these parameters, with the exception of the pool, buffer, and flush parameters, cannot be changed on a running repository without a full rebuild of the Resource Index.

connectorClassName
The name of the Trippi Connector class used to communicate with the triplestore. When using Kowari, the value should be as follows:
- org.trippi.impl.kowari.KowariConnector
remote
Tells the connector to communicate with Kowari in remote or local mode.
- true
  If remote is true, the host parameter must be specified.
  - host
    The hostname where Kowari is running.
  - port
    The rmi port Kowari is running on (default is 1099).
- false
  If remote is false, path parameter must be specified.
  - path
    The local path to the main triplestore directory.
serverName
The server name for rmi binding.
modelName
The name of the model to use.
poolInitialSize
The initial size of the session pool used for queries. Note: A value of 0 will cause the Resource Index to operate in synchronized mode: concurrent read/write requests are put in a queue and handled in FIFO order; this will severely impair performance and is only intended for debugging.
poolMaxGrowth
Maximum number of additional sessions the pool may add. If specified as -1, no limit will be placed on pool growth.
poolSpareSessions
The number of spare sessions to proactively make available. If unspecified, this defaults to 0, which means that additional sessions will only be created as needed. Note that if poolMaxGrowth is 0, the value of this parameter is inconsequential.
readOnly
Whether the triplestore should be read-only. Most Fedora repositories will set this to false.
- true
  No additional parameters need to be set if readOnly is true.
- false
  The following parameters must also be set if readOnly is false.
  - autocreate
    Whether to create the model if it doesn't already exist. At startup, the model will be automatically created. In addition, an XML schema datatyped model named "xsd" will also be automatically created.
    - true
    - false
  - autoTextIndex
    Whether to propagate adds/deletes to a full-text [Full-Text] model automatically. While a very useful feature, enabling full-text indexing adds significantly to object ingest times. If true, the text model will be named modelName-fullText. Note that if this is true and autoCreate is true, the text model will also be created if it doesn't already exist.
    - true
    - false
  - memoryBuffer
    Whether to use a memory buffer or a database buffer for write operations. The buffer is where triples are stored before they're actually written. Normally, a memory buffer will be sufficient. However, in certain server environments, a database buffer is the better option, offering higher capacity and persistence.
    - true
    - false
      - dbDriver
        The JDBC driver class name. This must be in the classpath.
        
        com.mysql.jdbc.Driver
        
        com.mckoi.JDBCDriver
        
        oracle.jdbc.driver.OracleDriver
      - dbURL
        The JDBC URL for the database. Examples: For MySQL, jdbc:mysql://localhost/mydb would use the local database named mydb. For McKoi, jdbc:mckoi://localhost:9157/ would use the local database at port 9157. For oracle, jdbc:oracle:thin:@localhost:1521:mydb would use the thin driver to connect to the local database named mydb at port 1521.
      - dbUsername
        The database username.
      - dbPassword
        The password for the db user.
      - dbTableName
        The table for buffered triple updates. This table must already exist with the following five columns: num (a large numeric type), action (char(1)), subject, predicate, and object (all large varchar or text types).
  - autoFlushDormantSeconds
    Seconds of buffer inactivity that will trigger an auto-flush. If this threshold is reached, flushing will occur in the background, during which time the buffer is still available for writing.
  - autoFlushBufferSize
    The size at which the buffer should be auto-flushed. If this threshold is reached, flushing will occur in the background, during which time the buffer is still available for writing.
  - bufferSafeCapacity
    The maximum size the buffer can reach before being forcibly flushed. If this threshold is reached, flushing will occur in the foreground and the buffer will be locked for writing until it is finished. This should be larger than autoFlushBufferSize.
  - bufferFlushBatchSize
    The number of updates to send to the triplestore at a time. This should be the same size as, or smaller than autoFlushBufferSize.

C. MPTStore Datastore Configuration

The example datastore configuration below would provide a local MPTStore triplestore backed by Postgres.



<datastore id="localPostgresMPTTriplestore">
  <comment>
    Example local MPTStore backed by Postgres.
    To use this triplestore for the Resource Index:
    1) In fedora.fcfg, change the "datastore" parameter of the
       ResourceIndex module to localPostgresMPTTriplestore.
    2) Login to your Postgres server as an administrative user and
       run the following commands:
         CREATE ROLE "fedoraAdmin" LOGIN PASSWORD 'fedoraAdmin'
           NOINHERIT CREATEDB
           VALID UNTIL 'infinity';
         CREATE DATABASE "riTriples"
           WITH ENCODING='SQL_ASCII'
           OWNER="fedoraAdmin";
    3) Make sure you can login to your Postgres server as fedoraAdmin.
    4) Download the appropriate Postgres JDBC 3 driver from 
       http://jdbc.postgresql.org/download.html
       and make sure it's accessible to your servlet container.
       If you're running Tomcat, putting it in common/lib/ will work.
  </comment>
  <param name="connectorClassName" value="org.trippi.impl.mpt.MPTConnector"/>
  <param name="ddlGenerator" value="org.nsdl.mptstore.impl.postgres.PostgresDDLGenerator"/>
  <param name="jdbcDriver" value="org.postgresql.Driver"/>
  <param name="jdbcURL" value="jdbc:postgresql://localhost/riTriples"/>
  <param name="username" value="fedoraAdmin"/>
  <param name="password" value="fedoraAdmin"/>
  <param name="poolInitialSize" value="3"/>
  <param name="poolMaxSize" value="10"/>
  <param name="backslashIsEscape" value="true"/>
  <param name="fetchSize" value="1000"/>
  <param name="autoFlushDormantSeconds" value="5"/>
  <param name="autoFlushBufferSize" value="1000"/>
  <param name="bufferFlushBatchSize" value="1000"/>
  <param name="bufferSafeCapacity" value="2000"/>
</datastore>

5. Search Interface

The Resource Index Search interface is exposed in a REST architectural style, providing a stateless query interface that accepts queries by value or by reference [Fedora Relationships].

The query interface to the Resource Index currently supports three RDF query languages, RDQL (Kowari-only), iTQL (Kowari-only), and SPO (Kowari and MPTStore). NOTE: As of Fedora 2.1, RDQL support is now deprecated and may be entirely absent in future releases. Support for SPARQL is planned for a future release.

Please consult the Resource Index Search documentation for more information.

5. Resource Index Demo

Demonstration objects that utilize the Resource Index are included in the Fedora distribution. Please see the Demo documentation for more information.

7. References

[Fedora Relationships]: Fedora: An Architecture for Complex Objects and their Relationships, Carl Lagoze, Sandy Payette, Edwin Shin, and Chris Wilper, 2004.
[Full-Text]: Kowari Full-Text Models
[RELS-EXT]: Fedora Relationship Ontology
[risearch]: Fedora Resource Index Search Service

7. Fedora Guide To Triples (and estimator for size of Resource Index)

To understand what triples are indexed in the Fedora Resource Index, and to estimate the size of your Kowari triplestore consult document named Triples in the Resource Index.