/
Configuring Pipeline Indexing Options

Configuring Pipeline Indexing Options

You can configure many general options that will apply to any Repository using a specific pipeline. These include Indexing, Thumbnail and Geometry settings as shown below:

Indexing

These options control the types of information included with each record that is indexed.

 

Check the boxes to select the types of information to index:

  • Index Referenced Data

In some cases, items in your index can refer to additional data stored in a separate location. This additional data is Referenced Data, and includes all of the items that your indexed data refers to or draws information from. 

  • Index Access Control Information

This setting enables the ability to add access control list (ACL) information from a file system directly to the index for every record where available. The ACL information is stored in "Acl Allow" and "Acl Deny" fields for indexed content items that this setting is applied to.

  • Index Extended File Attributes

This will extract extended file attributes from data records. Extended file attributes can include the author of a document, the character encoding of a plain-text document, a checksum, cryptographic hash or digital certificate.

  • Generate Indexing Sample

Analyzes the route of jobs as they are indexed and generates speed and performance metrics by sampling a document through the various stages of the indexing process.

NOTE: The results are stored in repos.json as the ‘sample’ property of a repository.

  • Index Debug Settings

Turn on this setting to pull more diagnostics information regarding the indexing process and where it will be stored in the index. This helps to diagnose potential indexing errors and also provides improvements to the overall performance of your instance.

  • Developer Mode

    • Index Job Info: enables the ability to access, store and query index job metadata for diagnostics and performance insights

    • Index Route Info: enables the ability to access, store and query index route data for diagnostics and debugging

    • Job Metrics Settings

      • Compute and Index Stage Metrics: generates aggregated metrics during the indexing process. When this option is checked, the ‘start’, ‘finished’ and ‘elapsed’ time for various indexing stages are indexed as part of the document. In particular, the generated fields in the indexed content item are: scan_full, post_scan, index, pre_index.

      • Index Pipeline Metrics: enables the ability to access, store and query the performance metrics of the indexing job

      • Index Start Date And Time: enables the ability to access, store and query the start date and time of the indexing job

      • Index Elapsed Time In Seconds: enables the ability to access, store and query the duration of the indexing job from start to finish (in seconds)

      • Index End Date And Time: enables the ability to access, store and query the end date and time of the indexing job

Thumbnails

These options control whether or not to create Thumbnails and when they should be generated. There are three options for creating Thumbnails:

 

  • Choose Default Image  to create Thumbnails using default settings for the type of content being indexed, depending on the type of content being indexed

  • Choose Do Not Build Images to skip creating Thumbnails (can be useful if the Repository is very large and creating Thumbnails would both slow down indexing as well as taking up large amounts of storage space)

  • Select Use Basemap to choose and configure a Basemap 

Geometry

This option controls how much detail is included in the geometry of a record.

 

Choose how much to generalize the Geometry of a feature by using the slider. The generalization value ranges from 0 to 1.

  • A value of 0 specifies no generalization and results in the original geometry with full detail (can be very noisy)

  • A value of 1 specifies full generalization and results smoothing of the bounding box with less detail (noise)

  • Values in between are scaled into that spectrum moving from gradually less geometric detail as values move from 0 to 1

NOTE: Making the geometry more generalized for a indexed feature may leave out desired geographic places of interest around the edges of the geometry on the map.