Data Insight Indexer checks

Table: Indexer checks lists various Data Insight Indexer checks that you must perform and the related remedial steps:

Table: Indexer checks

Checks

Steps

Number of filers and the shares that are serviced by each Indexer

  • Navigate to Settings > Filers, and filter on each Indexer.

Or

  • Check the idxcheck log for each Indexer.

Index integrity and space usage per Indexer

  • Check Idxcheck0.log under installdir/log folder checks for database integrity and index.

  • Go to the end of the log file to review the overall integrity check.

  • If the integrity check fails, review the index check for each share in the same file.

  • Review two key metrics: IndexDB Size and Segments Size. Others metrics indicate stale hash sizes.

    IndexDB Size is a sum of all index databases. It stores index information such as scan information (metadata and ACLs).

    Segment size is a sum of all segment sizes. It stores the audit events.

    Hashes are indexed against the two databases - indexDB and segments.

    ActIndex Size was introduced in Data Insight 4.0 to indicate activity in a social network map.

Inbox and Outbox chart

Figure: Inbox and Outbox chart illustrates the chart

  • Navigate to Settings > Servers> Statistics.

  • Ensure that the Inbox plot follows a smooth saw-tooth pattern.

  • If the number of files in inbox consistently rises, and so does CPU and memory usage, investigate if there is an Indexer issue.

  • Check schedule settings and number of threads to fine-tune the processing of incoming files.

Basic Settings

In Data Insight UI, navigate to Data Insight > Indexer > Advanced Settings, and fine-tune the following parameters as necessary:

  • Total Indexer threads (*): This number should not exceed the total CPU core count on the server. For a 4-CPU Indexer, ideally set this value to 'total cpu core count minus 1'. Allowing one core's worth of processor ensures that index writer processes cannot overcome a server and keeps resources available for query response and product communications. If the Indexer has more than 4 cores, leaving more cores available may be desirable if many reports are run. Disk subsystems start to lose the ability to keep up if there are more than 10 threads. If the server has more than 12 cores, do not set value to more than 10 on index writer. Specify a higher value only if the storage is very high performance.

  • Limit the maximum events that are processed in memory: Keeping this setting unlimited may lead to processing issues. Most of these issues should be resolved, but if the filer in question has excessively large number of events. (Application-generated workloads sometimes cause this type of load.) A limit of 1,000,000 is recommended.

  • Reconfirm deleted paths when reconciling full scan information: This Indexer-side setting enables Data Insight to reconfirm if paths now missing from full scan output are deleted.

  • Indexer schedule

  • Indexer integrity checking schedule

Audit Events Pre-Processor Settings

In the Data Insight UI, navigate to Data Insight Indexer>Advanced Settings

  • Audit events preprocessor schedule

  • Batch size

Accumulation of backlog in DATADIR/inbox

  • Look for the date of oldest files (backlog): If files are very old, it means that audit or scan file processing is not able to keep up with collection.

  • Execute the configcli list_jobs command to gather ADScanJob, ProcessEventsJob, and IndexWriterJob.

  • Collect Event Viewer logs. Collect the output of the indexcli -j command.

Audit/scan files in DATADIR/indexer/err

  • Review ldxcheck.log for index corruption.

  • Move files to Inbox and verify if they are consumed in the next run. If not, contact Symantec Support with the following information:

    • Event viewer logs

    • Index directory for the share or the site collection

    • installdir\log\commd.0.0.log

    • installdir\log\indexer\index-msu_id.log

    • Idxcheck.log (summary and integrity information for all shares)

    • datadir\indexer\err folder

    • Output of indexcli.exe -j command (To view the files queued for indexing.)

The following figure illustrates a smooth saw-tooth pattern in an Inbox and Outbox chart:

Figure: Inbox and Outbox chart

Inbox and Outbox chart