By default, Spotlight excludes many hidden, System, and developer directories from the metadata index for simplicity and ease of use by the majority of users. These exclusions include things like hidden UNIX directories and ADC Reference Documentation. It would be very helpful for some users to have some of these excluded areas available for Spotlight searching. While mdimport -f will allow you to manually force Spotlight to add a directory to its index, many of Spotlight's command-line tools make references to filter rules. I wondered where these rules might be kept, and how they might be altered.

I was hoping there would be a text-based configuration file somewhere, but this appeared at first glance to not be the case. It turns out the excluded directories are hardcoded into Spotlight's mds binary, located in /System: Library: Frameworks: CoreServices.framework: Versions: A: Frameworks: Metadata.framework: Versions: A: Support: mds. You can do a strings on the above binary to see them -- or here's the full list of excluded directories.

However, while looking in the .Spotlight-V100 directory at the root of the drive, I found a file called _rules.plist, which can be used to tweak the exclusions.

The contents of this file, by default, are (note that some rows have been split for a narrower display -- you shouldn't be copying and pasting this bit anyway!):

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN"
 "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
        <key>EXCLUDE</key>
        <array/>
        <key>INCLUDE</key>
        <array/>
        <key>NOTE</key>
        <string>Specify paths to include or exclude, preceeding
         rules which target user-homes with ~/</string>
</dict>
</plist>

To add excluded directories to the metadata index, _rules.plist can be modified by adding an array of directories that you'd like searched:

<array>
    <string>/private</string>
    <string>/usr</string>
</array>

Add this section directly below the <key>INCLUDE<key> line. In this example, the /private and /usr directories have been added to the metadata index. After this change, execute sudo mdutil -E / to remove the index and force the volume to reindex. After this, Spotlight has indexed and is able to search /private (including things like /etc) and /usr.

A side effect of this modification is that these areas now become searchable by the Finder's conventional "Find" functionality (Find...; Command-F; or search field in Finder window). Before, a search for httpd.conf yields no results (unless you happen to have other files named "httpd.conf" files around in non-excluded areas). After adding /private, which contains /etc (which is really a symbolic link to /private/etc), "httpd.conf" now appears in search results at its expected location of /private/etc/httpd/httpd.conf.

Further, be advised that explicitly excluded items may need to be explicitly included. For example, Spotlight specifically excludes /Developer/ADC Reference Library by default. Adding /Developer alone to the INCLUDE stanza may not index it; you may need to explicitly include /Developer/ADC Reference Library to cause it to be indexed. Note that I did not yet test this theory; this point is mentioned for information only.