Skip to content

Lookup replication issue with DA-ESS-ContentUpdate on distributed search #4229

Closed
#4282
Closed
Lookup replication issue with DA-ESS-ContentUpdate on distributed search#4229
#4282
Milestone
v6.8.0

Description

@Masoud00013
Masoud00013
opened on Aug 22, 2026

Hi,

I am facing an issue with one of the lookups provided by DA-ESS-ContentUpdate in a distributed Splunk Enterprise Security environment.

Environment
Splunk Enterprise: 10.4
Splunk Enterprise Security: 8.x
Search Head Cluster: 3 members
Multiple Indexers
Linux
The lookup is a CSV-based lookup provided by DA-ESS-ContentUpdate

The lookup is available and works correctly on the Search Head, but it is not available on the Search Peers/Indexers when the search is distributed.

Lookup

The lookup definition and CSV are both located under:

/opt/splunk/etc/apps/DA-ESS-ContentUpdate/

The lookup CSV is under:

/opt/splunk/etc/apps/DA-ESS-ContentUpdate/lookups/

and its definition is in the corresponding transforms.conf.

The lookup works correctly when executed locally on the Search Head, but distributed searches fail because the Search Peer does not have the lookup.

Investigation

We checked the effective distsearch.conf configuration using:

/opt/splunk/bin/splunk btool distsearch list replicationDenylist --debug

The following entry was found:

/opt/splunk/etc/apps/DA-ESS-ContentUpdate/default/distsearch.conf
[replicationDenylist]
excludeESCU = apps[/\]DA-ESS-ContentUpdate[/\]lookups[/\]...

This appears to exclude all lookup files under DA-ESS-ContentUpdate/lookups/ from Knowledge Bundle replication.

At the same time, the global configuration contains:

replicate.lookups = true

and the effective replication allowlist contains:

[replicationAllowlist]
other = (system|(apps/(?!pdfserver))|users(/_reserved)?//*)/(bin|lookups)/...

Therefore, lookups are generally allowed to be replicated, but the ESCU-specific denylist appears to explicitly exclude the entire DA-ESS-ContentUpdate/lookups/ directory.

We also checked:

/opt/splunk/bin/splunk btool distsearch list replicationSettings:refineConf --debug

which confirms:

replicate.lookups = true
replicate.transforms = true

However, the replicationDenylist entry above prevents the ESCU lookup from being included in the Knowledge Bundle.

Knowledge Bundle verification

We inspected the generated Knowledge Bundle and confirmed that the relevant lookup is not present in the bundle.

This explains why:

The lookup exists on the Search Head.
The lookup definition exists.
replicate.lookups = true.
The lookup still does not exist on the Search Peer.
Distributed searches cannot use the lookup.
Question

Is this exclusion intentional for ESCU lookups?

If so, what is the recommended/supported way to replicate a specific ESCU lookup to Search Peers while keeping the other ESCU lookup files excluded?

We specifically want to achieve the following:

DA-ESS-ContentUpdate/lookups/

lookup-A.csv                         -> NOT replicated
lookup-B.csv                         -> NOT replicated
...
cisco_secure_firewall_filetype_lookup.csv
                                      -> SHOULD be replicated

We do not want to replicate all ESCU lookups because of the potential Knowledge Bundle size and performance impact.

We also do not want to copy the lookup into a custom App, because doing so would cause the custom copy to become independent from future ESCU Content Updates. If Splunk updates this lookup in a future Content Update, we would want the updated official lookup to continue being used automatically.

Expected behavior

Ideally, we would like to keep the lookup managed by:

DA-ESS-ContentUpdate

so that it continues to receive future Content Updates, while allowing only this specific lookup to be included in Knowledge Bundle replication.

Is there a supported configuration or recommended approach for creating an exception to:

excludeESCU = apps[/\]DA-ESS-ContentUpdate[/\]lookups[/\]...

for a single lookup?

Thanks.

Activity

  1. nasbench commented on Aug 24, 2026

    @nasbench
    nasbench
    on Aug 24, 2026
    Contributor

    cc @pyth0n1c @cmcginley-splunk - You might wanna take a look at this.

  2. nasbench
    added this to the v6.8.0 milestone on Sep 22, 2026
  3. nasbench
    linked a pull request that will close this issueAdd Missing `local=t` From Lookup Calls #4282on Sep 22, 2026

  4. nasbench commented on Sep 22, 2026

    @nasbench
    nasbench
    on Sep 22, 2026
    Contributor

    Hey @Masoud00013 Thanks for raising this issue.

    Yes, this behavior is intentional, ESCU excludes its lookup directory from knowledge-bundle replication (as you found out):

    [replicationDenylist]
excludeESCU = apps[/\\]DA-ESS-ContentUpdate[/\\]lookups[/\\]...

    The purpose is to avoid replicating the entire lookup collection to every indexer. The app currently contains 76 lookup files totaling approximately 3.6 MB, with some individual files exceeding 2 MB. There are several reasons for this design, among them:

    • Replicating all lookup files increases knowledge-bundle size, transfer time, disk usage, and processing overhead on every search peer. (as you've pointed out already)
    • Frequently updated lookup files can cause additional knowledge-bundle churn and retransmission.
    • The lookups are generally intended to be applied late in the search, after indexers have filtered and reduced the event set.
    • Applying the lookup locally on the search head avoids unnecessary lookup processing and data copies on indexers.
    • Keeping the files on the search head preserves ownership and lifecycle management by DA-ESS-ContentUpdate.

    We have updated the affected production detections and macros to use local=t - See #4282

    With that said, If there is a specific lookup that needs to be available on indexers, Splunk supports a file-specific replication exception. The broad denylist can remain unchanged while allowing only the required lookup:

    [replicationSettings:fileSpecific]
apps/DA-ESS-ContentUpdate/lookups/<exact-lookup-file>.csv = allow

    For example:

    [replicationSettings:fileSpecific]
apps/DA-ESS-ContentUpdate/lookups/cisco_secure_firewall_filetype_lookup_20260513.csv = allow

    This should be added in an administrator-owned configuration, such as:

    $SPLUNK_HOME/etc/apps/site_distsearch_overrides/local/distsearch.conf

    Note: I did not test the above configuration, but it should work in theory

  5. Masoud00013 commented on Sep 26, 2026

    @Masoud00013
    Masoud00013
    on Sep 26, 2026
    Author

    Thank you for your attention and your reply. Based on the details in the distsearch.conf file, the approach you suggested might not work, as Splunk explicitly states that the Denylist takes precedence over the others.
    Nevertheless, I will certainly test this out.

  6. nasbench commented on Sep 29, 2026

    @nasbench
    nasbench
    on Sep 29, 2026
    Contributor

    Thank you for your attention and your reply. Based on the details in the distsearch.conf file, the approach you suggested might not work, as Splunk explicitly states that the Denylist takes precedence over the others. Nevertheless, I will certainly test this out.

    Hey thanks for the comment. You’re right that the denylist takes precedence over the allowlist. However, my suggestion was based on the docs, see this section

    [replicationSettings:fileSpecific]

<file_path> = allow|deny
* Whether the Splunk platform replicates a specific file identified by the
  file path under $SPLUNK_HOME/etc/.
* This setting supersedes any regex matches found under 'replicationDenyList'
  or 'replicationAllowlist'.
* Optional.
* No default.

    Specifically This setting supersedes any regex matches found under 'replicationDenyList'

    Please give it a try and let me know.

    Overall the issue should not happen since we're adding local=t to all prod rules as well as validation flows.

    But the solution above could work for those who really wants the replication.

  7. nasbench
    closed this as completedin #4282on Oct 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions