Description

API key based CCS allows for more granular control for what is allowed between clusters. ES|QL over CCS with API key based CCS has some unique requirements for how to set up the role that need to be documented.

There should not be any changes needed for the remote cluster's cross cluster API key. However, if users setup an cross cluster API key in 8.14 (or before), they will need to create a new one, or update the existing one in 8.15 to pick up the new permissions needed for ES|QL with ENRICH. (we are looking into removing the need for the user to do this, but as-is this will be required to use ES|QL with ENRICH over API key based CCS in 8.15+)

The documentation should focus on the role/security requirements for the local cluster (the cluster that initiates the query request to the remote cluster).

ES|QL over CCS with API key based CCS is first introduced (tech preview) in 8.14. The ENRICH keyword is not supported until 8.15.

The role in 8.14 for the local cluster will need to look like this:

POST /_security/role/remote1
{
  "indices": [
    { 
      "names" : [""],
      "privileges": ["read"]
    }
  ], 
  "remote_indices": [
    {
      "names": [ "logs-*" ],
      "privileges": [ "read","read_cross_cluster" ],
      "clusters" : ["my_remote_cluster"]
    }
  ]
}

Of note, is the need for read_cross_cluster. This is always required for ES|QL over CCS with API key based CCS. non-ES|QL queries only sometimes requires read_cross_cluster (depending on minimize round trips). This could be something users trip over when migrating to the ES|QL based queiries.

Also, of note is the local indices block with the empty names. This is better described in #108734 and is needed in 8.14, but are looking for ways to avoid this. This is only needed for role that can ONLY search the remote cluster. Typically users will have local permission + remote permissions.

The role needed in 8.15 to support the ENRICH keyword will need to look like this:

POST /_security/role/remote1
{
  "cluster": ["cross_cluster_search"], 
  "indices": [
    { 
      "names" : [""],
      "privileges": ["read"]
    }
  ], 
  "remote_indices": [
    {
      "names": [ "logs-*" ],
      "privileges": [ "read","read_cross_cluster" ],
      "clusters" : ["my_remote_cluster"]
    }
  ],
   "remote_cluster": [
        {
            "privileges": [
                "monitor_enrich"
            ],
            "clusters": [
                "my_remote_cluster"
            ]
        }
    ]
} 

of note is the local "cluster": ["cross_cluster_search"], This is needed because local enrich is a cluster level privilege and this permission covers enrich too (and technically, enrich is happening both remote and locally). I would suggest to document that this is always needed, ir-respective of ENRICH (and can be documented as required in 8.14 too). The fact it is only needed when using the ENRICH keyword is a bit misnomer and we should always recommend to set this for now and future use cases.

also of note is the new remote_cluster, which is new in 8.15, and is not documented at all yet. We need both ES|QL and security focused documentation since this is a security feature, but only currently used for ES|QL. This is needed to allow remote enrichment. With out this, users can not read from the .enrich indices on the remote cluster.

cc: @dnhatn @quux00 @leemthompo