Loading

Connect Elastic Cloud Enterprise deployments to Elastic Cloud on Kubernetes clusters

ECK ECE

These steps describe how to configure remote clusters between an Elasticsearch cluster in Elastic Cloud Enterprise (ECE) and an Elasticsearch cluster running within Elastic Cloud on Kubernetes (ECK). Once that's done, you'll be able to run CCS queries from Elasticsearch or set up CCR.

Note about terminology

In the case of remote clusters, the Elasticsearch cluster or deployment initiating the connection and requests is often referred to as the local cluster, while the Elasticsearch cluster or deployment receiving the requests is referred to as the remote cluster.

Before you start, consider the security model that you would prefer to use for authenticating remote connections between clusters, and follow the corresponding steps.

API key
For deployments based on Elastic Stack 8.14 or later, you can use an API key to authenticate and authorize cross-cluster operations to a remote cluster. This model uses a dedicated service endpoint, on port 9443 by default, and gives administrators fine-grained control over remote access. The API key is created on the remote cluster and defines the permissions available to all cross-cluster requests, while local user roles can further restrict, but not extend, those permissions.
TLS certificate (deprecated in Elastic Stack 9.0.0)
This model uses mutual TLS authentication over the Elasticsearch transport interface for cross-cluster operations. User authentication is performed on the local cluster and a user's role names are passed to the remote cluster for authorization. Because a superuser on the local cluster automatically gains full read access to the remote cluster, this model is only suitable for clusters within the same security domain.

Follow these steps to configure the API key security model for remote clusters. If you run into any issues, refer to Troubleshooting.

  • The local and remote deployments must be on Elastic Stack 8.14 or later.
  • Contrary to the certificate security model, the API key security model does not require that both local and remote clusters trust each other.

By default, the remote cluster server interface is deactivated on ECK-managed clusters. To use the API key–based security model for cross-cluster connections, you must first enable it on the remote Elasticsearch cluster:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: <cluster-name>
  namespace: <namespace>
spec:
  version: 9.2.1
  remoteClusterServer:
    enabled: true
  nodeSets:
    - name: default
      count: 3
      ...
      ...
Note

Enabling the remote cluster server triggers a restart of the Elasticsearch cluster.

When the remote cluster server is enabled, ECK automatically creates a Kubernetes service named <cluster-name>-es-remote-cluster that exposes the server internally on port 9443.

To allow clusters running outside your Kubernetes environment to connect to this Elasticsearch cluster, you must expose this service externally. The way to expose this service depends on your ECK version.

You can customize how the remote cluster service is exposed by overriding its service specification directly under spec.remoteClusterServer.service in the Elasticsearch resource. By default, this service listens on port 9443.

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: <cluster-name>
  namespace: <namespace>
spec:
  version: 9.2.1
  remoteClusterServer:
    enabled: true
    service:
      spec:
        type: LoadBalancer
  nodeSets:
    - name: default
      count: 3
      ...
      ...
  1. On cloud providers that support external load balancers, setting the type to LoadBalancer provisions a load balancer for your service. Alternatively, expose the service <cluster-name>-es-remote-cluster through one of the Kubernetes Ingress controllers that support TCP services.

In ECK 3.2 and earlier, you can't customize the service that ECK generates for the remote cluster interface, but you can create your own LoadBalancer service, Ingress object, or use another method available in your environment.

For example, for a cluster named quickstart, the following command creates a separate LoadBalancer service named quickstart-es-remote-cluster-lb, pointing to the ECK-managed service quickstart-es-remote-cluster:

kubectl expose service quickstart-es-remote-cluster \
  --name=quickstart-es-remote-cluster-lb \
  --type=LoadBalancer \
  --port=9443 --target-port=9443
  1. On cloud providers that support external load balancers, setting the type to LoadBalancer provisions a load balancer for your service. Alternatively, expose the service <cluster-name>-es-remote-cluster through one of the Kubernetes Ingress controllers that support TCP services.
Warning

If you change the service’s port, set targetPort explicitly to 9443, which is the default remote cluster server listening port. Otherwise, Kubernetes uses the same value for both fields, resulting in failed connections.

The certificate authority (CA) used by ECK to issue certificates for the remote cluster server interface is stored in the ca.crt key of the secret named <cluster_name>-es-transport-certs-public.

If the external connections reach the Elasticsearch Pods on port 9443 without any intermediate TLS termination, you need to retrieve this CA because it is required in the local cluster configuration to establish trust.

If TLS is terminated by any intermediate component and the certificate presented is not the ECK-managed one, use the CA associated with that component, or omit the CA entirely if it uses a publicly trusted certificate.

To save the transport CA certificate of a cluster named quickstart into a local file, run the following command:

kubectl get secret quickstart-es-transport-certs-public \
-o go-template='{{index .data "ca.crt" | base64decode}}' > eck_transport_ca.crt
Important

ECK-managed CA certificates are automatically rotated after one year by default, but you can configure a different validity period. When the CA certificate is rotated, ensure that this CA is updated in all environments where it's used to preserve trust.

  • On the deployment you will use as remote, use the Elasticsearch API or Kibana to create a cross-cluster API key. Configure it with access to the indices you want to use for cross-cluster search or cross-cluster replication.
  • Copy the encoded key (encoded in the response) to a safe location. You will need it in the next step.

The API key created previously will be used by the local deployment to authenticate with the corresponding set of permissions to the remote deployment. For that, you need to add the API key to the local deployment's keystore.

The steps to follow depend on whether the certificate authority (CA) presented by the remote cluster server, proxy, or load-balancing infrastructure is publicly trusted or private.

The CA is public

  1. Log into the Cloud UI.

  2. On the Deployments page, select your deployment.

    Narrow the list by name, ID, or choose from several other filters. To further define the list, use a combination of filters.

  3. From the navigation menu, select Security.

  4. Locate Remote Connections > Trust management > Connections using API keys and select Add API key.

    1. Fill both fields.

      • For the Remote cluster name, enter the alias of your choice. You will use this alias to connect to the remote cluster later. It must be lowercase and only contain letters, numbers, dashes and underscores.
      • For the Cross-cluster API key, paste the encoded cross-cluster API key.

    2. Click Add to save the API key.

  5. Restart the local deployment to reload the new setting. To do that, go to the deployment's main page, locate the Actions menu, and select Restart Elasticsearch.

    Note

    If the local deployment runs on version 8.14 or greater, you no longer need to perform this step because the keystore is reloaded automatically with the new API keys.

If you need to update the remote connection with different permissions later, refer to Change a cross-cluster API key used for a remote connection.

The CA is private (ECK-managed transport certificates)

When adding the CA certificate in the next steps, use either the ECK-managed transport CA obtained previously, or the CA of the component that terminates TLS connections to clients.

  1. Log in to the Cloud UI.

  2. On the Deployments page, select your deployment.

    Narrow the list by name, ID, or choose from several other filters. To further define the list, use a combination of filters.

  3. From the navigation menu, select Security.

  4. Select Remote Connections > Add trusted environment and choose Self-managed. Then click Next.

  5. Select API keys as authentication mechanism and click Next.

  6. When asked whether the Certificate Authority (CA) of the remote environment’s proxy or load-balancing infrastructure is public, select No, it is private.

  7. Add the API key:

    1. Fill both fields.

      • For the Remote cluster name, enter the alias of your choice. You will use this alias to connect to the remote cluster later. It must be lowercase and only contain letters, numbers, dashes and underscores.
      • For the Cross-cluster API key, paste the encoded cross-cluster API key.

    2. Click Add to save the API key.

    3. Repeat these steps for each API key you want to add. For example, if you want to use several clusters of the remote environment for CCR or CCS.

  8. Add the CA certificate of the remote environment.

  9. Provide a name for the trusted environment. That name will appear in the trust summary of your deployment's Security page.

  10. Select Create trust to complete the configuration.

  11. Restart the local deployment to reload the new settings. To do that, go to the deployment's main page, locate the Actions menu, and select Restart Elasticsearch.

    Note

    If the local deployment runs on version 8.14 or greater, you no longer need to perform this step because the keystore is reloaded automatically with the new API keys.

If you need to update the remote connection with different permissions later, refer to Change a cross-cluster API key used for a remote connection.

When using TLS certificates-based authentication, the first step is to establish trust between the two clusters, by adding the CA certificate and trust details of each environment into the other.

  1. Save the ECK CA certificate to a file. For a cluster named quickstart, run:

    kubectl get secret quickstart-es-transport-certs-public -o go-template='{{index .data "ca.crt" | base64decode}}' > eck.ca.crt

  2. Update the trust settings for the Elastic Cloud Enterprise deployment. Follow the steps provided in Access clusters of a self-managed environment, and specifically the first three steps in Specify the deployments trusted to be used as remote clusters using TLS certificate as security model.

    • Use the certificate file saved in the first step.
    • Select the Elastic Cloud pattern and enter default.es.local for the Scope ID.

  3. Select Save and then download the CA Certificate and trust.yml file. These files can also be retrieved in the Security page of the deployment. You will use these files in the next set of steps.

  1. Upload the Elastic Cloud Enterprise certificate (that you downloaded in the last step of the previous section) as a Kubernetes secret.

    kubectl create secret generic ce-aws-cert --from-file=<path to certificate file>

  2. Upload the trust.yml file (that you downloaded in the last step of the previous section) as a Kubernetes config map.

    kubectl create configmap quickstart-trust --from-file=<path to trust.yml>

  3. Edit the Elasticsearch kubernetes resource to ensure the following sections are included. This assumes the Elasticsearch deployment is named quickstart. Make sure to replace the CA-Certificate-Filename placeholder with the correct value. Note that these configuration changes are required for all nodeSets. Applying this change requires all pods in all nodeSets to be deleted and recreated, which might take quite a while to complete.

    spec:
  nodeSets:
  - config:
       xpack.security.transport.ssl.certificate_authorities:
       - /usr/share/elasticsearch/config/other/<CA-Certificate-Filename>
       xpack.security.transport.ssl.trust_restrictions.path:  /usr/share/elasticsearch/config/trust-filter/trust.yml
    podTemplate:
      spec:
        containers:
        - name: elasticsearch
           volumeMounts:
           - mountPath: /usr/share/elasticsearch/config/other
              name: ce-aws-cert
           - mountPath: /usr/share/elasticsearch/config/trust-filter
             name: quickstart-trust
        volumes:
        - name: ce-aws-cert
           secret:
             secretName: ce-aws-cert
        - configMap:
             name: quickstart-trust
           name: quickstart-trust

Expose the transport service (defaults to port 9300) of your ECK cluster to allow external Elasticsearch clusters to connect:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: <cluster-name>
spec:
  transport:
    service:
      spec:
        type: LoadBalancer
  1. On cloud providers which support external load balancers, setting the type field to LoadBalancer provisions a load balancer for your service. Alternatively, expose the service <cluster-name>-es-transport through one of the Kubernetes Ingress controllers that support TCP services.

On the local deployment, add the remote ECK cluster using Kibana or the Elasticsearch API with the following connection settings:

  • Remote address: Use the FQDN or IP address of the LoadBalancer service, or similar resource, you created to expose the remote cluster server interface (for API key-based authentication) or the transport interface (for TLS certificate-based authentication).

  • TLS server name: You can try leaving this field empty first. If the connection fails, and your environment is presenting the ECK-managed certificates during the TLS handshake, use <cluster-name>-es-remote-cluster.<namespace>.svc as the server name. For example, for a cluster named quickstart in the default namespace, use quickstart-es-remote-cluster.default.svc.

  1. Go to the Remote Clusters management page in the navigation menu or use the global search field.

  2. Select Add a remote cluster.

  3. In Select connection type, choose the authentication mechanism you prepared earlier (API keys or Certificates), and click Next.

  4. In Add connection information, fill in the following fields:

    • Remote cluster name: This cluster alias is a unique identifier that represents the connection to the remote cluster and is used to distinguish local and remote indices.

      When using API key authentication, this alias must match the Remote cluster name you configured when adding the API key in the Cloud UI.

    • Remote address: Enter the endpoint of the remote cluster, including the hostname, FQDN, or IP address, and the port.

      Make sure you use the correct port for your authentication method:

      • API keys: Use the port configured in the remote cluster interface of the remote cluster (defaults to 9443).
      • TLS Certificates: Use the Elasticsearch transport port (defaults to 9300).

      Starting with Kibana 9.2, this field also supports IPv6 addresses. When using an IPv6 address, enclose it in square brackets followed by the port number. For example: [2001:db8::1]:9443.

    • Configure advanced options (optional): Expand this section if you need to customize additional settings.

      • TLS server name: Specify a value if the certificate presented by the remote cluster is signed for a different name than the remote address.
      • Socket connections: Define the number of connections to open with the remote cluster.

  5. Click Next.

  6. In Confirm setup, click Add remote cluster (you have already established trust in a previous step).

To add a remote cluster, use the cluster update settings API. Configure the following fields:

  • Remote cluster alias: When using API key authentication, the cluster alias must match the one you configured when adding the API key in the Cloud UI as Remote cluster name.

  • mode: proxy

  • proxy_address: Enter the endpoint of the remote cluster, including the hostname, FQDN, or IP address, and the port. Both IPv4 and IPv6 addresses are supported.

    Make sure you use the correct port for your authentication method:

    • API keys: Use the port configured in the remote cluster interface of the remote cluster (defaults to 9443).
    • TLS Certificates: Use the Elasticsearch transport port (defaults to 9300).

    When using an IPv6 address, enclose it in square brackets followed by the port number. For example: [2001:db8::1]:9443.

  • server_name: Specify a value if the certificate presented by the remote cluster is signed for a different name than the proxy_address.

This is an example of the API call to add or update a remote cluster:

PUT /_cluster/settings
{
  "persistent": {
    "cluster": {
      "remote": {
        "alias-for-my-remote-cluster": {
          "mode":"proxy",
          "proxy_address": "<REMOTE_CLUSTER_ADDRESS>:9443",
          "server_name": "<REMOTE_CLUSTER_SERVER_NAME>"
        }
      }
    }
  }
}
  1. Align the alias with the remote cluster name used when adding the API key.

For a full list of available client connection settings in proxy mode, refer to the remote cluster settings reference.

If you're using the API key based security model, to use a remote cluster for cross-cluster replication or cross-cluster search, you need to create user roles with remote indices privileges on the local cluster. Refer to Configure roles and users.