use elasticsearch in gitlab-ci

For Elasticsearch 7, a working config looks like this:

test-elasticsearch:
  stage: test
  services:
    - name: "docker.elastic.co/elasticsearch/elasticsearch:7.10.1"
      alias: "elasticsearch"
      command: [ "bin/elasticsearch", "-Expack.security.enabled=false", "-Ediscovery.type=single-node" ]
  script:
    - curl "http://elasticsearch:9200/_cat/health"

However, you can also have a look whether there is an embedded elasticsearch available for your runtime, e.g. for Java there are "testcontainers": https://www.testcontainers.org/modules/elasticsearch/. That way it's also easier to run tests locally for devs, since they don't need to worry about spinning up elasticsearch first.

For those of you running rails apps using the searchkick gem, be sure to include the following in your gitlab.ci.yml file in addition to your other configurations. As mentioned by previous commenters, the hostname needs to be changed and the ELASTICSEARCH_URL environment variable is how you set it.

services:
  - elasticsearch:latest

variables:
  ELASTICSEARCH_URL "http://elasticsearch:9200"

And if you want to use the official elasticsearch image, you should be able to change the above to the following:

services:
  - name: docker.elastic.co/elasticsearch/elasticsearch:5.6.0
    alias: elasticsearch

variables:
  ELASTICSEARCH_URL "http://elasticsearch:9200"

For people coming here searching for a working example for elasticsearch 7.11 or higher:

elasticsearch 7.10.2 still has a Xmx and Xms in the config file, elasticsearch 7.11.0 has not. So to fix this for 7.11 and higher, you need your CI file to look like this:

  services:
    - name: docker.elastic.co/elasticsearch/elasticsearch:7.16.2
      alias: elastic
      command:
        - /bin/bash
        - -c
        - echo -Xms256m >> /usr/share/elasticsearch/config/jvm.options && echo -Xmx256m >> /usr/share/elasticsearch/config/jvm.options && /usr/local/bin/docker-entrypoint.sh elasticsearch -Ediscovery.type=single-node

Without this, my kubernetes killed the container because it used too much memory.

Expanding on the answer from @Pieter-Verloop, I needed to add something to wait for the Elasticsearch service to be ready on the default port of 9200. Gitlab has an issue where it does not know how to wait for services to be ready, sometimes.

My Gitlab CI file, in the most pared-down form I got it to, looks like this:

test-elasticsearch:
  stage: test
  services:
    - name: elasticsearch:7.16.3
      command:
        - bash
        - -c
        - docker-entrypoint.sh elasticsearch -Ediscovery.type=single-node
  script:
    - timeout 30 bash wait_for_service_up.sh elasticsearch:9200 || false
    - curl "http://elasticsearch:9200/_cat/health"

With a wait_for_service_up.sh script looking like:

#!/bin/bash
while [[
  "$(curl -s -o /dev/null -w ''%{http_code}'' $1)" != "200"
]]; do
  sleep 5;
done

The end result of this is that the CI step will wait for up to 30 seconds for a curl to elasticsearch:9200 to return a 200, which makes sure the service is ready before getting used in further work.

TLTR;

You might need to increase resources for ES container

The long answer:

We use Kubernetes runners in our project.

We had similar problem - we couldn't connect with elastic service even the alias was set in the correct way. We tried to use solutions from @bersling, @Pieter Verloop, @N N, @Jake Anderson without success.

After more debugging we found that the docker container started (docker ps in pipelines), but there were no logs (./kubectl -n <namespace> logs <runner_pod_name> svc-0).

Based on @Pieter Verloop answer we decided to check resources for the service (./kubectl -n <namespace> describe pod <runner_pod_name> and they were quite low.

    Init Containers:
      <pod_name>:
        Limits:
          cpu:     200m
          memory:  512Mi
        Requests:
          cpu:        100m
          memory:     256Mi

After the pod config update and with additional command from the @bersling answer it started working

values.yaml

      config: |
        [[runners]]
          [runners.kubernetes]
            # ...
            # These values might be overkill for your project
            # Check metrics to get the correct amount for your project
            service_cpu_limit = "900m"
            service_memory_limit = "2.3Gi"

gitlab-ci.yaml

      # ...
      command:
        - /bin/bash
        - -c
        - docker-entrypoint.sh elasticsearch -Ediscovery.type=single-node

The answer was to use the "elasticsearch" hostname. See here for hostnames depending o service used: https://docs.gitlab.com/ce/ci/docker/using_docker_images.html#accessing-the-services Thanks @jawad