ElasticSearch RestHighLevelClient throws java.io.IOException: Connection reset by peer

Question

Summary

When my Spring Boot app has been idle for a while, calling the search method of an ElasticsearchRepository (which uses the RestHighLevelClient for the underlying ES connection) results in a java.io.IOException: Connection reset by peer being thrown.

Detailed problem

I have a Spring Boot app (version 2.2.4.RELEASE) and an ElasticSearch (version 6.8.6). For communication with the ES instance I use Spring Data Elasticsearch (version 3.2.5.RELEASE), to be specific I use an ElasticsearchRepository.

My ES configuration class looks as follows:

@Configuration
@EnableElasticsearchRepositories
public class ElasticsearchConfiguration {

    private final RestHighLevelClient restHighLevelClient;

    @Autowired
    public ElasticsearchConfiguration(RestHighLevelClient restHighLevelClient) {
        this.restHighLevelClient = restHighLevelClient;
    }

    @Bean
    public ElasticsearchRestTemplate elasticsearchTemplate() {
        return new ElasticsearchRestTemplate(restHighLevelClient);
    }

}

I have got an REST endpoint which triggers a lookup in the ES by calling the below method:

@Service
@Transactional(readOnly = true)
public class SportsFacilityViewFilterService {

    private final SportsFacilityViewRepository sportsFacilityViewRepository;

    @Autowired
    public SportsFacilityViewFilterService(SportsFacilityViewRepository sportsFacilityViewRepository) {
        this.sportsFacilityViewRepository = sportsFacilityViewRepository;
    }

    /**
     * Retrieves all {@link SportsFacilityView}s stored in the search index that match the given filter parameters.
     *
     * @param activity filter parameter.
     * @param city     filter parameter.
     * @param date     filter parameter.
     * @param capacity filter parameter.
     * @return {@code List<SportsFacilityView>} the filtered sports facilities in a display optimized format.
     */
    @Retryable(include = IOException.class, maxAttempts = 1, backoff = @Backoff(delay = 15))
    @Transactional(readOnly = true)
    public List<SportsFacilityView> filterSportsFacilities(final String activity, final Integer capacity,
                                                           final String city, final LocalDate date) {
        // Use filter query to retrieve sports facility views
        final NativeSearchQuery searchQuery = new NativeSearchQueryBuilder()
            .withFilter(buildFilterQuery(activity, capacity, city))
            .withPageable(PageRequest.of(0, 50))
            .withSort(new FieldSortBuilder("vendorName").order(SortOrder.ASC))
            .build();
        final List<SportsFacilityView> sportsFacilities = Lists.newArrayList(sportsFacilityViewRepository.search(searchQuery));

        // Post-process filter results
        removeClosedSportsFacilities(sportsFacilities, date);
        updateMinimumPrice(sportsFacilities, date);

        // Return
        return sportsFacilities;
    }

}

I defined the SportsFacilityViewRepository as follows:

@Repository
public interface SportsFacilityViewRepository extends ElasticsearchRepository<SportsFacilityView, Long> {

}

This code works perfectly fine, but when the system has been idle for a while, the following exception is thrown when invoking the above illustrated method:

org.springframework.data.elasticsearch.ElasticsearchException: Error for search request with scroll: SearchRequest{searchType=DFS_QUERY_THEN_FETCH, indices=[sportsfacility], indicesOptions=IndicesOptions[ignore_unavailable=false, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false, ignore_throttled=true], types=[sportsfacilityview], routing='null', preference='null', requestCache=null, scroll=null, maxConcurrentShardRequests=0, batchedReduceSize=512, preFilterShardSize=128, allowPartialSearchResults=null, localClusterAlias=null, getOrCreateAbsoluteStartMillis=-1, source={"from":0,"size":50,"post_filter":{"bool":{"filter":[{"term":{"address.city":{"value":"Köln","boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},"version":true,"sort":[{"vendorName":{"order":"asc"}}]}}
        at org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate.doSearch(ElasticsearchRestTemplate.java:1153) ~[spring-data-elasticsearch-3.2.3.RELEASE.jar:3.2.3.RELEASE]
        at org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate.queryForPage(ElasticsearchRestTemplate.java:381) ~[spring-data-elasticsearch-3.2.3.RELEASE.jar:3.2.3.RELEASE]
        at org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate.queryForPage(ElasticsearchRestTemplate.java:376) ~[spring-data-elasticsearch-3.2.3.RELEASE.jar:3.2.3.RELEASE]
        at org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate.queryForPage(ElasticsearchRestTemplate.java:147) ~[spring-data-elasticsearch-3.2.3.RELEASE.jar:3.2.3.RELEASE]
        at org.springframework.data.elasticsearch.repository.support.AbstractElasticsearchRepository.search(AbstractElasticsearchRepository.java:259) ~[spring-data-elasticsearch-3.2.3.RELEASE.jar:3.2.3.RELEASE]
        at sun.reflect.GeneratedMethodAccessor260.invoke(Unknown Source) ~[na:na]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_222]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_222]
        at org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments.invoke(RepositoryComposition.java:371) ~[spring-data-commons-2.2.3.RELEASE.jar:2.2.3.RELEASE]
        at org.springframework.data.repository.core.support.RepositoryComposition.invoke(RepositoryComposition.java:204) ~[spring-data-commons-2.2.3.RELEASE.jar:2.2.3.RELEASE]
        at org.springframework.data.repository.core.support.RepositoryFactorySupport$ImplementationMethodExecutionInterceptor.invoke(RepositoryFactorySupport.java:657) ~[spring-data-commons-2.2.3.RELEASE.jar:2.2.3.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.doInvoke(RepositoryFactorySupport.java:621) ~[spring-data-commons-2.2.3.RELEASE.jar:2.2.3.RELEASE]
        at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.invoke(RepositoryFactorySupport.java:605) ~[spring-data-commons-2.2.3.RELEASE.jar:2.2.3.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:93) ~[spring-aop-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) ~[spring-aop-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at com.sun.proxy.$Proxy239.search(Unknown Source) ~[na:na]
        at sun.reflect.GeneratedMethodAccessor259.invoke(Unknown Source) ~[na:na]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_222]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_222]
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) ~[spring-aop-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) ~[spring-aop-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:139) ~[spring-tx-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) ~[spring-aop-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at com.sun.proxy.$Proxy239.search(Unknown Source) ~[na:na]
        at com.myapp.service.SportsFacilityViewFilterService.filterSportsFacilities(SportsFacilityViewFilterService.java:74) ~[app/:na]
        at com.myapp.service.SportsFacilityViewFilterService$$FastClassBySpringCGLIB$$d9ad847c.invoke(<generated>) ~[app/:na]
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) ~[spring-core-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:769) ~[spring-aop-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:747) ~[spring-aop-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.transaction.interceptor.TransactionInterceptor$$Lambda$1117.00000000674D8570.proceedWithInvocation(Unknown Source) ~[na:na]
        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:366) ~[spring-tx-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:99) ~[spring-tx-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:747) ~[spring-aop-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:689) ~[spring-aop-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at com.myapp.service.SportsFacilityViewFilterService$$EnhancerBySpringCGLIB$$492633ab.filterSportsFacilities(<generated>) ~[app/:na]
        at com.myapp.web.controller.publicapi.SportsFacilityFilterController.filterSportsFacilities(SportsFacilityFilterController.java:42) ~[app/:na]
        at com.myapp.web.controller.publicapi.SportsFacilityFilterController$$FastClassBySpringCGLIB$$13529b2a.invoke(<generated>) ~[app/:na]
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) ~[spring-core-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:769) ~[spring-aop-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:747) ~[spring-aop-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.validation.beanvalidation.MethodValidationInterceptor.invoke(MethodValidationInterceptor.java:120) ~[spring-context-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:747) ~[spring-aop-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:689) ~[spring-aop-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at com.myapp.web.controller.publicapi.SportsFacilityFilterController$$EnhancerBySpringCGLIB$$221bc350.filterSportsFacilities(<generated>) ~[app/:na]
        at sun.reflect.GeneratedMethodAccessor258.invoke(Unknown Source) ~[na:na]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_222]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_222]
        at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190) ~[spring-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138) ~[spring-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:106) ~[spring-webmvc-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:888) ~[spring-webmvc-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:793) ~[spring-webmvc-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040) ~[spring-webmvc-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943) ~[spring-webmvc-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) [spring-webmvc-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898) [spring-webmvc-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:634) [tomcat-embed-core-9.0.29.jar:9.0.29]
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) [spring-webmvc-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:741) [tomcat-embed-core-9.0.29.jar:9.0.29]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) [tomcat-embed-core-9.0.29.jar:9.0.29]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.29.jar:9.0.29]
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) [tomcat-embed-websocket-9.0.29.jar:9.0.29]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-9.0.29.jar:9.0.29]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.29.jar:9.0.29]
        at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:209) [spring-security-web-5.2.1.RELEASE.jar:5.2.1.RELEASE]
        at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:178) [spring-security-web-5.2.1.RELEASE.jar:5.2.1.RELEASE]
        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358) [spring-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271) [spring-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-9.0.29.jar:9.0.29]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.29.jar:9.0.29]
        at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) [spring-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) [spring-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-9.0.29.jar:9.0.29]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.29.jar:9.0.29]
        at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) [spring-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) [spring-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-9.0.29.jar:9.0.29]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.29.jar:9.0.29]
        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) [spring-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) [spring-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-9.0.29.jar:9.0.29]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.29.jar:9.0.29]
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202) [tomcat-embed-core-9.0.29.jar:9.0.29]
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [tomcat-embed-core-9.0.29.jar:9.0.29]
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:526) [tomcat-embed-core-9.0.29.jar:9.0.29]
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) [tomcat-embed-core-9.0.29.jar:9.0.29]
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) [tomcat-embed-core-9.0.29.jar:9.0.29]
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) [tomcat-embed-core-9.0.29.jar:9.0.29]
        at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:747) [tomcat-embed-core-9.0.29.jar:9.0.29]
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) [tomcat-embed-core-9.0.29.jar:9.0.29]
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:367) [tomcat-embed-core-9.0.29.jar:9.0.29]
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) [tomcat-embed-core-9.0.29.jar:9.0.29]
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:860) [tomcat-embed-core-9.0.29.jar:9.0.29]
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1591) [tomcat-embed-core-9.0.29.jar:9.0.29]
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-9.0.29.jar:9.0.29]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_222]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_222]
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-9.0.29.jar:9.0.29]
        at java.lang.Thread.run(Thread.java:819) [na:1.8.0_222]
Caused by: java.io.IOException: Connection reset by peer
        at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:964) ~[elasticsearch-rest-client-6.8.6.jar:6.8.6]
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:233) ~[elasticsearch-rest-client-6.8.6.jar:6.8.6]
        at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1764) ~[elasticsearch-rest-high-level-client-6.8.6.jar:6.8.6]
        at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1734) ~[elasticsearch-rest-high-level-client-6.8.6.jar:6.8.6]
        at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1696) ~[elasticsearch-rest-high-level-client-6.8.6.jar:6.8.6]
        at org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:1092) ~[elasticsearch-rest-high-level-client-6.8.6.jar:6.8.6]
        at org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate.doSearch(ElasticsearchRestTemplate.java:1151) ~[spring-data-elasticsearch-3.2.3.RELEASE.jar:3.2.3.RELEASE]
        ... 108 common frames omitted
Caused by: java.io.IOException: Connection reset by peer
        at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[na:1.8.0_222]
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[na:1.8.0_222]
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[na:1.8.0_222]
        at sun.nio.ch.IOUtil.read(IOUtil.java:197) ~[na:1.8.0_222]
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) ~[na:1.8.0_222]
        at org.apache.http.impl.nio.reactor.SessionInputBufferImpl.fill(SessionInputBufferImpl.java:231) ~[httpcore-nio-4.4.12.jar:4.4.12]
        at org.apache.http.impl.nio.codecs.AbstractMessageParser.fillBuffer(AbstractMessageParser.java:136) ~[httpcore-nio-4.4.12.jar:4.4.12]
        at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:241) ~[httpcore-nio-4.4.12.jar:4.4.12]
        at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81) ~[httpasyncclient-4.1.4.jar:4.1.4]
        at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39) ~[httpasyncclient-4.1.4.jar:4.1.4]
        at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114) ~[httpcore-nio-4.4.12.jar:4.4.12]
        at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) ~[httpcore-nio-4.4.12.jar:4.4.12]
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) ~[httpcore-nio-4.4.12.jar:4.4.12]
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) ~[httpcore-nio-4.4.12.jar:4.4.12]
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) ~[httpcore-nio-4.4.12.jar:4.4.12]
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) ~[httpcore-nio-4.4.12.jar:4.4.12]
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) ~[httpcore-nio-4.4.12.jar:4.4.12]
        ... 1 common frames omitted

I tried forcing a retry by annotating the method with @Retryable, but unfortunately this appears to be ignored, despite being picked up in other places of the program.

Can somebody please explain why I get this exception and how I can prevent it from occurring?

Update

Removing the @Repository annotation from SportsFacilityViewRepository made it possible to try/catch the code, respectively automatically retry by means of the @Retryable annotation. Obviously, the Spring Exception Translation mechanism applied due to the @Repository annotation being present prevented a try/catch block and Spring Retry to catch the exception that was shown on the console.

No, I use spring-data-elasticsearch of org.springframework.data version 3.2.5.RELEASE. Do you recommend trying the spring boot starter? — times29
– times29, Commented Mar 6, 2020 at 13:35
This idle connection problems seems to come up from time to time; I will create a ticket for a feature to be handle this. — Abacus
– Abacus, Commented Mar 6, 2020 at 21:14
what if you configure @Retryable to be triggered by the org.springframework.data.elasticsearch.ElasticsearchException class? — Abacus
– Abacus, Commented Mar 8, 2020 at 9:54

times29 · Accepted Answer · 2020-03-23 16:17:50Z

1

I finally found an easily viable workaround. I let Spring execute a scheduled task which executes the count method of the relevant ElasticsearchRepositories. By that means, the connection is kept alive so that no Exceptions can occur due to the system being idle for too long.

The code looks as follows:

@Component
@Slf4j
public class ElasticsearchConnectionTester {

    private final SportsFacilityViewRepository sportsFacilityViewRepository;

    @Autowired
    public ElasticsearchConnectionTester(SportsFacilityViewRepository sportsFacilityViewRepository) {
        this.sportsFacilityViewRepository = sportsFacilityViewRepository;
    }

    @Scheduled(fixedRate = 1800000, initialDelay = 1800000)
    public void keepConnectionAlive() {
        log.debug("Trying to ping Elasticsearch");
        try {
            final long noOfSportsFacilities = sportsFacilityViewRepository.count();
            log.debug("Ping succeeded for SportsFacilityViewRepository, it contains {} entities", noOfSportsFacilities);
        } catch (Exception e) {
            log.debug("Ping failed for SportsFacilityViewRepository");
        }
    }
}

The logging is optional, but it provides the necessary insight whether the chosen rate for the scheduled job is sufficient.

answered Mar 23, 2020 at 16:17

times29

3,4613 gold badges27 silver badges44 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Alexander.Furer Over a year ago

@Retryable seems to be more elegant and lower-cost solution , why did you go with keep-alive unnecessary calls ?

times29 Over a year ago

@Alexander.Furer it didn't work reliably. I couldn't find out why, but neither the try/catch nor the @Retryable was able to catch the exception. In my opinion, sending a request each 30 minutes isn't that high costly in comparision, especially when you consider the price of exceptions and their stack traces!

Alexander.Furer Over a year ago

It's not about sending request once in 30 minutes.. It's about keeping connection you don't need which might be expensive for server

times29 Over a year ago

@Alexander.Furer ah ok. Could you please elaborate how expensive it is for the server to keep up the connection?

Alexander.Furer Over a year ago

Depends : NIO or not. But your question is probably rhetorical - so, never mind.

DamCx · Accepted Answer · 2022-11-03 11:05:44Z

Here's some configuration:

 @Configuration
 public class ElasticSearchConfig extendsAbstractElasticsearchConfiguration {
    @Value("${spring.elasticsearch.rest.uris}")
    private String uris;

    @Value("${spring.elasticsearch.rest.username:#{\"\"}}")
    private String username;

    @Value("${spring.elasticsearch.rest.password:#{\"\"}}")
    private String password;

    @Override
    public RestHighLevelClient elasticsearchClient() {

        final ClientConfiguration clientConfiguration = ClientConfiguration.builder()
                .connectedTo(uris)
                .withBasicAuth(username, password)
                .withHttpClientConfigurer(clientBuilder -> {
                    clientBuilder.setKeepAliveStrategy((httpResponse, httpContext) -> 1000 * 60);
                    return clientBuilder;
                })
                .build();
        return RestClients.create(clientConfiguration).rest();

    }

}

StanZhao · Accepted Answer · 2023-02-13 09:31:04Z

0

Maybe you can config the client like this:

    @Override
    @Bean
    public RestHighLevelClient elasticsearchClient() {

        // Connect to elastic search in https://app.bonsai.io/
        final ClientConfiguration clientConfiguration = ClientConfiguration.builder()
                .connectedTo(hostAndPort)
                .withBasicAuth(userName, pwd)
                // config the connect
                .withConnectTimeout(5000)
                .withSocketTimeout(60000)
                .build();

        return RestClients.create(clientConfiguration).rest();
    }

answered Feb 13, 2023 at 9:31

StanZhao

12 bronze badges

Collectives™ on Stack Overflow

ElasticSearch RestHighLevelClient throws java.io.IOException: Connection reset by peer

Summary

Detailed problem

Update

3 Answers 3

5 Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

Summary

Detailed problem

Update

3 Answers 3

5 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related