After spending a week, here is what I got.
Overall flow :
use logger (slf4j) to add logs in your java file -> filebeat (oss version) -> aws OpenSearch
Step 1: Configure Spring Boot Application
- Add these dependency in your pom file
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
<version>1.2.11</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-logging</artifactId>
</dependency>
- Add logback.xml in your resources folder
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>/app/your-file-path/application.log</file>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>/app/your-file-path/application.%d{yyyy-MM-dd}.log</fileNamePattern>
<maxHistory>30</maxHistory>
</rollingPolicy>
<encoder>
<pattern>%d{yyyy-MM-dd HH:mm:ss} %-5level %logger{36} - %msg%n</pattern>
</encoder>
</appender>
<root level="info">
<appender-ref ref="FILE" />
</root>
</configuration>
Step 2: Install Filebeat (or logstash or both) on your ec2 instance.
If you have a better server than t2.micro, you can use Logstash here, which is better but needs more resources.
Full installation guide for filebeat - click here
check logs for any error:
sudo journalctl -u filebeat.
Make sure you install the oss(open-source) version otherwise, it will look for the license and will throw an error.
Step 3: Set up your opensearch. (which I guess you have already done).
Once filebeat start sending logs to your opensearch , navigate to Dashboard management > index pattern and type filebeat-* and follow the steps.
Hope it helps :)
