上节我们通过logstash收集日志输出到es主机中，如果遇到高峰期或者网络瓶颈很容易出现es主机接收日志丢失或者网络阻塞，拿么我们来避免一下这个问题，这时候我们就用到了消息队列服务或者REDIS服务。
本次我们使用redis服务
通过logstash收集主机日志输出到redis中，再由logstash从redis中心输入日志并输出到es集群中。这样一来日志将会被logstash从redis一个一个收走，解决高峰期文件丢失和网络阻塞问题。
一、安装redis服务
安装redis，本次使用ubunt最新版本
# apt install redis
修改配置文件，主要配置是修改redis密码，其他配置与公司优化相同
requirepass 123456 #密码设置为123456
重启服务
systemctl restart redis二、采集tomcat日志
配置文件如下
root@web1:/etc/logstash/conf.d# cat lhl-tomcat.conf
input {
file {
path => "/usr/local/tomcat/logs/localhost_access_log.2022-09-25.txt"
start_position => "beginning"
stat_interval => 3
type => "tomcat-access"
}
file {
path => "/usr/local/tomcat/logs/catalina.out"
start_position => "beginning"
stat_interval => 3
type => "tomcat-catalina"
}
}
output {
if [type] == "tomcat-access"{
redis{
host => ["10.0.0.124"] #redis主机ip列表中可以些多个ip
data_type => "list" #数据格式为列表
port => "6379" #redis端口
password => "123456" #redis密码
key => "lhl-tomcat-access" #写入redis的key的名称
db => "0" #redis编号，默认为0，
}}
if [type] == "tomcat-catalina"{
redis{
host => ["10.0.0.124"]
data_type => "list"
port => "6379"
password => "123456"
key => "lhl-tomcat-catalina"
db => "0"
}}
}
检查配置文件是否正常
root@web1:/etc/logstash/conf.d# /usr/share/logstash/bin/logstash -f lhl-tomcat.conf -t
Using JAVA_HOME defined java: /usr/local/jdk
WARNING, using JAVA_HOME while Logstash distribution comes with a bundled JDK
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2022-11-12 06:39:36.257 [main] runner - Starting Logstash {"logstash.version"=>"7.12.1", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc Java HotSpot(TM) 64-Bit Server VM 25.291-b10 on 1.8.0_291-b10 +indy +jit [linux-x86_64]"}
[WARN ] 2022-11-12 06:39:36.628 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2022-11-12 06:39:38.030 [LogStash::Runner] Reflections - Reflections took 39 ms to scan 1 urls, producing 23 keys and 47 values
Configuration OK
[INFO ] 2022-11-12 06:39:38.546 [LogStash::Runner] runner - Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
重启服务
root@web1:/etc/logstash/conf.d# systemctl restart logstash.service 三、测试logstash是否向redis写入数据
向日志中输入消息
进入redis中查看是否有写入的数据
有写入的key，则说明写入成功，如果没有值说明并没有写入redis中，检查一下配置文件
四、使用logstash收集redis中的日志
cat /etc/logstash/conf.d/logstashtoredis.conf
input {
if [type] == "tomcat-access"{
redis{
host => ["10.0.0.124"]
data_type => "list"
port => "6379"
password => "123456"
key => "lhl-tomcat-access"
db => "0"
}}
if [type] == "tomcat-catalina"{
redis{
host => ["10.0.0.124"]
data_type => "list"
port => "6379"
password => "123456"
key => "lhl-tomcat-catalina"
db => "0"
}}
}
output {
if [type] == "tomcat-access" {
elasticsearch{
hosts => ["10.0.0.121:9200"]
index => "lhl-tomcat-access-%{+YYYY.MM.dd}"
}}
if [type] == "tomcat-catalina" {
elasticsearch{
hosts => ["10.0.0.121:9200"]
index => "lhl-tomcat-catalina-%{+YYYY.MM.dd}"
}}
}
检查配置文件
root@logstash1:/etc/logstash/conf.d# /usr/share/logstash/bin/logstash -f logstashtoredis.conf -t
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2022-11-12 06:55:42.597 [main] runner - Starting Logstash {"logstash.version"=>"7.12.1", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.10+9 on 11.0.10+9 +indy +jit [linux-x86_64]"}
[WARN ] 2022-11-12 06:55:43.104 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2022-11-12 06:55:44.729 [LogStash::Runner] Reflections - Reflections took 30 ms to scan 1 urls, producing 23 keys and 47 values
[WARN ] 2022-11-12 06:55:45.049 [LogStash::Runner] elasticsearch - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2022-11-12 06:55:45.085 [LogStash::Runner] elasticsearch - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
Configuration OK
[INFO ] 2022-11-12 06:55:45.092 [LogStash::Runner] runner - Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
重启服务
root@web1:/etc/logstash/conf.d# systemctl restart logstash.service 五、查看redis数据是否被取走并查看es是否生成索引
数据已被logstash取走
查看es是是否生成索引
六、最后使用kibana匹配索引，并查看消息
到最后redis可能存储数据较多，出现内存占用较大，可以查看一下redis挤压了多少数据，如果数据较大那就增加logstash，让多个logstash去redis中获取数据。写一个脚本查看redis中的key的数据量。