UniDAC源码包在Delphi/C++Builder中的编译安装全攻略
2026/9/8 10:14:54
ELFK + kafka 架构部署
内存紧张，修改sever1、serever2、server3配置
[root@server1 ~]# cd /etc/elasticsearch/
[root@server1 elasticsearch]# vim elasticsearch.yml
bootstrap.memory_lock: false
[root@server1 elasticsearch]# vim jvm.options
-Xms512m
-Xmx512m
[root@server1 elasticsearch]# systemctl restart elasticsearch
server2、server3做同样配置
集群部署
再开启三台虚拟机server6、server7、server8
基础配置
互相解析
三台主机安装 JDK 环境
浏览器下载jdk-17.0.17_linux-x64_bin.rpm
[root@server6 ~]# rpm -ivh jdk-17.0.17_linux-x64_bin.rpm
[root@server6 ~]# java -version
三台主机安装 Kafka
浏览器下载kafka_2.12-3.9.1.tgz，https://kafka.apache.org/downloads
[root@server6 ~]# tar zxf kafka_2.12-3.9.1.tgz -C /opt
[root@server6 ~]# mv /opt/kafka_2.12-3.9.1 /opt/kafka
[root@server6 ~]# cd /opt/kafka/config/kraft
[root@server6 kraft]# cp server.properties{,.bak}备份配置文件
[root@server6 kraft]# mkdir /opt/kafka/logs 创建日志目录
三台主机 Kafka 配置文件修改
[root@server6 kraft]# vim server.properties
process.roles=broker,controller
node.id=176
controller.quorum.voters=176@192.168.164.176:9093,177@192.168.164.177:9093,178@192.168.164.178:9093
listeners=PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093
inter.broker.listener.name=PLAINTEXT
advertised.listeners=PLAINTEXT://192.168.164.176:9092,CONTROLLER://192.168.164.176:9093
controller.listener.names=CONTROLLER
listener.security.protocol.map=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL
num.network.threads=2
num.io.threads=2
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/opt/kafka/logs
num.partitions=3
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=3
transaction.state.log.replication.factor=3
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
注意server2、server3配置区别在于node.id、advertised.listeners
Kafka 集群初始化
server6 节点生成储目录唯一的 UUID
[root@server6 kraft]# KAFKA_CLUSTER_ID="$(/opt/kafka/bin/kafka-storage.sh random-uuid)"
[root@server6 kraft]# echo ${KAFKA_CLUSTER_ID}
K9eaFC2PSDSKruBsoJ-oTw
三台主机用该 uuid 格式化 kafka 存储目录
注意uuid
[root@server6 kraft]# /opt/kafka/bin/kafka-storage.sh format -t K9eaFC2PSDSKruBsoJ-oTw -c /opt/kafka/config/kraft/server.properties
[root@server7 kraft]# /opt/kafka/bin/kafka-storage.sh format -t K9eaFC2PSDSKruBsoJ-oTw -c /opt/kafka/config/kraft/server.properties
[root@server8 kraft]# /opt/kafka/bin/kafka-storage.sh format -t K9eaFC2PSDSKruBsoJ-oTw -c /opt/kafka/config/kraft/server.properties
三台节点启动 Kafka
server6上
[root@server6 kraft]# /opt/kafka/bin/kafka-server-start.sh -daemon /opt/kafka/config/kraft/server.properties
[root@server6 kraft]# jps
4261 Kafka
4332 Jps
server7上
[root@server7 kraft]# /opt/kafka/bin/kafka-server-start.sh -daemon /opt/kafka/config/kraft/server.properties
[root@server7 kraft]# jps
6039 Kafka
6109 Jps
server8上
[root@server8 kraft]# /opt/kafka/bin/kafka-server-start.sh -daemon /opt/kafka/config/kraft/server.properties
[root@server8 kraft]# jps
3667 Kafka
3688 Jps
Kafka 集群可用性验证
Kafka 集群任意节点创建 Topic
[root@server6 kraft]# /opt/kafka/bin/kafka-topics.sh --create --topic testtopics --bootstrap-server localhost:9092 --replication-factor 3 --partitions 3
Created topic testtopics.
--replication-factor 3：设置主题的副本因子为 3，即每个分区的副本数量为 3。副本用于数据冗余和高可用性，副本数需不能超过集群中 Broker 的总数。
--partitions 3：设置主题的分区数为 3。分区是 Kafka 并行处理消息的基本单位，影响吞吐量和扩展性。分区数一旦创建不可减少。
Kafka 集群任意节点查看 Topic
[root@server7 kraft]# /opt/kafka/bin/kafka-topics.sh --list --bootstrap-server localhost:9092
testtopics
Kafka 集群任意节点生产者测试
[root@server8 kraft]# /opt/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic testtopics
>111
>222
>333
Kafka 集群任意节点查看主题
[root@server7 kraft]# /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic testtopics --from-beginning
111
222
333
Processed a total of 3 messages
Kafka 集群任意节点查看 topic 详细信息
[root@server6 kraft]# /opt/kafka/bin/kafka-topics.sh --bootstrap-server server7:9092,server8:9092,server9:9092 --topic testtopics --describe
FIlebeat 配置文件修改
server4上
[root@server4 ~]# vim /etc/filebeat/filebeat.yml 注释掉logstash部分
#----------------------------- Kafka output --------------------------------
output.kafka:
hosts: ["192.168.164.176:9092", "192.168.164.177:9092", "192.168.164.178:9092"]
topic: "apache-logs"
partition.round_robin:
reachable_only: true
required_acks: 1
compression: gzip
[root@server4 ~]# filebeat test config -c /etc/filebeat/filebeat.yml
Config OK
[root@server4 ~]# filebeat test output -c /etc/filebeat/filebeat.yml
[root@server4 ~]# systemctl restart filebeat.service
Kafka 集群任意节点消费者测试
[root@server6 ~]# /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic apache-logs --from-beginning
Logstash 消费 Topic 消息
[root@server4 modules.d]# cd
[root@server5 conf.d]# cd /etc/logstash/conf.d
[root@server5 conf.d]# vim kafka.conf
input {
kafka {
bootstrap_servers => "192.168.164.176:9092,192.168.164.177:9092,192.168.164.178:9092"
topics => ["apache-logs"]
group_id => "logstash-consumer-group"
consumer_threads => 3
auto_offset_reset => "earliest"
codec => "json"
}
}
filter {
grok {
match => { "message" => "%{HTTPD_COMBINEDLOG}" }
}
}
output {
#stdout {}
elasticsearch {
hosts => ["192.168.164.171:9200","192.168.164.172:9200","192.168.164.173:9200"]
index => "apachelog-%{+YYYY.MM.dd}"
user => "elastic"
password => "westos"
}
}
[root@server5 conf.d]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/kafka.conf