BDE 项目地址： https://github.com/potonix/bde-over-bigip   。

BDE是基于 ELK 的大数据引擎，用于分析和可视化 BIG-IP产生的数据，本文主要是探究 BDE 各组件间传输数据时，对传输数据内容有无修改，便于我们更好的使用这些组件。

一条由BIG-IP 吐出的数据在BDE 中，主要流动路径是BIG-IP -> FLUENTD -> KAFKA -> LOGSTASH -> Elasticsearch ，下面我们通过tcpdump+wireshark 抓包的形式，看一下数据从传入FLUENTD 到流出LOGSTASH 变化情况。

用到的 tcpdump 命令为：tcpdump -i eh0 -w test.pcap，pcap文件直接用 vim 打开会乱码，需要将得到的 pcap文件放入 wireshark 进行分析。

1. 生成数据

使用python 脚本在host 中模拟BIG-IP 生成json 数据，通过udp 发送给FLUENTD 。

2. host -> Fluentd

host(centos): 172.20.0.1

Fluentd(docker): 172.20.0.7

抓包的数据如下：

{"stdout":"OK","server_local_port":84,"latency":3,"resp-status":404,"user-agent":"Gecko/20020508 Netscape6/6.1","delay_type":"init-delay","method":"POST","username":"zongzhaowei","client-ip":"187.7.30.161","server_local":"bigip-serverside-ip","timestamp":"2020-04-28T01:45:36.000Z","host":"bigip-vs-server","server_remote":"10.250.11.24","cookie":"","cdnumber":4,"delay_value":8,"sender":"zongzw 18","transmission":{"resource_name":"/pages/344","resource_size":344},"server_remote_port":80,"uri":"/pages/344","vs_name":"/Common/vs-l4-84","client_local":"bigip-clientside-ip","server-ip":"10.250.11.24","content-type":"application/json","client_remote_port":34345}

3. Fluentd -> KAFKA

KAFKA(docker): 172.20.0.4

抓包数据如下（去除了数据中的乱码），其中’general-topic’ 是在 fluentd 中为kafka 设置的topic名称，主体json文件内容无改变：

fluentd

general-topic

{"stdout":"OK","server_local_port":84,"latency":3,"resp-status":404,"user-agent":"Gecko/20020508 Netscape6/6.1","delay_type":"init-delay","method":"POST","username":"zongzhaowei","client-ip":"187.7.30.161","server_local":"bigip-serverside-ip","timestamp":"2020-04-28T01:45:36.000Z","host":"bigip-vs-server","server_remote":"10.250.11.24","cookie":"","cdnumber":4,"delay_value":8,"sender":"zongzw 18","transmission":{"resource_name":"/pages/344","resource_size":344},"server_remote_port":80,"uri":"/pages/344","vs_name":"/Common/vs-l4-84","client_local":"bigip-clientside-ip","server-ip":"10.250.11.24","content-type":"application/json","client_remote_port":34345}

4. KAFKA -> LOGSTASH

LOGSTASH(docker): 172.29.0.5

抓包数据如下，和原始数据保持一致：

{"stdout":"OK","server_local_port":84,"latency":3,"resp-status":404,"user-agent":"Gecko/20020508 Netscape6/6.1","delay_type":"init-delay","method":"POST","username":"zongzhaowei","client-ip":"187.7.30.161","server_local":"bigip-serverside-ip","timestamp":"2020-04-28T01:45:36.000Z","host":"bigip-vs-server","server_remote":"10.250.11.24","cookie":"","cdnumber":4,"delay_value":8,"sender":"zongzw 18","transmission":{"resource_name":"/pages/344","resource_size":344},"server_remote_port":80,"uri":"/pages/344","vs_name":"/Common/vs-l4-84","client_local":"bigip-clientside-ip","server-ip":"10.250.11.24","content-type":"application/json","client_remote_port":34345}

5. LOGSTASH -> Elasticsearch

Elasticsearch(docker) ：172.20.0.3

抓包数据如下，分为两个 json格式数据。第一个 json是 Logstash设置的关于 index的信息；第二个 json是经 Logstash处理过的数据，其中选中的 message字段是原生模拟产生的 json数据，其余字段都是根据 Logstash配置文件规则进行产生的：

6. 总结

json 数据从Fluentd 到kafka 再到logstash 都没有变化，使用的都是tcp ；然后在logstash 中原生json 变为新json 的message 字段（新json 还有其他logstash 自带或者用户定义的字段），再通过http 协议发送到ES 。