一种基于Thrift的日志收集分析系统

传统的方式是将日志记录到本地日志文件中,然后使用日志分析程序进行分析。而 PaaS 平台运行着由不同编程语言和开发框架编写的应用,其记录日志的方式不尽相同,使日志分析程序的通用性不佳;一个应用又会有多个实例在运行,导致同一个应用的日志分散在多个地方,不便于管理。本文设计了一种简单灵活的日志格式,使用 Thrift 作为远程调用框架,支持多种编程语言调用服务,将日志数据统一在 MongoDB 中进行存取和查询。开发者还可以指定字段进行统计,直接生成图表。系统部署则采用 Docker 容器的方式,可以方便地进行水平扩展。实际运行表明,运用相关技术构建的日志收集分析系统易于使用,功能丰富,能够灵活扩展,很好地满足了 PaaS 平台的需求。

A Log Collection and Analysis System Based on Thrift

Logging the records to local files and analyzing them with analysis programs are the traditional way. However, on PaaS, the diversity of log records the applications developed with varies programming languages makes the log analysis programs losing the generality. At the same time, the instances of each application generate log files in different machines, which increases the complexity of management. The log collection and analysis system supports for multiple languages to invocate the log service over Thrift remote procedure call with a flexible log format. Log data are stored in MongoDB. Developers can generate charts and graphs directly with the specified fields. The system can easily scaled out deploying with Docker containers. The experiment indicates that with these techniques, the log collection and analysis system is easy to use, feature-rich, flexible to expand, satisfying the needs of PaaS well.