微服务架构下的根因定位方法综述

doi:10.11871/jfdc.issn.2096-742X.2022.03.006

数据与计算发展前沿 ›› 2022, Vol. 4 ›› Issue (3): 78-89.

CSTR: 32002.14.jfdc.CN10-1649/TP.2022.03.006

doi: 10.11871/jfdc.issn.2096-742X.2022.03.006

• 专刊:先进智能计算平台及应用(下) • 上一篇下一篇

微服务架构下的根因定位方法综述

李思毅(),马诗雨(),崔丽月(),张圣林(),孙永谦(),张玉志()

南开大学,软件学院,天津 300350

收稿日期:2022-02-24 出版日期:2022-06-20 发布日期:2022-06-20
通讯作者: 张圣林
作者简介:李思毅,南开大学,软件学院,硕士研究生,研究方向为机器学习、智能运维等。
本文中主要负责基于根因分析方法和实践论文撰写。
LI Siyi is a master’s student in the College of Software at Nan-kai University. His research interests include machine learning and AIOps.
In this paper, he is responsible for the method and practical part of root cause analysis.
E-mail: lisiyimail@qq.com|马诗雨,南开大学,软件学院,本科生,研究方向为机器学习、智能运维等。
本文中主要负责关系学习方法和实践部分论文撰写。
MA Shiyu is an undergraduate student in the College of Soft-ware at Nankai University. Her research interests include mach-ine learning and AIOps.
In this paper, she is responsible for the method and practical part of the relational learning method.
E-mail: nkcs77@163.com|崔丽月,南开大学,软件学院,硕士研究生,研究方向为机器学习、智能运维等。
本文中主要负责文献的调研和论文修改。
CUI Liyue is a master’s student in the College of Software at Nankai University. Her research inter-ests include machine learning and AIOps.
In this paper, she is responsible for the related work investi-gation and paper revision.
E-mail: 2320190026@mail.nankai.edu.cn|张圣林,南开大学,软件学院,博士,副教授,主要研究数据中心网络中的故障检测、诊断和预测。发表SCI/EI 收录论文 15 篇以上。
本文中承担文献调研及指导。
ZHANG Shenglin, Ph.D., is currently an associate professor in the College of Software, Nankai University, Tianjin, China. His current rese-arch interests include failure detection, diagnosis and prediction in data center networks. He has published 15 papers that are in-dexed by SCI/EI.
In this paper, he is responsible for guidance and related work investigation.
E-mail: zhangsl@nankai.edu.cn|孙永谦,南开大学,软件学院,博士,讲师,主要研究异常检测、根本原因定位以及数据中心的高性能切换。
本文中主要设计文章架构并修改论文。
SUN Yongqian, Ph.D., is currently an assistant professor in the College of Software, Nankai Univer-sity, Tianjin, China. His research interests include anomaly detection, root cause localization, and high-performance swit-ching in datacenter.
In this paper, he is mainly responsible for the design of the paper architecture and paper revision.
E-mail: sunyongqian@nankai.edu.cn|张玉志,南开大学,软件学院,院长,博士,讲席教授,研究方向为人工智能。
本文中承担文献调研。
ZHANG Yuzhi, Ph.D., is currently a distinguished professor and the dean of the College of Software, Nankai University. His research interests include deep learning and other aspects in artificial intelligence.
In this paper, he is responsible for the related work investi-gation.
E-mail: zyz@nankai.edu.cn
基金资助:
国家重点研发计划(2021YFB0300104);国家自然科学基金青年项目(61902200);中国博士后科学基金面上项目(2019M651015)

Overview of Root Cause Localization Method in Microservice Architecture

LI Siyi(),MA Shiyu(),CUI Liyue(),ZHANG Shenglin(),SUN Yongqian(),ZHANG Yuzhi()

College of Software, Nankai University, Tianjin 300350, China

Received:2022-02-24 Online:2022-06-20 Published:2022-06-20
Contact: ZHANG Shenglin

摘要/Abstract

摘要：

【目的】在大规模云平台中,当微服务系统关键性能指标发生异常,要求运维人员面对告警风暴和纷繁复杂的异常指标及时梳理背后的异常关联,对异常进行准确的根因定位和快速的恢复。【方法】本文详细介绍在微服务架构下构建故障传播图的方式以及基于图推理的根因定位技术。结合云平台上运维及高可用的能力建设经验,对现有的根因定位方法进行梳理、总结。【结果】基于图推理的根因定位方法在大型数据中心显著提高了云上系统稳定性、可靠性。【局限】该方法依赖稳定的监控基础设施以及准确的指标异常检测能力。【结论】随着数字化转型的深入,微服务架构下的根因定位技术对大规模云平台的稳定性保障将会起到越来越大的作用。

关键词: 云原生, 微服务, 智能运维, 根因定位

Abstract:

[Objective] When the key performance indicators of the cloud native system are abnormal, operation and maintenance engineers are required to sort out the abnormal correlation behind the alarm storm and complex abnormal indicators in a timely manner, and perform accurate root cause localization and rapid recovery. [Methods] This paper introduces the way to build a fault propagation graph under a large-scale microservice architecture and a root cause localization technology based on graph reasoning. We investigate, compare and summarize the existing root cause localization methods based on the experience of operation and maintenance of the cloud platform and high-availability capacity building. [Results] The root cause localization method based on graph reasoning significantly improves the stability and reliability of the cloud system in large data centers. [Limitations] This method relies on a stable monitoring infrastructure and accurate anomaly detection capabilities for indicators. [Conclusions] With the deepening of digital transformation, the root cause localization technology of microservice indicators under the microservice architecture will play an increasingly important role in ensuring the stability of large-scale cloud platforms.

Key words: cloud native, microservices, AIOps, root cause localization

李思毅,马诗雨,崔丽月,张圣林,孙永谦,张玉志. 微服务架构下的根因定位方法综述[J]. 数据与计算发展前沿, 2022, 4(3): 78-89.

LI Siyi,MA Shiyu,CUI Liyue,ZHANG Shenglin,SUN Yongqian,ZHANG Yuzhi. Overview of Root Cause Localization Method in Microservice Architecture[J]. Frontiers of Data and Computing, 2022, 4(3): 78-89, https://cstr.cn/32002.14.jfdc.CN10-1649/TP.2022.03.006.

图/表 6

图1

图2

图3

图4

图5

表1

参考文献 25

[1]	蒋易民, 王恒, 胡泉, 于永恒, 蒋维杰, 杨祥合. 金融级IT架构[M]. 北京: 电子工业出版社, 2021:1-22.
[2]	Balalaie A, Heydarnoori A, Jamshidi P. Microservices architecture enables devops: Migration to a cloud-native architecture[J]. Ieee Software, 2016, 33(3): 42-52. doi: 10.1109/MS.2016.64
[3]	James Lewis, Martin Fowler. Microservices[EB/OL]. [2022-02-14]. https://martinfowler.com/articles/microser-vices.html.
[4]	中国信息通信研究院. 云计算发展白皮书(2018年)[EB/OL].[2022-02-14]. http://www.caict.ac.cn/kxyj/qwfb/bps/201808/P020180813540725575770.pdf.
[5]	Li W, Lemieux Y, Gao J, et al. Service mesh: Challenges, state of the art, and future research opportunities[C]// 2019 IEEE International Conference on Service-Oriented Sys-tem Engineering (SOSE), IEEE, 2019: 122-1225.
[6]	Brewer E A. Kubernetes and the path to cloud native[C]// Proceedings of the sixth ACM symposium on cloud com-puting, 2015: 167. https://kubernetes.io/docs/concepts/.
[7]	Keller A, Subramanian S. Best practices for deploying a CMDB in large-scale environments[C]// 2009 IFIP/IEEE International Symposium on Integrated Network Management. IEEE, 2009: 732-745.
[8]	Kalisch M, Bühlman P. Estimating high-dimensional directed acyclic graphs with the PC-algorithm[J]. Journal of Machine Learning Research, 2007, 8(3):613-636.
[9]	Tarjan R. Depth-first search and linear graph algorithms[J]. SIAM journal on computing, 1972, 1(2): 146-160. doi: 10.1137/0201010
[10]	Benesty J, Chen J, Huang Y, et al. Pearson correlation coefficient[M]//Noise reduction in speech processing. Springer, Berlin, Heidelberg, 2009: 1-4.
[11]	Xing W, Ghorbani A. Weighted pagerank algorithm[C]. InProceedings. Second Annual Conference on Communi-cation Networks and Services Research, IEEE, 2004:305-314.
[12]	Fouss F, Pirotte A, Renders J M, et al. Random-walk com-putation of similarities between nodes of a graph with application to collaborative recommendation[J]. IEEE Transactions on knowledge and data engineering, 2007, 19(3): 355-369. doi: 10.1109/TKDE.2007.46
[13]	Kim M, Sumbaly R, Shah S. Root cause detection in a service-oriented architecture[J]. ACM SIGMETRICS Performance Evaluation Review, 2013, 41(1): 93-104. doi: 10.1145/2494232.2465753
[14]	Wu L, Tordsson J, Elmroth E, et al. Microrca: Root cause localization of performance issues in microservices[C]// NOMS 2020-2020 IEEE/IFIP Network Operations and Management Symposium, IEEE, 2020: 1-9.
[15]	Jeh G, Widom J. Scaling personalized web search[C]// Proceedings of the 12th international conference on World Wide Web, 2003: 271-279.
[16]	Weng J, Wang J H, Yang J, et al. Root cause analysis of anomalies of multitier services in public clouds[J]. IEEE/ACM Transactions on Networking, 2018, 26(4): 1646-1659. doi: 10.1109/TNET.2018.2843805
[17]	Liu D, He C, Peng X, et al. MicroHECL: high-efficient root cause localization in large-scale microservice syste-ms[C]// 2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), IEEE, 2021: 338-347.
[18]	Wang H, Wu Z, Jiang H, et al. Groot: An Event-graph-based Approach for Root Cause Analysis in Industrial Settings[C]// 2021 36th IEEE/ACM International Confer-ence on Automated Software Engineering (ASE), IEEE, 2021: 419-429.
[19]	Wang P, Xu J, Ma M, et al. Cloudranger: Root cause identification for cloud native systems[C]// 2018 18th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID), IEEE, 2018: 492-502.
[20]	Ma M, Lin W, Pan D, et al. Ms-rank: Multi-metric and self-adaptive root cause diagnosis for microservice appl-ications[C]// 2019 IEEE International Conference on Web Services (ICWS), IEEE, 2019: 60-67.
[21]	Ma M, Lin W, Pan D, et al. ServiceRank: Root Cause Identification of Anomaly in Large-Scale Microservice Architecture[J]. IEEE Transactions on Dependable and Secure Computing, 2021: 1.
[22]	Meng Y, Zhang S, Sun Y, et al. Localizing failure root causes in a microservice through causality inference[C]// 2020 IEEE/ACM 28th International Symposium on Quality of Service (IWQoS), IEEE, 2020: 1-10.
[23]	Nie X, Zhao Y, Sui K, et al. Mining causality graph for automatic webbased service diagnosis[C]// 2016 IE-EE 35th International Performance Computing and Commu-nications Conference (IPCCC), IEEE, 2016: 1-8.
[24]	Lin J J, Chen P, Zheng Z. Microscope: Pinpoint perfor-mance issues with causal graphs in microservice envir-onments[C]// International Conference on ServiceOriented Computing, Springer, Cham, 2018: 3-20.
[25]	Chen P, Qi Y, Zheng P, et al. Causeinfer: Automatic and distributed performance diagnosis with hierarchical ca-usality graph in large distributed systems[C]// IEEE INF-OCOM 2014-IEEE Conference on Computer Communi-cations, IEEE, 2014:1887-1895.

论文	关系学习方法	根因分析方法
MonitorRank^[13]	系统信息	随机游走
MicroRCA^[14]	系统信息	PersonalizedPageRank^[15]
TON18^[16]	系统信息	随机游走
MicroHECL^[17]	系统信息	皮尔森相关系数
Groot^[18]	系统信息	PageRank
CloudRanger^[19]	PC算法	二阶随机游走
MSRank^[20]	PC算法	二阶随机游走
ServiceRank^[21]	PC算法	皮尔森相关系数
MicroCause^[22]	PCTS	TCORW
BRCA^[23]	相关系数分析	深度优先搜索
MicroScope^[24]	PC算法+系统信息	皮尔森相关系数
CauseInfer^[25]	PC算法+系统信息	深度优先搜索

微服务架构下的根因定位方法综述

Overview of Root Cause Localization Method in Microservice Architecture

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 6

参考文献 25

相关文章 1

编辑推荐

Metrics

本文评价