| Issue |
MATEC Web Conf.
Volume 173, 2018
2018 International Conference on Smart Materials, Intelligent Manufacturing and Automation (SMIMA 2018)
|
|
|---|---|---|
| Article Number | 01019 | |
| Number of page(s) | 5 | |
| Section | Modeling, Analysis, and Simulation of Intelligent Manufacturing Processes | |
| DOI | https://doi.org/10.1051/matecconf/201817301019 | |
| Published online | 19 June 2018 | |
Adaptive replica consistency policy for Kafka
1
Chongqing University of Posts and Telecommunications, 400065, Nan‘an District, Chongqing, P.R.China
2
Chongqing Mobile Internet Application Engineering Technology Research Center, 400065, Nan‘an District, Chongqing, P.R.China
* Corresponding author : 992828405@qq.com
With the rapid development of the Internet, such as storm, s4, sparkstreaming and other large data real-time computing framework, is widely used in real-time monitoring, real-time recommendation, real-time transaction analysis and other systems for real-time consumption of data streams, Kafka messaging system has been widely deployed. Aiming at the problem that the Kafka cluster needs a lot of network overhead, disk overhead and memory consumption to ensure the reliability of the message, the clustering load is increased, and a replica adaptive synchronization strategy based on the message heat and replica update frequency is proposed. It is proved that the Kafka cluster can guarantee the reliability of the message, and it can significantly reduce the overhead of the resource and improve the throughput of the cluster by using the method of dynamically adjusting the replica synchronization to reduce the system resource consumption while ensuring the reliability of the message. to ensure the system availability and high performance.
© The Authors, published by EDP Sciences, 2018
This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
Current usage metrics show cumulative count of Article Views (full-text article views including HTML views, PDF and ePub downloads, according to the available data) and Abstracts Views on Vision4Press platform.
Data correspond to usage on the plateform after 2015. The current usage metrics is available 48-96 hours after online publication and is updated daily on week days.
Initial download of the metrics may take a while.