| Issue |
MATEC Web Conf.
Volume 76, 2016
20th International Conference on Circuits, Systems, Communications and Computers (CSCC 2016)
|
|
|---|---|---|
| Article Number | 04040 | |
| Number of page(s) | 6 | |
| Section | Computers | |
| DOI | https://doi.org/10.1051/matecconf/20167604040 | |
| Published online | 21 October 2016 | |
Rack-Scale Storage Fabric: A Practical Way to Build Best-Fit Infrastructure for High-Performance Data Processing
1 Baidu. No. 10, Xibeiwang Road East, Beijing, PRC.
2 Intel APAC R&D Co. Ltd. No. 880, Zixing Road, Shanghai, PRC.
a Corresponding author: albert.hu@intel.com
+86-10-5679-5465
+86-21-6116-5000
This paper is to address the resource utilization problem for high-performance data processing applications in a large IDC (Internet Data Center) environment. On one hand, each application calls for a best-fit infrastructure with a specific compute-storage ratio, to achieve the highest resource utilization while meeting its performance requirement. And such a ratio varies among applications. On the other hand, IDCs have always been trying to unify the infrastructures for lower TCO (Total Cost of Ownership). Therefore, it’s getting harder and harder to adapt infrastructures to application needs. This issue results in significant waste of investment in large IDCs. Furthermore, the high-performance data processing applications always require the infrastructure to offer as high compute-storage performance as a DAS (Direct Attached Storage) server, which remains as a great challenge when addressing the resource utilization problem. This paper, as part of Baidu-Intel joint research program, first evaluates the state-of-the-art solutions, and then introduces a more practical infrastructure, the core of which is rack-scale storage fabric. This infrastructure disaggregates compute units and storage units by a SAS (Serial Attached SCSI) fabric, and allows to compose logical servers with arbitrary computer-storage ratios within a rack. And the experiments in Baidu’s research environment show that the logical servers exhibit the similar throughput/IOPS as DAS servers, and also their compute-storage ratios can best-fit the needs of different Hadoop applications.
© The Authors, published by EDP Sciences, 2016
This is an Open Access article distributed under the terms of the Creative Commons Attribution License 4.0, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
Current usage metrics show cumulative count of Article Views (full-text article views including HTML views, PDF and ePub downloads, according to the available data) and Abstracts Views on Vision4Press platform.
Data correspond to usage on the plateform after 2015. The current usage metrics is available 48-96 hours after online publication and is updated daily on week days.
Initial download of the metrics may take a while.