快速检索:      
引用本文:
【打印本页】   【下载PDF全文】   查看/发表评论  【EndNote】   【RefMan】   【BibTex】
←前一篇|后一篇→ 过刊浏览    高级检索
本文已被:浏览 316次   下载 270 本文二维码信息
码上扫一扫!
分享到: 微信 更多
基于Hive的性能优化研究
王康1, 陈海光1, 李东静2
1.上海师范大学 信息与机电工程学院, 上海 200234;2.南京航空航天大学 计算机科学与技术学院, 南京 211106
摘要:
主要从MapReduce作业调度和Hive性能调优两个方面对Hive的性能优化进行研究.对于MapReduce主要从编程模型切入,分析其执行过程,并从map端、reduce端进行参数调优.接着从Hive框架角度入手,分别从分区表和外部表以及常用数据文件的压缩、行式存储与列式存储等方面进行深入研究.实验结果表明,snappy压缩、orcfile/parquet存储格式对于列式查询,提高查询效率,对于大数据分析平台有较好的兼容性.
关键词:  数据仓库  作业调优  性能优化  压缩  存储格式
DOI:10.3969/J.ISSN.1000-5137.2017.04.011
分类号:TP301
基金项目:
Performance optimization research based on Hive
Wang Kang1, Chen Haiguang1, Li Dongjing2
1.The College of Information, Mechanical and Electrical Engineering, Shanghai Normal University, Shanghai 200234, China;2.College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing 211106, China
Abstract:
This paper research Hive performance optimization mainly from the two aspects of MapReduce scheduling and Hive performance tuning.MapReduce's programming model and its implementation process is analyzed,and parameters are tuned from the map side and reduce side.Then Hive's framework is researched from the aspects of the partition table,the external surface and common data file compression,the line storage and column type storage.The experimental results show that snappy compression and orcfile/parquet storage format can improve the efficiency of query for the column type queries, and has good compatibility.
Key words:  data warehouse  job optimization  performance optimization  compression  storage format