独立模式安装Hive
在三台CentOS虚拟机server1、server2、 server3上已经成功安装全分布模式的Hadoop集群,其中server1为NameNode, server2、server3为DataNode。本人在server1即NameNode上安装Hive,并使用MySQL数据库存放metadata。
1.首先配置MySQL
安装MySQL
[root@server1 ~]# yum install mysql mysql -server
一路选择yes就行
添加MySQL服务
[root@server1 ~]# /sbin/chkconfig --add mysqld
启动MySQL
[root@server1 ~]# service mysqld start
Starting mysqld: [ OK ]
用root账户在本地登录MySQL
[root@server1 ~]# mysql -u root
出现欢迎页面,进入MySQL monitor.
创建数据库实例hive
mysql > CREATE DATABASE hive;
创建用户hive
mysql > CREATE USER 'hive' IDENTIFIED BY 'hive';
给用户hive赋予相应的访问与读写权限
mysql > GRANT ALL ON hive.* TO hive@localhost ;
2.配置Hive
下载hive-0.8.1,并解压
[admin@server1 hive-0.8.1]$ pwd
/home/admin/hive-0.8.1
修改系统环境变量,在path中加入Hive执行路径
[root@server1 ~]# vim /etc/profile
...................
export HIVE_HOME=/home/admin/hive-0.8.1
export PATH=$PATH:$HIVE_HOME/bin
应用修改
[root@server1 ~]# source /etc/profile
下载MySQL的JDBC驱动包MySQL-connector-java-5.1.18-bin.jar,复制到hive-0.8.1的lib目录下
将Hive的conf下的文件hive-default.xml.template复制一份,重命名为hive-site.xml
[admin@server1 conf]$ cp hive-default.xml.template hive-site.xml
将复制的hive-site.xml 移动到所安装的Hadoop的conf目录下
[admin@server1 conf]$ mv hive-site.xml /home/admin/hadoop-0.20.2/conf
修改hive-site.xml,配置,主要修改以下属性
所连接的MySQL数据库实例
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://localhost:3306/hive</value>
<description>JDBC connect string for a JDBC metastore</description>
</property>
连接的MySQL数据库驱动
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
<description>Driver class name for a JDBC metastore</description>
</property>
连接的MySQL数据库用户名
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>hive</value>
<description>username to use against metastore database</description>
</property>
连接的MySQL数据库密码
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>hive</value>
<description>password to use against metastore database</description>
</property>
2.测试Hive
在控制台键入Hive,进入Hive操作界面
[admin@server1 ~]$ hive
Logging initialized using configuration in jar:file:/home/admin/hive-0.8.1/lib/hive-common-0.8.1.jar!/hive-log4j.properties
Hive history file=/tmp/admin/hive_job_log_admin_201212011113_1138680566.txt
hive>
显示存在的表:
hive> SHOW TABLES;
OK
Time taken: 0.063 seconds
建立表records:
hive> CREATE TABLE records (year STRING, temperature INT, quality INT)
> ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t';
OK
Time taken: 0.253 seconds
显示存在的表,多了一个records:
hive> SHOW TABLES;
OK
records
Time taken: 0.089 seconds
查看表records的定义
hive> DESCRIBE records;
OK
year string
temperature int
quality int
Time taken: 0.139 seconds
向表records导入数据
hive> LOAD DATA INPATH '/user/admin/in/ncdc/micro/sample.txt'
> INTO TABLE records;
Loading data to table default.records
OK
Time taken: 0.337 seconds
查看表records的数据
hive> SELECT * FROM records;
OK
1950 0 1
1950 22 1
1950 -11 1
1949 111 1
1949 78 1
Time taken: 0.264 seconds
计算records中每一年的最高温度
hive> SELECT year, MAX(temperature) FROM records GROUP BY year;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks not specified. Estimated from input data size: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Starting Job = job_201211240040_0037, Tracking URL = http://server1:50030/jobdetails.jsp?jobid=job_201211240040_0037
Kill Command = /home/admin/hadoop-0.20.2/bin/../bin/hadoop job -Dmapred.job.tracker=server1:9001 -kill job_201211240040_0037
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1
2012-12-01 11:30:57,089 Stage-1 map = 0%, reduce = 0%
2012-12-01 11:31:15,188 Stage-1 map = 100%, reduce = 0%
2012-12-01 11:31:24,235 Stage-1 map = 100%, reduce = 100%
Ended Job = job_201211240040_0037
MapReduce Jobs Launched:
Job 0: Map: 1 Reduce: 1 HDFS Read: 51 HDFS Write: 17 SUCESS
Total MapReduce CPU Time Spent: 0 msec
OK
1949 111
1950 22
Time taken: 47.238 seconds