1004-Hive的加载数据到hdfs和mysql上案例介绍

1、登录hive，然后创建t_order

hive> create table t_order(id int ,name string)
    > row format delimited
    > fields terminated by '\t';

2、准备基础数据

[hadoop@cloud04 ~]$ pwd
 /home/hadoop 
 
[hadoop@cloud04 ~]$ more order.log
 1     apple
 2     oracle
 3     banana

3、加载本地数据/home/hadoop/order.log到表t_order

hive> load data local inpath '/home/hadoop/order.log' into table t_order;
 Copying data from file:/home/hadoop/order.log
 Copying file: file:/home/hadoop/order.log
 Loading data to table jf01.t_order
 Table jf01.t_order stats: [num_partitions: 0, num_files: 1, num_rows: 0, total_size: 26, raw_data_size: 0]
 OK
 Time taken: 2.285 seconds

4、验证数据

4.1 登录hdfs，然后查看在hdfs上存储列表为：

[hadoop@cloud01 ~]$ hadoop fs -cat /user/hive/warehouse/jf01.db/t_order/order.log
 1     apple
 2     oracle
 3     banana

4.2 登录mysql验证数据

hive> select * from t_order;
 OK
 1     apple
 2     oracle
 3     banana
 Time taken: 0.149 seconds, Fetched: 3 row(s)

5、同等格式的文件上传至 /user/hive/warehouse/jf01.db/t_order，然后查询t_order 表

[hadoop@cloud04 ~]$ more order2.log 
 11     apple1
 21     oracle1
 31     banana1

[hadoop@cloud04 ~]$ hadoop fs -put order2.log  /user/hive/warehouse/jf01.db/t_order

5.1 查看所有的数据情况不转为M/R程序

hive> select * from t_order;
 OK
 1     apple
 2     oracle
 3     banana
 11     apple1
 21     oracle1
 31     banana1

5.2 查看指定的字段需要转为MapReduce程序

hive> select id,name from t_order;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1430138571101_0001, Tracking URL = http://cloud03:8088/proxy/application_1430138571101_0001/
Kill Command = /home/hadoop/app/hadoop-2.4.1/bin/hadoop job  -kill job_1430138571101_0001
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
2015-04-27 09:28:38,694 Stage-1 map = 0%,  reduce = 0%
2015-04-27 09:28:49,359 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 0.76 sec
2015-04-27 09:28:50,414 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 0.76 sec
MapReduce Total cumulative CPU time: 760 msec
Ended Job = job_1430138571101_0001
MapReduce Jobs Launched:
Job 0: Map: 1   Cumulative CPU: 0.76 sec   HDFS Read: 342 HDFS Write: 58 SUCCESS
Total MapReduce CPU Time Spent: 760 msec
OK
1     apple
2     oracle
3     banana
11     apple1
21     oracle1
31     banana1
Time taken: 28.233 seconds, Fetched: 6 row(s)
hive> select name from t_order;Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1430138571101_0002, Tracking URL = http://cloud03:8088/proxy/application_1430138571101_0002/
Kill Command = /home/hadoop/app/hadoop-2.4.1/bin/hadoop job  -kill job_1430138571101_0002
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
2015-04-27 09:29:17,821 Stage-1 map = 0%,  reduce = 0%
2015-04-27 09:29:26,376 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 0.9 sec
2015-04-27 09:29:27,465 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 0.9 sec
2015-04-27 09:29:28,512 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 0.9 sec
MapReduce Total cumulative CPU time: 900 msec
Ended Job = job_1430138571101_0002
MapReduce Jobs Launched:
Job 0: Map: 1   Cumulative CPU: 0.9 sec   HDFS Read: 342 HDFS Write: 43 SUCCESS
Total MapReduce CPU Time Spent: 900 msec
OK
apple
oracle
banana
apple1
oracle1
banana1
Time taken: 17.167 seconds, Fetched: 6 row(s)