mysql与hdfs的数据迁移（sqoop）

文章目录

• ​​1、查看mysql数据​​
• ​​2、查看mysql表​​
• ​​3、mysql表导入到hdfs​​
• ​​4、查看hdfs​​
• ​​5、清空dept表​​
• ​​6、数据从hdfs导出到MySQL表​​
• ​​7、查询dept表​​
• ​​8、Sqoop Import 增量导入到 HDFS​​

打开一个 dektop 终端，不要打开多个终端，所用数据看上一篇

1、查看mysql数据

hadoop@ddai-desktop:~$ sqoop list-databases --connect jdbc:mysql://ddai-master:3306/ --username sqoop --password Dai@123456

2、查看mysql表

hadoop@ddai-desktop:~$ sqoop list-tables --connect jdbc:mysql://ddai-master:3306/sqoop --username sqoop --password Dai@123456

3、mysql表导入到hdfs

打开hadoop，用上一篇创建的表

hadoop@ddai-desktop:~$ sqoop import --connect jdbc:mysql://ddai-master:3306/sqoop --username sqoop --password Dai@123456 --table dept -m 1 --target-dir /user/dept

4、查看hdfs

hadoop@ddai-desktop:~$ hdfs dfs -ls /user/dept

hadoop@ddai-desktop:~$ hdfs dfs -cat /user/dept/part-m-00000

5、清空dept表

hadoop@ddai-desktop:~$ mysql -hddai-master -usqoop -pDai@123456 
mysql> use sqoop;
mysql> truncate dept;

6、数据从hdfs导出到MySQL表

hadoop@ddai-desktop:~$ sqoop export --connect jdbc:mysql://ddai-master:3306/sqoop --username sqoop --password Dai@123456 --table dept -m 1 --export-dir /user/dept

7、查询dept表

hadoop@ddai-desktop:~$ mysql -hddai-master -usqoop -pDai@123456
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 24
Server version: 8.0.26-0ubuntu0.20.04.2 (Ubuntu)

Copyright (c) 2000, 2021, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> use sqoop;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> select * from dept;
+------+-----------+
| id   | name      |
+------+-----------+
| 1245 | 云计算    |
| 3512 | 物联网    |
| 3545 | 大数据    |
+------+-----------+
3 rows in set (0.00 sec)

mysql> quit
Bye

8、Sqoop Import 增量导入到 HDFS

hadoop@ddai-desktop:~$ mysql -hddai-master -usqoop -pDai@123456 
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 25
Server version: 8.0.26-0ubuntu0.20.04.2 (Ubuntu)

Copyright (c) 2000, 2021, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> use sqoop;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> insert into dept values(7854,'今天是2021七夕');
Query OK, 1 row affected (0.00 sec)

mysql> select * from dept;
+------+---------------------+
| id   | name                |
+------+---------------------+
| 1245 | 云计算              |
| 3512 | 物联网              |
| 3545 | 大数据              |
| 7854 | 今天是2021七夕      |
+------+---------------------+
4 rows in set (0.00 sec)

mysql> quit
Bye

hadoop@ddai-desktop:~$ sqoop import --connect jdbc:mysql://ddai-master:3306/sqoop --username sqoop --password Dai@123456 --table dept -m 1 --target-dir /user/dept -incremental append --check-column id