0
点赞
收藏
分享

微信扫一扫

python爬虫 - 爬取微博热搜数据

闲嫌咸贤 52分钟前 阅读 0
jenkinsdocker运维

jenkins

Jenkins是一个开源的、提供友好操作界面的持续集成(CI)工具，起源于Hudson（Hudson是商用的），主要用于持续、自动的构建/测试软件项目、监控外部任务的运行（这个比较抽象，暂且写上，不做解释）。Jenkins用Java语言编写，可在Tomcat等流行的servlet容器中运行，也可独立运行。通常与版本管理工具(SCM)、构建工具结合使用。常用的版本控制工具有SVN、GIT，构建工具有Maven、Ant、Gradle。

CI/CD

CI

image-20231108100010503

CD

image-20231108100239312

配置maven（linux）

创建maven文件

mkdir maven

上传maven解压后的程序

[root@localhost apache-maven-3.6.3]# ll
总用量 36
drwxr-xr-x. 2 root root    97 11月  9 10:08 bin
drwxr-xr-x. 2 root root    76 11月  9 10:08 boot
drwxr-xr-x. 3 root root    92 11月  9 10:10 conf
drwxr-xr-x. 4 root root  4096 11月  9 10:08 lib
-rw-r--r--. 1 root root 17504 11月  9 10:08 LICENSE
drwxr-xr-x. 2 root root     6 11月  9 15:41 maven-repository
-rw-r--r--. 1 root root  5141 11月  9 10:08 NOTICE
-rw-r--r--. 1 root root  2612 11月  9 10:08 README.txt

配置settings.xml

<localRepository>../maven-repository</localRepository>

image-20231109154355066

配置maven环境

M2_HOME=/usr/local/software/maven/apache-maven-3.6.3
export PATH=$PATH:${M2_HOME}/bin

让编辑生效

source /etc/profile

测试配置是否生效

[root@localhost conf]# mvn -version
-bash: /usr/local/software/apache-maven-3.6.3/bin/mvn: 权限不够

[root@localhost conf]# chmod a+x /usr/local/software/apache-maven-3.6.3/bin/mvn

[root@localhost apache-maven-3.6.3]# mvn -version
    
Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
Maven home: /usr/local/software/maven/apache-maven-3.6.3
Java version: 1.8.0_341, vendor: Oracle Corporation, runtime: /usr/local/software/jdk/jdk1.8.0_341/jre
Default locale: zh_CN, platform encoding: UTF-8
OS name: "linux", version: "3.10.0-1160.99.1.el7.x86_64", arch: "amd64", family: "unix"

jenkins安装(docker)配置

安装jenkins

查询jenkins
docker search jenkinsci/blueocean

image-20231108100811072

下载 jenkins镜像
docker pull jenkinsci/blueocean

image-20231108101121093

创建挂载文件jenkins
[root@localhost software]# mkdir jenkins
[root@localhost jenkins]# mkdir data jenkins_home
[root@localhost jenkins]# ls
data  jenkins_home
[root@localhost jenkins]# cd jenkins_home/
[root@localhost jenkins_home]# touch docker.sock
[root@localhost jenkins_home]# chmod 777 docker.sock 
[root@localhost jenkins_home]# tree
.
└── docker.sock
创建运行运行容器
docker run -it \
-u root \
--name jenkins \
--privileged \
-p 9055:8080 \
-p 50000:50000 \
--network wn_docker_net \
--ip 172.18.12.60 \
-v /etc/localtime:/etc/localtime \
-v /usr/local/software/jenkins/data:/var/jenkins_home \
-v /usr/local/software/jenkins/jenkins_home/docker.sock:/var/run/docker.sock \
-v /usr/local/software/apache-maven-3.6.3:/usr/local/maven \
-d jenkinsci/blueocean

image-20231108142959702

开放8087,50000端口
[root@localhost jenkins]# firewall-cmd --zone=public --add-port=8087/tcp --permanent 
success
[root@localhost jenkins]# firewall-cmd --zone=public --add-port=50000/tcp --permanent 
success
[root@localhost jenkins]# firewall-cmd --reload 
success

初次访问

访问管理页面

image-20231108143541342

查询jenkins的初始密码

image-20231108145119118

输入密码

image-20231108145155671

安装插件

image-20231108145326094

image-20231108145356497

image-20231108145451785

点击继续

创建第一个管理员用户

image-20231109140225123

配置实例

image-20231108150138670

重启应用

image-20231108150214874

后台手动重启

[root@localhost jenkins]# docker restart jenkins
登录系统

image-20231108153144059

image-20231108155052646

配置系统

升级最新版本

image-20231108155840689

image-20231108160008990

image-20231108161801809

image-20231108161843112

重启jenkins

image-20231108164839575

安装jenkins插件
设置镜像源

image-20231113150948023

image-20231108165048585

image-20231108165156418

安装Git插件

image-20231113151055304

image-20231113151110191

image-20231108170248831

重启jenkins容器

docker restart jenkins
安装gitlab插件

image-20231108170646012

系统配置

全局工具配置

image-20231109155157072

配置maven

image-20231113172348209

image-20231113172431231

image-20231113172555809

配置用户凭证
image-20231109155331438

image-20231109155355147

image-20231109155432433

image-20231109155604208

image-20231109155931214

image-20231109155955200

jenkins使用

gitlab创建maven项目

image-20231109160229627

image-20231109160254409

image-20231109160307786

image-20231109160419549

image-20231109160437086

image-20231109161042371

idea 导入gitlab项目

image-20231109161132018

image-20231109161246699

idea中隐藏不必要的文件/夹(可选)
image-20231109161421122

image-20231109161905444 image-20231109161952563

image-20231109162054047

image-20231109162119223

image-20231109162156893

image-20231109162215870
修改项目名称
<groupId>com.wnhz.bms</groupId>
<artifactId>wnhz-bms</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>jar</packaging>

<name>wnhz-bms</name>
<description>使用jenkins构建项目</description>

image-20231109162536683

创建项目提交到gitlab

gitlab提交报错

解决方案：需要修改git的remote用户名密码

image-20231109165015611

image-20231109165055082

再次提交

image-20231109165125130

gitlab

image-20231109165213762

jenkins构建

创建新项目(maven)

image-20231109165335323

配置（Configure）

image-20231109165633649

image-20231109165651422

image-20231109165709762

总览如下图：

image-20231109165534915

保存

image-20231109165800718

立即构建

image-20231109165851063

image-20231109165905370

点击#1进入构建详情

image-20231109165932608

image-20231109170049042

image-20231109170347439

jenkins的自动构建

在项目配置中配置构建触发器

image-20231110094352596

取消认证

image-20231110102342883

image-20231110102304395

登录gitlab
管理员登录 gitlaba

image-20231110100702357

image-20231110100630725

退出管理员

image-20231110100802990
当前用登录
image-20231110095529082

image-20231110095853120

image-20231110101225404

image-20231110102506970

image-20231110102523109

测试自动构建
push项目到gitlab

image-20231110102923061

查看jenkins是否自动构建

image-20231110102903373

image-20231110103016106

image-20231110111820189

jenkins发布

安装publisher ssh over插件

image-20231108171722390

配置ssh
image-20231110084123134

image-20231110084439553

image-20231110084517077

image-20231110084938792

image-20231208174642472

修改springboot项目的输出路径
<build>
    <plugins>
        <plugin>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-maven-plugin</artifactId>
            <configuration>
                <outputDirectory>../../out</outputDirectory>
            </configuration>
        </plugin>
    </plugins>
</build>
发布项目配置

点击项目设置post steps

image-20231110171830032

image-20231209173245868

测试发布

image-20231110172021959

image-20231209173428986

linux服务器

image-20231209173529186

image-20231110172039998

[root@localhost projects]# nohup java -jar wnhz-bms-0.0.1-SNAPSHOT.jar &
[1] 67855
[root@localhost projects]# nohup: 忽略输入并把输出追加到"nohup.out"

[root@localhost projects]# ls
nohup.out  wnhz-bms-0.0.1-SNAPSHOT.jar

[root@localhost projects]cat nohup.out

     /^\    /^\
    {  O}  {  O}
     \ /    \ /
     //     //       _------_
    //     //     ./~        ~-_
   / ~----~/     /              \
 /         :   ./       _---_    ~-
|  \________) :       /~     ~\   |
|        /    |      |  :~~\  |   |
|       |     |      |  \___-~    |
|        \ __/`^\______\.        ./
 \                     ~-______-~\.
 .|                                ~-_
/_________蜗牛杭州Jenkins测试__________~~
2023-11-10 17:15:43.926  INFO 28907 --- [           main] com.wnhz.bms.BmsApp                      : Starting BmsApp v0.0.1-SNAPSHOT using Java 1.8.0_341 on localhost.localdomain with PID 28907 (/usr/local/projects/wnhz-bms-0.0.1-SNAPSHOT.jar started by root in /usr/local/projects)
2023-11-10 17:15:44.062  INFO 28907 --- [           main] com.wnhz.bms.BmsApp                      : The following 1 profile is active: "dev"
2023-11-10 17:15:47.587  INFO 28907 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port(s): 9090 (http)
2023-11-10 17:15:47.610  INFO 28907 --- [           main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
2023-11-10 17:15:47.611  INFO 28907 --- [           main] org.apache.catalina.core.StandardEngine  : Starting Servlet engine: [Apache Tomcat/9.0.68]
2023-11-10 17:15:47.860  INFO 28907 --- [           main] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
2023-11-10 17:15:47.861  INFO 28907 --- [           main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 3472 ms
2023-11-10 17:15:49.326  INFO 28907 --- [           main] pertySourcedRequestMappingHandlerMapping : Mapped URL path [/v2/api-docs] onto method [springfox.documentation.swagger2.web.Swagger2ControllerWebMvc#getDocumentation(String, HttpServletRequest)]
2023-11-10 17:15:49.475  INFO 28907 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 9090 (http) with context path ''
2023-11-10 17:15:49.477  INFO 28907 --- [           main] d.s.w.p.DocumentationPluginsBootstrapper : Documentation plugins bootstrapped
2023-11-10 17:15:49.486  INFO 28907 --- [           main] d.s.w.p.DocumentationPluginsBootstrapper : Found 1 custom documentation plugin(s)
2023-11-10 17:15:49.558  INFO 28907 --- [           main] s.d.s.w.s.ApiListingReferenceScanner     : Scanning for api listing references
2023-11-10 17:15:49.887  INFO 28907 --- [           main] com.wnhz.bms.BmsApp                      : Started BmsApp in 8.873 seconds (JVM running for 10.226)
2023-11-10 17:16:19.245  INFO 28907 --- [nio-9090-exec-1] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring DispatcherServlet 'dispatcherServlet'
2023-11-10 17:16:19.245  INFO 28907 --- [nio-9090-exec-1] o.s.web.servlet.DispatcherServlet        : Initializing Servlet 'dispatcherServlet'
2023-11-10 17:16:19.247  INFO 28907 --- [nio-9090-exec-1] o.s.web.servlet.DispatcherServlet        : Completed initialization in 1 ms

swagger测试

image-20231110173326253

image-20231110172154445

关闭应用

image-20231110173416030

[root@localhost projects]# ps aux|grep 67855
root      67855  8.8  3.9 5856660 318416 pts/4  Sl   17:31   0:18 java -jar wnhz-bms-0.0.1-SNAPSHOT.jar
root      76563  0.0  0.0 112824   988 pts/4    S+   17:34   0:00 grep --color=auto 67855

[root@localhost projects]# kill -9 67855
[root@localhost projects]# ps aux|grep 67855
root      79602  0.0  0.0 112824   984 pts/4    S+   17:35   0:00 grep --color=auto 67855
[1]+  已杀死               nohup java -jar wnhz-bms-0.0.1-SNAPSHOT.jar

g-TG9kGPpx-1713174351305)]

关闭应用

[外链图片转存中…(img-19K8gVWG-1713174351306)]

[root@localhost projects]# ps aux|grep 67855
root      67855  8.8  3.9 5856660 318416 pts/4  Sl   17:31   0:18 java -jar wnhz-bms-0.0.1-SNAPSHOT.jar
root      76563  0.0  0.0 112824   988 pts/4    S+   17:34   0:00 grep --color=auto 67855

[root@localhost projects]# kill -9 67855
[root@localhost projects]# ps aux|grep 67855
root      79602  0.0  0.0 112824   984 pts/4    S+   17:35   0:00 grep --color=auto 67855
[1]+  已杀死               nohup java -jar wnhz-bms-0.0.1-SNAPSHOT.jar
举报

相关推荐

【经典爬虫案例】用Python爬取微博热搜榜！

Pythonpython爬虫python爬虫实战采集数据爬虫开发jQuery前端开发

爬取微博热搜榜并进行数据分析

华为云数据库

爬虫实战-微博评论爬取

数据bc3d计算机视觉人工智能yyds干货盘点

【爬虫实战】python微博热搜榜Top50

java设计模式观察者模式

Python爬虫 | 爬取微博和哔哩哔哩数据

python爬虫策略模式数据分析大数据

Python爬取热搜数据之炫酷可视化

可视化爬取热搜数据动态可视化定位追踪数据Python编程语言

Python 爬取微博指定博主所有内容

数据jsonIPSpark大数据

python爬虫获取微博吴某凡微博热评

Python爬虫实战Python数据分析python_爬虫

python爬取微博关键字

搜索pythonPython后端开发

Python 网络爬虫实战：手把手教你爬取微博评论区数据

爬虫python新浪微博数据jsonPython编程语言
0 条评论