Golang服务器热重启、热升级、热更新(safe and graceful hot-restart/reload http server)详解

服务端代码经常需要升级，对于线上系统的升级常用的做法是，通过前端的负载均衡（如nginx）来保证升级时至少有一个服务可用，依次（灰度）升级。

而另一种更方便的方法是在应用上做热重启，直接更新源码、配置或升级应用而不停服务。

这个功能在重要业务上尤为重要，会影响服务可用性、用户体验。

原理

热重启的原理比较简单，但是涉及到一些系统调用以及父子进程之间文件句柄的传递等等细节比较多。
处理过程分为以下几个步骤：

1. 监听信号（USR2..）
2. 收到信号时fork子进程（使用相同的启动命令），将服务监听的socket文件描述符传递给子进程
3. 子进程监听父进程的socket，这个时候父进程和子进程都可以接收请求
4. 子进程启动成功之后，父进程停止接收新的连接，等待旧连接处理完成（或超时）
5. 父进程退出，重启完成

细节

• 父进程将socket文件描述符传递给子进程可以通过命令行，或者环境变量等
• 子进程启动时使用和父进程一样的命令行，对于golang来说用更新的可执行程序覆盖旧程序
• server.Shutdown()优雅关闭方法是go>=1.8的新特性
• server.Serve(l)方法在Shutdown时立即返回，Shutdown方法则阻塞至context完成，所以Shutdown的方法要写在主goroutine中

代码

l, err = net.FileListener(f)
TCPListener

我的实现

package main

import (
   "net"
   "net/http"
   "time"
   "log"
   "syscall"
   "os"
   "os/signal"
   "context"
   "fmt"
   "os/exec"
   "flag"
)
var (
   listener net.Listener
   err error
   server http.Server
   graceful =  flag.Bool("g", false, "listen on fd open 3 (internal use only)")
)

type MyHandler struct {

}

func (*MyHandler)ServeHTTP(w http.ResponseWriter, r *http.Request){
   fmt.Println("request start at ", time.Now(),  r.URL.Path+"?"+r.URL.RawQuery,  "request done at ", time.Now(), "  pid:", os.Getpid())
   time.Sleep(10 * time.Second)
   w.Write([]byte("this is test response"))
   fmt.Println("request done at ", time.Now(), "  pid:", os.Getpid() )

}

func main() {
   flag.Parse()
   fmt.Println("start-up at " , time.Now(), *graceful)
   if *graceful {
      f := os.NewFile(3, "")
      listener, err = net.FileListener(f)
      fmt.Printf( "graceful-reborn  %v %v  %#v \n", f.Fd(), f.Name(), listener)
   }else{
      listener, err = net.Listen("tcp", ":1111")
      tcp,_ := listener.(*net.TCPListener)
      fd,_ := tcp.File()
      fmt.Printf( "first-boot  %v %v %#v \n ", fd.Fd(),fd.Name(), listener)
   }


   server := http.Server{
      Handler: &MyHandler{},
      ReadTimeout: 6 * time.Second,
   }
   log.Printf("Actual pid is %d\n", syscall.Getpid())
   if err != nil {
      println(err)
      return
   }
   log.Printf(" listener: %v\n",   listener)
   go func(){//不要阻塞主进程
      err := server.Serve(listener)
      if err != nil {
         log.Println(err)
      }
   }()

   //signals
   func(){
      ch := make(chan os.Signal, 1)
      signal.Notify(ch, syscall.SIGHUP, syscall.SIGTERM)
      for{//阻塞主进程， 不停的监听系统信号
         sig := <- ch
         log.Printf("signal: %v", sig)
         ctx, _ := context.WithTimeout(context.Background(), 20*time.Second)
         switch sig {
         case syscall.SIGTERM, syscall.SIGHUP:
            println("signal cause reloading")
            signal.Stop(ch)
            {//fork new child process
               tl, ok := listener.(*net.TCPListener)
               if !ok {
                  fmt.Println("listener is not tcp listener")
                  return
               }
               currentFD, err := tl.File()
               if err != nil {
                  fmt.Println("acquiring listener file failed")
                  return
               }
               cmd := exec.Command(os.Args[0], "-g")
               cmd.ExtraFiles, cmd.Stdout,cmd.Stderr = []*os.File{currentFD} ,os.Stdout, os.Stderr
               err = cmd.Start()

               if err != nil {
                  fmt.Println("cmd.Start fail: ", err)
                  return
               }
               fmt.Println("forked new pid : ",cmd.Process.Pid)
            }

            server.Shutdown(ctx)
            fmt.Println("graceful shutdown at ", time.Now())
         }

      }
   }()
}

qiangjian@sun-pro:/data1/works/IdeaProjects/go_core$ go  run src/wright/hotrestart/booter.go  
start-up at  2018-10-12 15:29:34.586269 +0800 CST m=+0.004439497 false
first-boot  5 tcp:[::]:1111-> &net.TCPListener{fd:(*net.netFD)(0xc00010e000)} 
 2018/10/12 15:29:34 Actual pid is 10771
2018/10/12 15:29:34  listener: &{0xc00010e000}
request start at  2018-10-12 15:29:40.287928 +0800 CST m=+5.705965906 /aa/bb?c=d request done at  2018-10-12 15:29:40.287929 +0800 CST m=+5.705966554   pid: 10771
2018/10/12 15:29:49 signal: terminated
signal cause reloading
forked new pid :  10775
start-up at  2018-10-12 15:29:49.689064 +0800 CST m=+0.001613279 true
graceful-reborn  3   &net.TCPListener{fd:(*net.netFD)(0xc0000ec000)} 
2018/10/12 15:29:49 Actual pid is 10775
2018/10/12 15:29:49  listener: &{0xc0000ec000}
request done at  2018-10-12 15:29:50.288525 +0800 CST m=+15.706330718   pid: 10771
2018/10/12 15:29:50 http: Server closed
request start at  2018-10-12 15:29:50.290622 +0800 CST m=+15.708426906 /aa/bb?c=d request done at  2018-10-12 15:29:50.290623 +0800 CST m=+15.708428113   pid: 10771
request start at  2018-10-12 15:29:50.290713 +0800 CST m=+0.603248262 /aa/bb?c=d request done at  2018-10-12 15:29:50.290714 +0800 CST m=+0.603249293   pid: 10775
request done at  2018-10-12 15:30:00.293988 +0800 CST m=+10.606290169   pid: 10775
request done at  2018-10-12 15:30:00.294043 +0800 CST m=+25.711615717   pid: 10771
request start at  2018-10-12 15:30:00.295554 +0800 CST m=+10.607856283 /aa/bb?c=d request done at  2018-10-12 15:30:00.295555 +0800 CST m=+10.607857307   pid: 10775
request start at  2018-10-12 15:30:00.29558 +0800 CST m=+10.607881997 /aa/bb?c=d request done at  2018-10-12 15:30:00.295581 +0800 CST m=+10.607883004   pid: 10775
graceful shutdown at  2018-10-12 15:30:00.79544 +0800 CST m=+26.213000502

ab -v -k -c2 -n100 '127.0.0.1:1111/aa/bb?c=d'
This is ApacheBench, Version 2.3 <$Revision: 1826891 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient)...^C

Server Software:        
Server Hostname:        127.0.0.1
Server Port:            1111

Document Path:          /aa/bb?c=d
Document Length:        21 bytes

Concurrency Level:      2
Time taken for tests:   48.292 seconds
Complete requests:      7
Failed requests:        0
Total transferred:      966 bytes
HTML transferred:       147 bytes
Requests per second:    0.14 [#/sec] (mean)
Time per request:       13797.702 [ms] (mean)
Time per request:       6898.851 [ms] (mean, across all concurrent requests)
Transfer rate:          0.02 [Kbytes/sec] received

kill 进程ID  #发送TERM信号

//还有一种方式去fork，和上面本质一样：
execSpec := &syscall.ProcAttr{
    Env:   os.Environ(),
    Files: []uintptr{os.Stdin.Fd(), os.Stdout.Fd(), os.Stderr.Fd(), lFd},
}
pid, err := syscall.ForkExec(os.Args[0], os.Args, execSpec)

可以看出： ab测试器Failed为0，且console中显示老请求处理完后才shutdown，即在kill触发reload后，请求无论是老进程的旧请求，还是fork子进程后的新请求，全都处理成功，没有失败的。

systemd & supervisor

父进程退出之后，子进程会挂到1号进程上面。这种情况下使用systemd和supervisord等管理程序会显示进程处于failed的状态。解决这个问题有两个方法：

• 使用pidfile，每次进程重启更新一下pidfile，让进程管理者通过这个文件感知到main pid的变更。
• 更通用的做法：起一个master来管理服务进程，每次热重启master拉起一个新的进程，把旧的kill掉。这时master的pid没有变化，对于进程管理者来说进程处于正常的状态。​​一个简洁的实现​​

FD复制时细节

请看：

​​http://man7.org/linux/man-pages/man2/fcntl.2.html#F_DUPFD_CLOEXEC​​

References

• ​​graceful​​
• ​​Graceful Restart in Golang​​
• ​​facebookgo/grace​​
• ​​endless​​
• ​​overseer​​