Py http.server 使用 HTTPServer 阻塞问题

language_py summary_2018/11

1、背景

• 在py中使用HTTPServer的时候，线上服务挂了，观察日志和查阅资源，发现是连接hive后请求阻塞，无法处理其它请求。且server_forever()没有服务端超时处理机制。

2、查阅资料

• py HTTPServer

// class http.server.HTTPServer(server_address, RequestHandlerClass)¶
This class builds on the TCPServer class by storing the server address as instance variables named server_name and server_port. The server is accessible by the handler, typically through the handler’s server instance variable.
// class http.server.ThreadingHTTPServer(server_address, RequestHandlerClass)
This class is identical to HTTPServer but uses threads to handle requests by using the ThreadingMixIn. This is useful to handle web browsers pre-opening sockets, on which HTTPServer would wait indefinitely.(多线程实现)

3、实践

// 单进程 HTTPServer
from http.server import BaseHTTPRequestHandler, HTTPServer
import time
class MyHandler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path == '/sleep':
            time.sleep(7)
            self.send_response(200)
            self.wfile.write(bytes('sleep 7s', 'UTF-8'))
        if self.path == '/':
            self.send_response(200)
            self.wfile.write(bytes('not sleep', 'UTF-8'))
if __name__ == '__main__':
    host = '0.0.0.0'
    port = 3004
    httpd = HTTPServer((host, port), MyHandler)
    try:
        httpd.serve_forever()
    except KeyboardInterrupt:
        httpd.server_close()
// 多线程 ThreadingMixIn
from http.server import BaseHTTPRequestHandler, HTTPServer
from socketserver import ThreadingMixIn
import time
class MyHandler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path == '/sleep':
            time.sleep(7)
            self.send_response(200)
            self.wfile.write(bytes('sleep 7s', 'UTF-8'))
        if self.path == '/':
            self.send_response(200)
            self.wfile.write(bytes('not sleep', 'UTF-8'))
# 混合类重写了process_request
class ThreadingHTTPServer(ThreadingMixIn, HTTPServer):
    pass
if __name__ == '__main__':
    host = '0.0.0.0'
    port = 3004
    httpd = ThreadingHTTPServer((host, port), MyHandler)
    try:
        httpd.serve_forever()
    except KeyboardInterrupt:
        httpd.server_close()

4、结论

// 结论：
1、单进程HTTPServer调用‘/sleep’会阻塞后面的请求7s
2、多线程ThreadingMixIn不会被阻塞，因为会建立一个线程去请求；当然在实际情况中，会设置线程的池的大小，避免线程被无止尽创建
// 原理：ThreadingMixIn源码，通过混合类重写了process_request方法，通过每次新建立一个线程来处理请求，达到并发处理请求的效果
class ThreadingMixIn:
    daemon_threads = False
    def process_request_thread(self, request, client_address):
        try:
            self.finish_request(request, client_address)
            self.shutdown_request(request)
        except:
            self.handle_error(request, client_address)
            self.shutdown_request(request)
    def process_request(self, request, client_address):
        t = threading.Thread(target = self.process_request_thread,
                             args = (request, client_address))
        t.daemon = self.daemon_threads
        t.start()
// 建议
1、官方文档已经说过：Warning http.server is not recommended for production. It only implements basic security checks.

5、参考文献

• python TCPServer, StreamRequestHandler设置超时时间timeout
• Python 网络编程：包括一些内部机制