Netty(十四)——EventLoop之式

蔚落 2022-04-12 14:08 362阅读 0赞

前边讲了ByteBuf、Channel、Unsafe、ChannelPipeline、ChannelHandler等核心的类。这篇来学习学习EventLoop（EventLoopGroup）——Netty的线程。Netty的线程模型是经过精心的设计，既提高了框架的并发性能，又能在很大程度上避免死锁，局部还是实现了无锁化设计。非常值得学习的。

**一，Reactor线程模型：**Netty线程模型本质上也是经典的Reactor线程模型，看下前边转过的一篇文章《[Reactor模式详解(转)][Reactor]》。其中包括了，Reactor单线程模型、Reactor多线程模型、Reactor主从多线程模型。大家可以参考Reactor线程模型图进行对比查看。

<table>  Reactor线程模型  
 <thead> 
  <tr> 
   <th>类型</th> 
   <th style="width:438px;">特点</th> 
   <th style="width:344px;">不足</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>Reactor单线程模型</td> 
   <td style="width:438px;"> <p>1，指所有的I/O操作都是在同一个NIO线程上完成，包括：a，接收客户端的TCP连接；b，向服务端发起TCP连接；c，读取通信对端的请求或应答消息；d，向通信对端发送消息或者应答消息。</p> <p>2，简单、容易理解并实现</p> </td> 
   <td style="width:344px;"> <p>1，一个NIO线程同时处理成百上千的链路，性能无法支撑；</p> <p>2，当NIO线程负载过重，处理速度变慢，导致客户端超时，超时后重发，更加重NIO线程的负载，最终导致大量消息积压和处理超时，成为系统瓶颈；</p> <p>3，可靠性问题：一旦NIO线程意外，或者进入死循环，会导致整个系统通信模块不可用，造成节点故障。</p> </td> 
  </tr> 
  <tr> 
   <td>Reactor多线程模型</td> 
   <td style="width:438px;"> <p>1，有一个专门NIO线程——acceptor线程用于监听服务端，接受客户端的TCP连接；</p> <p>2，网络IO操作——读写等由一个NIO线程池负责，包含一个任务队列和N个可用线程，由这些NIO线程负责消息的读取、解码、编码和发送等。</p> <p>3，一个NIO线程可以同时处理N条链路，但是一个链路只对应一个NIO线程，防止发生并发操作问题。</p> </td> 
   <td style="width:344px;">1，一个NIO线程负责监听和处理所有的客户端连接可能会存在性能问题。例如百万客户端连接，或者服务端对客户端握手进行安全认证（耗费性能），可能会出现性能不足。</td> 
  </tr> 
  <tr> 
   <td>主从Reactor多线程模型</td> 
   <td style="width:438px;"> <p>1，服务端用于接受客户端连接的不再是一个单独的NIO线程，而是一个独立的NIO线程池；</p> <p>2，sub reactor线程池用来负责SocketChannel的读写、编解码等工作。</p> </td> 
   <td style="width:344px;">1，相对于上边两种更加复杂吧。</td> 
  </tr> 
 </tbody> 
</table>

** 二，Netty线程模型**：Netty可以通过配置不同的启动参数，支持上边的几种线程模型的，看下原理图和代码吧：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpdWppYWhhbjYyOTYyOQ_size_16_color_FFFFFF_t_70][]

EventLoopGroup bossGroup = new NioEventLoopGroup();
                EventLoopGroup workerGroup = new NioEventLoopGroup();
                try{
    
                    ServerBootstrap b =new ServerBootstrap();
                    b.group(bossGroup,workerGroup).channel(NioServerSocketChannel.class).option(ChannelOption.SO_BACKLOG,100).handler(new LoggingHandler(LogLevel.INFO)).childHandler(new ChannelInitializer<SocketChannel>() {
                        @Override
                        protected void initChannel(SocketChannel socketChannel) throws Exception {
                            //解码
                            socketChannel.pipeline().addLast(MarshallingCodeFactory.buildMarshallingDecoder());
                            //编码
                            socketChannel.pipeline().addLast(MarshallingCodeFactory.buildMarshallingEncoder());
                            socketChannel.pipeline().addLast(new SubReqServerHandler());
                        }
                    });
                    ChannelFuture f =b.bind(port).sync();
                    f.channel().closeFuture().sync();

<table>  Netty线程职责  
 <thead> 
  <tr> 
   <th>类别</th> 
   <th>职责</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>用于接收Client请求的线程池</td> 
   <td> <p>1，接收Client的TCP连接，初始化Channel参数；</p> <p>2，将链路状态变更时间通知给ChannelPipeline；</p> </td> 
  </tr> 
  <tr> 
   <td>用于处理I/O操作的线程池</td> 
   <td> <p>1，异步读取通信对端的数据报，发送读时间到ChannelPipeline；</p> <p>2，异步发送消息到通信对端，调用ChannelPipeline的消息发送接口；</p> <p>3，执行系统调用Task；</p> <p>4，执行定时任务Task，例如链路空闲状态监测定时任务。</p> </td> 
  </tr> 
 </tbody> 
</table>

串行操作（无锁化设计）：上篇[Netty(十二)——ChannelPipeline之观][Netty_ChannelPipeline]、[Netty(十三)——ChannelHandler之意][Netty_ChannelHandler] 我们看到事件的处理是在ChannelPipeline中传输像职责链一样，经过ChannelHandler时进行处理，这种启动多个串行化的线程并行运行（避免锁的竞争），比一个队列一个工作线程性能更优。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpdWppYWhhbjYyOTYyOQ_size_16_color_FFFFFF_t_70 1][]

**       三，Netty实践建议：**

1，创建两个NioEventLoopGroup，用于逻辑隔离NIO Acceptor和NIO IO操作线程；

2，尽量不要在ChannelHandler中启动用户线程（解码后用于将POJO消息派发到后端业务线程的除外）；

3，解码要放在NIO线程调用的解码Handler中进行，不要切换到用户线程中完成消息的解码；

4，如果业务逻辑操作简单，没有复杂的业务逻辑计算，没有可能导致线程被阻塞的磁盘操作、数据库操作、网络操作等，可以直接在NIO线程上完成业务逻辑编排，不需要切换到用户线程。

5，如果业务逻辑处理复杂，不要在NIO线程上完成，建议将解码后的POJO消息封装成Task，派发到业务线程中执行，以保证NIO线程尽快被释放，处理其他的IO操作。

** 四，NioEventLoop的源码分析：**

**       **1，先看下NioEventLoop的类关系图：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpdWppYWhhbjYyOTYyOQ_size_16_color_FFFFFF_t_70 2][]

**       2**，看一下NioEventLoop的源码：

2.1，首先看下多路复用器Selector在NioEventLoop中的初始化。

/**
         * 一，聚合的多路复用器selector
         */
        Selector selector;
        private SelectedSelectionKeySet selectedKeys;
    
        private final SelectorProvider provider;
    
        /**
         * 二，初始化NioEventLoop，selector = openSelector();
         */
        NioEventLoop(NioEventLoopGroup parent, Executor executor, SelectorProvider selectorProvider) {
            super(parent, executor, false);
            if (selectorProvider == null) {
                throw new NullPointerException("selectorProvider");
            }
            provider = selectorProvider;
            selector = openSelector();
        }
    
        /**
         * 二-1，selector = openSelector();
         */
        private Selector openSelector() {
            final Selector selector;
            try {
                selector = provider.openSelector();
            } catch (IOException e) {
                throw new ChannelException("failed to open a new selector", e);
            }
            //如果没有开启selectedKeys优化开关，直接返回provider.openSelector()。
            if (DISABLE_KEYSET_OPTIMIZATION) {
                return selector;
            }
            
            //如果开启了，通过反射从selector中获取selectedKeys和publicSelectedKeys，将其设置为可写，通过反射的方式使用Netty构造的selectedKeys = selectedKeySet;替换JDK的。
            try {
                SelectedSelectionKeySet selectedKeySet = new SelectedSelectionKeySet();
    
                Class<?> selectorImplClass =
                        Class.forName("sun.nio.ch.SelectorImpl", false, PlatformDependent.getSystemClassLoader());
    
                // Ensure the current selector implementation is what we can instrument.
                if (!selectorImplClass.isAssignableFrom(selector.getClass())) {
                    return selector;
                }
    
                Field selectedKeysField = selectorImplClass.getDeclaredField("selectedKeys");
                Field publicSelectedKeysField = selectorImplClass.getDeclaredField("publicSelectedKeys");
    
                selectedKeysField.setAccessible(true);
                publicSelectedKeysField.setAccessible(true);
    
                selectedKeysField.set(selector, selectedKeySet);
                publicSelectedKeysField.set(selector, selectedKeySet);
    
                selectedKeys = selectedKeySet;
                logger.trace("Instrumented an optimized java.util.Set into: {}", selector);
            } catch (Throwable t) {
                selectedKeys = null;
                logger.trace("Failed to instrument an optimized java.util.Set into: {}", selector, t);
            }
    
            return selector;
        }

2.2，分析run方法的实现，这个方法基本调用到NioEventLoop中的所有封装方法，有的调用层级还比较深，不过耐心查看，跟下去，大概流程就清楚了。（这里我在注释中用数字表示了层级跟踪，例如4-1-1 为三级方法调用）

@Override
        protected void run() {
            //1-将wakeup还原false，并将之前的状态保存到oldWakeUp中
            boolean oldWakenUp = wakenUp.getAndSet(false);
            try {
                //2判断当前消息队列中是否有消息尚未处理，如果有，selectNow返回一次select操作。
                if (hasTasks()) {
                    selectNow();
                } else {
                    //3,轮询，看是否有准备就绪的Channel
                    select(oldWakenUp);
                    if (wakenUp.get()) {
                        selector.wakeup();
                    }
                }
    
                cancelledKeys = 0;
                needsToSelectAgain = false;
                final int ioRatio = this.ioRatio;
                //先后执行IO任务和非IO任务，两类任务的执行时间比由变量ioRatio控制，默认是非IO任务允许执行和IO任务相同的时间
                //Netty中控制IO运行比例占比-分析,得到就绪状态的SocketChannel
                if (ioRatio == 100) {
                    //4
                    processSelectedKeys();
                    //5
                    runAllTasks();
                } else {
                    final long ioStartTime = System.nanoTime();
    
                    processSelectedKeys();
    
                    final long ioTime = System.nanoTime() - ioStartTime;
                    runAllTasks(ioTime * (100 - ioRatio) / ioRatio);
                }
                //如果关闭，则进行优雅停机，调用closeAll，释放资源
                if (isShuttingDown()) {
                    //6,
                    closeAll();
                    if (confirmShutdown()) {
                        cleanupAndTerminate(true);
                        return;
                    }
                }
            } catch (Throwable t) {
                logger.warn("Unexpected exception in the selector loop.", t);
                try {
                    Thread.sleep(1000);
                } catch (InterruptedException e) {
                    // Ignore.
                }
            }
    
            scheduleExecution();
        }
    
    
        /**
         * 2-1@see {@link Queue#isEmpty()}
         */
        protected boolean hasTasks() {
            assert inEventLoop();
            return !taskQueue.isEmpty();
        }
        /**
         * 2-2 selectNow
         */
        void selectNow() throws IOException {
            try {
                selector.selectNow();
            } finally {
                // restore wakup state if needed
                if (wakenUp.get()) {
                    selector.wakeup();
                }
            }
        }
    
    
        /**
         * 3-1 select
         */
        private void select(boolean oldWakenUp) throws IOException {
            Selector selector = this.selector;
            try {
                int selectCnt = 0;
                //取当前系统的纳秒时间
                long currentTimeNanos = System.nanoTime();
                //调用delayNanos计算定时任务的触发时间
                long selectDeadLineNanos = currentTimeNanos + delayNanos(currentTimeNanos);
                //死循环
                for (;;) {
                    long timeoutMillis = (selectDeadLineNanos - currentTimeNanos + 500000L) / 1000000L;
                    //如果需要立即执行，或已经超时，则selector.selectNow();并退出当前循环
                    if (timeoutMillis <= 0) {
                        if (selectCnt == 0) {
                            selector.selectNow();
                            selectCnt = 1;
                        }
                        break;
                    }
                    
                    //将定时任务剩余的超时时间作为参数进行select操作，没完成一次select操作，对selectCnt+1
                    int selectedKeys = selector.select(timeoutMillis);
                    selectCnt ++;
                    //如果有下列情况，则进行退出循环
                    if (selectedKeys != 0 || oldWakenUp || wakenUp.get() || hasTasks() || hasScheduledTasks()) {
                        // - Selected something,
                        // - waken up by user, or
                        // - the task queue has a pending task.
                        // - a scheduled task is ready for processing
                        break;
                    }
                    //thread interrupted也break退出
                    if (Thread.interrupted()) {
                        // Thread was interrupted so reset selected keys and break so we not run into a busy loop.
                        // As this is most likely a bug in the handler of the user or it's client library we will
                        // also log it.
                        //
                        // See https://github.com/netty/netty/issues/2426
                        if (logger.isDebugEnabled()) {
                            logger.debug("Selector.select() returned prematurely because " +
                                    "Thread.currentThread().interrupt() was called. Use " +
                                    "NioEventLoop.shutdownGracefully() to shutdown the NioEventLoop.");
                        }
                        selectCnt = 1;
                        break;
                    }
    
                    long time = System.nanoTime();
                    if (time - TimeUnit.MILLISECONDS.toNanos(timeoutMillis) >= currentTimeNanos) {
                        // timeoutMillis elapsed without anything selected.
                        selectCnt = 1;
                    //如果死循环了，则进行重建Selector方式，让系统恢复正常rebuildSelector
                    } else if (SELECTOR_AUTO_REBUILD_THRESHOLD > 0 &&
                            selectCnt >= SELECTOR_AUTO_REBUILD_THRESHOLD) {
                        // The selector returned prematurely many times in a row.
                        // Rebuild the selector to work around the problem.
                        logger.warn(
                                "Selector.select() returned prematurely {} times in a row; rebuilding selector.",
                                selectCnt);
                        //3-1-1
                        rebuildSelector();
                        selector = this.selector;
    
                        // Select again to populate selectedKeys.
                        selector.selectNow();
                        selectCnt = 1;
                        break;
                    }
    
                    currentTimeNanos = time;
                }
                
                if (selectCnt > MIN_PREMATURE_SELECTOR_RETURNS) {
                    if (logger.isDebugEnabled()) {
                        logger.debug("Selector.select() returned prematurely {} times in a row.", selectCnt - 1);
                    }
                }
            } catch (CancelledKeyException e) {
                if (logger.isDebugEnabled()) {
                    logger.debug(CancelledKeyException.class.getSimpleName() + " raised by a Selector - JDK bug?", e);
                }
                // Harmless exception - log anyway
            }
        }
    
    
        /**
         * 3-1-1
         * Replaces the current {@link Selector} of this event loop with newly created {@link Selector}s to work
         * around the infamous epoll 100% CPU bug.
         */
        public void rebuildSelector() {
            //如果为其它线程发起，则为了避免多线程并发操作，将rebuildSelector()封装成task放到消息队列中，由NioEventLoop负责调用。
            if (!inEventLoop()) {
                execute(new Runnable() {
                    @Override
                    public void run() {
                        rebuildSelector();
                    }
                });
                return;
            }
    
            final Selector oldSelector = selector;
            final Selector newSelector;
    
            if (oldSelector == null) {
                return;
            }
    
            try {
                //创建新的Selector
                newSelector = openSelector();
            } catch (Exception e) {
                logger.warn("Failed to create a new Selector.", e);
                return;
            }
    
            // Register all channels to the new Selector.(将SocketChannel从旧的Selector移动到新的上)
            int nChannels = 0;
            for (;;) {
                try {
                    for (SelectionKey key: oldSelector.keys()) {
                        Object a = key.attachment();
                        try {
                            if (!key.isValid() || key.channel().keyFor(newSelector) != null) {
                                continue;
                            }
    
                            int interestOps = key.interestOps();
                            key.cancel();
                            SelectionKey newKey = key.channel().register(newSelector, interestOps, a);
                            if (a instanceof AbstractNioChannel) {
                                // Update SelectionKey
                                ((AbstractNioChannel) a).selectionKey = newKey;
                            }
                            nChannels ++;
                        } catch (Exception e) {
                            logger.warn("Failed to re-register a Channel to the new Selector.", e);
                            if (a instanceof AbstractNioChannel) {
                                AbstractNioChannel ch = (AbstractNioChannel) a;
                                ch.unsafe().close(ch.unsafe().voidPromise());
                            } else {
                                @SuppressWarnings("unchecked")
                                NioTask<SelectableChannel> task = (NioTask<SelectableChannel>) a;
                                invokeChannelUnregistered(task, key, e);
                            }
                        }
                    }
                } catch (ConcurrentModificationException e) {
                    // Probably due to concurrent modification of the key set.
                    continue;
                }
    
                break;
            }
    
            selector = newSelector;
    
            try {
                // time to close the old selector as everything else is registered to the new one 销毁旧的selector
                oldSelector.close();
            } catch (Throwable t) {
                if (logger.isWarnEnabled()) {
                    logger.warn("Failed to close the old Selector.", t);
                }
            }
    
            logger.info("Migrated " + nChannels + " channel(s) to the new Selector.");
        }
    
    
        /**
         * 4-1
         */
        private void processSelectedKeys() {
            //如果开启selectedKeys优化功能走processSelectedKeysOptimized，否则走processSelectedKeysPlain
            if (selectedKeys != null) {
                //4-1-1
                processSelectedKeysOptimized(selectedKeys.flip());
            } else {
                //4-1-2
                processSelectedKeysPlain(selector.selectedKeys());
            }
        }
    
        /**
         *4-1-1
         **/
        private void processSelectedKeysOptimized(SelectionKey[] selectedKeys) {
            for (int i = 0;; i ++) {
                final SelectionKey k = selectedKeys[i];
                if (k == null) {
                    break;
                }
                // null out entry in the array to allow to have it GC'ed once the Channel close
                // See https://github.com/netty/netty/issues/2363
                selectedKeys[i] = null;
    
                final Object a = k.attachment();
    
                if (a instanceof AbstractNioChannel) {
                    processSelectedKey(k, (AbstractNioChannel) a);
                } else {
                    @SuppressWarnings("unchecked")
                    NioTask<SelectableChannel> task = (NioTask<SelectableChannel>) a;
                    processSelectedKey(k, task);
                }
    
                if (needsToSelectAgain) {
                    // null out entries in the array to allow to have it GC'ed once the Channel close
                    // See https://github.com/netty/netty/issues/2363
                    for (;;) {
                        if (selectedKeys[i] == null) {
                            break;
                        }
                        selectedKeys[i] = null;
                        i++;
                    }
    
                    selectAgain();
                    // Need to flip the optimized selectedKeys to get the right reference to the array
                    // and reset the index to -1 which will then set to 0 on the for loop
                    // to start over again.
                    //
                    // See https://github.com/netty/netty/issues/1523
                    selectedKeys = this.selectedKeys.flip();
                    i = -1;
                }
            }
        }
    
        /**
         *4-1-2
         **/
        private void processSelectedKeysPlain(Set<SelectionKey> selectedKeys) {
            // check if the set is empty and if so just return to not create garbage by
            // creating a new Iterator every time even if there is nothing to process.
            // See https://github.com/netty/netty/issues/597
            if (selectedKeys.isEmpty()) {
                return;
            }
            //循环遍历selectedKeys，进行操作
            Iterator<SelectionKey> i = selectedKeys.iterator();
            for (;;) {
                //获取单个的进行，并从迭代器中删除
                final SelectionKey k = i.next();
                final Object a = k.attachment();
                i.remove();
    
                //如果为AbstractNioChannel类型，进行IO读写相关操作
                if (a instanceof AbstractNioChannel) {
                    //4-1-2-1
                    processSelectedKey(k, (AbstractNioChannel) a);
                } else {
                    //taks类型
                    @SuppressWarnings("unchecked")
                    NioTask<SelectableChannel> task = (NioTask<SelectableChannel>) a;
                    processSelectedKey(k, task);
                }
    
                if (!i.hasNext()) {
                    break;
                }
    
                if (needsToSelectAgain) {
                    selectAgain();
                    selectedKeys = selector.selectedKeys();
    
                    // Create the iterator again to avoid ConcurrentModificationException
                    if (selectedKeys.isEmpty()) {
                        break;
                    } else {
                        i = selectedKeys.iterator();
                    }
                }
            }
        }
    
        /**
         *4-1-2-1
         **/
        private static void processSelectedKey(SelectionKey k, AbstractNioChannel ch) {
            final AbstractNioChannel.NioUnsafe unsafe = ch.unsafe();
            //判断是否可用，不可用直接关闭返回
            if (!k.isValid()) {
                // close the channel if the key is not valid anymore
                unsafe.close(unsafe.voidPromise());
                return;
            }
    
            //进行readyOps判断并调用unsafe进行相应的操作，读、写
            try {
                int readyOps = k.readyOps();
                // Also check for readOps of 0 to workaround possible JDK bug which may otherwise lead
                // to a spin loop
                if ((readyOps & (SelectionKey.OP_READ | SelectionKey.OP_ACCEPT)) != 0 || readyOps == 0) {
                    unsafe.read();
                    if (!ch.isOpen()) {
                        // Connection already closed - no need to handle write.
                        return;
                    }
                }
                if ((readyOps & SelectionKey.OP_WRITE) != 0) {
                    // Call forceFlush which will also take care of clear the OP_WRITE once there is nothing left to write
                    ch.unsafe().forceFlush();
                }
                if ((readyOps & SelectionKey.OP_CONNECT) != 0) {
                    // remove OP_CONNECT as otherwise Selector.select(..) will always return without blocking
                    // See https://github.com/netty/netty/issues/924
                    int ops = k.interestOps();
                    ops &= ~SelectionKey.OP_CONNECT;
                    k.interestOps(ops);
    
                    unsafe.finishConnect();
                }
            } catch (CancelledKeyException ignored) {
                unsafe.close(unsafe.voidPromise());
            }
        }
    
    
        /**
         * 5-1 执行定时任务
         * Poll all tasks from the task queue and run them via {@link Runnable#run()} method.
         *
         * @return {@code true} if and only if at least one task was run
         */
        protected boolean runAllTasks() {
            //5-1-1
            fetchFromScheduledTaskQueue();
            Runnable task = pollTask();
            if (task == null) {
                return false;
            }
    
            for (;;) {
                try {
                    task.run();
                } catch (Throwable t) {
                    logger.warn("A task raised an exception.", t);
                }
    
                task = pollTask();
                if (task == null) {
                    lastExecutionTime = ScheduledFutureTask.nanoTime();
                    return true;
                }
            }
        }
    
        /**
         * 5-1-1将到时间的任务加入到taskqueue中供执行
         */
        private void fetchFromScheduledTaskQueue() {
            if (hasScheduledTasks()) {
                long nanoTime = AbstractScheduledEventExecutor.nanoTime();
                for (;;) {
                    Runnable scheduledTask = pollScheduledTask(nanoTime);
                    if (scheduledTask == null) {
                        break;
                    }
                    taskQueue.add(scheduledTask);
                }
            }
        }
    
    
        /**
         * 6-1 关闭所有链路，释放线程、各种资源
         */
        private void closeAll() {
            selectAgain();
            Set<SelectionKey> keys = selector.keys();
            Collection<AbstractNioChannel> channels = new ArrayList<AbstractNioChannel>(keys.size());
            for (SelectionKey k: keys) {
                Object a = k.attachment();
                if (a instanceof AbstractNioChannel) {
                    channels.add((AbstractNioChannel) a);
                } else {
                    k.cancel();
                    @SuppressWarnings("unchecked")
                    NioTask<SelectableChannel> task = (NioTask<SelectableChannel>) a;
                    invokeChannelUnregistered(task, k, null);
                }
            }
    
            for (AbstractNioChannel ch: channels) {
                ch.unsafe().close(ch.unsafe().voidPromise());
            }
        }

Netty的线程模型，Reactor的设计非常牛逼的，直接决定了软件的性能和并发处理能力。多学习，多思考，多反复，多总结……  继续中……

[Reactor]: https://blog.csdn.net/liujiahan629629/article/details/83684831
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpdWppYWhhbjYyOTYyOQ_size_16_color_FFFFFF_t_70]: /images/20220412/db2caea3a4df466fabcc51866b573f53.png
[Netty_ChannelPipeline]: https://blog.csdn.net/liujiahan629629/article/details/84574445
[Netty_ChannelHandler]: https://blog.csdn.net/liujiahan629629/article/details/84641300
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpdWppYWhhbjYyOTYyOQ_size_16_color_FFFFFF_t_70 1]: /images/20220412/d2bca6676bb1482c8526efff0c1d4469.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpdWppYWhhbjYyOTYyOQ_size_16_color_FFFFFF_t_70 2]: /images/20220412/b5b876b4020d4843bb8cd3f69f795af3.png