The C10K problem翻译

阳光穿透心脏的1/2处 2022-04-10 13:42 406阅读 0赞

来自[http://www.cnblogs.com/fll/archive/2008/05/17/1201540.html][http_www.cnblogs.com_fll_archive_2008_05_17_1201540.html]

# The C10K problem #

如今的web服务器需要同时处理一万个以上的客户端了，难道不是吗？毕竟如今的网络是个big place了。

现在的计算机也很强大了，你只需要花大概$1200就可以买一个1000MHz的处理器，2G的内存， 1000Mbit/sec的网卡的机器。让我们来看看--20000个客户，每个为50KHz，100Kbyes和 50Kbit/sec，那么没有什么比为这两万个客户端的每个每秒从硬盘读取4千字节然后发送到网络上 去更消耗资源的了。可以看出硬件不再是瓶颈了。 (That works out to $0.08 per client, by the way. Those $100/client licensing fees some operating systems charge are starting to look a little heavy!)

在1999年最繁忙的ftp站点，cdrom.com，尽管有G比特的网络带宽，却也只能同时处理10000个 客户端。在2001年，同样的速度可以被[几个ISP服务商][ISP]所提供，他们预期该趋势会因为大量的商业 用户而变得越来越普遍。

目前的瘦客户端模型也开始又变得流行起来了--服务器运行在Internet上，为数千个客户端服务。

基于以上一些考虑，这里就配置操作系统或者编写支持数千个网络客户端的代码问题提出一些 注意点，该论题是基于类Unix操作系统的--该系统是我的个人爱好，当然Windows也有占有一席之地。

## Contents ##

*  [The C10K problem][]
 *  [相关网站][Link 1]
 *  [须首先阅读的书籍][Link 2]
 *  [I/O框架][I_O]
 *  [I/O策略][I_O 1]
    
    1.  [Serve many clients with each thread, and use nonblocking I/O and **level-triggered** readiness notification][Serve many clients with each thread_ and use nonblocking I_O and _level-triggered_ readiness notification]
        
         *  [The traditional select()][The traditional select]
         *  [The traditional poll()][The traditional poll]
         *  [/dev/poll][dev_poll] (Solaris 2.7+)
         *  [kqueue][] (FreeBSD, NetBSD)
    2.  [Serve many clients with each thread, and use nonblocking I/O and readiness**change** notification][Serve many clients with each thread_ and use nonblocking I_O and readiness_change_ notification]
        
         *  [epoll][] (Linux 2.6+)
         *  [Polyakov's kevent][Polyakov_s kevent] (Linux 2.6+)
         *  [Drepper's New Network Interface][Drepper_s New Network Interface] (proposal for Linux 2.6+)
         *  [Realtime Signals][] (Linux 2.4+)
         *  [Signal-per-fd][]
         *  [kqueue][] (FreeBSD, NetBSD)
    3.  [Serve many clients with each thread, and use asynchronous I/O and completion notification][Serve many clients with each thread_ and use asynchronous I_O and completion notification]
    4.  [Serve one client with each server thread][]
        
         *  [LinuxThreads][] (Linux 2.0+)
         *  [NGPT][] (Linux 2.4+)
         *  [NPTL][] (Linux 2.6, Red Hat 9)
         *  [FreeBSD threading support][]
         *  [NetBSD threading support][]
         *  [Solaris threading support][]
         *  [Java threading support in JDK 1.3.x and earlier][]
         *  [Note: 1:1 threading vs. M:N threading][Note_ 1_1 threading vs. M_N threading]
    5.  [Build the server code into the kernel][]
 *  [Comments][]
 *  [Limits on open filehandles][]
 *  [Limits on threads][]
 *  [Java issues][] \[Updated 27 May 2001\]
 *  [Other tips][]
    
     *  [Zero-Copy][]
     *  [The sendfile() system call can implement zero-copy networking.][The sendfile_ system call can implement zero-copy networking.]
     *  [Avoid small frames by using writev (or TCP\_CORK)][Avoid small frames by using writev _or TCP_CORK]
     *  [Some programs can benefit from using non-Posix threads.][]
     *  [Caching your own data can sometimes be a win.][]
 *  [Other limits][]
 *  [Kernel Issues][]
 *  [Measuring Server Performance][]
 *  [Examples][]
    
     *  [Interesting select()-based servers][Interesting select_-based servers]
     *  [Interesting /dev/poll-based servers][Interesting _dev_poll-based servers]
     *  [Interesting kqueue()-based servers][Interesting kqueue_-based servers]
     *  [Interesting realtime signal-based servers][]
     *  [Interesting thread-based servers][]
     *  [Interesting in-kernel servers][]
 *  [Other interesting links][]

## Related Sites ##

2003年10月，Felix von Leitner整理了一个很好的[网站][Link 3]和一个 [presentation][]，该网站介绍了网络的可测量性，完成 了以不同网络系统调用和不同的操作系统为基准的性能比较。其中一项就是2.6版本的Linux内核 击败了2.4的内核，当然还有许多的图片可以给OS的开发者在平时提供点想法。  
(See also the [Slashdot][] comments; it'll be interesting to see whether anyone does followup benchmarks improving on Felix's results.)

## Book to Read First ##

如果你还没有读过W.Richard Stevens先生的[《Unix网络编程:第一卷》][Unix]的话，请尽快获取一份 拷贝，该书描述了许多关于编写高性能的服务器的I/O策略和各自的一些缺陷，甚至还讲述 了["thundering herd"][thundering herd]问题，同时你也可以阅读[Jeff Darcy写的关于高性能服务器设计][Jeff Darcy]的一些 notes。  
(Another book which might be more helpful for those who are \*using\* rather than \*writing\* a web server is [Building Scalable Web Sites][] by Cal Henderson.)

### I/O框架 ###

以下所列的为几个包装好的库，它们概要了几中常见的技巧，并且可以使你的代码与具体操作 系统隔离，从而具有更好的移植性。

*  [ACE][], 一个重量级的C++ I/O框架，用面向对象实现了一些I/O策略和其它有用的东西，特别的， 它的Reactor是用OO方式处理非阻塞I/O，而Proactor是用OO方式处理异步I/O的( In particular, his Reactor is an OO way of doing nonblocking I/O, and Proactor is an OO way of doing asynchronous I/O).
 *  [ASIO][] 一个C++的I/O框架，逐渐成为Boost库的一部分。it's like ACE updated for the STL era。
 *  [libevent][] 由Niels Provos用C编写的一个轻量级的I/O框架。它支持kqueue和select，并且很 快就可以支持poll和epoll(翻译此文时已经支持)。我想它应该是只采用了水平触发机制，该机制 有好处当然也有不好的地方。Niels给出了[一张图][Link 4] 来说明时间和连接数目在处理一个事件上的功能，从图上可以看出kqueue和sys\_epoll明显胜出。
 *  我本人也尝试过轻量级的框架(很可惜没有维持至今):
    
     *  [Poller][] 是一个轻量级的C++ I/O框架，它使用任何一种准备就绪API(poll, select, /dev/poll, kqueue, sigio)实现水平触发准备就绪API。以其它[不同的API为基准 ][Poller]，Poller的性能 好得多。该链接文档的下面一部分说明了如何使用这些准备就绪API。
     *  [rn][] 是一个轻量级的C I/O框架，也是我继Poller后的第二个框架。该框架可以很容易的被用 于商业应用中，也容易的适用于非C++应用中。它如今已经在几个商业产品中使用。
 *  Matt Welsh在2000年四月关于在构建服务器方面如何平衡工作线程和事件驱动技术的使用写了 一篇[论文][Link 5]，在该论文中描述了他自己的Sandstorm I/O框架。
 *  [Cory Nelson's Scale! library][Cory Nelson_s Scale_ library] \- 一个Windows下的异步套接字，文件和管道的库。

## I/O Strategies ##

网络软件设计者往往有很多种选择，以下列出一些：

*  是否处理多个I/O？如何处理在单一线程中的多个I/O调用？
    
     *  不处理，从头到尾使用阻塞和同步I/O调用，可以使用多线程或多进程来达到并发效果。
     *  使用非阻塞调用（如在一个设置O\_NONBLOCK选项的socket上使用write）读取I/O，当I/O完 成时发出通知（如poll，/dev/poll）从而开始下一个I/O。这种主要使用在网络I/O上，而不是磁盘的I/O上。
     *  使用异步调用（如aio\_write()）读取I/O，当I/O完成时会发出通知（如信号或者完成端口），可以同时使用在网络I/O和磁盘I/O上。
 *  如何控制对每个客户的服务?
    
     *  对每个客户使用一个进程（经典的Unix方法，自从1980年一直使用）
     *  一个系统级的线程处理多个客户，每个客户是如下一种：
        
         *  a user-level thread (e.g. GNU state threads, classic Java with green threads)
         *  a state machine (a bit esoteric, but popular in some circles; my favorite)
         *  a continuation (a bit esoteric, but popular in some circles)
     *  o一个系统级的线程对应一个客户端(e.g. classic Java with native threads)
     *  一个系统级的线程对应每一个活动的客户端(e.g. Tomcat with apache front end; NT完成端口; 线程池)
 *  是否使用标准的操作系统服务，还是把一些代码放入内核中（如自定义驱动，内核模块，VxD）。

下面的五种方式应该是最常用的了。

1.  [一个线程服务多个客户端，使用非阻塞I/O和**水平触发**的就绪通知][Serve many clients with each thread_ and use nonblocking I_O and _level-triggered_ readiness notification]
2.  [一个线程服务多个客户端，使用非阻塞I/O和就绪**改变**时通知][Serve many clients with each thread_ and use nonblocking I_O and readiness_change_ notification]
3.  [一个服务线程服务多个客户端，使用异步I/O][Serve many clients with each thread_ and use asynchronous I_O and completion notification]
4.  [一个服务线程服务一个客户端，使用阻塞I/O][Serve one client with each server thread]
5.  [把服务代码编译进内核][Build the server code into the kernel]

### 1. 一个线程服务多个客户端，使用非阻塞I/O和**水平触发**的就绪通知 ###

...把网络句柄设置为非阻塞模型，然后使用select()或poll()来告知哪个句柄已有数据在等待 处理。此模型是最传统的，在此模型下，由内核告知你某个文件描述符是否准备好，是否已经完 成你的任务自从上次内核告知已准备好以来（“水平触发”这个名字来源计算机硬件设计，与其 相对的是“[边缘触发][Serve many clients with each thread_ and use nonblocking I_O and readiness_change_ notification]”，Jonathon Lemon在它的[关于kqueue() 的论文][kqueue_]中介绍了这两个术语）。

注意：牢记内核的就绪通知仅仅只是个提示，当你试图从一个文件描述符读取数据时，该文件 描述符可能并没有准备好。这就是为什么需要在使用就绪通知的时候使用非阻塞模型的原因。

一个重要的瓶颈是read()或sendfile()从磁盘块读取时，如果该页当前并不在内存中。设置磁 盘文件描述符为非阻塞没有任何影响。同样的问题也发生在内存映射磁盘文件中。首先一个服务 需要磁盘I/O时，进程块和所有的客户端都必须等待，因此最初的非线程的性能就被消耗了。  
这也是异步I/O的目的，当然仅限于没有AIO的系统。处理磁盘I/O的工作线程或工作进程也可能遭遇此 瓶颈。一条途径就是使用内存映射文件，如果mincore()指明I/O必需的话，那么要求一个工作线 程来完成此I/O，然后继续处理网络事件。Jef Poskanzer提到Pai，Druschel和Zwaenepoel的 [Flash web服务器][Flash web]使用了这个方法，并且他们就此在[Usenix'99][Usenix_99]上做了一个演讲，看上去就好像 [FreeBSD和Solaris 中提供了mincore()一样，但是它并不是][FreeBSD_Solaris _mincore][Single Unix Specification][]的一部分，在Linux的2.3.51 的内核中提供了该方法，感谢[Chuck Lever][]。

在[2003.11的 freebsd-hackers list中，Vivek Pei][2003.11_ freebsd-hackers list_Vivek Pei]上报了一个不错的成果，他们利用系统剖析 工具剖析它们的Flash Web服务器，然后再攻击其瓶颈。其中找到的一个瓶颈就是mincore（猜测 毕竟不是好办法），另外一个就是sendfile在磁盘块访问时。他们修改了sendfile()，当需要读 取的页不在内存中时则返回类似EWOULDBLOCK的值，从而提高了性能。The end result of their optimizations is a SpecWeb99 score of about 800 on a 1GHZ/1GB FreeBSD box, which is better than anything on file at spec.org.

在非阻塞套接字的集合中，关于单一线程是如何告知哪个套接字是准备就绪的，以下列出了几 种方法:

*  **传统的select()**  
    遗憾的是，select()受限于FD\_SETSIZE个句柄。该限制被编译进了标准库和用户程序（有些 版本的C library允许你在用户程序编译时放宽该限制）。
    
    See [Poller\_select][Poller_select] ([cc][], [h][]) for an example of how to use select() interchangeably with other readiness notification schemes.
 *  **传统的poll()**  
    poll()虽然没有文件描述符个数的硬编码限制，但是当有数千个时速度就会变得很慢，因为 大多数的文件描述符在某个时间是空闲的，彻底扫描数千个描述符是需要花费一定时间的。
    
    有些操作系统（如Solaris 8）通过使用了poll hinting技术改进了poll()，该技术由[Niels Provos在1999年实现并利用基准测试程序][Niels Provos_1999]测试过。
    
    See [Poller\_poll][Poller_poll] ([cc][cc 1], [h][h 1], [benchmarks][]) for an example of how to use poll() interchangeably with other readiness notification schemes.
 *  **/dev/poll**  
    这是在Solaris中被推荐的代替poll的方法。
    
    /dev/poll的背后思想就是利用poll()在大部分的调用时使用相同的参数。使用/dev/poll时 ，首先打开/dev/poll得到文件描述符，然后把你关心的文件描述符写入到/dev/poll的描述符， 然后你就可以从/dev/poll的描述符中读取到已就绪的文件描述符。
    
    /dev/poll 在Solaris 7([see patchid 106541][]) 中就已经存在，不过在[Solaris 8][] 中才公开现身。[在750个客户端的情况下][750]，this has 10% of the overhead of poll()。
    
    关于/dev/poll在Linux上有多种不同的尝试实现，但是没有一种可以和epoll相比，不推荐在 Linux上使用/dev/poll。
    
    See [Poller\_devpoll][Poller_devpoll] ([cc][cc 2], [h][h 2] [benchmarks][] ) for an example of how to use /dev/poll interchangeably with many other readiness notification schemes. (Caution - the example is for Linux /dev/poll, might not work right on Solaris.)
 *  **kqueue()**  
    这是在FreeBSD系统上推荐使用的代替poll的方法(and, soon, NetBSD).
    
    [kqueue()][kqueue]即可以水平触发，也可以边缘触发，具体请看下面.

### 2. 一个线程服务多个客户端，使用非阻塞I/O和就绪**改变**时通知 ###

Readiness change notification（或边缘触发就绪通知）的意思就是当你给内核一个文件描述 符，一段时间后，如果该文件描述符从*没有就绪*到已经*准备就绪*，那么内核就会发出通知，告知 该文件描述符已经就绪，并且不会再对该描述符发出类似的就绪通知直到你在描述符上进行一些 操作使得该描述符不再就绪（如直到在send，recv或者accept等调用上遇到EWOULDBLOCK错误，或 者发送/接收了少于需要的字节数）。

当使用Readiness change notification时，必须准备好处理乱真事件，因为最常见的实现是只 要接收到任何数据包都发出就绪信号，而不管文件描述符是否准备就绪。

这是[水平触发][Serve many clients with each thread_ and use nonblocking I_O and _level-triggered_ readiness notification]的就绪通知的相对应的机制。It's a bit less forgiving of programming mistakes, since if you miss just one event, the connection that event was for gets stuck forever. 然而，我发现edge-triggered readiness notification可以使编写带OpenSSL的 非阻塞客户端更简单，可以试下。

[\[Banga, Mogul, Drusha '99\]][Banga_ Mogul_ Drusha _99]详细描述了这种模型.

有几种APIs可以使得应用程序获得“文件描述符已就绪”的通知:

*  **kqueue()** 这是在FreeBSD系统上推荐使用边缘触发的方法 (and, soon, NetBSD).
    
    FreeBSD 4.3及以后版本，[NetBSD（2002.10）][NetBSD_2002.10]都支持 [kqueue()/kevent()][kqueue_kevent]， 支持边沿触发和水平触发（请查看[Jonathan Lemon][] 的网页和他的BSDCon 2000关于[kqueue][kqueue_]的论文）。
    
    就像/dev/poll一样，你分配一个监听对象，不过不是打开文件/dev/poll，而是调用kqueue ()来获得。需要改变你所监听的事件或者获得当前事件的列表，可以在kqueue()返回的描述符上 调用kevent()来达到目的。它不仅可以监听套接字，还可以监听普通的文件的就绪，信号和I/O完 成的事件也可以.
    
    **Note:** 在2000.10，FreeBSD的线程库和kqueue()并不能一起工作得很好，当kqueue()阻塞时， 那么整个进程都将会阻塞，而不仅仅是调用kqueue()的线程。
    
    See [Poller\_kqueue][Poller_kqueue] ([cc][cc 3], [h][h 3], [benchmarks][]) for an example of how to use kqueue() interchangeably with many other readiness notification schemes.
    
    使用kqueue()的例程和库:
    
     *  [PyKQueue][] \-- 一个Python的kqueue()库.
     *  [Ronald F.Guilmette的echo的服务器例程][Ronald F.Guilmette_echo]; 另外可以查看他在 [2000.9.28在freebsd][2000.9.28_freebsd] 上发表的帖子。
 *  **epoll**  
    这是Linux 2.6的内核中推荐使用的边沿触发poll.
    
    2001.7.11， Davide Libenzi提议了一个实时信号的可选方法，他称之为[/dev/epoll< /a>， 该方法类似与实时信号就绪通知机制，但是结合了其它更多的事件，从而在大多数的事件获取上拥有更高的效率。][dev_epoll_ _a_]
    
    [epoll在将它的接口从一个/dev下的指定文件改变为系统调用sys\_epoll后就合并到2.5版本的 Linux内核开发树中，另外也提供了一个为2.4老版本的内核可以使用epoll的补丁。][dev_epoll_ _a_]
    
    [unifying epoll, aio, 2002 年万圣节前夕的Linux内核邮件列表就][dev_epoll_ _a_][统一epoll，aio和其它的event sources][epoll_aio_event sources] 展开了很久的争论，it may yet happen，but Davide is concentrating on firming up epoll in general first.
 *  Polyakov's kevent (Linux 2.6+) 的最后新闻：2006.2.9和2006.7.9，Evgeniy Polyakov发表了融合epoll和 aio的补丁，他的目标是支持网络AIO. See:
    
     *  [the LWN article about kevent][]
     *  [his July announcement][]
     *  [his kevent page][]
     *  [his naio page][]
     *  [some recent discussion][]
 *  Drepper的最新网络接口 (proposal for Linux 2.6+)  
    在2006 OLS上，Ulrich Drepper提议了一种最新的高速异步网络API. See:
    
     *  his paper, "[The Need for Asynchronous, Zero-Copy Network I/O][The Need for Asynchronous_ Zero-Copy Network I_O]"
     *  [his slides][]
     *  [LWN article from July 22][]
 *  **Realtime Signals实时信号**  
    Linux2.4内核中推荐使用的边沿触发poll.
    
    2.4的linux内核可以通过实时信号来分派套接字事件,示例如下:
    
        /* Mask off SIGIO and the signal you want to use. */ sigemptyset(&sigset); sigaddset(&sigset, signum); sigaddset(&sigset, SIGIO); sigprocmask(SIG_BLOCK, &m_sigset, NULL); /* For each file descriptor, invoke F_SETOWN, F_SETSIG, and set O_ASYNC. */ fcntl(fd, F_SETOWN, (int) getpid()); fcntl(fd, F_SETSIG, signum); flags = fcntl(fd, F_GETFL); flags |= O_NONBLOCK|O_ASYNC; fcntl(fd, F_SETFL, flags);
    
    当正常的I/O函数如read()或write()完成时，发送信号。要使用该段的话，在外层循环中编写 一个普通的poll()，在循环里面，当poll()处理完所有的描述符后，进入 [sigwaitinfo()][sigwaitinfo]循环。 如果sigwaitinfo()或sigtimedwait()返回了实时信号，那么siginfo.si\_fd和 siginfo\_si\_band给出的信息和调用poll()后pollfd.fd和pollfd.revents的几乎一样。如果你处 理该I/O，那么就继续调用sigwaitinfo()。  
    如果sigwaitinfo()返回了传统的SIGIO，那么信号队列溢出了，你必须通过临时[ 改变信号处理 程序为SIG\_DFL来刷新信号队列][_ _SIG_DFL]，然后返回到外层的poll()循环。  
    
    
    See [Poller\_sigio][Poller_sigio] ([cc][cc 4], [h][h 4]) for an example of how to use rtsignals interchangeably with many other readiness notification schemes.
    
    See [Zach Brown's phhttpd][Zach Brown_s phhttpd] 示例代码来如何直接使用这些特点. (Or don't; phhttpd is a bit hard to figure out...)
    
    \[[Provos, Lever, and Tweedie 2000][Provos_ Lever_ and Tweedie 2000]\] 描述了最新的phhttp的基准测试，使用了不同的sigtimewait()和sigtimedwait4()，这些调用可以使你只用一次调用便获得多个信号。 有趣的是，sigtimedwait4()的主要好处是它允许应用程序测量系统负载(so it could [behave appropriately][])（poll()也提供了同样的系统负载 测量）。
 *  **Signal-per-fd**  
    Signal-per-fd是由Chandra和Mosberger提出的对实时信号的一种改进，它可以减少甚至削除实 时信号的溢出通过oalescing redundant events。然而是它的性能并没有epoll好. 论文([www.hpl.hp.com/techreports/2000/HPL-2000-174.html][www.hpl.hp.com_techreports_2000_HPL-2000-174.html]) 比较了它和select()，/dev/poll的性能.  
    
    
    [Vitaly Luban在2001.5.18公布了一个实现Signal-per-fd的补丁][Vitaly Luban_2001.5.18_Signal-per-fd]; 授权见[www.luban.org/GPL/gpl.html][www.luban.org_GPL_gpl.html]. (到2001.9，在很重的负载情况下仍然存在稳定性问题，利用[dkftpbench][]测试在4500个用户时将引发问题.
    
    See [Poller\_sigfd][Poller_sigfd] ([cc][cc 5], [h][h 5]) for an example of how to use signal-per-fd interchangeably with many other readiness notification schemes.

### 3. 一个服务线程服务多个客户端，使用异步I/O ###

该方法目前还没有在Unix上普遍的使用，可能因为很少的操作系统支持异步I/O，或者因为它需 要重新修改应用程序(rethinking your applications)。 在标准Unix下，异步I/O是由["aio\_"接口][aio] 提供的，它把一个信号和值与每一个I/O操作关联起来。信号和其值的队列被有效地分配到用户的 进程上。异步I/O是POSIX 1003.1b实时标准的扩展，也属于Single Unix Specification,version 2.

AIO使用的是边缘触发的完成时通知，例如，当一个操作完成时信号就被加入队列（也可以使用 水平触发的完成时通知，通过调用[aio\_suspend()][aio_suspend]即可， 不过我想很少人会这么做）.

glibc 2.1和后续版本提供了一个普通的实现，仅仅是为了兼容标准，而不是为了获得性能上的提高。

Ben LaHaise编写的Linux AIO实现合并到了2.5.32的内核中，它并没有采用内核线程，而是使 用了一个高效的underlying api，但是目前它还不支持套接字（2.4内核也有了AIO的补丁，不过 2.5/2.6的实现有一定程序上的不同）。更多信息如下:

*  The page "[Kernel Asynchronous I/O (AIO) Support for Linux][Kernel Asynchronous I_O _AIO_ Support for Linux]" which tries to tie together all info about the 2.6 kernel's implementation of AIO (posted 16 Sept 2003)
 *  [Round 3: aio vs /dev/epoll][Round 3_ aio vs _dev_epoll] by Benjamin C.R. LaHaise (presented at 2002 OLS)
 *  [Asynchronous I/O Suport in Linux 2.5][Asynchronous I_O Suport in Linux 2.5], by Bhattacharya, Pratt, Pulaverty, and Morgan, IBM; presented at OLS '2003
 *  [Design Notes on Asynchronous I/O (aio) for Linux][Design Notes on Asynchronous I_O _aio_ for Linux] by Suparna Bhattacharya -- compares Ben's AIO with SGI's KAIO and a few other AIO projects
 *  [Linux AIO home page][] \- Ben's preliminary patches, mailing list, etc.
 *  [linux-aio mailing list archives][]
 *  [libaio-oracle][] \- library implementing standard Posix AIO on top of libaio. [First mentioned by Joel Becker on 18 Apr 2003][].

Suparma建议先看看[AIO的API][AIO_API].

RedHat AS和Suse SLES都在2.4的内核中提供了高性能的实现，与2.6的内核实现相似，但并不完全一样。

2006.2，在网络AIO有了一个新的尝试，具体请看Evgeniy Polyakov的基于[kevent][]的AIO.

1999， **[SGI为Linux实现了一个高速的AIO< /a>][SGI_Linux_AIO_ _a]**[，在到1.1版本时，据说可以很好的工作于磁盘I/O和网 络套接字，且使用了内核线程。目前该实现依然对那些不能等待Ben的AIO套接字支持的人来说是 很有用的。][SGI_Linux_AIO_ _a]

[O'Reilly 的][SGI_Linux_AIO_ _a]["POSIX.4: Programming for the Real World"][POSIX.4_ Programming for the Real World]一书对aio做了很好的介绍.

[这里][Link 6] 有一个指南介绍了早期的非标准的aio实现，可以看看，但是请记住你得把"aioread"转换为"aio\_read"。

注意AIO并没有提供无阻塞的为磁盘I/O打开文件的方法，如果你在意因打开磁盘文件而引起 sleep的话，[Linus建议][Linus]你在另外一个线程中调用open()而不是把希望寄托在对aio\_open()系统调用上。

在Windows下，异步I/O与术语"重叠I/O"和"IOCP"(I/O Completion Port,I/O完成端口)有一定联系。Microsoft的IOCP结合了 先前的如异步I/O(如aio\_write)的技术，把事件完成的通知进行排队(就像使用了aio\_sigevent字段的aio\_write),并且它 为了保持单一IOCP线程的数量从而阻止了一部分请求。（Microsoft's IOCP combines techniques from the prior art like asynchronous I/O (like aio\_write) and queued completion notification (like when using the aio\_sigevent field with aio\_write) with a new idea of holding back some requests to try to keep the number of running threads associated with a single IOCP constant.） 更多信息请看 Mark russinovich在sysinternals.com上的文章 [Inside I/O Completion Ports][Inside I_O Completion Ports]， Jeffrey Richter的书"Programming Server-Side Applications for Microsoft Windows 2000" ([Amazon][],[MSPress][]), [U.S. patent \#06223207][U.S. patent _06223207], or [MSDN][].

### 4. 一个服务线程服务一个客户端，使用阻塞I/O ###

... 让read()和write()阻塞. 这样不好的地方在于需要为每个客户端使用一个完整的栈，从而比较浪费内存。 许多操作系统仍在处理数百个线程时存在一定的问题。如果每个线程使用2MB的栈，那么当你在32位的机器上运行 512（2^30 / 2^21=512）个线程时，你就会用光所有的1GB的用户可访问虚拟内存（Linux也是一样运行在x86上的）。 你可以减小每个线程所拥有的栈内存大小，但是由于大部分线程库在一旦线程创建后就不能增大线程栈大小，所以这样做 就意味着你必须使你的程序最小程度地使用内存。当然你也可以把你的程序运行在64位的处理器上去。

Linux，FreeBSD和Solaris系统的线程库一直在更新，64位的处理器也已经开始在大部分的用户中所使用。 也许在不远的将来，这些喜欢使用一个线程来服务一个客户端的人也有能力服务于10000个客户了。 但是在目前，如果你想支持更多的客户，你最好还是使用其它的方法。

For an unabashedly pro-thread viewpoint, see [Why Events Are A Bad Idea (for High-concurrency Servers)][Why Events Are A Bad Idea _for High-concurrency Servers] by von Behren, Condit, and Brewer, UCB, presented at HotOS IX. Anyone from the anti-thread camp care to point out a paper that rebuts this one? :-)

#### LinuxThreads ####

[LinuxTheads][] 是标准Linux线程库的命名。 它从glibc2.0开始已经集成在glibc库中，并且高度兼容Posix标准，不过在性能和信号的支持度上稍逊一筹。

#### NGPT: Next Generation Posix Threads for Linux下一代LinuxPosix线程 ####

[NGPT][NGPT 1]是一个由IBM发起的项目，其目的是提供更好的Posix兼容的Linux线程支持。 现在已到2.2稳定版，并且运行良好...但是NGPT team [公布][Link 7] 他们正在把NGPT的代码基改为support-only模式，因为他们觉得这才是支持社区长久运行的最好的方式。 NGPT小组将继续改进Linux的线程支持，但主要关注NPTL方面。 (Kudos to the NGPT team for their good work and the graceful way they conceded to NPTL.)

#### NPTL: Native Posix Thread Library for Linux(Linux本地Posix线程库) ####

[NPTL][NPTL 1]是由 [Ulrich Drepper][] ( [glibc][]的主要维护人员)和 [Ingo Molnar][]发起的项目，目的是提供world-class的Posix Linux线程支持。

2003.10.5，NPTL作为一个add-on目录（就像linuxthreads一样）被合并到glibc的cvs树中，所以很有可能随glibc的下一次release而 一起发布。

Red Hat 9是最早的包含NPTL的发行版本（对一些用户来说有点不太方便，但是必须有人来打破这沉默\[break the ice\]...)

NPTL links:

*  [NPTL讨论的邮件列表][NPTL 2]
 *  [NPTL源码][NPTL 1]
 *  [NPTL的最初发表][NPTL 3]
 *  [最初的描述NPTL目标的白皮书][NPTL 4]
 *  [修改的NPTL的最后设计的白皮书][NPTL 5]
 *  [Ingo Molnar][Ingo Molnar 1]最初的基准测试表明可以处理10^6个线程
 *  [Ulrich的基准测试][Ulrich] 比较了LinuxThreads，NPTL和IBM的[NGPT][]的各自性能，结果看来NPTL比NGPT快的多。

这是我尝试写的描述NPTL历史的文章(也可以参考[Jerry Cooperstein的文章][Jerry Cooperstein]):

[2002.3，NGPT小组的Bill Abt，glibc的维护者Ulrich Drepper 和其它人召开了个会议][NPTL 4]来探讨LinuxThreads的发展，会议的一个idea就是要改进mutex的性能。 Rusty Russell [等人][Link 8] 随后实现了 [fast userspace mutexes (futexes)][fast userspace mutexes _futexes], （如今已在NGPT和NPTL中应用了）。 与会的大部分人都认为NGPT应该合并到glibc中。

然而Ulrich Drepper并不怎么喜欢NGPT，他认为他可以做得更好。 (对那些曾经想提供补丁给glibc的人来说，这应该不会令他们感到惊讶:-) 于是在接下来的几个月里，Ulrich Drepper, Ingo Molnar和其它人致力于glibc和内核的改变，然后就弄出了 Native Posix Threads Library (NPTL). NPTL使用了NGPT设计的所有内核改进（kernel enhancement），并且采用了几个最新的改进。 Ingo Molnar[描述了][Link 9] 一下的几个内核改进：

> *NPTL使用了三个由NGPT引入的内核特征: getpid()返回PID，CLONE\_THREAD和futexes; NPTL还使用了(并依赖)也是该项目的一部分的一个更为wider的内核特征集。*
> 
> *一些由NGPT引入内核的items也被修改，清除和扩展，例如线程组的处理(CLONE\_THREAD). \[the CLONE\_THREAD changes which impacted NGPT's compatibility got synced with the NGPT folks, to make sure NGPT does not break in any unacceptable way.\]*
> 
> *这些为NPTL开发的并且后来在NPTL中使用的内核特征都描述在设计白皮书中，[http://people.redhat.com/drepper/nptl-design.pdf][NPTL 5] ...*
> 
> *A short list: TLS support, various clone extensions (CLONE\_SETTLS, CLONE\_SETTID, CLONE\_CLEARTID), POSIX thread-signal handling, sys\_exit() extension (release TID futex upon VM-release), the sys\_exit\_group() system-call, sys\_execve() enhancements and support for detached threads.*
> 
> *There was also work put into extending the PID space - eg. procfs crashed due to 64K PID assumptions, max\_pid, and pid allocation scalability work. Plus a number of performance-only improvements were done as well.*
> 
> *In essence the new features are a no-compromises approach to 1:1 threading - the kernel now helps in everything where it can improve threading, and we precisely do the minimally necessary set of context switches and kernel calls for every basic threading primitive.*

NGPT和NPTL的一个最大的不同就是NPTL是1:1的线程模型，而NGPT是M:N的编程模型(具体请看下面). 尽管这样，[Ulrich的最初的基准测试][Ulrich 1] 还是表明NPTL比NGPT快很多。(NGPT小组期待查看Ulrich的测试程序来核实他的结果.)

#### FreeBSD线程支持 ####

FreeBSD支持LinuxThreads和用户空间的线程库。同样，M:N的模型实现KSE在FreeBSD 5.0中引入。 具体请查看[www.unobvious.com/bsd/freebsd-threads.html][www.unobvious.com_bsd_freebsd-threads.html].

2003.3.25, [Jeff Roberson 发表于freebsd-arch][Jeff Roberson _freebsd-arch]:

> *... 感谢Julian, David Xu, Mini, Dan Eischen,和其它的每一位参加了KSE和libpthread开发的成员所提供的基础， Mini和我已经开发出了一个1：1模型的线程实现，它可以和KSE并行工作而不会带来任何影响。It actually helps bring M:N threading closer by testing out shared bits. ...*

And 2006.7, [Robert Watson提议1:1的线程模型应该为FreeBSD 7.x的默认实现][Robert Watson_1_1_FreeBSD 7.x]:

> *我知道曾经讨论过这个问题，但是我认为随着7.x的向前推进，这个问题应该重新考虑。 在很多普通的应用程序和特定的基准测试中，libthr明显的比libpthread在性能上要好得多。 libthr是在我们大量的平台上实现的，而libpthread却只有在几个平台上。 最主要的是因为我们使得Mysql和其它的大量线程的使用者转换到"libthr"，which is suggestive, also! ... 所以strawman提议:让libthr成为7.x上的默认线程库。*

#### NetBSD线程支持 ####

根据Noriyuki Soda的描述:

> *内核支持M:N线程库是基于调度程序激活模型，合并于2003.1.18当时的NetBSD版本中。*

详情请看Nathan J. Williams, Wasabi Systems, Inc.在2002年的FREENIX上的演示 [An Implementation of Scheduler Activations on the NetBSD Operating System][]。

#### Solaris线程支持 ####

Solaris的线程支持还在进一步提高evolving... 从Solaris 2到Solaris 8，默认的线程库使用的都是M:N模型, 但是Solaris 9却默认使用了1:1线程模型. 查看[Sun多线程编程指南][Sun] 和[Sun的关于Java和Solaris线程的note][Sun_Java_Solaris_note].

#### Java在JDK 1.3.x及更早的线程支持 ####

大家都知道，Java一直到JDK1.3.x都没有支持任何处理网络连接的方法，除了一个线程服务一个客户端的模型之外。[Volanomark][]是一个不错的微型测试程序，可以用来测量在 某个时候不同数目的网络连接时每秒钟的信息吞吐量。在2003.5, JDK 1.3的实现实际上可以同时处理10000个连接，但是性能却严重下降了。 从[Table 4][] 可以看出JVMs可以处理10000个连接，但是随着连接数目的增长性能也逐步下降。

#### Note: 1:1 threading vs. M:N threading ####

在实现线程库的时候有一个选择就是你可以把所有的线程支持都放到内核中（也就是所谓的1：1的模型），也可以 把一些线程移到用户空间上去（也就是所谓的M：N模型）。从某个角度来说, M:N被认为拥有更好的性能，但是由于很难被正确的编写， 所以大部分人都远离了该方法。

*  [为什么Ingo Molnar相对于M：N更喜欢1：1][Ingo Molnar_M_N_1_1]
 *  [Sun改为1：1的模型][Sun_Java_Solaris_note]
 *  [NGPT][NGPT 1]是Linux下的M：N线程库.
 *  Although [Ulrich Drepper计划在新的glibc线程库中使用M：N的模型][NPTL 4], 但是还是[选用了1：1的模型.][NPTL 5]
 *  [MacOSX 也将使用1：1的线程.][MacOSX _1_1_.]
 *  [FreeBSD][]和 [NetBSD][An Implementation of Scheduler Activations on the NetBSD Operating System] 仍然将使用M：N线程，FreeBSD 7.0也倾向于使用1：1的线程（见上面描述），可能M：N线程的拥护者最后证明它是错误的。

### 5. 把服务代码编译进内核 ###

Novell和Microsoft都宣称已经在不同时期完成了该工作，至少NFS的实现完成了该工作。 [khttpd][]在Linux下为静态web页面完成了该工作， Ingo Molnar完成了["TUX" (Threaded linUX webserver)][TUX_ _Threaded linUX webserver] ，这是一个Linux下的快速的可扩展的内核空间的HTTP服务器。 Ingo在[2000.9.1宣布][2000.9.1] alpha版本的TUX可以在[ftp://ftp.redhat.com/pub/redhat/tux][ftp_ftp.redhat.com_pub_redhat_tux]下载, 并且介绍了如何加入其邮件列表来获取更多信息。  
在Linux内核的邮件列表上讨论了该方法的好处和缺点，多数人认为不应该把web服务器放进内核中， 相反内核加入最小的钩子hooks来提高web服务器的性能，这样对其它形式的服务器就有益。 具体请看 [Zach Brown的讨论][Zach Brown] 对比用户级别和内核的http服务器。 在2.4的linux内核中为用户程序提供了足够的权力（power），就像[X15][] 服务器运行的速度和TUX几乎一样，但是它没有对内核做任何改变。

## Comments ##

Richard Gooch曾经写了一篇讨论[I/O选项][I_O 2]的论文。

在2001, Tim Brecht和MMichal Ostrowski为使用简单的select的服务器 [做了各种策略的测度][Link 10] 测试的数据值得看一看。

在2003, Tim Brecht发表了 [userver的源码][userver], 该服务器是整合了Abhishek Chandra, David Mosberger, David Pariag和Michal Ostrowski所写的几个服务器而成的， 可以使用select(), poll(), epoll()和sigio.

回到1999.3, [Dean Gaudet发表][Dean Gaudet]:

> *我一直在问“为什么你们不使用基于select/event的模型，它明显是最快的。”...*

他们的理由是“太难理解了，并且其中关键部分（payoff）不清晰”，但是几个月后，当该模型变得易懂时人们就开始愿意使用它了。

Mark Russinovich写了 [一篇评论][Link 11]和 [文章][Link 12]讨论了在2.2的linux内核只能够I/O策略问题。 尽管某些地方似乎有点错误，不过还是值得去看。特别是他认为Linux2.2的异步I/O (请看上面的F\_SETSIG) 并没有在数据准备好时通知用户进程，而只有在新的连接到达时才有。 这看起来是一个奇怪的误解。 还可以看看 [早期的一些comments][comments], [Ingo Molnar在1999.4.30所举的反例][Ingo Molnar_1999.4.30], [Russinovich在1999.5.2的comments][Russinovich_1999.5.2_comments], Alan Cox的 [反例][Link 13]，和各种 [linux内核邮件][linux]. 我怀疑他想说的是Linux不支持异步磁盘I/O，这在过去是正确的，但是现在SGI已经实现了[KAIO][Serve many clients with each thread_ and use asynchronous I_O and completion notification]，它已不再正确了。

查看页面 [sysinternals.com][Inside I_O Completion Ports]和 [MSDN][MSDN 1]了解一下“完成端口”， 据说它是NT中独特的技术， 简单说，win32的"重叠I/O"被认为是太低水平而不方面使用，“完成端口”是提供了完成事件队列的封装，再加上魔法般的调度， 通过允许更多的线程来获得完成事件如果该端口上的其它已获得完成事件的线程处于睡眠中时（可能正在处理阻塞I/O），从而可以保持运行线程数目恒定（scheduling magic that tries to keep the number of running threads constant by allowing more threads to pick up completion events if other threads that had picked up completion events from this port are sleeping (perhaps doing blocking I/O).

查看[OS/400的I/O完成端口支持][OS_400_I_O].

在1999.9，在linux内核邮件列表上曾有一次非常有趣的讨论，讨论题目为 "[15,000 Simultaneous Connections][15_000 Simultaneous Connections]" (并且延续到[第二周][Link 14]). Highlights:

*  Ed Hall [发表了][Link 15]一些他自己的经验：他已经在运行Solaris的UP P2/333上完成>1000个连接每秒。 他的代码使用了一个很小的线程池（每个cpu 1或者2个线程池），每个线程池使用事件模型来管理大量的客户端连接。
 *  Mike Jagdis[posted an analysis of poll/select overhead][posted an analysis of poll_select overhead], and said "The current select/poll implementation can be improved significantly, especially in the blocking case, but the overhead will still increase with the number of descriptors because select/poll does not, and cannot, remember what descriptors are interesting. This would be easy to fix with a new API. Suggestions are welcome..."
 *  Mike [posted][] about his [work on improving select() and poll()][work on improving select_ and poll].
 *  Mike [posted a bit about a possible API to replace poll()/select()][posted a bit about a possible API to replace poll_select]: "How about a 'device like' API where you write 'pollfd like' structs, the 'device' listens for events and delivers 'pollfd like' structs representing them when you read it? ... "
 *  Rogier Wolff [suggested][] using "the API that the digital guys suggested",[http://www.cs.rice.edu/~gaurav/papers/usenix99.ps][http_www.cs.rice.edu_gaurav_papers_usenix99.ps]
 *  Joerg Pommnitz [pointed out][] that any new API along these lines should be able to wait for not just file descriptor events, but also signals and maybe SYSV-IPC. Our synchronization primitives should certainly be able to do what Win32's WaitForMultipleObjects can, at least.
 *  Stephen Tweedie [asserted][] that the combination of F\_SETSIG, queued realtime signals, and sigwaitinfo() was a superset of the API proposed in http://www.cs.rice.edu/~gaurav/papers/usenix99.ps. He also mentions that you keep the signal blocked at all times if you're interested in performance; instead of the signal being delivered asynchronously, the process grabs the next one from the queue with sigwaitinfo().
 *  Jayson Nordwick [compared][] completion ports with the F\_SETSIG synchronous event model, and concluded they're pretty similar.
 *  Alan Cox [noted][] that an older rev of SCT's SIGIO patch is included in 2.3.18ac.
 *  Jordan Mendelson [posted][posted 1] some example code showing how to use F\_SETSIG.
 *  Stephen C. Tweedie [continued][] the comparison of completion ports and F\_SETSIG, and noted: "With a signal dequeuing mechanism, your application is going to get signals destined for various library components if libraries are using the same mechanism," but the library can set up its own signal handler, so this shouldn't affect the program (much).
 *  [Doug Royer][] noted that he'd gotten 100,000 connections on Solaris 2.6 while he was working on the Sun calendar server. Others chimed in with estimates of how much RAM that would require on Linux, and what bottlenecks would be hit.

Interesting reading!

## Limits on open filehandles ##

*  Any Unix: the limits set by ulimit or setrlimit.
 *  Solaris: see [the Solaris FAQ, question 3.46][the Solaris FAQ_ question 3.46] (or thereabouts; they renumber the questions periodically).
 *  FreeBSD:  
      
    Edit /boot/loader.conf, add the line
    
        set kern.maxfiles=XXXX
    
    where XXXX is the desired system limit on file descriptors, and reboot. Thanks to an anonymous reader, who wrote in to say he'd achieved far more than 10000 connections on FreeBSD 4.3, and says
    
    > "FWIW: You can't actually tune the maximum number of connections in FreeBSD trivially, via sysctl.... You have to do it in the /boot/loader.conf file.  
    > The reason for this is that the zalloci() calls for initializing the sockets and tcpcb structures zones occurs very early in system startup, in order that the zone be both type stable and that it be swappable.  
    > You will also need to set the number of mbufs much higher, since you will (on an unmodified kernel) chew up one mbuf per connection for tcptempl structures, which are used to implement keepalive."
    
    Another reader says
    
    > "As of FreeBSD 4.4, the tcptempl structure is no longer allocated; you no longer have to worry about one mbuf being chewed up per connection."
    
    See also:
    
     *  [the FreeBSD handbook][]
     *  [SYSCTL TUNING][], [LOADER TUNABLES][], and [KERNEL CONFIG TUNING][] in 'man tuning'
     *  [The Effects of Tuning a FreeBSD 4.3 Box for High Performance][], Daemon News, Aug 2001
     *  [postfix.org tuning notes][], covering FreeBSD 4.2 and 4.4
     *  [the Measurement Factory's notes][the Measurement Factory_s notes], circa FreeBSD 4.3
 *  OpenBSD: A reader says
    
    > "In OpenBSD, an additional tweak is required to increase the number of open filehandles available per process: the openfiles-cur parameter in [/etc/login.conf][etc_login.conf] needs to be increased. You can change kern.maxfiles either with sysctl -w or in sysctl.conf but it has no effect. This matters because as shipped, the login.conf limits are a quite low 64 for nonprivileged processes, 128 for privileged."
 *  Linux: See [Bodo Bauer's /proc documentation][Bodo Bauer_s _proc documentation]. On 2.4 kernels:
    
        echo 32768 > /proc/sys/fs/file-max
    
    increases the system limit on open files, and
    
        ulimit -n 32768
    
    increases the current process' limit.
    
    On 2.2.x kernels,
    
        echo 32768 > /proc/sys/fs/file-max echo 65536 > /proc/sys/fs/inode-max
    
    increases the system limit on open files, and
    
        ulimit -n 32768
    
    increases the current process' limit.
    
    I verified that a process on Red Hat 6.0 (2.2.5 or so plus patches) can open at least 31000 file descriptors this way. Another fellow has verified that a process on 2.2.12 can open at least 90000 file descriptors this way (with appropriate limits). The upper bound seems to be available memory.  
    Stephen C. Tweedie [posted][posted 2] about how to set ulimit limits globally or per-user at boot time using initscript and pam\_limit.  
    In older 2.2 kernels, though, the number of open files per process is still limited to 1024, even with the above changes.  
    See also [Oskar's 1998 post][Oskar_s 1998 post], which talks about the per-process and system-wide limits on file descriptors in the 2.0.36 kernel.

## Limits on threads ##

On any architecture, you may need to reduce the amount of stack space allocated for each thread to avoid running out of virtual memory. You can set this at runtime with pthread\_attr\_init() if you're using pthreads.

*  Solaris: it supports as many threads as will fit in memory, I hear.
 *  Linux 2.6 kernels with NPTL: /proc/sys/vm/max\_map\_count may need to be increased to go above 32000 or so threads. (You'll need to use very small stack threads to get anywhere near that number of threads, though, unless you're on a 64 bit processor.) See the NPTL mailing list, e.g. the thread with subject "[Cannot create more than 32K threads?][Cannot create more than 32K threads]", for more info.
 *  Linux 2.4: /proc/sys/kernel/threads-max is the max number of threads; it defaults to 2047 on my Red Hat 8 system. You can set increase this as usual by echoing new values into that file, e.g. "echo 4000 > /proc/sys/kernel/threads-max"
 *  Linux 2.2: Even the 2.2.13 kernel limits the number of threads, at least on Intel. I don't know what the limits are on other architectures. [Mingo posted a patch for 2.1.131 on Intel][] that removed this limit. It appears to be integrated into 2.3.20.
    
    See also [Volano's detailed instructions for raising file, thread, and FD\_SET limits in the 2.2 kernel][Volano_s detailed instructions for raising file_ thread_ and FD_SET limits in the 2.2 kernel]. Wow. This document steps you through a lot of stuff that would be hard to figure out yourself, but is somewhat dated.
 *  Java: See [Volano's detailed benchmark info][Volano_s detailed benchmark info], plus their [info on how to tune various systems][] to handle lots of threads.

## Java issues ##

Up through JDK 1.3, Java's standard networking libraries mostly offered the [one-thread-per-client model][]. There was a way to do nonblocking reads, but no way to do nonblocking writes.

In May 2001, [JDK 1.4][] introduced the package [java.nio][] to provide full support for nonblocking I/O (and some other goodies). See [the release notes][] for some caveats. Try it out and give Sun feedback!

HP's java also includes a [Thread Polling API][].

In 2000, Matt Welsh implemented nonblocking sockets for Java; his performance benchmarks show that they have advantages over blocking sockets in servers handling many (up to 10000) connections. His class library is called [java-nbio][]; it's part of the [Sandstorm][] project. Benchmarks showing [performance with 10000 connections][] are available.

See also [Dean Gaudet's essay][Dean Gaudet_s essay] on the subject of Java, network I/O, and threads, and the [paper][Link 5] by Matt Welsh on events vs. worker threads.

Before NIO, there were several proposals for improving Java's networking APIs:

*  Matt Welsh's [Jaguar system][] proposes preserialized objects, new Java bytecodes, and memory management changes to allow the use of asynchronous I/O with Java.
 *  [Interfacing Java to the Virtual Interface Architecture][], by C-C. Chang and T. von Eicken, proposes memory management changes to allow the use of asynchronous I/O with Java.
 *  [JSR-51][] was the Sun project that came up with the java.nio package. Matt Welsh participated (who says Sun doesn't listen?).

## Other tips ##

*  Zero-Copy  
    Normally, data gets copied many times on its way from here to there. Any scheme that eliminates these copies to the bare physical minimum is called "zero-copy".
    
     *  [Thomas Ogrisegg's zero-copy send patch][Thomas Ogrisegg_s zero-copy send patch] for mmaped files under Linux 2.4.17-2.4.20. Claims it's faster than sendfile().
     *  [IO-Lite][] is a proposal for a set of I/O primitives that gets rid of the need for many copies.
     *  [Alan Cox noted that zero-copy is sometimes not worth the trouble][Link 13] back in 1999. (He did like sendfile(), though.)
     *  Ingo [implemented a form of zero-copy TCP][] in the 2.4 kernel for TUX 1.0 in July 2000, and says he'll make it available to userspace soon.
     *  [Drew Gallatin and Robert Picco have added some zero-copy features to FreeBSD][]; the idea seems to be that if you call write() or read() on a socket, the pointer is page-aligned, and the amount of data transferred is at least a page, \*and\* you don't immediately reuse the buffer, memory management tricks will be used to avoid copies. But see [followups to this message on linux-kernel][] for people's misgivings about the speed of those memory management tricks.
        
        According to a note from Noriyuki Soda:
        
        > *Sending side zero-copy is supported since NetBSD-1.6 release by specifying "SOSEND\_LOAN" kernel option. This option is now default on NetBSD-current (you can disable this feature by specifying "SOSEND\_NO\_LOAN" in the kernel option on NetBSD\_current). With this feature, zero-copy is automatically enabled, if data more than 4096 bytes are specified as data to be sent.*
     *  The sendfile() system call can implement zero-copy networking.  
        The sendfile() function in Linux and FreeBSD lets you tell the kernel to send part or all of a file. This lets the OS do it as efficiently as possible. It can be used equally well in servers using threads or servers using nonblocking I/O. (In Linux, it's poorly documented at the moment; [use \_syscall4 to call it][use _syscall4 to call it]. Andi Kleen is writing new man pages that cover this. See also [Exploring The sendfile System Call][] by Jeff Tranter in Linux Gazette issue 91.) [Rumor has it][], ftp.cdrom.com benefitted noticeably from sendfile().
        
        A zero-copy implementation of sendfile() is on its way for the 2.4 kernel. See [LWN Jan 25 2001][].
        
        One developer using sendfile() with Freebsd reports that using POLLWRBAND instead of POLLOUT makes a big difference.
        
        Solaris 8 (as of the July 2001 update) has a new system call 'sendfilev'. [A copy of the man page is here.][]. The Solaris 8 7/01 [release notes][] also mention it. I suspect that this will be most useful when sending to a socket in blocking mode; it'd be a bit of a pain to use with a nonblocking socket.
 *  Avoid small frames by using writev (or TCP\_CORK)  
    A new socket option under Linux, TCP\_CORK, tells the kernel to avoid sending partial frames, which helps a bit e.g. when there are lots of little write() calls you can't bundle together for some reason. Unsetting the option flushes the buffer. Better to use writev(), though...
    
    See [LWN Jan 25 2001][] for a summary of some very interesting discussions on linux-kernel about TCP\_CORK and a possible alternative MSG\_MORE.
 *  Behave sensibly on overload.  
    \[[Provos, Lever, and Tweedie 2000][Provos_ Lever_ and Tweedie 2000]\] notes that dropping incoming connections when the server is overloaded improved the shape of the performance curve, and reduced the overall error rate. They used a smoothed version of "number of clients with I/O ready" as a measure of overload. This technique should be easily applicable to servers written with select, poll, or any system call that returns a count of readiness events per call (e.g. /dev/poll or sigtimedwait4()).
 *  Some programs can benefit from using non-Posix threads.  
    Not all threads are created equal. The clone() function in Linux (and its friends in other operating systems) lets you create a thread that has its own current working directory, for instance, which can be very helpful when implementing an ftp server. See Hoser FTPd for an example of the use of native threads rather than pthreads.
 *  Caching your own data can sometimes be a win.  
    "Re: fix for hybrid server problems" by Vivek Sadananda Pai (vivek@cs.rice.edu) on [new-httpd][], May 9th, states:
    
    > "I've compared the raw performance of a select-based server with a multiple-process server on both FreeBSD and Solaris/x86. On microbenchmarks, there's only a marginal difference in performance stemming from the software architecture. The big performance win for select-based servers stems from doing application-level caching. While multiple-process servers can do it at a higher cost, it's harder to get the same benefits on real workloads (vs microbenchmarks). I'll be presenting those measurements as part of a paper that'll appear at the next Usenix conference. If you've got postscript, the paper is available at[http://www.cs.rice.edu/~vivek/flash99/][Flash web]"

## Other limits ##

*  Old system libraries might use 16 bit variables to hold file handles, which causes trouble above 32767 handles. glibc2.1 should be ok.
 *  Many systems use 16 bit variables to hold process or thread id's. It would be interesting to port the [Volano scalability benchmark][Volano_s detailed benchmark info] to C, and see what the upper limit on number of threads is for the various operating systems.
 *  Too much thread-local memory is preallocated by some operating systems; if each thread gets 1MB, and total VM space is 2GB, that creates an upper limit of 2000 threads.
 *  Look at the performance comparison graph at the bottom of[http://www.acme.com/software/thttpd/benchmarks.html][http_www.acme.com_software_thttpd_benchmarks.html]. Notice how various servers have trouble above 128 connections, even on Solaris 2.6? Anyone who figures out why, let me know.  
    Note: if the TCP stack has a bug that causes a short (200ms) delay at SYN or FIN time, as Linux 2.2.0-2.2.6 had, and the OS or http daemon has a hard limit on the number of connections open, you would expect exactly this behavior. There may be other causes.

## Kernel Issues ##

For Linux, it looks like kernel bottlenecks are being fixed constantly. See [Linux Weekly News][], [Kernel Traffic][], [the Linux-Kernel mailing list][], and [my Mindcraft Redux page][].

In March 1999, Microsoft sponsored a benchmark comparing NT to Linux at serving large numbers of http and smb clients, in which they failed to see good results from Linux. See also [my article on Mindcraft's April 1999 Benchmarks][my Mindcraft Redux page] for more info.

See also [The Linux Scalability Project][]. They're doing interesting work, including [Niels Provos' hinting poll patch][Niels Provos_ hinting poll patch], and some work on the [thundering herd problem][thundering herd].

See also [Mike Jagdis' work on improving select() and poll()][work on improving select_ and poll]; here's [Mike's post][posted] about it.

[Mohit Aron (aron@cs.rice.edu) writes that rate-based clocking in TCP can improve HTTP response time over 'slow' connections by 80%.][Mohit Aron _aron_cs.rice.edu_ writes that rate-based clocking in TCP can improve HTTP response time over _slow_ connections by 80_.]

## Measuring Server Performance ##

Two tests in particular are simple, interesting, and hard:

1.  raw connections per second (how many 512 byte files per second can you serve?)
2.  total transfer rate on large files with many slow clients (how many 28.8k modem clients can simultaneously download from your server before performance goes to pot?)

Jef Poskanzer has published benchmarks comparing many web servers. See[http://www.acme.com/software/thttpd/benchmarks.html][http_www.acme.com_software_thttpd_benchmarks.html] for his results.

I also have [a few old notes about comparing thttpd to Apache][] that may be of interest to beginners.

[Chuck Lever keeps reminding us][] about [Banga and Druschel's paper on web server benchmarking][Banga and Druschel_s paper on web server benchmarking]. It's worth a read.

IBM has an excellent paper titled [Java server benchmarks][] \[Baylor et al, 2000\]. It's worth a read.

# Examples #

## Interesting select()-based servers ##

*  [thttpd][] Very simple. Uses a single process. It has good performance, but doesn't scale with the number of CPU's. Can also use kqueue.
 *  [mathopd][]. Similar to thttpd.
 *  [fhttpd][]
 *  [boa][]
 *  [Roxen][]
 *  [Zeus][], a commercial server that tries to be the absolute fastest. See their [tuning guide][].
 *  The other non-Java servers listed at[http://www.acme.com/software/thttpd/benchmarks.html][http_www.acme.com_software_thttpd_benchmarks.html]
 *  [BetaFTPd][]
 *  [Flash-Lite][] \- web server using IO-Lite.
 *  [Flash: An efficient and portable Web server][Flash web] \-- uses select(), mmap(), mincore()
 *  [The Flash web server as of 2003][] \-- uses select(), modified sendfile(), async open()
 *  [xitami][] \- uses select() to implement its own thread abstraction for portability to systems without threads.
 *  [Medusa][] \- a server-writing toolkit in Python that tries to deliver very high performance.
 *  [userver][] \- a small http server that can use select, poll, epoll, or sigio

## Interesting /dev/poll-based servers ##

*  *N. Provos, C. Lever*, ["Scalable Network I/O in Linux,"][Scalable Network I_O in Linux] May, 2000. \[FREENIX track, Proc. USENIX 2000, San Diego, California (June, 2000).\] Describes a version of thttpd modified to support /dev/poll. Performance is compared with phhttpd.

## Interesting kqueue()-based servers ##

*  [thttpd][] (as of version 2.21?)
 *  Adrian Chadd says "I'm doing a lot of work to make squid actually LIKE a kqueue IO system"; it's an official Squid subproject; see[http://squid.sourceforge.net/projects.html\#commloops][http_squid.sourceforge.net_projects.html_commloops]. (This is apparently newer than[Benno][]'s [patch][].)

## Interesting realtime signal-based servers ##

*  Chromium's X15. This uses the 2.4 kernel's SIGIO feature together with sendfile() and TCP\_CORK, and reportedly achieves higher speed than even TUX. The [source is available][]under a community source (not open source) license. See [the original announcement][] by Fabio Riccardi.
 *  Zach Brown's [phhttpd][] \- "a quick web server that was written to showcase the sigio/siginfo event model. consider this code highly experimental and yourself highly mental if you try and use it in a production environment." Uses the [siginfo][Realtime Signals] features of 2.3.21 or later, and includes the needed patches for earlier kernels. Rumored to be even faster than khttpd. See [his post of 31 May 1999][] for some notes.

## Interesting thread-based servers ##

*  [Hoser FTPD][]. See their [benchmark page][].
 *  [Peter Eriksson's phttpd][Peter Eriksson_s phttpd] and
 *  [pftpd][]
 *  The Java-based servers listed at [http://www.acme.com/software/thttpd/benchmarks.html][http_www.acme.com_software_thttpd_benchmarks.html]
 *  Sun's [Java Web Server][] (which has been [reported to handle 500 simultaneous clients][])

## Interesting in-kernel servers ##

*  [khttpd][]
 *  ["TUX" (Threaded linUX webserver)][TUX_ _Threaded linUX webserver] by Ingo Molnar et al. For 2.4 kernel.

## Other interesting links ##

*  [Jeff Darcy's notes on high-performance server design][Jeff Darcy]
 *  [Ericsson's ARIES project][Ericsson_s ARIES project] \-- benchmark results for Apache 1 vs. Apache 2 vs. Tomcat on 1 to 12 processors
 *  [Prof. Peter Ladkin's Web Server Performance][Prof. Peter Ladkin_s Web Server Performance] page.
 *  [Novell's FastCache][Novell_s FastCache] \-- claims 10000 hits per second. Quite the pretty performance graph.
 *  Rik van Riel's [Linux Performance Tuning site][]

--------------------

## Changelog ##

$Log: c10k.html,v $ Revision 1.212 2006/09/02 14:52:13 dank added asio Revision 1.211 2006/07/27 10:28:58 dank Link to Cal Henderson's book. Revision 1.210 2006/07/27 10:18:58 dank Listify polyakov links, add Drepper's new proposal, note that FreeBSD 7 might move to 1:1 Revision 1.209 2006/07/13 15:07:03 dank link to Scale! library, updated Polyakov links Revision 1.208 2006/07/13 14:50:29 dank Link to Polyakov's patches Revision 1.207 2003/11/03 08:09:39 dank Link to Linus's message deprecating the idea of aio_open Revision 1.206 2003/11/03 07:44:34 dank link to userver Revision 1.205 2003/11/03 06:55:26 dank Link to Vivek Pei's new Flash paper, mention great specweb99 score

--------------------

*Copyright 1999-2006 Dan Kegel*  
dank@kegel.com  
Last updated: 2 Sept 2006  
[\[Return to www.kegel.com\]][Return to www.kegel.com] ![pv.aspx_id_24][]

[http_www.cnblogs.com_fll_archive_2008_05_17_1201540.html]: http://www.cnblogs.com/fll/archive/2008/05/17/1201540.html
[ISP]: http://www.senteco.com/telecom/ethernet.htm
[The C10K problem]: http://www.cnblogs.com/fll/admin/EditPosts.aspx?pg=2#top
[Link 1]: http://www.cnblogs.com/fll/admin/EditPosts.aspx?pg=2#related
[Link 2]: http://www.cnblogs.com/fll/admin/EditPosts.aspx?pg=2#books
[I_O]: http://www.cnblogs.com/fll/admin/EditPosts.aspx?pg=2#frameworks
[I_O 1]: http://www.cnblogs.com/fll/admin/EditPosts.aspx?pg=2#strategies
[Serve many clients with each thread_ and use nonblocking I_O and _level-triggered_ readiness notification]: http://www.cnblogs.com/fll/admin/EditPosts.aspx?pg=2#nb
[The traditional select]: http://www.cnblogs.com/fll/admin/EditPosts.aspx?pg=2#nb.select
[The traditional poll]: http://www.cnblogs.com/fll/admin/EditPosts.aspx?pg=2#nb.poll
[dev_poll]: http://www.cnblogs.com/fll/admin/EditPosts.aspx?pg=2#nb./dev/poll
[kqueue]: http://www.cnblogs.com/fll/admin/EditPosts.aspx?pg=2#nb.kqueue
[Serve many clients with each thread_ and use nonblocking I_O and readiness_change_ notification]: http://www.cnblogs.com/fll/admin/EditPosts.aspx?pg=2#nb.edge
[epoll]: http://www.cnblogs.com/fll/admin/EditPosts.aspx?pg=2#nb.epoll
[Polyakov_s kevent]: http://www.cnblogs.com/fll/admin/EditPosts.aspx?pg=2#nb.kevent
[Drepper_s New Network Interface]: http://www.cnblogs.com/fll/admin/EditPosts.aspx?pg=2#nb.newni
[Realtime Signals]: http://www.cnblogs.com/fll/admin/EditPosts.aspx?pg=2#nb.sigio
[Signal-per-fd]: http://www.cnblogs.com/fll/admin/EditPosts.aspx?pg=2#nb.sigfd
[Serve many clients with each thread_ and use asynchronous I_O and completion notification]: http://www.cnblogs.com/fll/admin/EditPosts.aspx?pg=2#aio
[Serve one client with each server thread]: http://www.cnblogs.com/fll/admin/EditPosts.aspx?pg=2#threaded
[LinuxThreads]: http://www.cnblogs.com/fll/admin/EditPosts.aspx?pg=2#threads.linuxthreads
[NGPT]: http://www.cnblogs.com/fll/admin/EditPosts.aspx?pg=2#threads.ngpt
[NPTL]: http://www.cnblogs.com/fll/admin/EditPosts.aspx?pg=2#threads.nptl
[FreeBSD threading support]: http://www.cnblogs.com/fll/admin/EditPosts.aspx?pg=2#threads.freebsd
[NetBSD threading support]: http://www.cnblogs.com/fll/admin/EditPosts.aspx?pg=2#threads.netbsd
[Solaris threading support]: http://www.cnblogs.com/fll/admin/EditPosts.aspx?pg=2#threads.solaris
[Java threading support in JDK 1.3.x and earlier]: http://www.cnblogs.com/fll/admin/EditPosts.aspx?pg=2#threads.java
[Note_ 1_1 threading vs. M_N threading]: http://www.cnblogs.com/fll/admin/EditPosts.aspx?pg=2#1:1
[Build the server code into the kernel]: http://www.cnblogs.com/fll/admin/EditPosts.aspx?pg=2#kio
[Comments]: http://www.cnblogs.com/fll/admin/EditPosts.aspx?pg=2#comments
[Limits on open filehandles]: http://www.cnblogs.com/fll/admin/EditPosts.aspx?pg=2#limits.filehandles
[Limits on threads]: http://www.cnblogs.com/fll/admin/EditPosts.aspx?pg=2#limits.threads
[Java issues]: http://www.cnblogs.com/fll/admin/EditPosts.aspx?pg=2#java
[Other tips]: http://www.cnblogs.com/fll/admin/EditPosts.aspx?pg=2#tips
[Zero-Copy]: http://www.cnblogs.com/fll/admin/EditPosts.aspx?pg=2#zerocopy
[The sendfile_ system call can implement zero-copy networking.]: http://www.cnblogs.com/fll/admin/EditPosts.aspx?pg=2#sendfile
[Avoid small frames by using writev _or TCP_CORK]: http://www.cnblogs.com/fll/admin/EditPosts.aspx?pg=2#writev
[Some programs can benefit from using non-Posix threads.]: http://www.cnblogs.com/fll/admin/EditPosts.aspx?pg=2#nativethreads
[Caching your own data can sometimes be a win.]: http://www.cnblogs.com/fll/admin/EditPosts.aspx?pg=2#caching
[Other limits]: http://www.cnblogs.com/fll/admin/EditPosts.aspx?pg=2#limits.other
[Kernel Issues]: http://www.cnblogs.com/fll/admin/EditPosts.aspx?pg=2#kernel
[Measuring Server Performance]: http://www.cnblogs.com/fll/admin/EditPosts.aspx?pg=2#benchmarking
[Examples]: http://www.cnblogs.com/fll/admin/EditPosts.aspx?pg=2#examples
[Interesting select_-based servers]: http://www.cnblogs.com/fll/admin/EditPosts.aspx?pg=2#examples.nb.select
[Interesting _dev_poll-based servers]: http://www.cnblogs.com/fll/admin/EditPosts.aspx?pg=2#examples.nb./dev/poll
[Interesting kqueue_-based servers]: http://www.cnblogs.com/fll/admin/EditPosts.aspx?pg=2#examples.nb.kqueue
[Interesting realtime signal-based servers]: http://www.cnblogs.com/fll/admin/EditPosts.aspx?pg=2#examples.nb.sigio
[Interesting thread-based servers]: http://www.cnblogs.com/fll/admin/EditPosts.aspx?pg=2#examples.threaded
[Interesting in-kernel servers]: http://www.cnblogs.com/fll/admin/EditPosts.aspx?pg=2#examples.kio
[Other interesting links]: http://www.cnblogs.com/fll/admin/EditPosts.aspx?pg=2#links
[Link 3]: http://bulk.fefe.de/scalability/
[presentation]: http://bulk.fefe.de/scalable-networking.pdf
[Slashdot]: http://developers.slashdot.org/developers/03/10/19/0130256.shtml?tid=106&tid=130&tid=185&tid=190
[Unix]: http://www.amazon.com/exec/obidos/ASIN/013490012X/
[thundering herd]: http://www.citi.umich.edu/projects/linux-scalability/reports/accept.html
[Jeff Darcy]: http://pl.atyp.us/content/tech/servers.html
[Building Scalable Web Sites]: http://www.amazon.com/gp/product/0596102356
[ACE]: http://www.cs.wustl.edu/~schmidt/ACE.html
[ASIO]: http://asio.sf.net/
[libevent]: http://monkey.org/~provos/libevent
[Link 4]: http://monkey.org/~provos/libevent/libevent-benchmark.jpg
[Poller]: http://www.kegel.com/dkftpbench/Poller_bench.html
[rn]: http://www.cnblogs.com/fll/admin/rn/
[Link 5]: http://www.cs.berkeley.edu/~mdw/papers/events.pdf
[Cory Nelson_s Scale_ library]: http://svn.sourceforge.net/viewcvs.cgi/*checkout*/int64/scale/readme.txt
[kqueue_]: http://people.freebsd.org/~jlemon/papers/kqueue.pdf
[Flash web]: http://www.cs.rice.edu/~vivek/flash99/
[Usenix_99]: http://www.usenix.org/events/usenix99/technical.html
[FreeBSD_Solaris _mincore]: http://www.freebsd.org/cgi/man.cgi?query=mincore
[Single Unix Specification]: http://www.unix-systems.org/
[Chuck Lever]: http://www.citi.umich.edu/projects/citi-netscape/status/mar-apr2000.html
[2003.11_ freebsd-hackers list_Vivek Pei]: http://marc.theaimsgroup.com/?l=freebsd-hackers&m=106718343317930&w=2
[Poller_select]: http://www.cnblogs.com/fll/admin/dkftpbench/doc/Poller_select.html
[cc]: http://www.cnblogs.com/fll/admin/dkftpbench/dkftpbench-0.45/Poller_select.cc
[h]: http://www.cnblogs.com/fll/admin/dkftpbench/dkftpbench-0.45/Poller_select.h
[Niels Provos_1999]: http://www.humanfactor.com/cgi-bin/cgi-delegate/apache-ML/nh/1999/May/0415.html
[Poller_poll]: http://www.cnblogs.com/fll/admin/dkftpbench/doc/Poller_poll.html
[cc 1]: http://www.cnblogs.com/fll/admin/dkftpbench/dkftpbench-0.45/Poller_poll.cc
[h 1]: http://www.cnblogs.com/fll/admin/dkftpbench/dkftpbench-0.45/Poller_poll.h
[benchmarks]: http://www.cnblogs.com/fll/admin/dkftpbench/Poller_bench.html
[see patchid 106541]: http://sunsolve.sun.com/pub-cgi/retrieve.pl?patchid=106541&collection=fpatches
[Solaris 8]: http://docs.sun.com/ab2/coll.40.6/REFMAN7/@Ab2PageView/55123?Ab2Lang=C&Ab2Enc=iso-8859-1
[750]: http://www.sun.com/sysadmin/ea/poll.html
[Poller_devpoll]: http://www.cnblogs.com/fll/admin/dkftpbench/doc/Poller_devpoll.html
[cc 2]: http://www.cnblogs.com/fll/admin/dkftpbench/dkftpbench-0.45/Poller_devpoll.cc
[h 2]: http://www.cnblogs.com/fll/admin/dkftpbench/dkftpbench-0.45/Poller_devpoll.h
[Banga_ Mogul_ Drusha _99]: http://www.cs.rice.edu/~druschel/usenix99event.ps.gz
[NetBSD_2002.10]: http://kerneltrap.org/node.php?id=472
[kqueue_kevent]: http://www.freebsd.org/cgi/man.cgi?query=kqueue&apropos=0&sektion=0&manpath=FreeBSD+5.0-current&format=html
[Jonathan Lemon]: http://people.freebsd.org/~jlemon/
[Poller_kqueue]: http://www.cnblogs.com/fll/admin/dkftpbench/doc/Poller_kqueue.html
[cc 3]: http://www.cnblogs.com/fll/admin/dkftpbench/dkftpbench-0.45/Poller_kqueue.cc
[h 3]: http://www.cnblogs.com/fll/admin/dkftpbench/dkftpbench-0.45/Poller_kqueue.h
[PyKQueue]: http://people.freebsd.org/~dwhite/PyKQueue/
[Ronald F.Guilmette_echo]: http://www.monkeys.com/kqueue/echo.c
[2000.9.28_freebsd]: http://groups.yahoo.com/group/freebsd-questions/message/223580
[dev_epoll_ _a_]: http://www.xmailserver.org/linux-patches/nio-improve.html
[epoll_aio_event sources]: http://marc.theaimsgroup.com/?l=linux-kernel&m=103607925020720&%20w=2
[the LWN article about kevent]: http://lwn.net/Articles/172844/
[his July announcement]: http://lkml.org/lkml/2006/7/9/82
[his kevent page]: http://tservice.net.ru/~s0mbre/old/?section=projects&item=kevent
[his naio page]: http://tservice.net.ru/~s0mbre/old/?section=projects&item=naio
[some recent discussion]: http://thread.gmane.org/gmane.linux.network/37595/focus=37673
[The Need for Asynchronous_ Zero-Copy Network I_O]: http://people.redhat.com/drepper/newni.pdf
[his slides]: http://people.redhat.com/drepper/newni-slides.pdf
[LWN article from July 22]: http://lwn.net/Articles/192410/
[sigwaitinfo]: http://www.opengroup.org/onlinepubs/007908799/xsh/sigwaitinfo.html
[_ _SIG_DFL]: http://www.cs.helsinki.fi/linux/linux-kernel/Year-1999/1999-41/0644.html
[Poller_sigio]: http://www.cnblogs.com/fll/admin/dkftpbench/doc/Poller_sigio.html
[cc 4]: http://www.cnblogs.com/fll/admin/dkftpbench/dkftpbench-0.45/Poller_sigio.cc
[h 4]: http://www.cnblogs.com/fll/admin/dkftpbench/dkftpbench-0.45/Poller_sigio.h
[Zach Brown_s phhttpd]: http://www.cnblogs.com/fll/admin/EditPosts.aspx?pg=2#phhttpd
[Provos_ Lever_ and Tweedie 2000]: http://www.citi.umich.edu/techreports/reports/citi-tr-00-7.ps.gz
[behave appropriately]: http://www.cnblogs.com/fll/admin/EditPosts.aspx?pg=2#overload
[www.hpl.hp.com_techreports_2000_HPL-2000-174.html]: http://www.hpl.hp.com/techreports/2000/HPL-2000-174.html
[Vitaly Luban_2001.5.18_Signal-per-fd]: http://boudicca.tux.org/hypermail/linux-kernel/2001week20/1353.html
[www.luban.org_GPL_gpl.html]: http://www.luban.org/GPL/gpl.html
[dkftpbench]: http://www.cnblogs.com/fll/admin/dkftpbench
[Poller_sigfd]: http://www.cnblogs.com/fll/admin/dkftpbench/doc/Poller_sigfd.html
[cc 5]: http://www.cnblogs.com/fll/admin/dkftpbench/dkftpbench-0.45/Poller_sigfd.cc
[h 5]: http://www.cnblogs.com/fll/admin/dkftpbench/dkftpbench-0.45/Poller_sigfd.h
[aio]: http://www.opengroup.org/onlinepubs/007908799/xsh/realtime.html
[aio_suspend]: http://www.opengroup.org/onlinepubs/007908799/xsh/aio_suspend.html
[Kernel Asynchronous I_O _AIO_ Support for Linux]: http://lse.sourceforge.net/io/aio.html
[Round 3_ aio vs _dev_epoll]: http://www.linuxsymposium.org/2002/view_txt.php?text=abstract&talk=11
[Asynchronous I_O Suport in Linux 2.5]: http://archive.linuxsymposium.org/ols2003/Proceedings/All-Reprints/Reprint-Pulavarty-OLS2003.pdf
[Design Notes on Asynchronous I_O _aio_ for Linux]: http://sourceforge.net/docman/display_doc.php?docid=12548&group_id=8875
[Linux AIO home page]: http://www.kvack.org/~blah/aio/
[linux-aio mailing list archives]: http://marc.theaimsgroup.com/?l=linux-aio
[libaio-oracle]: http://www.ocfs.org/aio/
[First mentioned by Joel Becker on 18 Apr 2003]: http://marc.theaimsgroup.com/?l=linux-aio&m=105069158425822&w=2
[AIO_API]: http://www.dafscollaborative.org/tools/dafs_api.pdf
[kevent]: http://www.cnblogs.com/fll/admin/EditPosts.aspx?pg=2#kevent
[SGI_Linux_AIO_ _a]: http://oss.sgi.com/projects/kaio/
[POSIX.4_ Programming for the Real World]: http://www.oreilly.com/catalog/posix4/
[Link 6]: http://sunsite.nstu.nsk.su/sunworldonline/swol-03-1996/swol-03-aio.html
[Linus]: http://www.ussg.iu.edu/hypermail/linux/kernel/0102.1/0124.html
[Inside I_O Completion Ports]: http://www.sysinternals.com/ntw2k/info/comport.shtml
[Amazon]: http://www.amazon.com/exec/obidos/ASIN/0735607532
[MSPress]: http://www.microsoft.com/mspress/books/toc/3402.asp
[U.S. patent _06223207]: http://patft.uspto.gov/netacgi/nph-Parser?Sect1=PTO1&Sect2=HITOFF&d=PALL&p=1&u=/netahtml/srchnum.htm&r=1&f=G&l=50&s1=%276223207%27.WKU.&OS=PN/6223207&RS=PN/6223207
[MSDN]: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/filesio_4z1v.asp
[Why Events Are A Bad Idea _for High-concurrency Servers]: http://www.usenix.org/events/hotos03/tech/vonbehren.html
[LinuxTheads]: http://pauillac.inria.fr/~xleroy/linuxthreads/
[NGPT 1]: http://www-124.ibm.com/pthreads/
[Link 7]: http://www-124.ibm.com/pthreads/docs/announcement
[NPTL 1]: http://people.redhat.com/drepper/nptl/
[Ulrich Drepper]: http://people.redhat.com/drepper/
[glibc]: http://www.gnu.org/software/libc/
[Ingo Molnar]: http://people.redhat.com/mingo/
[NPTL 2]: https://listman.redhat.com/mailman/listinfo/phil-list
[NPTL 3]: http://lwn.net/Articles/10465/
[NPTL 4]: http://people.redhat.com/drepper/glibcthreads.html
[NPTL 5]: http://people.redhat.com/drepper/nptl-design.pdf
[Ingo Molnar 1]: http://marc.theaimsgroup.com/?l=linux-kernel&m=103230439008204&w=2
[Ulrich]: http://marc.theaimsgroup.com/?l=linux-kernel&m=103269598000900&w=2
[Jerry Cooperstein]: http://www.onlamp.com/pub/a/onlamp/2002/11/07/linux_threads.html
[Link 8]: http://marc.theaimsgroup.com/?l=linux-kernel&m=103284847815916&w=2
[fast userspace mutexes _futexes]: http://marc.theaimsgroup.com/?l=linux-kernel&m=102196625921110&w=2
[Link 9]: https://listman.redhat.com/pipermail/phil-list/2002-September/000013.html
[Ulrich 1]: https://listman.redhat.com/pipermail/phil-list/2002-September/000009.html
[www.unobvious.com_bsd_freebsd-threads.html]: http://www.unobvious.com/bsd/freebsd-threads.html
[Jeff Roberson _freebsd-arch]: http://docs.freebsd.org/cgi/getmsg.cgi?fetch=121207+0+archive/2003/freebsd-arch/20030330.freebsd-arch
[Robert Watson_1_1_FreeBSD 7.x]: http://marc.theaimsgroup.com/?l=freebsd-threads&m=115191979412894&w=2
[An Implementation of Scheduler Activations on the NetBSD Operating System]: http://web.mit.edu/nathanw/www/usenix/freenix-sa/
[Sun]: http://docs.sun.com/db/doc/805-5080
[Sun_Java_Solaris_note]: http://java.sun.com/docs/hotspot/threads/threads.html
[Volanomark]: http://www.volano.com/report/
[Table 4]: http://www.volano.com/report/#nettable
[Ingo Molnar_M_N_1_1]: http://marc.theaimsgroup.com/?l=linux-kernel&m=103284879216107&%20w=2
[MacOSX _1_1_.]: http://developer.apple.com/technotes/tn/tn2028.html#MacOSXThreading
[FreeBSD]: http://people.freebsd.org/~julian/
[khttpd]: http://www.fenrus.demon.nl/
[TUX_ _Threaded linUX webserver]: http://slashdot.org/comments.pl?sid=00/07/05/0211257&cid=218
[2000.9.1]: http://marc.theaimsgroup.com/?l=linux-kernel&m=98098648011183&w=2
[ftp_ftp.redhat.com_pub_redhat_tux]: ftp://ftp.redhat.com/pub/redhat/tux
[Zach Brown]: http://www.linuxhq.com/lnxlists/linux-kernel/lk_9906_03/msg01041.html
[X15]: http://www.cnblogs.com/fll/admin/EditPosts.aspx?pg=2#x15
[I_O 2]: http://www.atnf.csiro.au/~rgooch/linux/docs/io-events.html
[Link 10]: http://www.hpl.hp.com/techreports/2001/HPL-2001-314.html
[userver]: http://www.hpl.hp.com/research/linux/userver/
[Dean Gaudet]: http://marc.theaimsgroup.com/?l=apache-httpd-dev&m=92100977123493&w=2
[Link 11]: http://linuxtoday.com/stories/5499.html
[Link 12]: http://www.winntmag.com/Articles/Index.cfm?ArticleID=5048
[comments]: http://www.dejanews.com/getdoc.xp?AN=431444525
[Ingo Molnar_1999.4.30]: http://www.dejanews.com/getdoc.xp?AN=472893693
[Russinovich_1999.5.2_comments]: http://www.linuxhq.com/lnxlists/linux-kernel/lk_9905_01/msg00089.html
[Link 13]: http://www.linuxhq.com/lnxlists/linux-kernel/lk_9905_01/msg00263.html
[linux]: http://www.dejanews.com/dnquery.xp?ST=PS&QRY=threads&DBS=1&%20format=threaded&showsort=score&maxhits=100&LNG=ALL&groups%20=fa.linux.kernel+&fromdate=jun+1+1998
[MSDN 1]: http://msdn.microsoft.com/library/techart/msdn_scalabil.htm
[OS_400_I_O]: http://www.as400.ibm.com/developer/v4r5/api.html
[15_000 Simultaneous Connections]: http://www.cs.helsinki.fi/linux/linux-kernel/Year-1999/1999-36/0160.html
[Link 14]: http://www.cs.helsinki.fi/linux/linux-kernel/Year-1999/1999-37/0612.html
[Link 15]: http://www.linuxhq.com/lnxlists/linux-kernel/lk_9909_01/msg00807.html
[posted an analysis of poll_select overhead]: http://www.linuxhq.com/lnxlists/linux-kernel/lk_9909_01/msg00831.html
[posted]: http://www.linuxhq.com/lnxlists/linux-kernel/lk_9909_01/msg00964.html
[work on improving select_ and poll]: http://www.purplet.demon.co.uk/linux/select/
[posted a bit about a possible API to replace poll_select]: http://www.linuxhq.com/lnxlists/linux-kernel/lk_9909_01/msg00971.html
[suggested]: http://www.linuxhq.com/lnxlists/linux-kernel/lk_9909_01/msg00979.html
[http_www.cs.rice.edu_gaurav_papers_usenix99.ps]: http://www.cs.rice.edu/~gaurav/papers/usenix99.ps
[pointed out]: http://www.linuxhq.com/lnxlists/linux-kernel/lk_9909_02/msg00001.html
[asserted]: http://www.linuxhq.com/lnxlists/linux-kernel/lk_9909_02/msg01198.html
[compared]: http://www.linuxhq.com/lnxlists/linux-kernel/lk_9909_03/msg00002.html
[noted]: http://www.linuxhq.com/lnxlists/linux-kernel/lk_9909_03/msg00043.html
[posted 1]: http://www.linuxhq.com/lnxlists/linux-kernel/lk_9909_03/msg00093.html
[continued]: http://www.linuxhq.com/lnxlists/linux-kernel/lk_9909_03/msg00095.html
[Doug Royer]: http://www.linuxhq.com/lnxlists/linux-kernel/lk_9909_04/msg00900.html
[the Solaris FAQ_ question 3.46]: http://www.wins.uva.nl/pub/solaris/solaris2/Q3.46.html
[the FreeBSD handbook]: http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/configtuning-kernel-limits.html
[SYSCTL TUNING]: http://www.freebsd.org/cgi/man.cgi?query=tuning#SYSCTL+TUNING
[LOADER TUNABLES]: http://www.freebsd.org/cgi/man.cgi?query=tuning#LOADER+TUNABLES
[KERNEL CONFIG TUNING]: http://www.freebsd.org/cgi/man.cgi?query=tuning#KERNEL+CONFIG+TUNING
[The Effects of Tuning a FreeBSD 4.3 Box for High Performance]: http://www.daemonnews.org/200108/benchmark.html
[postfix.org tuning notes]: http://www.postfix.org/faq.html#moby-freebsd
[the Measurement Factory_s notes]: http://www.measurement-factory.com/docs/FreeBSD/
[etc_login.conf]: http://www.freebsd.org/cgi/man.cgi?query=login.conf&manpath=OpenBSD+3.1
[Bodo Bauer_s _proc documentation]: http://asc.di.fct.unl.pt/~jml/mirror/Proc/
[posted 2]: http://www.linuxhq.com/lnxlists/linux-kernel/lk_9909_02/msg01092.html
[Oskar_s 1998 post]: http://www.dejanews.com/getdoc.xp?AN=313316592
[Cannot create more than 32K threads]: https://listman.redhat.com/archives/phil-list/2003-August/msg00005.html
[Mingo posted a patch for 2.1.131 on Intel]: http://www.linuxhq.com/lnxlists/linux-kernel/lk_9812_02/msg00048.html
[Volano_s detailed instructions for raising file_ thread_ and FD_SET limits in the 2.2 kernel]: http://www.volano.com/linux.html
[Volano_s detailed benchmark info]: http://www.volano.com/benchmarks.html
[info on how to tune various systems]: http://www.volano.com/server.html
[one-thread-per-client model]: http://www.cnblogs.com/fll/admin/EditPosts.aspx?pg=2#threaded.java
[JDK 1.4]: http://java.sun.com/j2se/1.4/
[java.nio]: http://java.sun.com/j2se/1.4/docs/guide/nio/
[the release notes]: http://java.sun.com/j2se/1.4/relnotes.html#nio
[Thread Polling API]: http://www.devresource.hp.com/JavaATC/JavaPerfTune/pollapi.html
[java-nbio]: http://www.cs.berkeley.edu/~mdw/proj/java-nbio/
[Sandstorm]: http://www.cs.berkeley.edu/~mdw/proj/sandstorm/
[performance with 10000 connections]: http://www.cs.berkeley.edu/~mdw/proj/sandstorm/iocore-bench/
[Dean Gaudet_s essay]: http://arctic.org/~dean/hybrid-jvm.txt
[Jaguar system]: http://www.cs.berkeley.edu/~mdw/proj/jaguar/
[Interfacing Java to the Virtual Interface Architecture]: http://www.cs.cornell.edu/Info/People/chichao/papers.htm
[JSR-51]: http://java.sun.com/aboutJava/communityprocess/jsr/jsr_051_ioapis.html
[Thomas Ogrisegg_s zero-copy send patch]: http://marc.theaimsgroup.com/?l=linux-kernel&m=104121076420067&w=2
[IO-Lite]: http://www.usenix.org/publications/library/proceedings/osdi99/full_papers/pai/pai_html/pai.html
[implemented a form of zero-copy TCP]: http://boudicca.tux.org/hypermail/linux-kernel/2000week36/0979.html
[Drew Gallatin and Robert Picco have added some zero-copy features to FreeBSD]: http://people.freebsd.org/~ken/zero_copy/
[followups to this message on linux-kernel]: http://boudicca.tux.org/hypermail/linux-kernel/2000week39/0249.html
[use _syscall4 to call it]: http://www.dejanews.com/getdoc.xp?AN=422899634
[Exploring The sendfile System Call]: http://www.linuxgazette.com/issue91/tranter.html
[Rumor has it]: http://www.dejanews.com/getdoc.xp?AN=423005088
[LWN Jan 25 2001]: http://lwn.net/2001/0125/kernel.php3
[A copy of the man page is here.]: http://www.cnblogs.com/fll/admin/sendfilev.txt
[release notes]: http://www.sun.com/software/solaris/fcc/ucc-details701.html
[new-httpd]: http://www.humanfactor.com/cgi-bin/cgi-delegate/apache-ML/nh/1999/
[http_www.acme.com_software_thttpd_benchmarks.html]: http://www.acme.com/software/thttpd/benchmarks.html
[Linux Weekly News]: http://lwn.net/
[Kernel Traffic]: http://www.kt.opensrc.org/
[the Linux-Kernel mailing list]: http://marc.theaimsgroup.com/?l=linux-kernel
[my Mindcraft Redux page]: http://www.cnblogs.com/fll/admin/mindcraft_redux.html
[The Linux Scalability Project]: http://www.citi.umich.edu/projects/citi-netscape/
[Niels Provos_ hinting poll patch]: http://linuxwww.db.erau.edu/mail_archives/linux-kernel/May_99/4105.html
[Mohit Aron _aron_cs.rice.edu_ writes that rate-based clocking in TCP can improve HTTP response time over _slow_ connections by 80_.]: http://www.linuxhq.com/lnxlists/linux-kernel/lk_9910_02/msg00889.html
[a few old notes about comparing thttpd to Apache]: http://www.alumni.caltech.edu/~dank/fixing-overloaded-web-server.html
[Chuck Lever keeps reminding us]: http://linuxhq.com/lnxlists/linux-kernel/lk_9906_02/msg00248.html
[Banga and Druschel_s paper on web server benchmarking]: http://www.cs.rice.edu/CS/Systems/Web-measurement/paper/paper.html
[Java server benchmarks]: http://www.research.ibm.com/journal/sj/391/baylor.html
[thttpd]: http://www.acme.com/software/thttpd/
[mathopd]: http://mathop.diva.nl/
[fhttpd]: http://www.fhttpd.org/
[boa]: http://www.boa.org/
[Roxen]: http://www.roxen.com/
[Zeus]: http://www.zeustech.net/
[tuning guide]: http://support.zeustech.net/faq/entries/tuning.html
[BetaFTPd]: http://ca.us.mirrors.freshmeat.net/appindex/1999/02/17/919251275.html
[Flash-Lite]: http://www.cs.rice.edu/~vivek/iol98/
[The Flash web server as of 2003]: http://www.cs.princeton.edu/~yruan/debox/
[xitami]: http://www.imatix.com/html/xitami/
[Medusa]: http://www.nightmare.com/medusa/medusa.html
[Scalable Network I_O in Linux]: http://www.citi.umich.edu/techreports/reports/citi-tr-00-4.pdf
[http_squid.sourceforge.net_projects.html_commloops]: http://squid.sourceforge.net/projects.html#commloops
[Benno]: http://www.advogato.org/person/benno/
[patch]: http://netizen.com.au/~benno/squid-select.tar.gz
[source is available]: http://www.chromium.com/cgi-bin/crosforum/YaBB.pl
[the original announcement]: http://boudicca.tux.org/hypermail/linux-kernel/2001week21/1624.html
[phhttpd]: http://www.zabbo.net/phhttpd/
[his post of 31 May 1999]: http://www.cs.helsinki.fi/linux/linux-kernel/Year-1999/1999-22/0453.html
[Hoser FTPD]: http://www.zabbo.net/hftpd/
[benchmark page]: http://www.zabbo.net/hftpd/bench.html
[Peter Eriksson_s phttpd]: http://ca.us.mirrors.freshmeat.net/appindex/1999/02/06/918317238.html
[pftpd]: http://ca.us.mirrors.freshmeat.net/appindex/1999/02/06/918313631.html
[Java Web Server]: http://jserv.javasoft.com/
[reported to handle 500 simultaneous clients]: http://archives.java.sun.com/cgi-bin/wa?A2=ind9901&L=jserv-interest&F=&S=&P=47739
[Ericsson_s ARIES project]: http://www2.linuxjournal.com/lj-issues/issue91/4752.html
[Prof. Peter Ladkin_s Web Server Performance]: http://nakula.rvs.uni-bielefeld.de/made/artikel/Web-Bench/web-bench.html
[Novell_s FastCache]: http://www.novell.com/bordermanager/ispcon4.html
[Linux Performance Tuning site]: http://linuxperf.nl.linux.org/
[Return to www.kegel.com]: http://www.kegel.com/
[pv.aspx_id_24]: http://counter.csdn.net/pv.aspx?id=24