python内存管理写的简单没啥深度.......... 不建议看....

太过爱你忘了你带给我的痛 2023-02-09 14:20 58阅读 0赞

### `写的很浅显 有不对的地方见谅一下 欢迎大家提出批评或者给出指导或者xxx` ###

--------------------

### 内存管理 ###

[内容摘选自这里的官文][Link 1]

内存分配结构如下

<table> 
 <thead> 
  <tr> 
   <th>-</th> 
   <th>-</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>3层</td> 
   <td>这里也有两部分内容<br>具体对象的内存<br>python自身核心内存</td> 
  </tr> 
  <tr> 
   <td>2层</td> 
   <td>这一层有两部分内容<br>python对象分配器<br>内部缓冲区</td> 
  </tr> 
  <tr> 
   <td>1层</td> 
   <td>python原始内存分配器<br>使用PyMem_ API相关API<br>在PyMem管理器的控制下</td> 
  </tr> 
  <tr> 
   <td>0层</td> 
   <td>使用底层通用的内存分配器比如c的malloc为python进程分配内存</td> 
  </tr> 
 </tbody> 
</table>

第二层作用

*  负责所有python对象的分配和回收(PyObject\_New/Del)  
    除非某个特定对象的分配器实现了特定的分配规则 比如int的实现就是一个简单的free list
 *  `垃圾回收的处理也是在第二层进行`

Here is an introduction to the layers of the Python memory architecture,
    showing where the object allocator is actually used (layer +2), It is
    called for every object allocation and deallocation (PyObject_New/Del),
    unless the object-specific allocators implement a proprietary allocation
    scheme (ex.: ints use a simple free list). This is also the place where
    the cyclic garbage collector operates selectively on container objects.
    
        Object-specific allocators
        _____   ______   ______       ________
       [ int ] [ dict ] [ list ] ... [ string ]       Python core         |
    +3 | <----- Object-specific memory -----> | <-- Non-object memory --> |
        _______________________________       |                           |
       [   Python's object allocator   ]      |                           |
    +2 | ####### Object memory ####### | <------ Internal buffers ------> |
        ______________________________________________________________    |
       [          Python's raw memory allocator (PyMem_ API)          ]   |
    +1 | <----- Python memory (under PyMem manager's control) ------> |   |
        __________________________________________________________________
       [    Underlying general-purpose allocator (ex: C library malloc)   ]
     0 | <------ Virtual memory allocated for the python process -------> |
    
       =========================================================================
        _______________________________________________________________________
       [                OS-specific Virtual Memory Manager (VMM)               ]
    -1 | <--- Kernel dynamic storage allocation & management (page-based) ---> |
        __________________________________   __________________________________
       [                                  ] [                                  ]
    -2 | <-- Physical memory: ROM/RAM --> | | <-- Secondary storage (swap) --> |

*  超过512bytes将会由系统分配`但是这个具体系统分配是哪一层我还不清楚`
 *  block的大小是按照8bytes来区分的
 *  pool是大小都相同的block的池
 *  pool会被不同的分配器共享 因此可以最小化的保留特定大小的空间
 *  请求会从pool中请求一个特定大小的block空间 并且pool的大小
 *  pool的的大小被设置为4k 理想上是与电脑的虚拟内存页面大小一致
 *  arena的大小是256kb 也就是256k/4b=64个pool的大小

这种方式主要缺点是  
随着时间的退化 为了free lists产生了大量的保留内存  
为了避免这种情况 我们将单独分离每个pool中的free list并且动态共享所有pool的free list空间

下面的表格表示如下 如果我们请求一个1-8bytes大小的空间 将得到一个8bytes大小的block  
然后类推

/*
     * Allocation strategy abstract:
     *
     * For small requests, the allocator sub-allocates <Big> blocks of memory.
     * Requests greater than SMALL_REQUEST_THRESHOLD bytes are routed to the
     * system's allocator.
     *
     * Small requests are grouped in size classes spaced 8 bytes apart, due
     * to the required valid alignment of the returned address. Requests of
     * a particular size are serviced from memory pools of 4K (one VMM page).
     * Pools are fragmented on demand and contain free lists of blocks of one
     * particular size class. In other words, there is a fixed-size allocator
     * for each size class. Free pools are shared by the different allocators
     * thus minimizing the space reserved for a particular size class.
     *
     * This allocation strategy is a variant of what is known as "simple
     * segregated storage based on array of free lists". The main drawback of
     * simple segregated storage is that we might end up with lot of reserved
     * memory for the different free lists, which degenerate in time. To avoid
     * this, we partition each free list in pools and we share dynamically the
     * reserved space between all free lists. This technique is quite efficient
     * for memory intensive programs which allocate mainly small-sized blocks.
     *
     * For small requests we have the following table:
     *
     * Request in bytes     Size of allocated block      Size class idx
     * ----------------------------------------------------------------
     *        1-8                     8                       0
     *        9-16                   16                       1
     *       17-24                   24                       2
     *       25-32                   32                       3
     *       33-40                   40                       4
     *       41-48                   48                       5
     *       49-56                   56                       6
     *       57-64                   64                       7
     *       65-72                   72                       8
     *        ...                   ...                     ...
     *      497-504                 504                      62
     *      505-512                 512                      63
     *
     *      0, SMALL_REQUEST_THRESHOLD + 1 and up: routed to the underlying
     *      allocator.
     */
    
    #define SMALL_REQUEST_THRESHOLD 512
    #define NB_SMALL_SIZE_CLASSES   (SMALL_REQUEST_THRESHOLD / ALIGNMENT)
    
    /*
     * The system's VMM page size can be obtained on most unices with a
     * getpagesize() call or deduced from various header files. To make
     * things simpler, we assume that it is 4K, which is OK for most systems.
     * It is probably better if this is the native page size, but it doesn't
     * have to be.  In theory, if SYSTEM_PAGE_SIZE is larger than the native page
     * size, then `POOL_ADDR(p)->arenaindex' could rarely cause a segmentation
     * violation fault.  4K is apparently OK for all the platforms that python
     * currently targets.
     */
    #define SYSTEM_PAGE_SIZE        (4 * 1024)
    #define SYSTEM_PAGE_SIZE_MASK   (SYSTEM_PAGE_SIZE - 1)
    
    /*
     * The allocator sub-allocates <Big> blocks of memory (called arenas) aligned
     * on a page boundary. This is a reserved virtual address space for the
     * current process (obtained through a malloc()/mmap() call). In no way this
     * means that the memory arenas will be used entirely. A malloc(<Big>) is
     * usually an address range reservation for <Big> bytes, unless all pages within
     * this space are referenced subsequently. So malloc'ing big blocks and not
     * using them does not mean "wasting memory". It's an addressable range
     * wastage...
     *
     * Arenas are allocated with mmap() on systems supporting anonymous memory
     * mappings to reduce heap fragmentation.
     */
    #define ARENA_SIZE              (256 << 10)     /* 256KB */
    
    /*
     * Size of the pools used for small blocks. Should be a power of 2,
     * between 1K and SYSTEM_PAGE_SIZE, that is: 1k, 2k, 4k.
     */
    #define POOL_SIZE               SYSTEM_PAGE_SIZE        /* must be 2^N */
    #define POOL_SIZE_MASK          SYSTEM_PAGE_SIZE_MASK
    
    #define MAX_POOLS_IN_ARENA  (ARENA_SIZE / POOL_SIZE)
    #if MAX_POOLS_IN_ARENA * POOL_SIZE != ARENA_SIZE
    #   error "arena size not an exact multiple of pool size"
    #endif

文档中接下来有`struct pool_header`和`struct arena_object`的结构 这里就不表示了  
具体的话可以自己点击上面的链接搜索关键字看看具体如何实现的

文仅仅简述一下后面的文档

pool和arena会互相关联(pool中有一个arenaindex arena中有一个pool\_address)  
一旦pool被关联之后 pool会有三种状态 使用中 满 空 这就没啥好解释了哈

Pools are carved off an arena's highwater mark (an arena_object's pool_address
    member) as needed.  Once carved off, a pool is in one of three states forever
    after:
    
    used == partially used, neither empty nor full
    full == all the pool's blocks are currently allocated
    empty == all the pool's blocks are currently available for allocation

后续的内容在写就要看代码拉拉  
暂时先搁置搁置啦  
有兴趣的去看上面的链接自己看啦

[Link 1]: https://gitee.com/mirrors/cpython/blob/master/Objects/obmalloc.c