内存上下文的设计思路可以参考src/backend/utils/mmgr/README。
https://www.pgcon.org/2019/schedule/attachments/514_introduction-memory-contexts.pdf
http://www.light-pg.com/docs/lightdb/13.3-22.2/lt_cheat_funcs.html
可以通过lt_cheat_funcs(该扩展对运行时无明显的侵入和干扰,可以在生产使用)扩展实时查询,如下:
zjh@postgres=# create extension lt_cheat_funcs ;
CREATE EXTENSION
zjh@postgres=# select * from pg_stat_get_memory_context();
name | parent | level | total_bytes | total_nblocks | free_bytes | free_chunks | used_bytes
--------------------------+--------------------+-------+-------------+---------------+------------+-------------+------------
TopMemoryContext | | 0 | 510640 | 12 | 23720 | 19 | 486920
dynahash | TopMemoryContext | 1 | 8192 | 1 | 1448 | 0 | 6744
不仅可以查询当前实例的MemoryContext树,还可以在每个语句执行结束前打印内存上下文。
shared_preload_libraries='lt_stat_statements,lt_stat_activity,lt_cheat_funcs,lt_prewarm,lt_cron,ltaudit,lt_hint_plan,lt_show_plans,lt_standby_forward,lt_pathman'
lt_cheat_funcs.log_memory_context=on
2022-09-06 09:32:13.807665T ltsql zjh@postgres [local] client backend SELECT 00000[2022-09-06 09:28:47 CST] 0 [84905] STATEMENT: select 1;
TopMemoryContext: 494256 total in 11 blocks; 9760 free (11 chunks); 484496 used
dynahash: 8192 total in 1 blocks; 1448 free (0 chunks); 6744 used
TopTransactionContext: 8192 total in 1 blocks; 7784 free (2 chunks); 408 used
RowDescriptionContext: 8192 total in 1 blocks; 6888 free (0 chunks); 1304 used
MessageContext: 32768 total in 3 blocks; 21504 free (6 chunks); 11264 used
dynahash: 8192 total in 1 blocks; 552 free (0 chunks); 7640 used
dynahash: 16384 total in 2 blocks; 4592 free (2 chunks); 11792 used
TransactionAbortContext: 32768 total in 1 blocks; 32504 free (0 chunks); 264 used
dynahash: 8192 total in 1 blocks; 552 free (0 chunks); 7640 used
TopPortalContext: 8192 total in 1 blocks; 7656 free (0 chunks); 536 used
PortalContext: 1024 total in 1 blocks; 592 free (0 chunks); 432 used
dynahash: 16384 total in 2 blocks; 3504 free (2 chunks); 12880 used
CacheMemoryContext: 524288 total in 7 blocks; 104920 free (0 chunks); 419368 used
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used
index info: 2048 total in 2 blocks; 496 free (1 chunks); 1552 used
index info: 3072 total in 2 blocks; 1160 free (2 chunks); 1912 used
index info: 2048 total in 2 blocks; 952 free (2 chunks); 1096 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 3072 total in 2 blocks; 1128 free (1 chunks); 1944 used
index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 3072 total in 2 blocks; 1096 free (2 chunks); 1976 used
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used
index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used
index info: 3072 total in 2 blocks; 1160 free (2 chunks); 1912 used
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used
index info: 2048 total in 2 blocks; 952 free (2 chunks); 1096 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 3072 total in 2 blocks; 1096 free (2 chunks); 1976 used
index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used
index info: 3072 total in 2 blocks; 1096 free (2 chunks); 1976 used
index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 3072 total in 2 blocks; 1160 free (2 chunks); 1912 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used
index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used
index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 2048 total in 2 blocks; 656 free (2 chunks); 1392 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 3072 total in 2 blocks; 1160 free (2 chunks); 1912 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 3072 total in 2 blocks; 1128 free (1 chunks); 1944 used
index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used
index info: 3072 total in 2 blocks; 1160 free (2 chunks); 1912 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used
index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 2048 total in 2 blocks; 952 free (2 chunks); 1096 used
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used
index info: 3072 total in 2 blocks; 1160 free (2 chunks); 1912 used
index info: 2048 total in 2 blocks; 952 free (2 chunks); 1096 used
index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used
index info: 2048 total in 2 blocks; 448 free (1 chunks); 1600 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 3072 total in 2 blocks; 776 free (1 chunks); 2296 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 2048 total in 2 blocks; 656 free (2 chunks); 1392 used
index info: 2048 total in 2 blocks; 656 free (2 chunks); 1392 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used
index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used
index info: 2048 total in 2 blocks; 656 free (2 chunks); 1392 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 1024 total in 1 blocks; 0 free (0 chunks); 1024 used
index info: 3072 total in 2 blocks; 1160 free (2 chunks); 1912 used
index info: 2048 total in 2 blocks; 952 free (2 chunks); 1096 used
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used
index info: 2048 total in 2 blocks; 688 free (2 chunks); 1360 used
index info: 2048 total in 2 blocks; 952 free (2 chunks); 1096 used
index info: 2048 total in 2 blocks; 656 free (2 chunks); 1392 used
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used
index info: 2048 total in 2 blocks; 824 free (0 chunks); 1224 used
WAL record construction: 49768 total in 2 blocks; 6360 free (0 chunks); 43408 used
dynahash: 8192 total in 1 blocks; 2616 free (0 chunks); 5576 used
MdSmgr: 8192 total in 1 blocks; 7760 free (0 chunks); 432 used
dynahash: 8192 total in 1 blocks; 552 free (0 chunks); 7640 used
dynahash: 16384 total in 2 blocks; 6656 free (3 chunks); 9728 used
dynahash: 8192 total in 1 blocks; 552 free (0 chunks); 7640 used
dynahash: 104120 total in 2 blocks; 2616 free (0 chunks); 101504 used
ErrorContext: 8192 total in 1 blocks; 7928 free (7 chunks); 264 used
lt_cheat_funcs还有一个参数lt_cheat_funcs.exit_on_segv,控制segment fault的时候是否触发master进程宕机。因为这种概率发生极小,所以默认可以不用开启。一旦有异常某个参数值导致了越界,可以考虑临时补救。
lt_cheat_funcs对性能的影响。基于ltbench测试可知,其对性能的影响几乎可以忽略不计。
https://jnidzwetzki.github.io/2022/05/28/postgres-memory-context.html
共享内存的使用姿势 https://pgsql-hackers.postgresql.narkive.com/6VuZjbMh/shared-memory-and-memory-context-question
主要在两个c文件中:src/backend/utils/mmgr/aset.c是实现,src/backend/utils/mmgr/mcxt.c对外接口。如果想要干预或监控memorycontext,可以通过注册回调hook进行。
要想掌握memorycontext,还得理解发生out of memory的情况以及如何解决,如下:
client backend BIND 53200[2022-10-12 11:09:13 CST] 14856195 [402043] ERROR: out of memory
client backend BIND 53200[2022-10-12 11:09:13 CST] 14856195 [402043] DETAIL: Failed on request of size 48 in memory context "MessageContext".
client backend BIND 53200[2022-10-12 11:09:13 CST] 14856195 [402043] STATEMENT: insert into tbfundprftflow4 (ta_client,last_asset
client backend PARSE 53200[2022-10-12 11:09:55 CST] 0 [402230] ERROR: out of memory
client backend PARSE 53200[2022-10-12 11:09:55 CST] 0 [402230] DETAIL: Failed on request of size 40 in memory context "CachedPlanQuery".
client backend PARSE 53200[2022-10-12 11:09:55 CST] 0 [402230] STATEMENT: insert into tbfundprftflow15 (ta_client,last_asset,curr_income_ratio
client backend PARSE 53200[2022-10-12 11:09:13 CST] 0 [402043] ERROR: out of memory
client backend PARSE 53200[2022-10-12 11:09:13 CST] 0 [402043] DETAIL: Failed on request of size 65536 in memory context "ErrorContext".
从源码可以看出,如果allocset中没有空闲chunk或申请的内存特别大(超过AllocSetContext.allocChunkLimit)时,都会调用c malloc申请内存,如下:
/*
* Time to create a new regular (multi-chunk) block?
*/
if (block == NULL)
{
Size required_size;
/*
* The first such block has size initBlockSize, and we double the
* space in each succeeding block, but not more than maxBlockSize.
*/
blksize = set->nextBlockSize;
set->nextBlockSize <<= 1;
if (set->nextBlockSize > set->maxBlockSize)
set->nextBlockSize = set->maxBlockSize;
/*
* If initBlockSize is less than ALLOC_CHUNK_LIMIT, we could need more
* space... but try to keep it a power of 2.
*/
required_size = chunk_size + ALLOC_BLOCKHDRSZ + ALLOC_CHUNKHDRSZ;
while (blksize < required_size)
blksize <<= 1;
/* Try to allocate it */
block = (AllocBlock) malloc(blksize);
lightdb 22.4开始将会打印具体的值),此时通过malloc分配。如果malloc失败,就会直接返回NULL,没有打印实际申请多大的内存失败。从memorycontext本身申请是永远不会失败的。所以就要分析malloc失败的原因。
malloc()函数分配内存失败的常见原因:
1. 内存不足。
2. 在前面的程序中出现了内存的越界访问,导致malloc()分配函数所涉及的一些信息被破坏。下次再使用malloc()函数申请内存就会失败,返回空指针NULL(0)。
3. 没有连续内存能够满足申请。http://t.zoukankan.com/huty-p-8518846.html
4. 根据vm.overcommit_ratio和vm.overcommit_memory分析每个进程能够使用的内存大小。可以通过sar -r查看Committed_AS,通过/proc/meminfo查看Committed_AS、CommitLimit。如下: