Hsd150mx15tft

HANA Database is unresponsiveIndexserver traces show numerous memory map errors, ResourceContainerShrink warnings and massive fragmentation of available memory : [26452]{-1}[-1/-1] 2023-07-05 20:24:02.920349 e OSMemory         mmMallocProxy.cpp(02207) : Failed: munmap MM [0x00007ee252b5a000, 0x00007ee252b6a000[ (65536b | 0x0000000000000000), error 12[26452]{-1}[-1/-1] 2023-07-05 20:24:02.920593 e OSMemory         mmMallocProxy.cpp(02207) : Failed: munmap MM [0x00007ee252b5a000, 0x00007ee252b6a000[ (65536b | 0x0000000000000000), error 12[26538]{-1}[-1/-1] 2023-07-05 20:24:04.001173 e OSMemory         mmMallocProxy.cpp(02207) : Failed: munmap MM [0x00007ee252b5a000, 0x00007ee252b6a000[ (65536b | 0x0000000000000000), error 12[26538]{-1}[-1/-1] 2023-07-05 20:24:04.001464 e OSMemory         mmMallocProxy.cpp(02207) : Failed: munmap MM [0x00007ee252b5a000, 0x00007ee252b6a000[ (65536b | 0x0000000000000000), error 1 ......[26452]{-1}[-1/-1] 2023-07-05 20:23:21.322703 w ResMan           ResourceContainerShrink.cpp(02478) : Information about shrink at 2023-07-05 20:23:21 317 Local:Reason for shrink: IPMM (other process). Request ID= 64225ShrinkCaller 1: 0x00007f253a7a3a86 in ResourceManager::ShrinkInformation::updateShrinkInformation(char const*, unsigned long, bool)+0xb2 at ResourceContainerShrink.cpp:725 (libhdbbasis.so) 2: 0x00007f253a79d164 in ResourceManager::ResourceContainerImpl::genericShrink(unsigned long, unsigned long, char const*, unsigned long, MemoryManager::ReclaimMemoryTaskType)+0x640 at ResourceContainerShrink.cpp:2477 (libhdbbasis.so) 3: 0x00007f253a8aff26 in MemoryManager::CompactionThread::compactCompactors(unsigned long, bool&, char const*, unsigned long, MemoryManager::ReclaimMemoryTaskType)+0x582 at mmCompactionTrigger.cpp:955 (libhdbbasis.so) 4: 0x00007f253a8ab7a6 in MemoryManager::CompactionThread::doCompactionIntern(MemoryManager::Precharge&, unsigned long, unsigned long, unsigned long, unsigned long&, char const*, unsigned long, MemoryManager::ReclaimMemoryTaskType, bool)+0xfc2 at mmCompactionTrigger.cpp:1116 (libhdbbasis.so) 5: 0x00007f253a8ac7b5 in MemoryManager::CompactionThread::doIpmmCompactionIntern(unsigned long, int, unsigned long, unsigned long, unsigned long, MemoryManager::ProcessSlot::RequestedTask)+0x1b1 at mmCompactionTrigger.cpp:1704 (libhdbbasis.so) 6: 0x00007f253a8ad1ec in MemoryManager::CompactionThread::run(void*&)+0x188 at mmCompactionTrigger.cpp:1659 (libhdbbasis.so) 7: 0x00007f253aa0d3fd in Execution::Thread::staticMainImp(Execution::Thread*)+0x529 at Thread.cpp:522 (libhdbbasis.so) 8: 0x00007f253a9feb39 in Execution::pthreadFunctionWrapper(Execution::Thread*)+0x165 at ThreadInterposition.cpp:639 (libhdbbasis.so) 9: 0x00007f253a07e4f9 in start_thread+0xd5 (libpthread.so.0)10: 0x00007f2538758ecf in __GI___clone+0x3b (libc.so.6)IPMM short info:GLOBAL_ALLOCATION_LIMIT (GAL) = 281.87gb (302660976640b), SHARED_MEMORY = 3.71gb (3993792512b), CODE_SIZE = 3.29gb (3539582976b), OOM_RESERVATION_ALLOCATOR_SIZE = 96.14mb (100810752b)PID=26383 (hdbindexserver), PAL=300.42gb (322578493440b), AB=265.42gb (285003878400b), UA=0, U=52.64gb (56527229591b), FSL=0b, FMC=0b[Some processes may have been skipped]Total allocated memory= 281.99gb (302786670592b)Total used memory     = 67.33gb (72297484846b)Sum AB                = 295253295104Sum Used              = 64764109358Heap memory fragmentation: 76% (this value may be high if defragmentation does not help solving the current memory request)Top "M_HEAP_MEMORY" allocators (component, name, size). Ordered descending by inclusive_size_in_use. 1: System:                        /                                                                    52.64gb (56527229591b) 2: System:                        Pool                                                                 49.63gb (53292362661b) 3: System:                        Pool/PersistenceManager                                              20.12gb (21609728720b)   'oom_memory_release.trc' files are observed 'indexserver_host.30003.rtedump.20230705-162124.0026383.oom_memory_release.trc' [MEMORY_OOM]  Information about current out of memory situation: (2023-07-05 16:21:54 432 Local)OUT OF MEMORY occurred.Host: <>Executable: hdbindexserverPID: 26383Current callstack: 1: 0x00007f253a8ed185 in MemoryManager::AllocatorConfigurationSPI::releaseMemoryMostProbablyDueToAWrongConfiguration()+0x21 at mmAllocatorConfigurationImpl.cpp:1675 (libhdbbasis.so) 2: 0x00007f253a890944 in MemoryManager::HugeAlignmentPool::allocateHugeAlignedNoThrowImpl(MemoryManager::MemoryPool**, ltt::allocator_statistics&, unsigned long, unsigned long, void const*)+0xae0 at mmHugeAlignmentAllocator.cpp:1327 (libhdbbasis.so) 3: 0x00007f253a85412b in MemoryManager::MemoryPool::allocateAligned(unsigned long, unsigned long, unsigned short, ltt::allocator_statistics&, bool&, bool, void const*)+0x1c7 at mmMemoryPool.cpp:1722 (libhdbbasis.so) 4: 0x00007f253a8262f6 in MemoryManager::PoolAllocator::allocateAlignedNoThrowImpl(unsigned long, unsigned long, void const*)+0x52 at mmPoolAllocator.cpp:1811 (libhdbbasis.so) 5: 0x00007f2547cfd89d in RowEngine::Version::PagedAllocatorPartition::allocateAlignedNoThrow(unsigned long, unsigned long, bool)+0x1e9 at memory.cpp:336 (libhdbrskernel.so) 6: 0x00007f2547cff727 in RowEngine::Version::PartitionedMVCCAllocator::allocateAlignedNoThrowImpl(unsigned long, unsigned long, void const*)+0x63 at MVCCAllocatorImpl.cc:158 (libhdbrskernel.so) 7: 0x00007f254713e5d8 in ltt::allocator::allocate(unsigned long, void const*)+0x24 at memory.cpp:236 (libhdbrskernel.so) 8: 0x00007f254713fe77 in ltt::handle_mem_ref::create(unsigned long, void const*, ltt::allocator&) const+0x43 at handle_ref.cpp:18 (libhdbrskernel.so) 9: 0x00007f2547d07237 in RowEngine::Version::BasicMVCCManager::acquireVersionContext(ptime::Transaction*)+0x63 at MVCCManagerImpl.cc:410 (libhdbrskernel.so)10: 0x00007f2547e42142 in RowEngine::RSContext::getVersionContextHandle()+0x60 at RSTransactionDataContainer.cc:472 (libhdbrskernel.so)11: 0x00007f25480fe4b3 in ptime::VersionMgr::createAndRegisterVersionContext(ptime::Transaction*)+0x150 at VersionMgr.cc:1493 (libhdbrskernel.so)12: 0x00007f2548007025 in ptime::Transaction::start_logging(long)+0xa61 at transmgmt.cc:1446 (libhdbrskernel.so)13: 0x00007f2548007578 in ptime::Transaction::set_write_trans(bool, long)+0xa4 at transmgmt.cc:1099 (libhdbrskernel.so)14: 0x00007f25480095d0 in ptime::Transaction::start(ptime::Transaction::ReadUpdateTrans, unsigned long, bool, bool, unsigned long)+0x840 at transmgmt.cc:1020 (libhdbrskernel.so)15: 0x00007f2552294a79 in TableConsistencyCheck::ColumnTablePartJobNode::runCheck(Execution::Context&, Execution::JobObject&)+0xe5 at Request.cc:502 (libhdbtableconsistencycheck.so)16: 0x00007f255229687b in TableConsistencyCheck::LocalTableJobNode::run(Execution::Context&, Execution::JobObject&)+0x47 at Request.cc:192 (libhdbtableconsistencycheck.so)17: 0x00007f253aa6a74c in Execution::JobObjectImpl::run(Execution::JobWorker*)+0x2028 at JobExecutorImpl.cpp:1195 (libhdbbasis.so)18: 0x00007f253aa54122 in Execution::JobWorker::runJob(ltt::smartptr_handle&)+0x320 at JobExecutorThreads.cpp:383 (libhdbbasis.so)19: 0x00007f253aa4e5c8 in Execution::JobWorker::run(void*&)+0x764 at JobExecutorThreads.cpp:1283 (libhdbbasis.so)20: 0x00007f253aa0d3fd in Execution::Thread::staticMainImp(Execution::Thread*)+0x529 at Thread.cpp:522 (libhdbbasis.so)21: 0x00007f253a9feb39 in Execution::pthreadFunctionWrapper(Execution::Thread*)+0x165 at ThreadInterposition.cpp:639 (libhdbbasis.so)22: 0x00007f253a07e4f9 in start_thread+0xd5 (libpthread.so.0)23: 0x00007f2538758ecf in __GI___clone+0x3b (libc.so.6)Memory consumption information:[No information available.]Out of memory reasons:!:! <1> 2x BigBlockAllocator failure, --- precharge ok ---.    /data/jenkins_prod/workspace/awduyghiom/s/Basis/MemoryManager/impl/mmMemoryPool.cpp (2276)Could not return 112038625280b to operating system.This is a configuration problem of your operating system:Please increase /proc/sys/vm/max_map_countMemory Configuration:global_allocation_limit: 288768 (-> 282gb (302795194368b))allocationlimit:  (-> 300.42gb (322578493440b))     'OPERATING_SYSTEM_LIMIT' OOM files are present 'indexserver_hostname.30003.rtedump.20230705-192033.0026383.oom.trc' [MEMORY_OOM]  Information about current out of memory situation: (2023-07-05 19:20:34 613 Local)OUT OF MEMORY occurred.Host: <>Executable: hdbindexserverPID: 26383Failed to allocate 1.12kb (1152b).Allocation failure type: OPERATING_SYSTEM_LIMITCurrent callstack: 1: 0x00007f253a824d35 in MemoryManager::PoolAllocator::notifyOOMImpl(unsigned long, unsigned long, bool, ltt::allocation_failure_type, bool)+0x1b1 at mmPoolAllocator.cpp:2284 (libhdbbasis.so) 2: 0x00007f253a6e17ad in ltt::allocator::notifyOOM(unsigned long, unsigned long, bool, ltt::allocation_failure_type, bool) [clone .constprop.0]+0x59 at memory.cpp:130 (libhdbbasis.so) 3: 0x00007f253a825bd0 in MemoryManager::PoolAllocator::oomHandling(unsigned long, unsigned long, bool)+0x60 at mmPoolAllocator.cpp:1222 (libhdbbasis.so) 4: 0x00007f253a826420 in MemoryManager::PoolAllocator::allocateNoThrowImpl(unsigned long, void const*)+0xd0 at mmPoolAllocator.cpp:1733 (libhdbbasis.so) 5: 0x00007f253a716c28 in ltt::allocator::allocate(unsigned long, void const*)+0x24 at memory.cpp:236 (libhdbbasis.so) 6: 0x00007f253a718957 in ltt::handle_mem_ref::create(unsigned long, void const*, ltt::allocator&) const+0x43 at handle_ref.cpp:18 (libhdbbasis.so) 7: 0x00007f253a9fb497 in Execution::TimerThread::TimerCallback::processTime(Synchronization::LockHandle&, bool)+0xb3 at JobExecutor.hpp:1226 (libhdbbasis.so) 8: 0x00007f253a9fbddd in Execution::TimerThreadSingleton::run(void*&)+0x129 at TimerThread.cpp:444 (libhdbbasis.so) 9: 0x00007f253aa0d3fd in Execution::Thread::staticMainImp(Execution::Thread*)+0x529 at Thread.cpp:522 (libhdbbasis.so)10: 0x00007f253a9feb39 in Execution::pthreadFunctionWrapper(Execution::Thread*)+0x165 at ThreadInterposition.cpp:639 (libhdbbasis.so)11: 0x00007f253a07e4f9 in start_thread+0xd5 (libpthread.so.0)12: 0x00007f2538758ecf in __GI___clone+0x3b (libc.so.6)   The '/proc/self/maps' section reveals that the operating system layer configurations for Linux THP is enabled and that the virtual map memory limit is incredibly low  : /sys/kernel/mm/transparent_hugepage/enabled:[always] madvise never/proc/sys/vm/max_map_count:65530