Oracle TimesTen Database, was not using Huge/Large pages. Well at least when I executed a ttStatus command, I saw the following:
Shared Memory KEY 0x0c0504dd ID 1838710795 (LOCKED) when I expected to see
Shared Memory KEY 0x0c0504dd ID 1838710795 (Large Pages,LOCKED)
There are 11 connections to the data store Shared Memory KEY 0x0c0504dd ID 1838710795 (LOCKED) PL/SQL Memory KEY 0x0d0504dd ID 1838743564 Address 0x7fa0000000 Type PID Context Connection Name ConnID Subdaemon 15971 0x0000000001e30470 Manager 142 Subdaemon 15971 0x0000000001e87990 Rollback 141 Subdaemon 15971 0x0000000001f6ff70 Flusher 140 Subdaemon 15971 0x0000000001f85310 Monitor 139 Subdaemon 15971 0x0000000001f9a6b0 Deadlock Detector 138 Subdaemon 15971 0x0000000001faf930 Checkpoint 134 Subdaemon 15971 0x0000000001fc4cd0 Aging 137 Subdaemon 15971 0x00007f53cc0008c0 IndexGC 133 Subdaemon 15971 0x00007f53d0040b20 AsyncMV 136 Subdaemon 15971 0x00007f53d0055da0 HistGC 132 Subdaemon 15971 0x00007f53d8040c60 Log Marker 135
RAM residence policy: Manual Data store is manually loaded into RAM Replication policy : Manual Cache Agent policy : Manual PL/SQL enabled.
My ttendaemon.options file was configured correctly (see snippet below):
-linuxLargePageAlignment 2
-bash-3.2$ cat ttendaemon.options # Commented values are default values -maxsupportlogfiles 10 -maxuserlogfiles 10 -linuxLargePageAlignment 2 -verbose -server <TimesTen port number> # show the date in the ttmesg.log and tterrors.log file -showdate # Allow TimesTen transaction logs to be access via NFS mount point #-allowNetworkFiles
Memory allocations were correct
What could be the problem? One key was the tterrors.log file. I found this line:
-- OS reports not enough large pages available, ####### pages requested
Lets take a look and see what’s going on in the shared memory.
-bash-3.2$ ipcs -m ------ Shared Memory Segments -------- key shmid owner perms bytes nattch status 0x00000000 2523137 root 644 80 2 0x00000000 2555906 root 644 16384 2 0x00000000 2588675 root 644 280 2 0x00000000 1833172998 ttowner 660 285636296704 1 dest 0x00000000 1833205767 ttowner 660 67108864 1 dest 0x00000000 1746075656 ttowner 644 790528 3 dest 0x00000000 1746108425 ttowner 644 790528 3 dest 0x00000000 1746141194 ttowner 644 790528 3 dest
Turns out that we had some stale memory segments sitting out there, which was enough to prevent TimesTen from from using large/huge pages. In the status column we see “dest”. Dest means that this memory segment is marked to be destroyed. Because this hasn’t/never happened, there is not enough memory, and we need to go ahead and manually free up this space:
-bash-3.2$ grep 1833172998 /proc/*/maps 2>/dev/null | awk -F"/" '{print "kill -9 "$3}'
kill -9 15304
-bash-3.2$ kill -9 15304
Once this space has been freed up, reload the TimesTen database and ttstatus should return:
Shared Memory KEY 0x0c0504dd ID 1838710795 (LARGE PAGES,LOCKED) PL/SQL Memory KEY 0x0d0504dd ID 1838743564 Address 0x7fa0000000 Type PID Context Connection Name ConnID Subdaemon 15971 0x0000000001e30470 Manager 142 Subdaemon 15971 0x0000000001e87990 Rollback 141 Subdaemon 15971 0x0000000001f6ff70 Flusher 140 Subdaemon 15971 0x0000000001f85310 Monitor 139 Subdaemon 15971 0x0000000001f9a6b0 Deadlock Detector 138 Subdaemon 15971 0x0000000001faf930 Checkpoint 134 Subdaemon 15971 0x0000000001fc4cd0 Aging 137 Subdaemon 15971 0x00007f53cc0008c0 IndexGC 133 Subdaemon 15971 0x00007f53d0040b20 AsyncMV 136 Subdaemon 15971 0x00007f53d0055da0 HistGC 132 Subdaemon 15971 0x00007f53d8040c60 Log Marker 135
RAM residence policy: Manual Data store is manually loaded into RAM Replication policy : Manual Cache Agent policy : Manual PL/SQL enabled.
Now this may not be the only reason TimesTen won’t use Large Pages, but it will get you started.
Enjoy!
