What "deleted" means in /proc/$pid/maps? - linux-kernel

Question

I downloaded libhugetlbfs.so and have a simple test source :
int glbarr[1024*1024]={0} ;
int main()
{
char * ptr ;
ptr = (char *) malloc( 1024 * 1024 * 1 ) ;
printf(" press any key to go on \n");
getchar() ;
for(int idx=0;idx<100;idx++){
char strtmp[64] = {0} ;
sprintf(strtmp,"%020d",idx) ;
strcpy( ptr+1024*idx , strtmp ) ;
} //for
for(int idx=0;idx<100;idx++){
glbarr[idx] = idx ;
}
printf(" press any key to go on \n");
getchar() ;
} // main
then set the env :
export LD_PRELOAD=libhugetlbfs.so
export HUGETLB_MORECORE=yes
export HUGETLB_DEBUG=1
At last , execute test_malloc.exe :
INFO: Found pagesize 2048 kB
INFO: Detected page sizes:
INFO: Size: 2048 kB (default) Mount: /mnt/SharedMem_2M
INFO: Parsed kernel version: [3] . [10] . [0]
INFO: Feature private_reservations is present in this kernel
INFO: Feature noreserve_safe is present in this kernel
INFO: Feature map_hugetlb is present in this kernel
INFO: Kernel has MAP_PRIVATE reservations. Disabling heap prefaulting.
INFO: Kernel supports MAP_HUGETLB
INFO: HUGETLB_SHARE=0, sharing disabled
INFO: HUGETLB_NO_RESERVE=no, reservations enabled
INFO: Segment 0 (phdr 2): 0x400000-0x400a04 (filesz=0xa04) (prot = 0x5)
INFO: Segment 1 (phdr 3): 0x600de0-0xa01080 (filesz=0x274) (prot = 0x3)
DEBUG: symbol to copy at 0x601060: stdin
DEBUG: Total memsz = 0x400ca4, memsz of largest segment = 0x4002a0
INFO: libhugetlbfs version: 2.20
INFO: Mapped hugeseg at 0x2aaaaac00000. Copying 0xa04 bytes and 0 extra bytes from 0x400000...done
INFO: Prepare succeeded
INFO: Mapped hugeseg at 0x2aaaaac00000. Copying 0x274 bytes and 0x14 extra bytes from 0x600de0...done
INFO: Prepare succeeded
INFO: setup_morecore(): heapaddr = 0x1c00000
INFO: hugetlbfs_morecore(2101248) = ...
INFO: heapbase = 0x1c00000, heaptop = 0x1c00000, mapsize = 0, delta=2101248
INFO: Attempting to map 4194304 bytes
INFO: ... = 0x1c00000
INFO: hugetlbfs_morecore(0) = ...
INFO: heapbase = 0x1c00000, heaptop = 0x1e01000, mapsize = 400000, delta=-2093056
INFO: ... = 0x1e01000
and the /proc/pidof test_malloc.exe/maps :
00400000-00600000 r-xp 00000000 00:2b 6019488 /mnt/SharedMem_2M/libhugetlbfs.tmp.uI55WD (deleted)
00600000-00c00000 rw-p 00000000 00:2b 6123885 /mnt/SharedMem_2M/libhugetlbfs.tmp.VUALYM (deleted)
01c00000-02000000 rw-p 00000000 00:0d 6123886 /anon_hugepage (deleted)
numastat -m showes hugepage really in use for 8M ,
What bother me is what do "deleted" means in maps output ?!
either in /mnt/SharedMem_2M or /anon_hugepage
Edit :
and the debug information :
INFO: Found pagesize 2048 kB
INFO: Detected page sizes:
INFO: Size: 2048 kB (default) Mount: /mnt/SharedMem_2M
INFO: Parsed kernel version: [3] . [10] . [0]
INFO: Feature private_reservations is present in this kernel
INFO: Feature noreserve_safe is present in this kernel
INFO: Feature map_hugetlb is present in this kernel
INFO: Kernel has MAP_PRIVATE reservations. Disabling heap prefaulting.
INFO: Kernel supports MAP_HUGETLB
INFO: HUGETLB_SHARE=0, sharing disabled
INFO: HUGETLB_NO_RESERVE=no, reservations enabled
INFO: Segment 0 (phdr 3): 0x600de0-0xa01080 (filesz=0x274) (prot = 0x3)
DEBUG: symbol to copy at 0x601060: stdin
DEBUG: Total memsz = 0x4002a0, memsz of largest segment = 0x4002a0
INFO: libhugetlbfs version: 2.20
INFO: Mapped hugeseg at 0x2aaaaac00000. Copying 0x274 bytes and 0x14 extra bytes from 0x600de0...done
INFO: Prepare succeeded
INFO: setup_morecore(): heapaddr = 0x2200000
INFO: hugetlbfs_morecore(2101248) = ...
INFO: heapbase = 0x2200000, heaptop = 0x2200000, mapsize = 0, delta=2101248
INFO: Attempting to map 4194304 bytes
INFO: ... = 0x2200000
INFO: hugetlbfs_morecore(0) = ...
INFO: heapbase = 0x2200000, heaptop = 0x2401000, mapsize = 400000, delta=-2093056
INFO: ... = 0x2401000
and the maps :
00400000 default file=/home/marschen/test/posix-memalign/test_malloc.exe mapped=1 N0=1 kernelpagesize_kB=4
00600000 default file=/mnt/SharedMem_2M/libhugetlbfs.tmp.85Y41e\040(deleted) huge anon=1 dirty=1 N0=1 kernelpagesize_kB=2048
02200000 default file=/anon_hugepage\040(deleted) huge anon=1 dirty=1 N0=1 kernelpagesize_kB=2048

Answer 1

Temporary file deletion (unlinking) is normal for libhugetlbfs when it uses hugetlb fs pseudo filesystem (grep hugetlbfs /proc/filesystems) for getting mmaps backed with hugetlb pages.
For example, there is hugetlbfs_unlinked_fd function of libhugetlbfs/hugeutils.c
https://github.com/libhugetlbfs/libhugetlbfs/blob/e44180072b796c0e28e53c4d01ef6279caaa2a99/hugeutils.c#L1033
int hugetlbfs_unlinked_fd_for_size(long page_size)
{
const char *path;
char name[PATH_MAX+1];
int fd;
path = hugetlbfs_find_path_for_size(page_size);
..
name[sizeof(name)-1] = '\0';
strcpy(name, path);
strncat(name, "/libhugetlbfs.tmp.XXXXXX", sizeof(name)-1);
/* FIXME: deal with overflows */
fd = mkstemp64(name);
....
unlink(name);
return fd;
}
Temporary file name is randomly generated in mkstemp function; it also creates the file and opens it. Then this file is unlinked (man 2 unlink) from the filesystem (file name is marked as deleted in the directory, there is still inode and file data, but other programs can't access this file by name).
While unlinked fd is opened, it can be used to work with hugetlb mmap and to store data. Only when this fd is closed, file data will be actually deleted by fs.
Early unlinking of mktemp files is often used: When a file created with mkstemp() is deleted?
Some useful information can be also listed in HOWTO of libhugetlbfs project
https://github.com/libhugetlbfs/libhugetlbfs/blob/master/HOWTO