How does printing to terminal affect memory usage? Would it just keep eating up memory until OOM?
What about in a docker container?
How does the OS handle the memory of prints to terminal?
Does the OS flush the terminal at a certain point?
When you call printf for printing to the terminal, the standard library will use line buffering and wait until a newline character to write the output. The size will depend on the implementation (might be 8K). See: In C, what's the size of stdout buffer?
. But this memory usage doesn't grow over time.
When written (via a write syscall), the buffer will get copied through pipes and ptys to end up in the terminal emulator, which then displays it on the screen. Aside from the scrollback buffer of the terminal emulator, it does not accumulate anywhere along this path.
Most terminal emulators will have a limit for the scrollback buffer, that defaults to a few thousand lines. The old lines would be presumably deallocated after this limit. Some terminal emulators provide an option to remove the limit, which means it could grow until OOM (I believe on macOS, the Terminal app actually handles this event to clear the scrollback buffer) and the terminal emulator might be killed by the OOM-killer. From the OS perspective, it is not different than any other inter-process communication.
A container may only affect the creation of the pipes. It is still the process calling the printf, sending the resulting buffer to the terminal emulator process, through the kernel.
Related
I have read about user space and kernel space and how a program's execution path can bring it from user space to kernel space, I suppose an example of this is if my program runs like this
Poco::Net::SocketAddress sender;
char buffer[64000];
.
.
.
socket.receiveFrom(buffer, sizeof(buffer), sender);
since this call requires accessing the network card, I think it should go into kernel space.
My question is:
What happens as the program makes the socket.receivefrom(...) call
Does the thread go to sleep and give up its core since it is going
to kernel space and only gets woken up when the char buffer has been
written
Does the thread go directly to kernel space and come back to user space after writing into the char buffer
No. The thread goes to execute kernel code, at kernel-permissions (ring 0 in an x86). The thread might go to "sleep" (i.e. the CPU might go and execute a different program, or will go idle, this depends on what the scheduler decides) inside the kernel. However, it might not go to sleep at all, if, for example, data is already available from the network card. From a user perspective, you know that when the call returns you have your data in the buffer, and you might expect the call to take a while.
It depends on the scheduler. You might get an interrupt at any time and execute something else. But generally, yes, you go to the kernel and back.
I am running into an issue spawning a large number of processes (200) under MacOS X Mountain Lion (though I'm sure this issue is not version specific). What I am doing is launching processes (in my test it is /bin/cat) which have their STDIN, STDOUT, and STDERR connected to pipes -- the other end of which is the spawning (parent) process.
The parent process writes data into the STDIN of the processes, which is piped to the [/bin/cat] child processes, which in turn spit the data back out of STDOUT and is read by the parent process. /bin/cat is just used for testing.
I am actually using kqueue to be notified when there is space available in the STDIN pipe. When kqueue notifies you with a EVFILT_WRITE event that space is available, the event also tells you exactly how many bytes can be written without blocking.
This all works well, and I can easily spawn 100 child (/bin/cat) processes, and everything flows through the pipes without blocking (all day long). However, when I crank up the number of processes to 200 everything grinds to a halt when the single kqueue service thread blocks on a write() call to one of the STDIN pipes. The event says that there is 16384 bytes available (basically an empty pipe) but when the program tries to write exactly 16384 bytes into the pipe, the write() blocks anyway.
Initially I thought I was running into a max. open files issue, but I've jacked up the ulimit for open files to 8192, so that is not the issue. What I have discovered from some googling is that on OS X, STDIN/STDOUT/STDERR are not in fact "files" (or "pipes") but are actually devices. When the process is hung, running lsof on the command-line also hangs with a warning about not being able to stat() the file system:
lsof: WARNING: can't stat() hfs file system /
Output information may be incomplete.
assuming "dev=1000008" from mount table
As soon as I kill the process, lsof completes. This reinforces the notion that STDIN/OUT/ERR are in fact devices and I'm running into some kind of limit.
Does anyone have an idea of what limit I am running into, for example is there a limit on the number of "device" that can be created? Can this be increased somehow?
Just to answer my own question here. This appears to be related to how MacOS X will dynamically expand a pipe from 16K to 32K to 64K (and so on). My basic workaround was to prevent the pipe from expanding. It appears that whenever you fill the pipe completely the OS will expand it. So, when the kqueue triggers that I can write into the pipe, and indicates that I have 16384 bytes available to write, I simply write 16384 - 1 bytes. Basically, whatever it tells me I have available, I write at most (available - 1) bytes. This prevents the pipe from expanding, and is preventing my code from encountering the condition where a write() to the pipe would block (even though the pipe is non-blocking).
Free Pascal heaptrc keepreleased is described as "useful if you suspect that the same memory block is released twice" but is it possible to detect usage of previously freed memory (object method call of freed object) with it? If it is impossible - can it be detected with other tools?
Yes, it should do that. The idea is the following:
an used allocation has a different .sig then $AAAAAAAA or $DEADBEEF. On freemem the sig is checked (see around line 593 in trunk) against sig $AAAAAAA IF useCRC is false.
The keepreleased prevents blocks from being reused, which would change the signature to something else then $AAAAAAAA. It will print something like:
Marked memory at $12345678 released
to the file descriptor ptext. The error standard files can be set and directed using various other variables. It looks fairly complicated, but that is probably to deal with consoleless GUI applications
Some other variables (like haltonerror) govern if the application is halted on such corruption
An alternate (but very slow) way is using valgrind (fpc option -gv), but I only have run valgrind on *nix, and as said it is extremely slow, so not for very heavy processing apps.
When I type any key, normally, it is immediately echoed back to the std output i.e.my screen.
If I have to enter a password, that says that it will not echo back, I cannot see the keys that I type.
How does this work.
Does each key press go to the kernel immediately(without me pressing ENTER), and then the kernel decides to echo them or not?
Like , I press 'A', it goes to the kernel; kernel echoes it; I get it on my screen. Now I hit 'B'...same sequence again...; Now I have 'AB' on my screen (my command) and hit ENTER; My command goes to the kernel and is finally executed.
Is there any other explanation? What happens in the background during the key presses?
The terminal driver in the kernel can be put in several modes (there are actually many more flags than this, and these days "cbreak" is actually the opposite of a different flag, so this is simplified).
The "cbreak" mode means that the process that is attempting to read from the terminal will receive keyboard input as soon as it is available. When cbreak mode is off, the data is stored by the kernel in a buffer until enter is pressed, and certain keys like backspace are handled by the kernel (when you press backspace, it removes the character from the buffer and - if echo mode is on - writes "backspace-space-backspace" to the terminal to overwrite the character with a blank space).
Echo mode means that whenever the user presses a key, the kernel will immediately echo it back to the screen. When it is off, nothing will be echoed to the screen, and the program will need to write it to the terminal if it wants you to see it.
There are several typical situations:
In programs that do advanced input handling (like most shells, or something like a full screen program), cbreak is on and echo is off; the program will write the character to the terminal itself if it is not part of a special key escape sequence.
In most situations [the default with a simple program that reads stdin and writes stdout], echo is on and cbreak is off. When you type, it behaves as I described above, all of this is handled by the kernel until you hit enter and it sends it to the application. Input editing is limited to backspace [and ctrl-u, ctrl-w], trying to use the arrow keys will just put escape sequences like ^[[D in the input line.
When reading a password, echo is off and cbreak is off. Input works just like the default case, except the kernel does not copy input to the screen.
The program that is running tells the kernel what mode to have it in with the termios functions. You can use the stty command to do the same in a shell environment, but be aware that this may interfere with the shell's own input handling or with what programs you run expect the default state to be.
Your keyboard generates electrical signals that are eventually interpreted as keycodes that correspond to letters - 'A', 'B', function keys F1, F2 etc. This all happen in the keyboard driver that is handled by the kernel. That keyboard driver has a buffer to receive all the keypresses from the keyboard and sends that to the kernel that in turn direct them to processes that is currently having the focus. What to do with the sequence of keys are totally decided by the individual application, such as whether to display the keys or not.
echo program is part of coreutils. You can download its sources here. Look at src/echo.c it's quite small. You can see that echo uses fputc or putchar calls. Those calls deal with standard stream called stdout. The architecture of standard streams is quite complicated and it's beyond of this post. You can found it using for example google.
I needed to recover the partition table I deleted accidentally. I used an application named TestDisk. Its simply mind blowing. I reads each cylinder from the disk. I've seen similar such applications which work with MBR & partitioning.
I'm curious.
How do they read
clusters/cylinders/sectors from the
disk? Is there some kind of API for this?
Is it again OS dependent? If so whats the way to for Linux & for windows?
EDIT:
Well, I'm not just curious I want a hands on experience. I want to write a simple application which displays each LBA.
Cylinders and sectors (wiki explanation) are largely obsoleted by the newer LBA (logical block addressing) scheme for addressing drives.
If you're curious about the history, use the Wikipedia article as a starting point. If you're just wondering how it works now, code is expected to simply use the LBA address (which works largely the same way as a file does - a linear array of bytes arranged in blocks)
It's easy due to the magic of *nix special device files. You can open and read /dev/sda the same way you'd read any other file.
Just use open, lseek, read, write (or pread, pwrite). If you want to make sure you're physically fetching data from a drive and not from kernel buffers you can open with the flag O_DIRECT (though you must perform aligned reads/writes of 512 byte chunks for this to work).
For *nix, there have been already answers (/dev directory); for Windows, there are the special objects \\.\PhisicalDriveX, with X as the number of the drive, which can be opened using the normal CreateFile API. To actually perform reads or writes you have then to use the DeviceIoControl function.
More info can be found in "Physical Disks and Volumes" section of the CreateFile API documentation.
I'm the OP. I'm combining Eric Seppanen's & Matteo Italia's answers to make it complete.
*NIX Platforms:
It's easy due to the magic of *nix special device files. You can open and read /dev/sda the same way you'd read any other file.
Just use open, lseek, read, write (or pread, pwrite). If you want to make sure you're physically fetching data from a drive and not from kernel buffers you can open with the flag O_DIRECT (though you must perform aligned reads/writes of 512 byte chunks for this to work).
Windows Platform
For Windows, there are the special objects \\.\PhisicalDriveX, with X as the number of the drive, which can be opened using the normal CreateFile API. To perform reads or writes simply call ReadFile and WriteFile (buffer must be aligned on sector size).
More info can be found in "Physical Disks and Volumes" section of the CreateFile API documentation.
Alternatively you can also you DeviceIoControl function which sends a control code directly to a specified device driver, causing the corresponding device to perform the corresponding operation.
On linux, as root, you can save your MBR like this (Assuming you drive is /dev/sda):
dd if=/dev/sda of=mbr bs=512 count=1
If you wanted to read 1Mb from you drive, starting at the 10th MB:
dd if=/dev/sda of=1Mb bs=1Mb count=1 skip=10