I'm building an embedded system that talks RS-232 to a serial terminal, "full-duplex" style, so the host echos what the terminal sends.
I know printables (at least ASCII 0x20 to 0x7E) are normally echoed, but which control characters (if any) are normally echoed in a case like this?
Is there some Posix or other standard for this? How does Linux do it?
For example, if I type a ^C at the terminal, should the ^C be echoed by the host? What about ^G (bell)? Etc?
I'll try to answer my own question. Here's what I'm planning to do:
Printables (ASCII 0x20 to 0x7E) are echoed.
CR is echoed as CR LF (because the Enter key on terminals normally
sends CR, and an ANSI terminal requires CR to move the cursor to the
left, and LF to move it to the next line).
BS (backspace, 0x08) and DEL (0x7F) are treated identically and are
echoed as "\b \b" (in C syntax) - that is, backspace, space,
backspace, to erase the last character on the terminal.
All other control characters are not echoed. (Not to say they're not processed, but they're not automatically echoed. What they do is outside the scope of what I'm asking about.)
My reasoning is that the remaining control characters are usually meant to do something, and that something is meant to happen at the host, not at the terminal.
For example DC1/DC3 (^Q/^S) are often used as flow control ("XON/XOFF") - it doesn't make sense to echo the ^S (XOFF) back to the terminal, since the purpose is to flow control the host. Echoing XOFF back to the terminal would flow control the terminal, clearly not what was intended. So echoing this makes no sense.
Similarly, ANSI escape sequences sent by the terminal (cursor up/down/left/right, etc.) should not be echoed.
Bottom line - echo printables only. Control characters as a rule should not be echoed, except on a case-by-case basis depending on their function (carriage return, backspace, etc.).
I'd like comments on whether or not this is the right thing to do (and why).
Related
Struggling to get using xtermjs, and have some questions which aren't covered in the official documentation, at least I didn't find.
I understand that when I use some app within the terminal, for example, Vim terminal need to be switched to alternate buffer, after I quit from the app, terminal switched back to the normal buffer. Is this right?
To switch between buffers (and overall to control terminal behavior) I need to use a control sequence. It isn't something unique to xterm.js, but the common pattern and control sequence is unified between terminals?
Control sequence to switch to the alternate buffer is CSI ? Pm h with parameter 47 according to documentation:
DECSET DEC Private Set Mode CSI ? Pm h Set various terminal attributes.
Where
paramAction 47 - Use Alternate Screen Buffer.
How to use this control sequence with xterm.js, for example, I want to switch to alternate buffer. What string should be used in terminal.write(...)?
Yes, see description here in this question Using the “alternate screen” in a bash script
The alternate screen is used by many "user-interactive" terminal applications like vim, htop, screen, alsamixer, less, ... It is like a different buffer of the terminal content, which disappears when the application exits, so the whole terminal gets restored and it looks like the application hasn't output anything
Yes, ANSI escape code
ANSI escape sequences are a standard for in-band signaling to control the cursor location, color, and other options on video text terminals and terminal emulators. Certain sequences of bytes, most starting with Esc (ASCII character 27) and '[', are embedded into the text, which the terminal looks for and interprets as commands, not as character codes.
Control sequence to switch to alternate buffer: CSI ? 47 h
Control sequence to switch to regular buffer: CSI ? 47 l
Code to apply control sequence to switch to alternate buffer:
terminal.write("\x9B?47h"); //CSI ? 47 h
I have a Linux distribution, and I'd like to be able to detects a wide range of keys, but I can't figure out how to distinguish the press of the Escape key, versus a key detected as keypress.
If I use this:
require 'io/console'
puts STDIN.raw { STDIN.getc }
and press, say, Arrow up, the interpreter will return immediately \e, then on subsequent getc calls, without pressing any key, it will return the remaining chars of the escape sequence ([ and A).
The problem is that, after the first getc call, I don't know if the user has actually pressed Esc, or another key which generates an escape sequence.
Is it possible to make the distinction between the two cases, without multiple getc invocations?
I don't think so, because the character input is done at a higher level than the physical keyboard. That is, by design, an ESC character read by getc is an ESC character, no matter how on the keyboard it was input (or not input on the keyboard at all, for example, by redirection).
I don't think Ruby out of the box has a way to detect hardware events such as keyboard presses. There is IOCTL for lower level device access, but I don't know how to use it for your purpose. But even if you could, how would you handle alternate keyboard layouts? What you expect to be an ESC key might be mapped to a different character.
There is a read_nonblock method on $stdin that you might use to see if there is an additional character in the buffer immediately after reading the ESC key. If so, it's likely that a special character other than ESC was pressed. If not, it's likely the ESC key was pressed.
I have a little program that prints out a direcory structure.
It works fine except when the direcory names contain german umlaut characters.
In this case int prints a blank line after the directory line.
I'm running Python 3.50 on Windows 7 64bit.
This Code ...
class dm():
...
def print(self, rootdir=None, depth=0):
if rootdir is None:
rootdir = self.initialdir
if rootdir in self.dirtree:
print('{}{} ({} files)'.format(' '*depth,
rootdir,
len(self.dirtree[rootdir]['files'])))
for _dir in self.dirtree[rootdir]['dirs']:
self.print(os.path.join(rootdir, _dir), depth+1)
else:
pass
...produces the following output:
B:\scratch (11 files)
B:\scratch\Test1 (3 files)
B:\scratch\Test1 - Kopie (0 files)
B:\scratch\Test1 - Übel (0 files)
B:\scratch\Test2 (3 files)
B:\scratch\Test2\Test21 (0 files)
This is so with codepage set to 65001. If i change the codepage to e.g. 850 then the blank line disappears but of course the "Ü" isn't printed correctly.
The structure self.dirtree is a dict of dicts of lists, is parsed with os.walk and seems OK.
Python or Windows? Any suggestions?
Marvin
There are several bugs when using codepage 65001 (UTF-8) -- all of which are due to the Windows console (i.e. conhost.exe), not Python. The best solution is to avoid this buggy codepage, and instead use the wide-character API, such as by loading win_unicode_console.
You're experiencing a bug that exists in the legacy console that was used prior to Windows 10. (It's still available in Windows 10 if you select the option "Use legacy console".) The console decodes the UTF-8 buffer to UTF-16 and reports back that it writes b'\xc3\x9c' (i.e. "Ü" encoded as UTF-8) as one character, but it's supposed to report back the number of bytes that it writes, which is two. Python's buffered sys.stdout sees that apparently one byte wasn't written, so it dutifully writes the last byte of the line again, which is b'\n'. That's why you get an extra newline. The result can be far worse if a written buffer has many non-ASCII characters, especially codes above U+07FF that get encoded as three UTF-8 bytes.
There's a worse bug if you try to paste "Ü" into the interactive REPL. This bug is still present even in Windows 10. In this case a process is reading the console's wide-character (UTF-16) input buffer encoded as UTF-8. The console does the conversion via WideCharToMultiByte with a buffer that assumes one Unicode character is a single byte in the target codepage. But that's completely wrong for UTF-8, in which one UTF-16 code may map to as many as three bytes. In this case it's two bytes, and the console only allocates one byte in the translation buffer. So WideCharToMultiByte fails, but does the console try to increase the translation buffer size? No. Does it fail the call? No. It actually returns back that it 'successfully' read 0 bytes. To Python's REPL that signals EOF (end of file), so the interpreter just exits as if you had entered Ctrl+Z at the prompt.
When I type any key, normally, it is immediately echoed back to the std output i.e.my screen.
If I have to enter a password, that says that it will not echo back, I cannot see the keys that I type.
How does this work.
Does each key press go to the kernel immediately(without me pressing ENTER), and then the kernel decides to echo them or not?
Like , I press 'A', it goes to the kernel; kernel echoes it; I get it on my screen. Now I hit 'B'...same sequence again...; Now I have 'AB' on my screen (my command) and hit ENTER; My command goes to the kernel and is finally executed.
Is there any other explanation? What happens in the background during the key presses?
The terminal driver in the kernel can be put in several modes (there are actually many more flags than this, and these days "cbreak" is actually the opposite of a different flag, so this is simplified).
The "cbreak" mode means that the process that is attempting to read from the terminal will receive keyboard input as soon as it is available. When cbreak mode is off, the data is stored by the kernel in a buffer until enter is pressed, and certain keys like backspace are handled by the kernel (when you press backspace, it removes the character from the buffer and - if echo mode is on - writes "backspace-space-backspace" to the terminal to overwrite the character with a blank space).
Echo mode means that whenever the user presses a key, the kernel will immediately echo it back to the screen. When it is off, nothing will be echoed to the screen, and the program will need to write it to the terminal if it wants you to see it.
There are several typical situations:
In programs that do advanced input handling (like most shells, or something like a full screen program), cbreak is on and echo is off; the program will write the character to the terminal itself if it is not part of a special key escape sequence.
In most situations [the default with a simple program that reads stdin and writes stdout], echo is on and cbreak is off. When you type, it behaves as I described above, all of this is handled by the kernel until you hit enter and it sends it to the application. Input editing is limited to backspace [and ctrl-u, ctrl-w], trying to use the arrow keys will just put escape sequences like ^[[D in the input line.
When reading a password, echo is off and cbreak is off. Input works just like the default case, except the kernel does not copy input to the screen.
The program that is running tells the kernel what mode to have it in with the termios functions. You can use the stty command to do the same in a shell environment, but be aware that this may interfere with the shell's own input handling or with what programs you run expect the default state to be.
Your keyboard generates electrical signals that are eventually interpreted as keycodes that correspond to letters - 'A', 'B', function keys F1, F2 etc. This all happen in the keyboard driver that is handled by the kernel. That keyboard driver has a buffer to receive all the keypresses from the keyboard and sends that to the kernel that in turn direct them to processes that is currently having the focus. What to do with the sequence of keys are totally decided by the individual application, such as whether to display the keys or not.
echo program is part of coreutils. You can download its sources here. Look at src/echo.c it's quite small. You can see that echo uses fputc or putchar calls. Those calls deal with standard stream called stdout. The architecture of standard streams is quite complicated and it's beyond of this post. You can found it using for example google.
Is it possible in any way to send the key "C-(" to Emacs over a VT100/xterm terminal (Mac OS X Terminal)? Is there an escape sequence that could be sent to achieve the equivalent?
I suspect the fundamental problem is that the concept of combining control with the character "(" (and other such characters that are produced using shift) does not exist.
Note: Using Cocoa Emacs is not an option. And the reason for needing "C-(" is that paredit.el uses it amongst other key combinations, and it would be preferable to not remap it (because it makes sense to have it on "C-(").
A VT100 terminal couldn't do that, because there is no ^( control character corresponding to (. However, xterm has the so-called "modifyOtherKeys" mode, which does allow to send unique keycodes for combinations like that.
To enable it, set the modifyOtherKeys resource, e.g. in ~/.Xdefaults:
XTerm*vt100.modifyOtherKeys: 1
With that, Ctrl+( will send the following keycode:
^[[27;6;40~
That's rather long though, so another format for keycodes like that was introduced, which can be enabled by setting the formatOtherKeys resource:
XTerm*vt100.formatOtherKeys: 1
With that, Ctrl+( sends:
^[[40;6u
In both of these keycodes, the 40 is the decimal ASCII code for (, and the 6 represents the Ctrl.
See man xterm and http://invisible-island.net/xterm/ctlseqs/ctlseqs.html for further details. No idea whether Terminal.app supports any of it.