I have an Executable which shows progress bar on terminal. when i redirect the output to a file in th ebelow manner
the below command is embedded in a script file
usr/bin/exec >> log.txt
this is terminal out put
Progress [====================================> ] 58%
But the log.txt looks some thing like this
Progress [=> ] 0
Progress [====> ] 5
Progress [========> ] 10
can i redirect the output in same way as it is shown on terminal.
A related answer on a different site, suggests the col command.
To suppress overwritten output on standard out prior to writing to the file:
usr/bin/exec | col -b >> log.txt
To view the contents of a file while suppressing overwritten characters:
col -b < log.txt | less
Your progress bar probably works by using a carriage return to signal that the current line should be overwritten. The col utility buffers the output so that it can print only the last character for each position.
There's no way to answer this with certainty without knowing exactly what your progress-bar program is doing, but...
The way such a progress bar is typically done is by outputting the current progress update, but then putting an ascii CR (carriage return) at the end of the line without an LF (line feed). This resets the display cursor to the beginning of the line without moving the cursor down to the next line so that each subsequent progress update overwrites the previous one. So probably, if you're redirecting that to a file, you'll have the same content in the file.
If you were to view such a file with vi, you'll actually see the embedded CRs, indicated as "^M" (but note that LF characters are simply interpreted as end-of-line and aren't actually displayed). If you view the file with another editor, it might interpret CR characters as a valid end-of-line indicator and show you each line as a separate line with or without LF characters.
You don't say how you're displaying the log.txt file. I would expect that if the program were using the embedded CR method, then using "cat log.txt" would show the same output as in interactive use: that is, each progress update "line" would overwrite the previous one and only the last one would be visible at the end.
It's also possible that the program outputting the progress bar is sensitive to whether it's displaying to a terminal device or not: i.e. it might be calling isatty() and modifying its output accordingly.
It might be instructive to run this command on your final log.txt file to see the actual binary content:
od -tx1z log.txt
A CR (aka ^M) will show up as "0d". An LF (aka ^J) will show up as 0a.
Related
I want to download a huge file from s3 to a server so I'm using nohup.
Problem is, the process outputs "self-updating reports", which are cool in the terminal but are horrible when written to a file as a single long line.
My questions are:
How should I handle this output to avoid a 84MB text file of just one line?
Given that I have such a file, how can I read its "bottom line" effectively?
thanks!
The "self-updating" text is just a carriage return character, which when printed to the terminal moves the cursor back to the beginning of line, allowing you to overwrite any previous text on the line.
To remove everything up through the last carriage return character,
awk '{ sub(/.*\r/, ""); }1' file
or the same with any regex tool (sed 's/.*\r//' file would work if your sed recognizes the nonstandard \r escape).
Some tools have an option to turn off progress updates; if you are using the standard aws s3 cp, try adding the option --no-progress.
I am using Windows and MobaXterm.
I created a .bash_profile file in the ~ directory and the following line
alias sbp="source ~/.bash_profile"
is the only code in that file.
However, when I was trying to do sbp, I got an error.
This works on my Mac and it used to work on my old Windows computer (but that one has some water damage so it broke down). Why does this not work now?
Thanks in advance!
From the way that error message is garbled I'm pretty sure that the .bash_profile file you created has DOS/Windows-style line endings, consisting of a carriage return character followed by a newline character. Unix tools expect unix-style line endings consisting of just a newline; if they see DOS/Windows-style endings, they'll treat the carriage return as part of the content of the line. In this case, bash will treat the carriage return as part of the alias definition, and therefore part of the filename to filename to source. Try running alias sbp | cat -vt to print the alias with invisible characters shown; my guess is it'll print alias sbp='source ~/.bash_profile^M' (where the ^M is cat -vt's way of representing the carriage return).
Solution: convert the file to unix format, and either switch to a text editor that knows how to save in unix format, or change your settings in the current editor to do it. For conversion, there are a number of semi-standard tools like dos2unix and fromdos. If you don't have any of those, this answer has some other options.
BTW, the reason the error message is garbled is that the CR gets printed as part of the error message, and the terminal treats that as an instruction to go back to the beginning of the line; it then prints the rest of the message over top of the beginning of the message. It's a little like this:
-bash: /home/dir/path/.bash_profile
: No such file or directory
...but with the second line printed over the first, so it comes out as:
: No such file or directory_profile
I have an output file (namely a log from screen) containing several control characters. Inside the screen, I have programs running that use control characters to refresh certain lines (examples would be top or anything printing progress bars).
I would like to output a tail of this file using PHP. If I simply read in that file and echo its contents (either using PHP functions or through calling tail, the output is messy and much more than these last lines as it also includes things that have been overwritten. If I instead run tail in the command line, it returns just what I want because the terminal evaluates the control characters.
So my question is: Is there a way to evaluate the control characters, getting the output that a terminal would show me, in a way that I could then use elsewhere (e.g., write to a file)?
#5gon12eder's answer got rid of some control characters (thanks for that!) but it did not handle the carriage return part that was even more important to me.
I figured out that I could just delete anything from the beginning of a line to the last carriage return inside that line and simply keep everything after that, so here is my sed command accomplishing that:
sed 's/^.*\r\([^\r]\+\)\r\?$/\1\r/g'
The output can then be further cleaned using #5gon12eder's answer:
cat screenlog.0 | sed 's/^.*\r\([^\r]\+\)\r\?$/\1\r/g' | sed 's,\x1B\[[0-9?;]*[a-zA-Z],,g'
Combined, this looks exactly like I wanted.
I'm not sure what you mean by “evaluating” the control characters but you could remove them easily.
Here is an example using sed but if you are already using PHP, its internal regex processing functionality seems more appropriate. The command
$ sed 's,\x1B\[[0-9?;]*[a-zA-Z],,g' file.dat
will dump the contents of file.dat to standard output with all ANSI escape sequences removed. (And I'm pretty sure that nothing else is removed except if your file contains invalid escape sequences in which case the operation is ill-defined anyway.)
Here is a little demo:
$ echo -e "This is\033[31m a \033[umessy \033[46mstring.\033[0m" > file.dat
$ cat file.dat
# The output of the above command is not shown to protect small children
# that might be browsing this site.
$ reset # your terminal
$ sed 's,\x1B\[[0-9?;]*[a-zA-Z],,g' file.dat
This is a messy string.
The less program has some more advanced logic built in to selectively replace some escape sequences. Read the man page for the relevant options.
I'm trying to use vim with -s option to run a script that replaces some lines in a file like this (text.txt):
test1
ab
ac
ae
test2
sd
Script file is like this (script):
:silent %s/test1\zs\_.\+\zetest2/\=substitute(submatch(0), '\n\(\w\)', '\n#\1', 'g')/g
:wq
It comments out lines between test1 and test2. Which is what I want. What I don't want though is output before and after prompt. I run it and get:
user#hostname: ~/vimtest$ vim -s script text.txt
^[[?1;2cuser#hostname: ~/vimtest$ 1;2c
So this ^[[?1;2c is bad news already but 1;2c is in the input as if I already typed it. If I hit enter it gives me a bash error. So I have to remove these symbols each time the script is used. Any ideas?
It seems like vim (or some vim startup script) is trying to figure out what type of terminal you are using. The ^[[?1;2c, with the last few characters left in the input buffer, is almost certainly part of your terminal emulator's response to a DA (Device Attributes) query. You can see this yourself by typing in bash:
printf '\033[c'
or, to see the complete return, pause a bit:
printf '\033[c'; sleep 0.1; echo
The response \033[?1;2c means "I'm a VT100 with Advanced Video Option.", which is what xterm and many other console programs respond. (The Linux console itself responds \033[?6c, which means "I'm a VT102.")
The reason that only 1;2c is left in the console input buffer, by the way, is that the initial escape code \033[? was ignored when it was read. The readline library will ignore it without echoing it, whereas normal console input will echo it and then ignore it; that's why the two shell commands above differ.
I can't reproduce this problem with my vim installation, so I don't really even know where to start looking. But you might try to see if disabling all startup files helps:
vim -u NONE -s script text.txt
If that helps, start disabling installed extensions one by one until you find the one which is causing the problem.
:%s/test1\zs\_.\+\ze\ntest2/\=substitute(submatch(0), '\n', '\n#', 'g')/g
:wq
this is tested here, it changed the input file in required way.
Some changes done based on your command:
add \n after \ze
in substitute() function we can just handle the \n, we don't need to capture the word after the \n
I noticed that you tagged the question with bash, so I thought a shell-solution should be accepted too.
awk '/test1/{p=1;print;next}/test2/{p=0;print;next}{$0=(p?"#":"")$0}7' file
this awk oneliner should do that for you. vim is very powerful editor, I love vim. But if you want to do some automatic transformation, I prefer a script or a proper text processing tool. On a linux box you can always find one. It is easier to test and debug.
Test with your input:
kent$ cat f
test1
ab
ac
ae
test2
sd
kent$ awk '/test1/{p=1;print;next}/test2/{p=0;print;next}{$0=(p?"#":"")$0}7' f
test1
#ab
#ac
#ae
test2
sd
If you want to save the text back to your file, you can :
awk '...' file > tmp.file && mv tmp.file file
I'd like to have a blank line after my bash prompt and before the output on my Mac. It should look like this would:
echo; ls
Can I add a newline to my bash prompt and then go back up one line to wait for user input? Is there something obvious I'm missing?
I know this is old but for someone like me who came across this while googling for it. This is how you do this...
It's actually pretty simple!
Check out this link --> Cursor Movement
Basically to move up N number of lines:
echo -e "\033[<N>A HELLO WORLD\n"
Just change the "< N >" to however many lines you want to go back...
For instance, to move up 5 lines it would be "/033[5A"
To my knowledge this is not possible unless you delve into more low-level stuff like full-screen emulators like curses.
This is a bit of a stab in the dark, but you may be able to use VT102 terminal codes to control the cursor without having to use Curses. The relevant VT102 commands that you'd be interested in all consist of sending ESC, then [, then the specific command parameters.
For instance, to move the cursor up one line, one needs to output:
ESC [ 1 A
0x1B 0x5B 0x31 0x41
Be warned that the VT102 documentation generally uses octal, so keep an ascii table handy if you're using hex.
All of this advice is given without having tested it -- I don't know if VT102 commands can be embedded into your bash prompt, but I thought it might be worth a shot.
Edit: Yeah -- looks like a lot of people use VT102 formatting codes in their bash prompts. To translate my above example into something Bash would recognize, putting:
\e[1A
into your prompt should move the cursor up one line.
This is very possible. If your bash has C-v set as the readline quoted-insert command, you can simply add the following to your ~/.inputrc:
RETURN: "\C-e\C-v\n\C-v\n\n"
This wil make bash (readline, actually) insert two verbatim newlines before a regular interpreted newline. By default, only one is inserted, which is what causes output to start on the line after the prompt.
You can test if C-v is set to quoted-insert by typing it in bash (that's Ctrl+V) followed by e.g. an up arrow. This should print ^[[A or something similar. If it doesn't, you can bind it in ~/.inputrc too:
C-v: quoted-insert
RETURN: "\C-e\C-v\n\C-v\n\n"
~/.inputrc can be created if it doesn't exist. The changes will not take effect in running bashes unless you issue a readline re-read-init-file command (by default on C-x C-r). Be careful though. If you do something wrong, enter will no longer issue commands, and fixing your mistake could prove to be difficult. If you should do something wrong, C-o will by default also accept the line.
Adding a newline followed by moving the cursor back to the regular prompt (like you described) is possible, but will not have the effect you intend. The newline you inserted would simply be overwritten by the application output, since you moved the cursor back in front of it.
This works:
trap echo DEBUG
It doesn't add an extra newline if you hit return at an empty prompt.
The command above will cause a newline to be output for every member of a pipeline or multi-command line such as:
$ echo foo; echo bar
\n
foo
\n
bar
To prevent that so that only one extra newline is output before all command output:
PROMPT_COMMAND='_nl=true'; trap -- '$_nl && [[ $BASH_COMMAND != $PROMPT_COMMAND ]] && echo; _nl=false' DEBUG
The DEBUG trap is performed before each command so before the first command it checks to see if the flag is true and, if so, outputs a newline. Then it sets the flag to false so each command afterwards on the line doesn't trigger an extra newline.
The contents of $PROMPT_COMMAND are executed before the prompt is output so the flag is set to true - ready for the next cycle.
Because pressing enter on an empty command line still triggers the execution of the contents of $PROMPT_COMMAND the test in the trap also checks for those contents as the current command and doesn't perform the echo if they match.
I believe (but haven't tried) if you put '\n\b' in the prompt string it would do that.
In general, if you want to find out the codes to do anything a terminal can do, read the terminfo man page.
In this case, the cursor up one line code can be determined by:
tput cuu1
If you redirect the tput output to a file, you can see what control characters are used.
Bash also supports the PROMPT_COMMAND variable, allowing you to run arbitrary commands before each prompt is issued.