I am running a for loop inside a while loop.
File passed as parameter has the contents:
peter
roger
casie
I am trying to create a path to test existence of files a,b,c,d,e
I am expecting the output to be
/peter/a
/peter/b
and so on.
Instead I am getting
/aeter
/beter
etc.
What do I need to understand here? Please find the code below -
CODE:
while read fileLine; do
x=$fileLine
for i in a b c d e
do
echo /$x/$i
done
done < $1
Apparently your input file uses the windows end-of-line format of \r\n. The read removes the \n but leaves the \r. When the string /$x/$i is printed, the "carriage" is returned to the beginning of the line at the end of the x string, printing the slash over top of the slash from x and printing the letter from i over the first letter of x.
You may be able to fix it by replacing your x=$fileLine line with
x=${fileLine%?}
which should remove the last character.
Related
I'm looking for a bit of help here. I'm a complete newbie!
I need to look in a file for a code matching the pattern A00000_00_A and append a count to it, so the first time it appears it is replaced with A00000_00_A_001, second time A00000_00_A_002 etc. The output needs to be written back to the same file. Each file only contains 1 code, but it appears multiple times.
After some digging I have found-
perl -pi -e 's/Q\d{4,5}'_'\d{2}_./$&.'_'.++$A /ge' /users/documents/*.xml
but the issue is the counter does not reset in each file.
That is, the output of the first file is say Q00390_01_A_1 to Q00390_01_A_7, while the second file is Q00391_01_A_8 to Q00391_01_A_10.
What I want is Q00390_01_A_1 to Q00390_01_A_7 in the first file and Q00391_01_A_1 to Q00391_01_A_2 in the second.
Does anyone have any idea on how to edit the above code to make it do that? I'm a total newbie so ideally an edit to what I have would be brilliant. Thanks
cd /users/documents/
for f in *.xml;do
perl -pi -e 's/facs=.(Q|M)\d{4,5}_\d{2}_\w/$&._.sprintf("%04d",++$A) /ge' $f
done
This matches the string facs= and any character, then "Q" or "M" followed by either four or five digits, then an underscore, then two digits, another underscore, and a word character. The entire match is then concatenated with an underscore and the value of $A zero padded to four digits.
I'm getting really strange output for this program. What is the "Carriage Return" doing, and how to remove it - missing single quote in the end? Why is the letter "T" missing? How to write code to correct this?
code i'm using
#!/bin/bash
export DATABASE_LIST="/opt/halogen/crontab/etc/db_stat_list.cfg"
export v3=""
while read -r USERID ORACLE_SID2
do
v3="This is '${ORACLE_SID2}' "
echo $v3
done < <(tac $DATABASE_LIST)
output
'his is 'OT1SL80
'his is 'OT1SL010
The file I'm reading from is not corrupt and is small one with two lines
[oracle#ot1sldbm001v test2]$ cat /opt/halogen/crontab/etc/db_stat_list.cfg
asp_dba/dba OT1SL010
asp_dba/dba OT1SL80
Thank you
Your DATABASE_LIST file is in DOS/Windows format, with carriage return + linefeed at the end of each line. Unix uses just linefeed as a line terminator, so unix tools treat the carriage return as part of the content of the line. You can keep this from being a problem by telling the read command to treat the carriage return as whitespace (like spaces, tabs, etc), since read automatically removes whitespace from the beginning and end of lines:
...
while IFS="$IFS"$'\r' read -r USERID ORACLE_SID2
...
Note that since this assignment to IFS (which basically lists the whitespace characters) is a prefix to the read command, it only applies to that one command and doesn't have to be set back to normal afterward.
While implementing a script, I am facing the following issue :
when putting the multi-line result of a command into a variable, it seems the last (empty) line of my multi-line string disappear.
This line is "empty", but however, I can not lose the carriage return it contains (because I am concatenating blocks of code saved in DB and containing "\n" character into a human-readable string... If I lose some of the "\n", I will lose a part of my code indentation)
Here is the code to illustrate my issue :
test="A
B
";
test2=`echo "$test"`;
echo "||$test2||";
This returns
||A
B||
while I was expecting :
||A
B
||
--> the last (empty) line has disappeared... and a carriage return is thus missing in my human-readable code.
This issue only occurs when the last line of my multi-line string is empty...
Do you know
Why this last line disappears ?
How I can ensure my last empty line is saved in my multi-line string variable ?
Note that I can of course not use the easiest solution
test2="$test";
because the complete process is rather :
test="^A\n\nB\n^"
test2="`echo "$test" | sed -e 's/\^//g'`";
but I tried to simplify the issue the most I could.
Command substitutions always trim trailing newlines -- that's in accordance with design and specification. If you don't want that, you can append a fixed sigil character to your output and trim it, such that the newlines you want to preserve are before the sigil:
test="A
B
"
test_wip=$(printf '%sEND' "$test")
test2=${test_wip%END}
Instead of trying to work around the issues that arise from assigning the output from echo to a variable (eg, stripping of trailing \n's), consider using ksh's built in string processing in this case, eg:
$ test="^A\n\nB\n^"
$ test2="${test//^}"
$ echo "||${test2}||"
||A
B
||
//^ : remove all ^ characters
I have a huge file, a really huge file (some 600+MB of text). In fact they are jsons. Each json is on a new line and only comes in a few flavours.
They look like:
{"text":{"some nested words":"Some more","something else":"Yeah more stuff","some list":["itemA","ItemB","itemEtc"]},"One last object":{"a thing":"and it's value"}}
And what I want is it go through with sed, suck out the text, and for each nexted pair put in some indent, so we get:
{
-{
--[]
-}
--{}
-}
}
(I'm not 100% sure I got the nesting right on the output, I think it's right)
Is this possible? I saw this, which was the closest I could imagine it being, but that gets rid of the brackets two.
I've noticed the answer there uses braching, so I think I need that, and I'll need to do some kind of s/pattern/newline+tab/space/g type command but I can't figure out how or what to make that...
Could someone help please? It needn't be pure sed but that is prefered.
This will not be pretty... =) Here is my solution as a sed script. Notice that it requires that the first line notifies the shell how to invoke sed to execute our script. As you can see, the "-n" flag is used so we force sed only to print what we explicitly command it to through the "p" or "P" commands. The "-f" option tells sed to read the commands from a file, with the name following the option. As the file name of the script is concatenated by the shell into the final command, it will properly read commands from the script (ie. if you run "./myscript.sed" the shell will execute "/bin/sed -nf myscript.sed").
#!/bin/sed -nf
s/[^][{}]//g
t loop
: loop
t dummy
: dummy
s/^\s*[[{]/&/
t open
s/^\s*[]}]/&\
/
t close
d
: open
s/^\(\s*\)[[]\s*[]]/\1[]\
/
s/^\(\s*\)[{]\s*[}]/\1{}\
/
t will_loop
b only_open
: will_loop
P
s/.*\n//
b loop
: only_open
s/^\s*[[{]/&\
/
P
s/.*\n//
s/[][{}]/ &/g
b loop
: close
s/ \([][{}]\)/\1/g
P
s/.*\n//
b loop
Before we start, we must first strip everything into brackets and square brackets. That's the responsibility of the first "s" command. It tells sed to replace every character that isn't a bracket or a square bracket with nothing, ie. remove it. Notice that the square brackets in the match represent a group of characters to match, but when the first character inside them is a "^", it will actually match any character except the ones specified after the "^". Because we want to match the closing square bracket and we need to close with a square bracket the group of characters to ignore, we tell that a closing square bracket should be included in the group by making it the first character following the "^". We can then specify the rest of the characters: opening square bracket, open bracket and close bracket (group of ignored characters: "][{}"), and then close the group with the closing square bracket. I tried to detail more here because this can be confusing.
Now for the actual logic. The algorithm is pretty simple:
while line isn't empty
if line starts with optional spaces followed by [ or {
if after the [ or { there are optional spaces followed by a respective ] or }
print the respective pair, with only the indentation spaces, followed by a newline
else
print the opening square or normal bracket, followed by a newline
remove what was printed from the pattern space (a.k.a. the buffer)
add a space before every open or close bracket (normal or square)
end-if
else
remove a space before every open or close bracket (normal or square)
print the closing square or normal bracket, followed by a newline
remove what was printed from the pattern space
end-if
end-while
But there are a couple of quirks. First of all, sed doesn't support a "while" loop or an "if" statement directly. The closest we can get to is the "b" and "t" commands. The "b" command branches (jumps) to a predefined label, similar to a C goto statement. The "t" also branches to a predefined label, but only if a substitution has happened since the start of the script running on the current line or since the last "t" command. Labels are written with the ":" command.
Because it is very likely that the first command actually performs at least one substitution, the first "t" command that follows it will cause a branch. Because we need to test for some other substitutions, we need to make sure that the next "t" command won't automatically succeed because of that first command. That is why we start with a "t" command to a line just above it (ie. if it branches or not, it will still continue at the same point), so we can "reset" the internal flag used by "t" commands.
Because the "loop" label will be branched to from at least one "b" command, it is possible that the same flag will be set when the "b" is executed, because only "t" commands can clear it. Therefore, we need to do the same workaround to reset the flag, this time by using a "dummy" label.
We now start the algorithm by checking for the presence of an open square bracket or an open close bracket. Because we only want to test for their presence, we must replace the match with itself, which is what "&" represents, and sed will automatically set the internal flag for the "t" command if the match succeeds. If the match succeeds, we use the "t" command to branch into into "open" label.
If it doesn't succeed, we need to see if we match a close square or normal the bracket. The command is nearly identical, but now we append a newline after the closing bracket. We do this by adding an escaped newline (ie. a backslash followed by an actual newline) after where we place the match (ie. after the "&"). Similarly to above, we use the "t" command to branch to the "close" label if the match succeeds. If it doesn't succeed, we will consider the line as invalid, and promptly empty the pattern space (buffer) and restart the script on the next line, all with the single "d" command.
Entering the "open" label, we will first handle the case of a pair of matching open and close brackets. If we do match them, we will print them with the indentation spaces preceding them, without any spaces between them, and ending with a newline. There is one specific command for each type of bracket pair (square or normal), but they are analogous. Because we have to keep track of how many indentation spaces there are we must store them in a special "variable". We do this by using the group capture, which will store the part of the match that starts after the "(" and ends before the ")". Therefore, we use it to capture the spaces after the start of the line and before the open bracket. We then proceed to match the open bracket followed by spaces and the respective close bracket. When we write the replacement, we make sure to reinsert the spaces by using the special variable "\1", which contains the data stored by the first group capture in the match. We then write the respective pair of open and close brackets and the escaped newline.
If we managed to do any of the replacements, we must print what we have just written, remove it from the pattern space and restart the loop with the remaining characters of the line. Because of this, we first branch with the "t" command to the "will_loop" label. Otherwise, we branch to the "only_open" label, which will handle the case of only an open bracket, without the consecutive respective close bracket.
Inside the "will_loop" label, we just print everything in the pattern space up to the first newline (which we manually added) with the "P" command. We then manually remove everything up to that first newline, so we can proceed with the rest of the line. This is similar to what the "D" command does, but without restarting the execution of the script. Finally we branch to the start of the loop again.
Inside the "only_open" label, we match an open bracket in a similar fashion as previously, but now we rewrite it appended with a newline. We then print that line and remove it from the pattern space. Now we replace all brackets (open or close, square or normal) with itself preceded by a single space character. This is so we can increment the indentation. Finally we branch to the beginning of the loop again.
The final label "close" will handle a closing bracket. We first remove every single space before a bracket, effectively decrementing the indentation. To do this, we need to use captures, because although we want to match the space and the bracket that follows, we only want to write back the bracket. Finally, print everything up to the newline that we manually added before entering the "close" label, remove what we printed from pattern space and restart the loop.
Some observations:
This doesn't check for the syntactic correctness of the code (ie. {{[}] would be accepted)
It will add and remove indentation as brackets are encountered, regardless of their type. This means that when it adds an indentation, it will remove it even if the encountered close bracket is not of the same type.
Hope this helps, and sorry for the long post =)
This might work for you (GNU sed):
sed 's/[^][{}]//g;s/./&\n/g;s/.$//' file |
sed -r '/[[{]/{G;s/(.)\n(.*)/\2\1/;x;s/^/\t/;x;b};x;s/.//;x;G;s/(.)\n(.*)/\2\1/' |
sed -r '$!N;s/((\{).*(\}))|((\[).*(\]))/\2\5\3\6/;P;D'
Explanation:
The first sed command produces a stream of curly/square brackets each on its own line
The second sed command indents each bracket
The third sed command reduces those paired brackets to a single line
If your happy with correctly indented brackets, the third command can be omitted.
I think you're expected output should look like:
{
-{
--[]
-}
-{
-}
}
Here's one way using GNU awk:
awk -f script.awk file.txt
Contents of script.awk:
BEGIN {
FS=""
flag = 0
}
{
for (i=1; i<=NF; i++) {
if ($i == "{" || $i == "[") {
flag = flag + 1
build_tree(flag, $i)
printf (flag <=2) ? "\n" : ""
}
if ($i == "}" || $i == "]") {
flag = flag - 1
printf (flag >= 2) ? $i : \
build_tree(flag + 1, $i); \
printf "\n"
}
}
}
function build_tree (num, brace) {
for (j=1; j<=num - 1; j++) {
printf "-"
}
printf brace
}
I know this is an ancient thread and nobody is looking anyway, but there's a simpler way now.
cat file.txt | jq '.' | sed 's/ /-/g' | tr -dc '[[]{}()]\n-' | sed '/^-*$/d'
There are 2 spaces in the first sed.
I know that Esc + . gives you the last argument of the last command.
But I'm interested in first argument of the last command.
Is there a key binding to do so?
On the same lines, is there a generic way of getting the nth argument from the last command?
I know that in a bash script, you can use $0, $1 etc., but these don't work on the commandline.
Also, what about iterating through the 0th argument of previous commands, like we can do with the last argument by continuously pressing Esc + .?
!$ gets the last element of the previous command line argument.
Just as M-. (meta-dot or esc-dot or alt-dot) is the readline function yank-last-arg, M-C-y (meta-control-y or esc-ctrl-y or ctrl-alt-y) is the readline function yank-nth-arg. Without specifying n, it yanks the first argument of the previous command.
To specify an argument, press Escape and a number or hold Alt and press a number. You can do Alt--to begin specifying a negative number then release Alt and press the digit (this will count from the end of the list of arguments.
Example:
Enter the following command
$ echo a b c d e f g
a b c d e f g
Now at the next prompt, type echo (with a following space), then
Press Alt-Ctrl-y and you'll now see:
$ echo a
without pressing Enter yet, do the following
Press Alt-3 Alt-Ctrl-y
Press Alt-- 2 Alt-Ctrl-y
Now you will see:
$ echo ace
By the way, you could have put the echo on the line by selecting argument 0:
Press Alt-0 Alt-Ctrl-y
Edit:
To answer the question you added to your original:
You can press Alt-0 then repeatedly press Alt-. to step through the previous commands (arg 0). Similarly Alt-- then repeating Alt-. would allow you to step through the previous next-to-last arguments.
If there is no appropriate argument on a particular line in history, the bell will be rung.
If there is a particular combination you use frequently, you can define a macro so one keystroke will perform it. This example will recall the second argument from previous commands by pressing Alt-Shift-Y. You could choose any available keystroke you prefer instead of this one. You can press it repeatedly to step through previous ones.
To try it out, enter the macro at a Bash prompt:
bind '"\eY": "\e2\e."'
To make it persistent, add this line to your ~/.inputrc file:
"\eY": "\e2\e."
Unfortunately, this doesn't seem to work for arg 0 or negative argument numbers.
To use the first argument, you can use !^ or !:1
Example:
$ echo a b c d e
a b c d e
$ echo !^
echo a
a
$ echo a b c d e
a b c d e
$ echo !:1
echo a
a
Since your question is about using any other arguments, here are some useful ones:
!^ first argument
!$ last argument
!* all arguments
!:2 second argument
!:2-3 second to third arguments
!:2-$ second to last arguments
!:2* second to last arguments
!:2- second to next to last arguments
!:0 the command
!! repeat the previous line
The first four forms are more often used. The form !:2- is somewhat counter-intuitive, as it doesn't include the last argument.
I liked #larsmans answer so much I had to learn more. Adding this
answer to help others find the man page section and know what to
google for:
$ man -P 'less -p ^HISTORY\ EXPANSION' bash
<...>
Word Designators
Word designators are used to select desired words from the event.
A : separates the event specification from the word designator.
It may be omitted if the word designator begins with a ^, $, *, -,
or %. Words are numbered from the beginning of the line, with the
first word being denoted by 0 (zero). Words are inserted into the
current line separated by single spaces.
0 (zero)
The zeroth word. For the shell, this is the command word.
n The nth word.
^ The first argument. That is, word 1.
$ The last argument.
% The word matched by the most recent ‘?string?’ search.
x-y A range of words; ‘-y’ abbreviates ‘0-y’.
* All of the words but the zeroth.
This is a synonym for ‘1-$’.
It is not an error to use * if there is just one word in
the event; the empty string is returned in that case.
x* Abbreviates x-$.
x- Abbreviates x-$ like x*, but omits the last word.
If a word designator is supplied without an event
specification, the previous command is used as the event.
Tested on Ubuntu 18.04
To insert previous arguments:
Alt+.: insert last argument from last command.
Alt+#+.: insert #nth last argument from last command.
Alt+- , # , Alt+., zsh: Alt+-+#+.: insert #nth first argument from last command.
In Linux you can repeat commands to go back in history
Example:
Last command is:
mv foo bar
Alt+0+.: insert first argument of last command = mv
Alt+2+.: insert last 2nd argument of last command = foo
up , Ctrl+w: last command without the last word = mv foo
General shortcuts
Ctrl+w: removes last word from cursor
Alt+d: removes next word from cursor
Ctrl+k: cuts everything after cursor
Ctrl+u, zsh: Alt+w: cuts everything before cursor
zsh: Ctrl+u: cuts the entire command (In bash you can combine Ctrl+u , Ctrl+k)
Ctrl+y: paste characters previously cut with Ctrl+u and Ctrl+k
Ctrl+_: undo last edit (very useful when exceeding Ctrl+w)
Ctrl+left: move to last word
Ctrl+right: move to next word
home or Ctrl+a: move to start of line
end or Ctrl+e: move to end of line
To iterate through the arguments in a previous command
only works in zsh
run or add this to your ~/.zshrc
autoload -Uz copy-earlier-word
zle -N copy-earlier-word
bindkey "^[:" copy-earlier-word
Now use Alt+. to go as back as you want, then use Alt+: to iterate through the arguments
Assuming last command is
echo 1 2 3 4 5
Alt+.: 5
Alt+.+:: 4
Alt+.+:+:: 3
Alt+.+:+:+:: 2
Alt+.+:+:+:+:: 1
Alt+.+:+:+:+:+:: echo
source: https://stackoverflow.com/a/34861762/3163120
To see all shortcuts available
bash: bind -lp
zsh: bindkey -L
I'm keeping this up-to-date here: https://github.com/madacol/knowledge/blob/master/bash-zsh_TerminalShorcuts.md
!^ may be the command for the first argument. i'm not sure if there is a way to get the nth.
You can also get arguments from any command in your history!
$ echo a b c d e f g
a b c d e f g
$ echo build/libs/jenkins-utils-all-0.1.jar
build/libs/jenkins-utils-all-0.1.jar
$ history | tail -5
601 echo build/libs/jenkins-utils-all-0.1.jar
602 history | tail -10
603 echo a b c d e f g
604 echo build/libs/jenkins-utils-all-0.1.jar
605 history | tail -5
$ echo !-3:4
echo d
d
$ echo !604:1
echo build/libs/jenkins-utils-all-0.1.jar
build/libs/jenkins-utils-all-0.1.jar
!^ will get you the first param, !$ will get you the last param, !:n will get you the nth element.
Basically it has a use in yanking previous (command's) arguments.
For instance, if the following command is issued:
echo Hello, world how are you today?
then, Hello, will be the first argument, and today? the sixth, that is the last one; meaning it can be referenced by typing:
Alt+6 followed by Ctrl-Alt-6
Ctrl is traditionally denoted as a hat character ^ prepended to keys names, and Alt as M- that is Meta prefix.
So the above shortcut can be redefined as ^My to yank.
Also, there is hats substitution shortcut in the command line:
echo Hello, world!
^Hello^Bye
Bye, world!
to substitute the previous command's first matched string, meaning:
Hello, world! Hello, people!
^Hello^Bye
would result in:
Bye, world! Hello, people!
leaving the second match (hello) unchanged.
Note: Do not leave space between hats, or the operation won't work.
The above is just a shortcut for:
!:s/Hello/Bye
event-level(*) substitution for the first found (matched) string in the previous command, while prefixing the first part with the g switch will apply to the whole line globally:
echo Hello, world! Hello, people!
!:gs/Hello/Bye
Bye, world! Bye, people!
as usually being done in other related commands such as sed, vi, and in regex (regular expression) - a standart way to search (match string).
No, you can't do !:sg/Hello/Bye or !:s/Hello/Bye/g here, that's the syntax!
! is for events; event might be understood as command output or operation done in the commands history.
That's what I understood by using it myself and trying things on my own from what I read from various sources including manual pages, blogs, and forums.
Hope it will shed some light into mysterious ways of bash, the Bourne-Again shell (a play on sh shell, which itself is called Bourne shell after its inventor's last name), what is default shell in many distributions including servers (server OS's).
The method described at the end of the accepted answer also works with the zeroth argument for me. I have these lines in my ~/.inputrc:
"\en": "\e0\e."
"\em": "\e1\e."
"\e,": "\e2\e."
\e2\e. has the advantage over \e2\e\C-y that it cycles through previous commands if it is pressed repeatedly instead of inserting the second argument of the previous command multiple times.
To insert the whole previous command, you can type !!\e^. \e^ is history-expand-line.
If you are on a mac you will tend to get extended characters with ctrl+letter. I have my right-of-space-bar-option key defined as meta in my terminal (iTerm2) set up. This means I use the key to navigate by word and pull parameters from previous commands.
For pasting 1th argument, press and hold down Alt key, and while it is down, hit the '1' key followed by the '.' key.
For pasting n-th argument, replace the '1' key above with the corresponding number key.
If this does not work, your terminal emulator may be catching the Alt key before it gets to shell. Some terminals (xfce4-terminal) allow turning off the "Alt-" shortcuts in the configuration file.
Credit to Jonas Eberle, I've fished this out from his comment to another answer here.