I am trying to write a shell script that reads the system configuration status of linux and saves the result as CSV.
The result to be saved is the hostname, setting criterion ID, and actual setting value.
The most problematic part when saving as csv is the newline.
For example,
#!/bin/bash
hostname='linux01'
id='SRV-01'
result="*result*\n"
result=$result`cat /etc/passwd | head -n 5`
echo "$hostname, $id, $value" >> test.csv
When the above script exists, the CSV format is broken due to newline.
When script result is as below
linux01, SRV-01, *result*
root:x:0:0:root:/root:/bin/zsh
daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
bin:x:2:2:bin:/bin:/usr/sbin/nologin
sys:x:3:3:sys:/dev:/usr/sbin/nologin
sync:x:4:65534:sync:/bin:/bin/sync
I want to save to csv file this change.
linux01, SRV-01, *result*\nroot:x:0:0:root:/root:/bin/zsh\ndaemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin\nbin:x:2:2:bin: /bin:/usr/sbin/nologin\nsys:x:3:3:sys:/dev:/usr/sbin/nologin\nsync:x:4:65534:sync:/bin:/bin/sync
First, I used sed as a test to change \n to \\n, but when outputting with echo, the \n string is changed to newline.
echo `cat /etc/passwd | head -n 5 | sed -z 's/\n/\\\n/g'`
It is saved as a \n string in the variable, but it is judged that "\n" is changed to 0x0a in echo.
Ask if there is a way to change the echo output to the string "\n" instead of newline.
You can replace the the newlines.
echo "$hostname, $id, ${result//$'\n'/\\n}" >> test.csv
Related
I have two strings saved in a bash variable delimited by :. I want to get extract the second string, prepend that with THIS_VAR= and append it to a file named saved.txt
For example if myVar="abc:pqr", THIS_VAR=pqr should be appended to saved.txt.
This is what I have so far,
myVar="abc:pqr"
echo $myVar | cut -d ':' -f 2 >> saved.txt
How do I prepend THIS_VAR=?
printf 'THIS_VAR=%q\n' "${myVar#*:}"
See Shell Parameter Expansion and run help printf.
The more general solution in addition to #konsolebox's answer is piping into a compound statement, where you can perform arbitrary operations:
echo This is in the middle | {
echo This is first
cat
echo This is last
}
I wrote a bash script to read multiple inputs from the user
Here is the command:
read -a choice
In this way, I can put all the inputs in the choice variable as an array so that I can extract them using an index.
The problem is that when one of the inputs, which is a string has space in it, like
user1 google.com "login: myLogin\npassword: myPassword"
the read command will split the quoted string into 3 words. How can I stop this from happening?
bash doesn't process quotes in user input. The only thing I can think of is to use eval to execute an array assignment.
IFS= read -r input
eval "choice=($input)"
Unfortunately this is dangerous -- if the input contains executable code, it will be executed by eval.
You can use a tab instead of space as a field delimiter. For instance :
$ IFS=$'\t' read -a choice
value1 value2 a value with many words ## This is typed
$ echo ${choice[2]}
a value with many words
Regards!
Given risk of using eval, and the fact the input seems to have only two types of tokens: unquoted, and quoted, consider using scripting engine that will put all text into proper format that will be easy to read.
It's not clear from the example what other quoting rules are used. Example assume 'standard' escaped that can be processed with bash #E processor.
The following uses Perl one liner to generate TAB delimited tokens (hopefully, raw tabs can not be part of the input, but other character can be used instead).
input='user1 google.com "login: myLogin\npassword: myPassword"'
tsv_input=$(perl -e '$_ = " $ARGV[0]" ; print $2 // $3, "\t" while ( /\s+("([^"]*)"|(\S*))/g) ;' "$input")
IFS=$'\t' read -d '' id domain values <<< $(echo -e "${tsv_input#E}")
Or using a function to get more readable code
function data_to_tsv {
# Translate to TSV
local tsv_input=$(perl -e '$_ = " $ARGV[0]" ; print $2 // $3, "\t" while ( /\s+("([^"]*)"|(\S*))/g) ;' "$1")
# Process escapes
echo -n "${tsv_input#E}"
}
input='user1 google.com "login: myLogin\npassword: myPassword"'
IFS=$'\t' read -d '' id domain values <<< $(data_to_tsv "$input")
I'm writing a for loop in bash to run a command and I need to add a comma after one of my variables. I can't seem to do this without an extra space added. When I move "," right next to $bams then it outputs *.sorted,
#!/bin/bash
bams=*.sorted
for i in $bams
do echo $bams ","
done;
Output should be this:
'file1.sorted','file2.sorted','file3.sorted'
The eventual end goal is to be able to insert a list of files into a --flag in the format above. Not sure how to do that either.
First, a literal answer (if your goal were to generate a string of the form 'foo','bar','baz', rather than to run a program with a command line equivalent to somecommand --flag='foo','bar','baz', which is quite different):
shopt -s nullglob # generate a null result if no matches exist
printf -v var "'%s'," *.sorted # put list of files, each w/ a comma, in var
echo "${var%,}" # echo contents of var, with last comma removed
Or, if you don't need the literal single quotes (and if you're passing your result to another program on its command line with the single quotes being syntactic rather than literal, you absolutely don't want them):
files=( *.sorted ) # put *.sorted in an array
IFS=, # set the comma character as the field separator
somecommand --flag "${files[*]}" # run your program with the comma-separated list
try this -
lst=$( echo *.sorted | sed 's/ /,/g' ) # stack filenames with commas
echo $lst
if you really need the single-ticks around each filename, then
lst="'$( echo *.sorted | sed "s/ /','/g" )'" # commas AND quotes
#!/bin/bash
bams=*.sorted
for i in $bams
do flag+="${flag:+,}'$i'"
done
echo $flag
I am looping over a CSV file. Each line of the file is formatted something like this (it's Open Street Maps data):
planet_85.287_27.665_51a5fb91,AcDbEntity:AcDbPolyline,{ [name] Purano
Bus Park-Thimi [type] route [route] microbus [ref] 10 } { [Id] 13.0
[Srid] 3857 [FieldsTableId]
This follows the format:
Layer,SubClasses,ExtendedEntity,Linetype,EntityHandle,Text
I want to add a new column for Name. I can find the name in a line by cutting off everything before [name] and after [. This code successfully creates a new-line delineated file of all of the names (which I open as a CSV and then copy-paste into the original file as a new column).
cat /path/to/myfile.csv | while read line
do
if [[ ${line} == *"name"* ]]
then
printf "$(echo $line | LC_ALL=C sed 's/^.*name\]//g'| LC_ALL=C cut -f1 -d'[') \n"
else
printf "\n"
fi
done >/path/to/newrow.csv
This system is clearly suboptimal - I would far prefer to print the entire final row. But when I replace that printf line with this:
printf "$line,$(echo $line | LC_ALL=C sed 's/^.*name\]//g'| LC_ALL=C cut -f1 -d'[') \n"
It prints the line but not the name. I've tried printing them in separate print statements, printing the line and then echoing the name, saving the name in a variable and then printing, and a number of other techniques, and each time I either a) only print the line, or b) print the name on a new line, which breaks the CSV format.
What am I doing wrong? How can I print the full original line with the name appended as a new column at the end?
NOTE: I am running this in Terminal on macOS Sierra on a MacBook Pro 15" Retina.
If I understand correctly, you want to extract the name between [name] and [type], and append as the new last CSV column. You can do that using capture groups:
sed -e 's/.*\[name\] \(.*\) \[type\].*/&,\1/' < input
Notice the \(.*\) in the middle. That captures the text between [name] and [type].
In the replacement string, & stands for the matched string, which is the entire line, as the pattern starts and ends with .*.
Next the , is a literal comma, and \1 stands for the content of the first capture group, the part matched within \(...\).
How to I concatenate stdin to a string, like this?
echo "input" | COMMAND "string"
and get
inputstring
A bit hacky, but this might be the shortest way to do what you asked in the question (use a pipe to accept stdout from echo "input" as stdin to another process / command:
echo "input" | awk '{print $1"string"}'
Output:
inputstring
What task are you exactly trying to accomplish? More context can get you more direction on a better solution.
Update - responding to comment:
#NoamRoss
The more idiomatic way of doing what you want is then:
echo 'http://dx.doi.org/'"$(pbpaste)"
The $(...) syntax is called command substitution. In short, it executes the commands enclosed in a new subshell, and substitutes the its stdout output to where the $(...) was invoked in the parent shell. So you would get, in effect:
echo 'http://dx.doi.org/'"rsif.2012.0125"
use cat - to read from stdin, and put it in $() to throw away the trailing newline
echo input | COMMAND "$(cat -)string"
However why don't you drop the pipe and grab the output of the left side in a command substitution:
COMMAND "$(echo input)string"
I'm often using pipes, so this tends to be an easy way to prefix and suffix stdin:
echo -n "my standard in" | cat <(echo -n "prefix... ") - <(echo " ...suffix")
prefix... my standard in ...suffix
There are some ways of accomplish this, i personally think the best is:
echo input | while read line; do echo $line string; done
Another can be by substituting "$" (end of line character) with "string" in a sed command:
echo input | sed "s/$/ string/g"
Why i prefer the former? Because it concatenates a string to stdin instantly, for example with the following command:
(echo input_one ;sleep 5; echo input_two ) | while read line; do echo $line string; done
you get immediatly the first output:
input_one string
and then after 5 seconds you get the other echo:
input_two string
On the other hand using "sed" first it performs all the content of the parenthesis and then it gives it to "sed", so the command
(echo input_one ;sleep 5; echo input_two ) | sed "s/$/ string/g"
will output both the lines
input_one string
input_two string
after 5 seconds.
This can be very useful in cases you are performing calls to functions which takes a long time to complete and want to be continuously updated about the output of the function.
You can do it with sed:
seq 5 | sed '$a\6'
seq 5 | sed '$ s/.*/\0 6/'
In your example:
echo input | sed 's/.*/\0string/'
I know this is a few years late, but you can accomplish this with the xargs -J option:
echo "input" | xargs -J "%" echo "%" "string"
And since it is xargs, you can do this on multiple lines of a file at once. If the file 'names' has three lines, like:
Adam
Bob
Charlie
You could do:
cat names | xargs -n 1 -J "%" echo "I like" "%" "because he is nice"
Also works:
seq -w 0 100 | xargs -I {} echo "string "{}
Will generate strings like:
string 000
string 001
string 002
string 003
string 004
...
The command you posted would take the string "input" use it as COMMAND's stdin stream, which would not produce the results you are looking for unless COMMAND first printed out the contents of its stdin and then printed out its command line arguments.
It seems like what you want to do is more close to command substitution.
http://www.gnu.org/software/bash/manual/html_node/Command-Substitution.html#Command-Substitution
With command substitution you can have a commandline like this:
echo input `COMMAND "string"`
This will first evaluate COMMAND with "string" as input, and then expand the results of that commands execution onto a line, replacing what's between the ‘`’ characters.
cat will be my choice: ls | cat - <(echo new line)
With perl
echo "input" | perl -ne 'print "prefix $_"'
Output:
prefix input
A solution using sd (basically a modern sed; much easier to use IMO):
# replace '$' (end of string marker) with 'Ipsum'
# the `e` flag disables multi-line matching (treats all lines as one)
$ echo "Lorem" | sd --flags e '$' 'Ipsum'
Lorem
Ipsum#no new line here
You might observe that Ipsum appears on a new line, and the output is missing a \n. The reason is echo's output ends in a \n, and you didn't tell sd to add a new \n. sd is technically correct because it's doing exactly what you are asking it to do and nothing else.
However this may not be what you want, so instead you can do this:
# replace '\n$' (new line, immediately followed by end of string) by 'Ipsum\n'
# don't forget to re-add the `\n` that you removed (if you want it)
$ echo "Lorem" | sd --flags e '\n$' 'Ipsum\n'
LoremIpsum
If you have a multi-line string, but you want to append to the end of each individual line:
$ ls
foo bar baz
$ ls | sd '\n' '/file\n'
bar/file
baz/file
foo/file
I want to prepend my sql script with "set" statement before running it.
So I echo the "set" instruction, then pipe it to cat. Command cat takes two parameters : STDIN marked as "-" and my sql file, cat joins both of them to one output. Next I pass the result to mysql command to run it as a script.
echo "set #ZERO_PRODUCTS_DISPLAY='$ZERO_PRODUCTS_DISPLAY';" | cat - sql/test_parameter.sql | mysql
p.s. mysql login and password stored in .my.cnf file