Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 2 years ago.
Improve this question
i have a file that have variables and values
objective: open the file and replace all id by input id
[FILE]
var1 = 2
id = 3
var3 = 5
id = 12
var4 = 5
and i can't replace the id values to new ones.
here's my code, any help or something will help. thanks
#!/bin/bash
filename=$1
uuid=$2
input="./$filename"
# awk -v find="id " -v field="5" -v newval="abcd" 'BEGIN {FS=OFS="="} {if ($1 == find) $field=newval; print $1}' $input
while IFS= read -r line
do
awk -v find="id " -v field="5" -v newval="abcd" 'BEGIN {FS=OFS="="} {if ($1 == find) $field=newval;}' $input
echo $line
done < "$input"
expected output
execute
./myscript.sh file.cnf 77
expected output:
[FILE]
var1 = 2
id = 77
var3 = 5
id = 77
var4 = 5
I think sed is the right tool for this. You can even use its -i switch and update the file in-place.
$ cat file.txt
var1 = 2
id = 3
var3 = 5
id = 12
var4 = 5
$ NEW_ID=1234
$ sed -E "s/(id\s*=\s*)(.+)/\1${NEW_ID}/g" file.txt
var1 = 2
id = 1234
var3 = 5
id = 1234
var4 = 5
The string inside the quotes is a sed script for substituting some text with different text, and its general form is s/regexp/replacement/flags where "regexp" stands for "regular expression".
In the above example, the script looks for the string "id = ..." with any number of spaces or tabs around the "=" character. I divided the regexp into 2 groups (using parentheses) because we only want to replace the part to the right of the "=" character, and I don't think sed allows partial substitutions, so as a workaround I used \1 in the "replacement", which inserts the contents of the 1st group. The ${NEW_ID} actually gets evaluated by the shell so the value of the variable ("1234") is already part of the string by the time sed processes it. The g at the end stands for "global" and is probably redundant in this case. It makes sure that all occurrences of the regex on every line will get replaced; otherwise sed would only replace the first occurrence on each line.
Not sure. Bash scripts are extremely sensitive. I'm guessing your touch is what is causing this issue for a couple of reasons.
First whenever you touch a file name is should not consist of an operand or prefix unliss it is part of the shell script and $filename is shell or inline block quote. Touch is usually used for binaries or high priority data objects.
Second I'd try changing input and adjusted to $done and instead of echoing the $line echo the entire script using esac or end if instead of a do while loop.
Related
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 1 year ago.
Improve this question
Assume this: It needs to pass a file name as an argument.
This is the only text I’m showing. The remaining text has more data (not shown). The problem: The text is semi-clean, full of whitespace, tabs, Unicode, isn't clean and has to be like this (my needs), so copy/paste this exact text doesn't work (formatted by markup):
I have some text like this:
*** *
more text with spaces and tabs
*****
1
Something here and else, 2000 edf, 60 pop
Usd324.32 2 Usd534.22
2
21st New tetx that will like to select with pattern, 334 pop
Usd162.14
*** *
more text with spaces and tabs, unicode
*****
I'm trying to grab this explicit text:
1 Something here and else, 2000 edf, 60 pop Usd324.32
because of the newline and whitespace, the next command only grabs 1:
grep -E '1\s.+'
Also, I have been trying to make it with new concatenations:
grep -E '1\s|[A-Z].+'
But it doesn't work. grep begins to select a similar pattern in different parts of the text:
awk '{$1=$1}1' #done already
tr -s "\t\r\n\v" #done already
tr -d "\t\b\r" #done already
How can I grab:
grab one newline
grab the whole second line after one newline
grab the number $Usd324.34 and remove Usd
You can use this sed:
sed -En '/^1/ {N;N;s/[[:blank:]]*Usd([^[:blank:]]+)[^\n]*$/\1/; s/\n/ /gp;}' file
1 Something here and else, 2000 edf, 60 pop 324.32
Or this awk would also work:
awk '$0 == 1 {
printf "%s", $0
getline
printf " %s ", $0
getline
sub(/Usd/, "")
print $1
}' file
1 Something here and else, 2000 edf, 60 pop 324.32
Pure Bash:
#! /bin/bash
exec <<EOF
*** *
more text with spaces and tabs
*****
1
Something here and else, 2000 edf, 60 pop
Usd324.32 2 Usd534.22
2
21st New tetx that will like to select with pattern, 334 pop
Usd162.14
*** *
more text with spaces and tabs, unicode
*****
EOF
while read -r line1; do
if [[ $line1 =~ ^1$ ]]; then
read -r line2
read -r line3col1 dontcare
printf '%s %s %s\n' "$line1" "$line2" "${line3col1#Usd}"
fi
done
I have a program that prints data into the console like so (separated by space):
variable1 value1
variable2 value2
variable3 value3
varialbe4 value4
EDIT: Actually the output can look like this:
data[variable1]: value1
pre[variable2] value2
variable3: value3
flag[variable4] value4
In the end I want to search for a part of the name e.g. for variable2 or variable3 but only get value2 or value3 as output.
EDIT: This single value should then be stored in a variable for further processing within the bash script.
I first tried to put all the console output into a file and process it from there with e.g.
# value3_var="$(grep "variable3" file.log | cut -d " " -f2)"
This works fine but is too slow. I need to process ~20 of these variables per run and this takes ~1-2 seconds on my system. Also I need to do this for ~500 runs. EDIT: I actually do not need to automatically process all of the ~20 'searches' automatically with one call of e.g. awk. If there is a way to do it automaticaly, it's fine, but ~20 calls in the bash script are fine here too.
Therefore I thought about putting the console output directly into a variable to remove the slow file access. But this will then eliminate the newline characters which then again makes it more complicated to process:
# console_output=$(./programm_call)
# echo $console_output
variable1 value1 variable2 value2 variable3 value3 varialbe4 value4
EDIT: IT actually looks like this:
# console_output=$(./programm_call)
# echo $console_output
data[variable1]: value1 pre[variable2] value2 variable3: value3 flag[variable4] value4
I found a solution for this kind of string arangement, but these seem only to work with a text file. At least I was not able to use the string stored in $console_output with these examples
How to print the next word after a found pattern with grep,sed and awk?
So, how can I output the next word after a found pattern, when providing a (long) string as variable?
PS: grep on my system does not know the parameter -P...
I'd suggest to use awk:
$ cat ip.txt
data[variable1]: value1
pre[variable2] value2
variable3: value3
flag[variable4] value4
$ cat var_list
variable1
variable3
$ awk 'NR==FNR{a[$1]; next}
{for(k in a) if(index($1, k)) print $2}' var_list ip.txt
value1
value3
To use output of another command as input file, use ./programm_call | awk '...' var_list - where - will indicate stdin as input.
This single value should then be stored in a variable for further processing within the bash script.
If you are doing further text processing, you could do it within awk and thus avoid a possible slower bash loop. See Why is using a shell loop to process text considered bad practice? for details.
Speed up suggestions:
Use LC_ALL=C awk '..' if input is ASCII (Note that as pointed out in comments, this doesn't apply for all cases, so you'll have to test it for your use case)
Use mawk if available, that is usually faster. GNU awk may still be faster for some cases, so again, you'll have to test it for your use case
Use ripgrep, which is usually faster than other grep programs.
$ ./programm_call | rg -No -m1 'variable1\S*\s+(\S+)' -r '$1'
value1
$ ./programm_call | rg -No -m1 'variable3\S*\s+(\S+)' -r '$1'
value3
Here, -o option is used to get only the matched portion. -r is used to get only the required text by replacing the matched portion with the value from the capture group. -m1 option is used to stop searching input once the first match is found. -N is used to disable line number prefix.
Exit after the first grep match, like so:
value3_var="$(grep -m1 "variable3" file.log | cut -d " " -f2)"
Or use Perl, also exiting after the first match. This eliminates the need for a pipe to another process:
value3_var="$(perl -le 'print $1, last if /^variable3\s+(.*)/' file.log)"
If I'm understanding your requirements correctly, how about feeding
the output of programm_call directly to the awk script instead of
assinging a shell variable.
./programm_call | awk '
# the following block is invoked line by line of the input
{
a[$1] = $2
}
# the following block is executed after all lines are read
END {
# please modify the print statement depending on your required output format
print "variable1 = " a["variable1"]
print "variable3 = " a["variable3"]
}'
Output:
variable1 = value1
variable3 = value3
As you see, the script can process all (~20) variables at once.
[UPDATE]
Assumptions including the provided information:
The ./program_call prints approx. 50 pairs of "variable value"
variable and value are delimited by blank character(s)
variable may be enclosed with [ and ]
variable may be followed by :
We have interest with up to 20 variables out of the ~50 pairs
We use just one of the 20 variables at once
We don't want to invoke ./program_call whenever accessing just one variable
We want to access the variable values from within bash script
We may use an associative array to fetch the value via the variable name
Then it will be convenient to read the variable-value pairs directly within
bash script:
#!/bin/bash
declare -A hash # declare an associative array
while read -r key val; do # read key (variable name) and value
key=${key#*[} # remove leading "[" and the characters before it
key=${key%:} # remove trailing ":"
key=${key%]} # remove trailing "]"
hash["$key"]="$val" # store the key and value pair
done < <(./program_call) # feed the output of "./program_call" to the loop
# then you can access the values via the variable name here
foo="${hash["variable2"]}" # the variable "foo" is assigned to "value2"
# do something here
bar="${hash["variable3"]}" # the variable "bar" is assigned to "value3"
# do something here
Some people criticize that bash is too slow to process text lines,
but we process just about 50 lines in this case. I tested a simulation by
generating 50 lines, processing the output with the script above,
repeating the whole process 1,000 times. It completed within a few seconds. (Meaning one batch ends within a few milliseconds.)
This is how to do the job efficiently AND robustly (your approach and all other current answers will result in false matches from some input and some values of the variables you want to search for):
$ cat tst.sh
#!/usr/bin/env bash
vars='variable2 variable3'
awk -v vars="$vars" '
BEGIN {
split(vars,tmp)
for (i in tmp) {
tags[tmp[i]":"]
tags["["tmp[i]"]"]
tags["["tmp[i]"]:"]
}
}
$1 in tags || ( (s=index($1,"[")) && (substr($1,s) in tags) ) {
print $2
}
' "${#:--}"
$ ./tst.sh file
value2
value3
$ cat file | ./tst.sh
value2
value3
Note that the only loop is in the BEGIN section where it populates a hash table (tags[]) with the strings from the input that could match your variable list so that while processing the input it doesn't have to loop, it just does a hash lookup of the current $1 which will be very efficient as well as robust (e.g. will not fail on partial matches or even regexp metachars).
As shown, it'll work whether the input is coming from a file or a pipe. If that's not all you need then edit your question to clarify your requirements and improve your example to show a case where this does not do what you want.
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 4 years ago.
Improve this question
I have some trouble with this script.
#!/bin/bash
i=1
for i in $(cat rows.txt);
do
j = "$(sed -ne "$[i-1]p" LOB_final.html)"
echo $j
sed -ne "$[i-1],$[i+4]p" LOB_final.html >> ./cards/$j.txt
done;
I have an other file that contains the right row-numbers (not exactly, [row-1] is the relevant row). This row contains a string with spaces included and should be the name of the file.
Script works so far as expected, but initializing j don't work.
Does anyone have a tip?
Thank you.
EDIT: The goal was, to write any six rows (1 before and 4 after ans the given row) in a file. The file should be named with the 1 before row of the given row (a string with white spaces included).
Questions is cleared, thanks to all.
check this, have fixed quite a few issues with your script
#!/bin/bash
i=1
for i in $(cat rows.txt);
do
j="$(sed -ne "$((i-1))p" LOB_final.html)"
echo "$j"
sed -ne "$((i-1)),$((i+4))p" LOB_final.html >> ./cards/"$j".txt
done;
Removed space which was there in assigning j
$[var] replaced with $((var))
Expanded variables in double quotes
You seem to want to get the filename from line i-1 and then to store that line together with the next five lines in a file with that name.
Using awk:
awk 'NR == FNR { rows[$0]; next }
FNR + 1 in rows { name = sprintf("./cards/%s.txt", $0); left = 6 }
left > 0 { print >name; --left }' rows.txt LOB_final.html
This has been tested on some toy data, but since I don't know what your data looks like I can't say for certain that it will work without (minor) modifications.
It has the benefit that it will not parse the whole of LOB_final.html twice for each row number read from rows.txt, which your original code does. In fact, it only ever reads each file once.
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 8 years ago.
Improve this question
A B 5 C Hello: XYZ 'Main Search String', Searching In: '[Name] = "I just want this" AND [State] = Active ('NAME', 'ACT')
How to extract this string: I just want this
grep "Main Search String" filename | awk -F"tab" '{print $6}' | ??
Using awk
awk -F\" '/Main Search String/ {print $2}' file
I just want this
using grep
grep -Po "\[Name\] = \"\K[^\"]*" file
grep -Po "(?<=\[Name\] = \")[^\"]*" file
sed -n 's/.*\[Name\] = "\([^"]*\)".*/\1/p' YourFile
assuming the content is betwween " and for the tag [Name] on any line of your input file
If you want more fun :)
$ cat /tmp/delme
A B 5 C Hello: XYZ 'Main Search String', Searching In: '[Name] = "I just want this" AND [State] = Active ('NAME', 'ACT')
$ cat /tmp/delme | perl -e 'while (<>) { if ( /.*[Name] = \"(.*)\"/ ) { print "$1\n"; } };'
I just want this
With this you can use the whole regex and pattern match things provided by perl...and perl is on almost every system as I know.Give it a try! ;)
EDITED for pattern found case:
$ cat /tmp/delme | perl -e 'while (<>) { print "TEXT: ";print; if ( /.*Main Search String.*\[Name] = "(.*)\"/ ) { print "RESULT: $1\n"; } };'
TEXT: A B 5 C Hello: XYZ 'Main Search String', Searching In: '[Name] = "I just want this" AND [State] = Active ('NAME', 'ACT')
RESULT: I just want this
TEXT: A B 5 C Hello: XYZ 'Not Search String', Searching In: '[Name] = "I just want this" AND [State] = Active ('NAME', 'ACT')
Got it in 13 characters plus the file name:
cut -d\" -f 2 text.txt
This command cuts the text into columns separated by the double-quotes. (-d\") That creates three columns. It prints out the second column with your answer (-f 2).
Like Lutz Horn said, you need to provide a bigger data set to test any of these commands. Many of them will fail if they do not represent the complete pattern they need to parse.
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 8 years ago.
Improve this question
I file /tmp/txt
contents of the file: aaa aaa aaa _bbb bbb bbb
I need to save the file /tmp/txt_left: aaa aaa aaa
I need to save the file /tmp/txt_right: bbb bbb bbb
!!! attention seeking solutions without the use of variables !!!
awk -F '_' '{print $1> "/tmp/txt_left"; print $2 > "/tmp/txt_right" }' /tmp/txt
You could try cutting the line, slitting on the underscore
Cat /tmp/txt | cut -d_ -f 1 > txt_left
A sed way:
Shorter and quicker:
sed -ne $'h;s/_.*$//;w /tmp/txt_left\n;g;s/^.*_//;w /tmp/txt_right' /tmp/txt
Explained: It could be written:
sed -ne '
h; # hold (copy current line in hold space)
s/_.*$//; # replace from _ to end of line by nothing
w /tmp/txt_left
# Write current line to file
# (filename have to be terminated by a newline)
g; # get (copy hold space to current line buffer)
s/^.*_//; # replace from begin of line to _ by nothing
w /tmp/txt_right
# write
' /tmp/txt
Bash as bash
This is not a real variable, I use first argument element for doing the job and restore argument list once finish:
set -- "$(</tmp/txt)" "$#"
echo >>/tmp/txt_right ${1#*_}
echo >>/tmp/txt_left ${1%_*}
shift
I unshift the string at first place in argument line,
do operation on $1, than shift the argument line so no variable is used and in fine, the argument line return in his original state
... and this is a pure bash solution ;-)
Using bash process substitution, tee, and cut:
tee -a >(cut -d _ -f 0 > /tmp/txt_left) >(cut -d _ -f 1 >/tmp/txt_right) < /tmp/txt