I have some code using GNU-parallel which should replace text in an input file with a series of strings of the form vclist_2d_*.txt where * is a number between 1 and 10000.
FILES=(vclist_2d_*.txt)
parallel -j1 '
sed -i "s/50pc\/vclist_2d_.*/50pc\/{}'\''/" 1759_input.py
sed -i "s/schedule_analysis\/vclist_2d_.*/schedule_analysis\/{}'\\_temp\/1759_cs_output.spc''\''/" 1759_input.py
' ::: ${FILES[#]}
The first sed command successful replaces whichever vclist_2d_* file is already in 1759_input with the next one in the list FILES as defined by {}. However, the second sed command needs to replace the vclist_2d_* and append to this the text _temp/1759_cs_output.spc'
However, with the code above two things happen:
the vclist name never gets replaced with the next one in the list
the text .temp/1759_cs_output.spc gets appended rather than _temp/1759_cs_output.spc
I've tried several variations of the above none of which were successful. I'm not sure why this works successfully for the first sed but not the second. I thought maybe _ needed escaping but that didn't help.
I don't quite understand what you're doing with the single quotes: I am going to assume that your regex pattern is too greedy and you need to add a quote that got consumed. I'll change .* to [^']0 -- i.e. zero or more non-quote characters.
You're doing twice as much work as required: put both substitutions into a single sed call
parallel -j1 '
sed -i "
s#\(50pc\)/vclist_2d_[^'\'']*#\1/{}#
s#\(schedule_analysis\)/vclist_2d_[^'\'']*#\1/{}_temp/1759_cs_output.spc#
" 1759_input.py
' ::: "${FILES[#]}"
I used a different delimiter for the s/// command in order to reduce backslashes
Related
The name in the title says it all. However, I'm absolutely the worst with the sed command. So I'm trying to edit the following file:
/var/www/html/phpMyAdmin/config.inc.php
I want to edit the line that says
$cfg['Servers'][$i]['AllowRoot'] = false;
into the following
$cfg['Servers'][$i]['AllowRoot'] = true;
It has so many special characters and whatnot and I have no prior knowledge of how sed works. So here's some commands I've tried to specifically edit that one line.
sed -i "/*.AllowRoot.*/\$cfg['Servers'][\$i]['AllowRoot'] = true;/" /var/www/html/phpMyAdmin/config.inc.php
sed -i "/*.AllowRoot.*/$cfg['Servers'][$i]['AllowRoot'] = true;/" /var/www/html/phpMyAdmin/config.inc.php
# this one finds the line successfully and prints it so I know it's got the right string:
sed -n '/AllowRoot/p' /var/www/html/phpMyAdmin/config.inc.php
sed -i "s/'AllowRoot|false'/'AllowRoot|true'/" /var/www/html/phpMyAdmin/config.inc.php
I have absolutely no idea what I'm doing and I'm not learning a whole lot besides the feeling that the last command splits up 'AllowRoot|false' makes sure that both must be present in the sentence to come back as a result. So to my logic, I thought changing the word false into true would make that happen, but nothing. The other commands return... bizarre results at best, one even emptying the file. Or that's one of the commands I had not written down here, I've lost track after 50 attempts. What is the solution here?
The [ and ] need to be escaped to match literal brackets, instead of inadvertently starting a bracket expression. This should work:
$ sed -i "/\$cfg\['Servers'\]\[\$i\]\['AllowRoot'\]/s/false/true/" /var/www/html/phpMyAdmin/config.inc.php
There is not many things to escape in sed. Main problem in your line is / which you have chosen as delimiter (most common, but not required). I suggest you use # and the following will work:
sed -i "s#$cfg['Servers'][$i]['AllowRoot'] = false;<br />#$cfg['Servers'][$i]['AllowRoot'] = true;<br />#g" input.txt
however you need to think about bash interpreter as well. $i and $cfg will be interpreted as variables. My suggestion is that when you want to match a string like this to put the sed expression in a text file like this:
cat allow_root_true.sed
s#['Servers'][]['AllowRoot'] = false;<br />#['Servers'][]['AllowRoot'] = true;<br />#g
and run the command using sed -f like this:
sed -i -f allow_root_true.sed input.txt
Warning -i will change the input file
sed can't do literal string matching which is why you need to escape so many characters (see Is it possible to escape regex metacharacters reliably with sed), but awk can:
$ awk -v str="\$cfg['Servers'][\$i]['AllowRoot']" 'index($0,str){sub(/false/,"true")} 1' file
//some text here
$cfg['Servers'][$i]['AllowRoot'] = true;<br />
//some more text here
Run code snippetHide resultsExpand snippet
In the above we only have to escape the $s to protect them from the shell since the string is enclosed in "s to allow it to include 's.
Input:-
echo "1234ABC89,234" # A
echo "0520001DEF78,66" # B
echo "46545455KRJ21,00"
From the above strings, I need to split the characters to get the alphabetic field and the number after that.
From "1234ABC89,234", the output should be:
ABC
89,234
From "0520001DEF78,66", the output should be:
DEF
78,66
I have many strings that I need to split like this.
Here is my script so far:
echo "1234ABC89,234" | cut -d',' -f1
but it gives me 1234ABC89 which isn't what I want.
Assuming that you want to discard leading digits only, and that the letters will be all upper case, the following should work:
echo "1234ABC89,234" | sed 's/^[0-9]*\([A-Z]*\)\([0-9].*\)/\1\n\2/'
This works fine with GNU sed (I have 4.2.2), but other sed implementations might not like the \n, in which case you'll need to substitute something else.
Depending on the version of sed you can try:
echo "0520001DEF78,66" | sed -E -e 's/[0-9]*([A-Z]*)([,0-9]*)/\1\n\2/'
or:
echo "0520001DEF78,66" | sed -E -e 's/[0-9]*([A-Z]*)([,0-9]*)/\1$\2/' | tr '$' '\n'
DEF
78,66
Explanation: the regular expression replaces the input with the expected output, except instead of the new-line it puts a "$" sign, that we replace to a new-line with the tr command
Where do the strings come from? Are they read from a file (or other source external to the script), or are they stored in the script? If they're in the script, you should simply reformat the data so it is easier to manage. Therefore, it is sensible to assume they come from an external data source such as a file or being piped to the script.
You could simply feed the data through sed:
sed 's/^[0-9]*\([A-Z]*\)/\1 /' |
while read alpha number
do
…process the two fields…
done
The only trick to watch there is that if you set variables in the loop, they won't necessarily be visible to the script after the done. There are ways around that problem — some of which depend on which shell you use. This much is the same in any derivative of the Bourne shell.
You said you have many strings like this, so I recommend if possible save them to a file such as input.txt:
1234ABC89,234
0520001DEF78,66
46545455KRJ21,00
On your command line, try this sed command reading input.txt as file argument:
$ sed -E 's/([0-9]+)([[:alpha:]]{3})(.+)/\2\t\3/g' input.txt
ABC 89,234
DEF 78,66
KRJ 21,00
How it works
uses -E for extended regular expressions to save on typing, otherwise for example for grouping we would have to escape \(
uses grouping ( and ), searches three groups:
firstly digits, + specifies one-or-more of digits. Oddly using [0-9] results in an extra blank space above results, so use POSIX class [[:digit:]]
the next is to search for POSIX alphabetical characters, regardless if lowercase or uppercase, and {3} specifies to search for 3 of them
the last group searches for . meaning any character, + for one or more times
\2\t\3 then returns group 2 and group 3, with a tab separator
Thus you are able to extract two separate fields per line, just separated by tab, for easier manipulation later.
I've seen many search and replace threads based on the assumption that 1. you either know what string or substring you are explicitly looking for or 2. you know the exact position it is at within the string or 3. both combined.
In my situation I have one csv file containing one column and 1M rows. e.g.
1,google.com
2,yahoo.com
3,twitter.com
4,xyz.com
For every column, I want to replace every character (the incrementing integers) up to and including the comma with the http semicolon dble forward slash dubdubdub
So far I have the following
HTTPSTRING="http://www."
cat X.csv << Will this ensure that the while block is executed on this file?
while IFS=, read line
do {$line/(.*?),/HTTPSTRING} << This is where I am having trouble
done
exit 0
and I would likea text file containing one URL per line e.g.
http://www.google.com
...
http://www.${999,999_more_urls}
Thank you so much in advance
Lewis
This does a greedy match, which would be problematic if you ever have any commas other than the one that separates the initial integer from the characters you want to retain. But it works on your sample X.csv file, producing a Y.csv file that meets your output specification.
HTTPSTRING="http://www."
while read line
do
echo ${line/*,/$HTTPSTRING}
done < X.csv > Y.csv
exit 0
For what it's worth, if you put this in a script, you can take the file input/input redirection parts out of the code itself, and instead apply them when calling the script.
If you're not strictly limited to bash itself, you might want to consider using sed. Either of these should do what you want, differing only in whether you prefer to escape the slashes in your string or use a non-standard delimiter:
sed 's/[0-9]*,/http:\/\/www./' X.csv > Y.csv
sed 's~[0-9]*,~http://www.~' X.csv > Y.csv
Your script is close. You can pipe the output of cat directly to the while loop, but it's better to use input redirection ( < X.csv). Using IFS=, before read will split the line into fields separated by a comma, but you are just missing a variable to hold the second field.
HTTPSTRING="http://www."
while IFS=, read number domain
do
echo "$HTTPSTRING$domain"
done < X.csv
You could use commands only, there is no need for an explicit Bash loop :
cut -d',' -f2 < X.csv | sed 's_^_http://www._' > Y.txt
Notice that the usual / used after the s in sed is replaced by _ because it is included in the string to replace. ^ matches the start of the line.
I'm running sed as a part of a shell script to clean up bind logs for insertion into a database.
One of the sed commands is the following:
sed -i 's/-/:/g' $DPath/named.query.log
This turns out to be problematic as it disrupts any resource requests that also include a dash (I'm using : as a delimiter for an awk statement further down).
My question is how do I limit the sed command above to only the first ten characters of the line? I haven't seen a specific switch that does this, and I'm nowhere near good enough with RegEx to even start on developing one that works. I can't just use regex to match the preceding numbers because it's possible that the pattern could be part of a resource request. Heck, I can't even use pattern matching for ####-##-## because, again, it could be part of the resource.
Any ideas are much appreciated.
It's [almost always] simpler with awk:
awk '{target=substr($0,1,10); gsub(/-/,":",target); print target substr($0,11)}' file
I think the shortest solution, and perhaps the simplest, is provided by sed itself, rather than awk[ward]:
sed "h;s/-/:/g;G;s/\(..........\).*\n........../\1/"
Explanation:
(h) copy everything to the hold space
(s) do the substitution (to the entire pattern space)
(G) append the hold space, with a \n separator
(s) delete the characters up to the tenth after the \n, but keep the first ten.
Some test code:
echo "--------------------------------" > foo
sed -i "h;s/-/:/g;G;s/\(..........\).*\n........../\1/" foo
cat foo
::::::::::----------------------
I'm not sure how make sed do it per se, however, I do know that you can feed sed the first 10 characters then paste the rest back in, like so:
paste -d"\0" <(cut -c1-10 $DPath/named.query.log | sed 's/\-/:/g') <(cut -c11- $DPath/named.query.log)
You can do the following:
cut -c 1-10 $DPath/named.query.log | sed -i 's/-/:/g'
The cut statemnt takes only the first 10 chars of each line in that file. The output of that should be piped in a file. As of now it will just output to your terminal
I have a series of text files that I want to convert to markdown. I want to remove any leading spaces and add a hash sign to the first line in every file. If I run this:
sed -i.bak '1s/ *\(.*\)/\#\1/g' *.md
It alters the first line of the first file and processes them all, leaving the rest of the files unchanged.
What am I missing that will search and replace something on the n-th line of multiple files?
Using bash on OSX 10.7
The problem is that sed by default treats any number of files as a single stream, and thus line-number offsets are relative to the start of the first file.
For GNU sed, you can use the -s (--separate) flag to modify this behavior:
sed -s -i.bak '1s/^ */#/' *.md
...or, with non-GNU sed (including the one on Mac OS X), you can loop over the files and invoke once per each:
for f in *.md; do sed -i.bak '1s/^ */#/' "$f"; done
Note that the regex is a bit simplified here -- no need to match parts of the line that you aren't going to change.
XARgs will do the trick for you:
http://en.wikipedia.org/wiki/Xargs
Remove the *.md from the end of your sed command, then use XArgs to gather your files one at a time and send them to your sed command as a single entity, sorry I don't have time to work it out for you but the wikiPedia article should show you what you need to know.
sed -rsi.bak '1s/^/#/;s/^[ \t]+//' *.md
You don't need g(lobally) at the end of the command(s), because you wan't to replace something at the begin of line, and not multiple times.
You use two commands, one to modify line 1 (1s...), seperated from the second command for the leading blanks (and tabs? :=\t) with a semicolon. To remove blanks in the first line, switch the order:
sed -rsi.bak 's/^[ \t]+//;1s/^/#/' *.md
Remove the \t if you don't need it. Then you don't need a group either:
sed -rsi.bak 's/^ +//;1s/^/#/' *.md
-r is a flag to signal special treatment of regular expressions. You don't need to mask the plus in that case.