I'm new to sed command. I was reading the build script for some source code and I found this for loop in it.
for x in '*.la'
do
sed -i -e 's|^\(libdir=.\).*\(/opt/toolchains\)|\1\2|' x
done
I'm not able to understand the function this for loop is doing. Can anyone help.
It's iterating over a series of files in the current directory ending with ".la" and for each file found, it's editing the contents using sed to convert lines of the form:
libdir=X[zero-or-more-chars]/opt/toolchains
(where X is any character) into lines of the form:
libdir=X/opt/toolchains.
In other words, it's removing the [zero-or-more-chars] part of those lines.
Actually, this looks buggy because I would expect the sed command to reference $x, not x. I have a feeling you lost the $ somehow in the copy/paste step (or perhaps it's simply a bug).
Related
I am trying to use sed to add some text, at every line end, for all .txt files in a directory. This is the exact command I use: find . -name "*.txt" -exec sed -i 's/$/:orig/' {} +
Expected:
https://pbs.twimg.com/media/EUr539_UMAAFqZM.jpg:orig
https://pbs.twimg.com/media/ENTrymcUwAAnd6_.jpg:orig
https://pbs.twimg.com/media/EIzzcrFUYAAgfUo.jpg:orig
That is also what I actually get when I run it in my laptop with Linux Mint 19.2. But when I try it in my Windows PC, running sed through Ubuntu in WSL, what I get is this:
https://pbs.twimg.com/media/EUr539_UMAAFqZM.jpg
:orig
https://pbs.twimg.com/media/ENTrymcUwAAnd6_.jpg
:orig
https://pbs.twimg.com/media/EIzzcrFUYAAgfUo.jpg:orig
If I cat the files in question while still in the Ubuntu terminal, what's displayed is more like this (there's some weird whitespace that makes it look like columns in SO, but generally they all look pretty chaotic):
:orig://pbs.twimg.com/media/EUr539_UMAAFqZM.jpg :orig://pbs.twimg.com/media/ENTrymcUwAAnd6_.jpg https://pbs.twimg.com/media/EIzzcrFUYAAgfUo.jpg:orig
I understand Windows and Linux text is formatted differently and that line ends in particular are problematic, though I am uncertain if that is of any importance here.
Can anyone shed light on this behavior? How can I get the command to behave consistently?
The problem is that your files end in CRLF but the WSL sed uses just LF and the line end. You can get around this with a three step process, if you know it's a CRLF-style file:
get rid of the CR;
do your change;
put the CR back.
That would go something like: sed -i -e 's/\r$//' -e 's/$/:orig/' -e 's/$/\r/'.
However, that this won't work on UNIX-style files since the first substitution will do nothing but the third will put a CR character at the end of each line, even though it wasn't there originally. If you want something that will work on both types of files, this should do it:
sed -E 's/(\r)?$/:orig\1/'
This captures the optional CR at the end of the line and puts it back in the substitution (if it's not in the original line, it won't put it back).
I'm trying to extract user's crontabs so as to view them together. The initial problem I ran into was that the crontab file (From crontab -l) contains a lot of commented lines placed there by the system to explain the file's function. I stole a sed snippet to deal with this that deletes lines starting with comments and replaces the rest of lines following comments with blanks. (As best as I understand it.)
Here's an example crontab I'd like to capture:
0 0 5 * * /home/thornegarvin/myscript.sh
The sed code I'm using is: (With croneditor.temp containing the crontab)
sed '/^[[:blank:]]*#/d;s/#.*//' croneditor.temp
I think that the command is matching the *s in the file as comments and then deleting the line, but I'm not sure if that's why the command is failing.
I need a version of this command or another one entirely that works as I intended (Grabbing crontabs from the output of crontab -l).
I have an output file (namely a log from screen) containing several control characters. Inside the screen, I have programs running that use control characters to refresh certain lines (examples would be top or anything printing progress bars).
I would like to output a tail of this file using PHP. If I simply read in that file and echo its contents (either using PHP functions or through calling tail, the output is messy and much more than these last lines as it also includes things that have been overwritten. If I instead run tail in the command line, it returns just what I want because the terminal evaluates the control characters.
So my question is: Is there a way to evaluate the control characters, getting the output that a terminal would show me, in a way that I could then use elsewhere (e.g., write to a file)?
#5gon12eder's answer got rid of some control characters (thanks for that!) but it did not handle the carriage return part that was even more important to me.
I figured out that I could just delete anything from the beginning of a line to the last carriage return inside that line and simply keep everything after that, so here is my sed command accomplishing that:
sed 's/^.*\r\([^\r]\+\)\r\?$/\1\r/g'
The output can then be further cleaned using #5gon12eder's answer:
cat screenlog.0 | sed 's/^.*\r\([^\r]\+\)\r\?$/\1\r/g' | sed 's,\x1B\[[0-9?;]*[a-zA-Z],,g'
Combined, this looks exactly like I wanted.
I'm not sure what you mean by “evaluating” the control characters but you could remove them easily.
Here is an example using sed but if you are already using PHP, its internal regex processing functionality seems more appropriate. The command
$ sed 's,\x1B\[[0-9?;]*[a-zA-Z],,g' file.dat
will dump the contents of file.dat to standard output with all ANSI escape sequences removed. (And I'm pretty sure that nothing else is removed except if your file contains invalid escape sequences in which case the operation is ill-defined anyway.)
Here is a little demo:
$ echo -e "This is\033[31m a \033[umessy \033[46mstring.\033[0m" > file.dat
$ cat file.dat
# The output of the above command is not shown to protect small children
# that might be browsing this site.
$ reset # your terminal
$ sed 's,\x1B\[[0-9?;]*[a-zA-Z],,g' file.dat
This is a messy string.
The less program has some more advanced logic built in to selectively replace some escape sequences. Read the man page for the relevant options.
Due to processes out of my control I need run multiple SH files which contains lengthy CURL commands. Problem is that whichever process created these commands seems to have included one line of whitespace at the very end. If I call it as is - it fails. If I physically open the file and hit backspace on the first full empty line and save the file - it works perfectly.
Any way to put some kind of command into the SH file so that it removes any unnecessary stuff?
More info would be helpful, but the following might work:
If you need to put something into each of the files that contain the curl commands as you mention, you could try putting exit as the last line of the curl script (also depends on how you're calling the 'curl files'
exit
If you can run a separate script against the files that have a blank line, perhaps sed the blank lines away?
sed -i s/^\s$// $fileWithLineOfSpaces
edit:
Or (after thinking about it), perhaps simply delete the last line of the file....
sed -i '$d' $file
To process a bunch of data and get it ready to be inserted into our database, we generate a bunch of shell scripts. Each of them has about 15 lines, one for each table that the data is going. One a recent import batch, some of the import files failed going into one particular table. So, I have a bunch of shell scripts (about 600) where I need to comment out the first 7 lines, then rerun the file. There are about 6000 shell scripts in this folder, and nothing about a particular file can tell me if it needs the edit. I've got a list of which files that I pulled from the database output.
So how do I write a bash script (or anything else that would work better) to take this list of file names and for each of them, comment out the first 7 lines, and run the script?
EDIT:
#!/usr/bin/env sh
cmd1
cmd2
cmd3
cmd4
cmd5
cmd6
cmd7
cmd8
Not sure how readable that is. Basically, the first 7 lines (not counting the first line) need to have a # added to the beginning of them. Note: the files have been edited to make each line shorter and partially cut off copying out of VIM. But in the main part of each file, there is a line starting with echo, then a line starting with sqlldr
Using sed, you can specify a line number range in the file to be changed.
#!/bin/bash
while read line
do
# add a comment to beginning of lines 1 - 7 and rename the script
sed '3,9 s/^/#/' $line > $line.new
exec $line.new
done < "filelist.txt"
You may wish to test this before running it on all of those scripts...
EDIT: changed the lines numbers to reflect comments.
Roughly speaking:
#!/bin/sh
for file in "$#"
do
out=/tmp/$file.$$
sed '2,8s/^/#/' < $file > $out
$SHELL $out
rm -f $out
done
Assuming you don't care about checking for race conditions etc.
ex seems made for what you want to do.
For instance, for editing one file, with a here document:
#!/bin/sh
ex test.txt << END
1,12s/^/#/
wq
END
That'll comment out the first 12 lines in "test.txt". For your example you could try "$FILE" or similar (including quotes!).
Then run them the usual way, i.e. ./"$FILE"
edit: $SHELL "$FILE" is probably a better approach to run them (from one of the above commenters).
Ultimately you're going to want to use the linux command sed. Whatever logic you need to place in the script, you know. But your script will ultimately call sed. http://lowfatlinux.com/linux-sed.html