I was trying to understand the usage of IFS but there is something I couldn't find any information about.
My example code:
#!/bin/sh
# (C) 2016 Ergin Bilgin
IFS=!
for LINE in $(last -a | sed '$ d')
do
echo $LINE | awk '{print $1}'
done
unset IFS
I use this code to print last users line by line. I totally understand the usage of IFS and in this example when I use default IFS, it reads word by word inside of my loop. And when I use IFS=! it reads line by line as I wish. The problem here is I couldn't find anything about that "!" on anywhere. I don't remember where I learned that. When I google about achieving same kind of behaviour, I see other values which are usually strings.
So, what is the meaning of that "!" and how it gives me the result I wish?
Thanks
IFS=! is merely setting a non-existent value for IFS so that you can iterate input line by line. Having said that using for loop here is not recommended, better to use read in a while loop like this to print first column i.e. username:
last | sed '$ d' | while read -r u _; do
echo "$u"
done
As you are aware, if the output of last had a !, the script would split the input lines on that character.
The output format of last is not standardized (not in POSIX for instance), but you are unlikely to find a system where the first column contains anything but the name of whatever initiated an action. For instance, I see this:
tom pts/8 Wed Apr 27 04:25 still logged in michener.jexium-island.net
tom pts/0 Wed Apr 27 04:15 still logged in michener.jexium-island.net
reboot system boot Wed Apr 27 04:02 - 04:35 (00:33) 3.2.0-4-amd64
tom pts/0 Tue Apr 26 16:23 - down (04:56) michener.jexium-island.net
continuing to
reboot system boot Fri Apr 1 15:54 - 19:03 (03:09) 3.2.0-4-amd64
tom pts/0 Fri Apr 1 04:34 - down (00:54) michener.jexium-island.net
wtmp begins Fri Apr 1 04:34:26 2016
with Linux, and different date-formats, origination, etc., on other machines.
By setting IFS=!, the script sets the field-separator to a value which is unlikely to occur in the output of last, so each line is read into LINE without splitting it. Normally, lines are split on spaces.
However, as you see, the output of last normally uses spaces for separating columns, and it is fed into awk which splits the line anyway — with spaces. The script could be simplified in various ways, e.g.,:
#!/bin/sh
for LINE in $(last -a | sed -e '$ d' -e 's/ .*//')
do
echo $LINE
done
which is (starting from the example in the question) adequate if the number of logins is not large enough to exceed your command-line. While checking for variations in last output, I noticed one machine with about 9800 lines from several years. (The other usual motivations given for not using for-loops are implausible in this instance). As a pipe:
#!/bin/sh
last -a | sed -e 's/ .*//' -e '/^$/d' | while IFS= read LINE
do
echo $LINE
done
I changed the sed expression (which OP likely copied from some place such as Bash - remove the last line from a file) because it does not work.
Finally, using the -a option of last is unnecessary, since all of the additional information it provides is discarded.
Related
I feel like there's probably good ways to do this in bash, but I'm struggling to find direct explanations for the best tools to do something like the following:
Given an input of string data from git log
filter commits down to only those between a set of tags
then format each commit, pulling convention based snippets of data
output the formatted snippets
So far, I've found that:
set $(git tag -l | sort -V | tail -2)
currentVersion=$2
previousVersion=$1
will give me variables for relevant tags. I can then do this:
$(git log v9.5.3..) | {what now?}
to pipe all commits from the previous tag to current. But I'm not sure on the next step?
Will the commits coming from the pipe be considered an array?
If not, how do I differentiate each commit distinctly?
Should I run a function against the piped input data?
Am I thinking about this completely wrong?
If this were Javascript, I'd run a loop over what would assuredly be an array input, regex the snippets I want from the commit, then output a formatted string with the snippets, probably in a map method or something. But I'm not sure if this is how I should be thinking in Bash with pipes?
Expecting data for each commit like:
commit xxxxxxxxxx
Author: xxxx xxxx <xxx#xxx.xxx>
Date: Thu Jul 29 xx:xx:xx 2021 +0000
Subject of the commit
A multiline description of the commit, the contents of which are not
really relevant for what I need, but still useful for consideration.
{an issue id}
And right now I'd be looking to grab:
the commit hash
the author
the date
the subject
the issue id
Would appreciate any insight as to the normal way to do this sort of thing with bash, with pipes, etc. I'd love to get my head right with Bash and do it here, rather than retreat back to my comfort zone of JS. Thanks!
Alright, I spent some time and found a solution that works for me. I'm (again) very much not a bash script'er, so I'm still curious if there's better ways to do this, but this works:
PREVIOUS_VERSION=${1:-$(git tag | tail -n 2 | head -n 1)}
CURRENT_VERSION=$(git tag | tail -n 1)
URL="https://your-hosting-domain-here/q/"
echo "-----RELEASE: $CURRENT_VERSION-----"
echo ""
parse_commits() {
while read LINE
do
if grep -q "Author:" <<< "$LINE"; then
echo "$LINE"
read DATE_LINE; echo "$DATE_LINE"
read SUBJECT_LINE; echo "Subject: $SUBJECT_LINE"
fi
if grep -q "Change-Id:" <<< "$LINE"; then
CHANGE_ID=$(echo "$LINE" | awk '{print $NF}')
echo "$URL$CHANGE_ID"
echo ""
fi
done
}
git log $PREVIOUS_VERSION.. | strings | parse_commits
I'll explain for anyone curious as I was as to how this could be done:
PREVIOUS_VERSION=${1:-$(git tag | tail -n 2 | head -n 1)}
This is simply a means within Bash to assign a variable to the incoming argument, with a fallback if it's not defined.
git log $PREVIOUS_VERSION.. | strings | parse_commits
This uses a git method to output all commits since the given version. We then pipe those commits to Bash "strings" command, which translates the input stream(?)/string(?) into a set of lines, and then pipe that to our custom function.
while read LINE
This starts a while loop, using the Bash command "read" which is super useful for what I needed to do. Essentially, it reads one line from input, and assigns it to the given arg as a variable. So this reads a line, and assigns it to variable: LINE.
if grep -q "Author:" <<< "$LINE"; then
This is a conditional that uses Bash command grep, which will search a file for the given string. We don't have a file, we have a string as a variable $LINE, but we can turn that into a temporary file using the Bash operator <<< which does exactly that. So this line runs the internal block if the given LINE as a file contains the substring "Author".
read DATE_LINE; echo "$DATE_LINE"
Once we've found our desired position after the Author: line (and echo'ed it), we simply read the next line, assign it to variable DATE_LINE and immediately echo that as well. We do the same for the subject line.
Up until now, we probably could have used simpler commands to achieve a similar result, but now we get to the tricky part (for me at least, not knowing much about Bash).
CHANGE_ID=$(echo "$LINE" | awk '{print $NF}')
After a similar conditional grep'ing for a substring Change-Id:, we snag the second word in the LINE by echo'ing it, and piping that to Bash awk command, which was the best way I could find for grabbing a substring. The awk command has a special keyword NF that equates to the count of words in the string. By using $NF we are referencing the last word, since for example the last word of 5 words would be $5. We set that to a variable, then echo it out with a given format (a url in my case).
The ultimate output looks like this:
-----RELEASE: v9.6.0-----
Author: xxxxxx xxxx <xxxx#xxxx.com>
Date: Fri Jul 30 xx:xx:xx 2021 +0000
Subject: The latest commit subject since last tag
https://your-hosting-domain-here/q/xxxxxxxxxxxxxxx
Author: xxxxxx xxxx <xxxx#xxxx.com>
Date: Thu Jul 29 xx:xx:xx 2021 +0000
Subject: The second latest commit subject
https://your-hosting-domain-here/q/xxxxxxxxxxxxxxx
... (and so on)
Hope that was helpful to someone, and if not, to future me :)
I have a file that contain data in the format
2 Mar 1 1234 141.98.80.59
1 Mar 1 1234 171.239.249.233
5 Mar 1 admin 116.110.119.156
4 Mar 1 admin1 177.154.8.15
2 Mar 1 admin 141.98.80.63
2 Mar 1 Admin 141.98.80.63
i tried this command to convert into csv format but it is giving me the output with extra (,) in the front
cat data.sql | tr -s '[:blank:]' ',' > data1.csv
,2,Mar,1,1234,141.98.80.59
,1,Mar,1,1234,171.239.249.233
,5,Mar,1,admin,116.110.119.156
,4,Mar,1,admin1,177.154.8.15
,2,Mar,1,admin,141.98.80.63
,2,Mar,1,Admin,141.98.80.63
In my file there is 6 character space is there in-front on every record
how can i remove extra comma from the front
how [to] remove extra comma from the front using awk:
$ awk -v OFS=, '{$1=$1}1' file
Output:
2,Mar,1,1234,141.98.80.59
1,Mar,1,1234,171.239.249.233
5,Mar,1,admin,116.110.119.156
...
Output with #EdMorton's version proposed in comments:
2,Mar,1,1234,141.98.80.59
1,Mar,1,1234,171.239.249.233
5,Mar,1,admin,116.110.119.156
...
The improved version of your current method is:
cat data.sql | sed -E -e 's/^[[:blank:]]+//g' -e 's/[[:blank:]]+/,/g' > data1.csv
But do be aware that replacing spaces/commas isnt a real way of changing this format into a CSV. If there are/were any commas and/or spaces present in the actual data this approach would fail.
The fact that your example source file has the .sql extension suggests that perhaps you get this file by exporting a database, and have already stripped parts of the file away with other tr statements ? If that is the case, a better approach would be to export to CSV (or another format) directly
edit: Made sed statement more portable as recommended by per Quasímodo in the comments.
Using Miller is
mlr --n2c -N remove-empty-columns ./input.txt >./output.txt
The output will be
2,Mar,1,1234,141.98.80.59
1,Mar,1,1234,171.239.249.233
5,Mar,1,admin,116.110.119.156
4,Mar,1,admin1,177.154.8.15
2,Mar,1,admin,141.98.80.63
2,Mar,1,Admin,141.98.80.63
I have following string in a variable:
-rw-r--r-- 0 1068 1001 4870 Dec 6 11:58 1.zip -rw-r--r-- 0 1068 1001 20246 Dec 6 11:59 10.zip
I'm trying to loop over this string with a for loop and get the following result:
Dec 6 11:58 1.zip
Dec 6 11:59 10.zip
Does anyone have the proper sed command to do this?
So let me make my question a little more clear. I do an sftp command with -b file and in there I do an ls -l *.zip. The result of this goes into a file. At first, I used a sed command to clear the first 2 lines since these are irrelevant information for me. I now only have the ls results, but they are on one line. In my example, there were just 2 zip files but there can be a lot more.
ListOfFiles=$(sed '1,2d' $LstFile) #delete first 2 lines
for line in $ListOfFiles
do
$line=$(echo "${line}" | sed (here i want the command to ony print zip file and date)
done
Notes on the revised scenario
The question has been modified to include a shell fragment:
ListOfFiles=$(sed '1,2d' $LstFile) #delete first 2 lines
for line in $ListOfFiles
do
$line=$(echo "${line}" | sed # I want to print only file name and date
done
Saving the results into a variable, as in the first line, is simply the wrong way to deal with it. You can use a simple adaptation of the code in my original answer (below) to achieve your requirement simply — very simply using awk, but it is possible using sed with a simple adaptation of the original code, if you're hung up on using sed.
awk variant
awk 'NR <= 2 { next } { print $6, $7, $8, $9 }' $LstFile
The NR <= 2 { next } portion skips the first two lines; the rest is unchanged, except that the data source is the list file you downloaded.
sed variant
sed -nE -e '1,2d' -e 's/^([^ ]+[ ]+){5}([^ ]+([ ]+[^ ]+){3})$/\2/p' $LstFile
In practice, the 1,2d command is unlikely to be necessary, but it is safer to use it, just in case one of the first two lines has 9 fields. (Yes, I could avoid using the -e option twice — no, I prefer to have separate commands in separate options; it makes it easier to read IMO.)
An answer for the original question
If you treat this as an exercise in string manipulation (disregarding legitimate caveats about trying to parse the output from ls reliably), then you don't need sed. In fact, sed is almost definitely the wrong tool for the job — awk would be a better choice — but the shell alone could be used. For example, assuming the data is in the string $variable, you could use:
set -- $variable
echo $6 $7 $8 $9
echo $15 $16 $17 $18
This gives you 18 positional parameters and prints the 8 you're interested in. Using awk, you might use:
echo $variable | awk '{ print $6, $7, $8, $9; print $15, $16, $17, $18 }'
Both these automatically split a string at spaces and allow you to reference the split elements with numbers. Using sed, you don't get that automatic splitting, which makes the job extremely cumbersome.
Suppose the variable actually holds two lines, so:
echo "$variable"
reports:
-rw-r--r-- 0 1068 1001 4870 Dec 6 11:58 1.zip
-rw-r--r-- 0 1068 1001 20246 Dec 6 11:59 10.zip
The code above assumed that the contents of $variable was a single line (though it would work unchanged if the variable contained two lines), but the code below assumes that it contains two lines. In fact, the code below would work if $variable contained many lines, whereas the set and awk versions are tied to '18 fields in the input'.
Assuming that the -E option to sed enables extended regular expressions, then you could use:
variable="-rw-r--r-- 0 1068 1001 4870 Dec 6 11:58 1.zip
-rw-r--r-- 0 1068 1001 20246 Dec 6 11:59 10.zip"
echo "$variable" |
sed -nE 's/^([^[:space:]]+[[:space:]]+){5}([^[:space:]]+([[:space:]]+[^[:space:]]+){3})$/\2/p'
That looks for a sequence of not white space characters followed by a sequence of white space characters, repeated 5 times, followed by a sequence of not white space characters and 3 sets of a sequence of white space followed by a sequence of not white space. The grouping parentheses — thus picking out fields 1-5 into \1 (which is ignored), and fields 6-9 into \2 (which is preserved), and then prints the result. If you decide you can assume no tabs etc, you can simplify the sed command to:
echo "$variable" | sed -nE 's/^([^ ]+[ ]+){5}([^ ]+([ ]+[^ ]+){3})$/\2/p'
Both of those produce the output:
Dec 6 11:58 1.zip
Dec 6 11:59 10.zip
Dealing with the single line variant of the input is excruciating — sufficiently so that I'm not going to show it.
Note that with the two-line value in $variable, the awk version could become:
echo "$variable" | awk '{ print $6, $7, $8, $9 }'
This will also handle an arbitrary number of lines.
Note how it is crucial to understand the difference between echo $variable and echo "$variable". The first treats all white space sequences as equivalent to a single blank but the other preserves the internal spacing. And capturing output such as with:
variable=$(ls -l 1*.zip)
preserves the spacing (especially the newline) in the assignment (see Capturing multiple line output into a Bash variable). Thus there's a moderate chance that the sed shown would work for you, but it isn't certain because you didn't answer clarifications sought before this answer was posted.
as others said you shouldn't really be parsing ls output.. Otherwise a dummy way to do it using awk to print out the columns you're interested in :
awk '{print $6, $7, $8, $9 "\n" $15, $16, $17, $18}' <<< $your_variable
I wanna change unix epoch to normal date
i'm trying:
sed < file.json -e 's/\([0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]/`date -r \1`/g'
any hint?
With the lack of information from your post, I can not give you a better answer than this but it is possible to execute commands using sed!
You have different ways to do it you can use
directly sed e instruction followed by the command to be
executed, if you do not pass a command to e then it will treat the content of the pattern buffer as external command.
use a simple substitute command with sed and pipe the output to sh
Example 1:
echo 12687278 | sed "s/\([0-9]\{8,\}\)/date -d #\1/;e"
Example 2:
echo 12687278 | sed "s/\([0-9]\{8,\}\)/date -d #\1/" |sh
Test 1 (with Japanese locale LC_TIME=ja_JP.UTF-8):
Test 2 (with Japanese locale LC_TIME=ja_JP.UTF-8):
Remarks:
I will let you adapt the date command accordingly to your system specifications
Since modern dates are longer than 8 characters, the sed command uses an
open ended length specifier of at least 8, rather than exactly 8.
Allan has a nice way to tackle dynamic arguments: write a script dynamically and pipe it to a shell! It works. It tends to be a bit more insecure because you could potentially pipe unintentional shell components to sh - for example if rm -f some-important-file was in the file along with the numbers , the sed pipeline wouldn't change that line, and it would also be passed to sh along with the date commands. Obviously, this is only a concern if you don't control the input. But mistakes can happen.
A similar method I much prefer is with xargs. It's a bit of a head trip for new users, but very powerful. The idea behind xargs is that it takes its input from its standard in, then adds it to the command comprised of its own non-option arguments and runs the command(s). For instance,
$ echo -e "/tmp\n/usr/lib" | xargs ls -d
/tmp /usr/lib
Its a trivial example of course, but you can see more exactly how this works by adding an echo:
echo -e "/tmp\n/usr/lib" | xargs echo ls -d
ls -d /tmp /usr/lib
The input to xargs becomes the additional arguments to the command specified in xargs's own arguments. Read that twice if necessary, or better yet, fiddle with this powerful tool, and the light bulb should come on.
Here's how I would approach what you're doing. Of course I'm not sure if this is actually a logical thing to do in your case, but given the detail you went into in your question, it's the best I can do.
$ cat dates.txt
Dates:
1517363346
I can run a command like this:
$ sed -ne '/^[0-9]\{8,\}$/ p' < dates.txt | xargs -I % -n 1 date -d #%
Tue Jan 30 19:49:06 CST 2018
Makes sense, because I used the commnad echo -e "Dates:\ndate +%s" > dates.txt to make the file a few minutes before I wrote this post! Let's go through it together and I'll break down what I'm doing here.
For one thing, I'm running sed with -n. This tells it not to print the lines by default. That makes this script work if not every line has an 8+ digit "date" in it. I also added anchors to the start (^) and end ($) of the regex so the line had only the approprate digits ( I realize this may not be perfect for you, but without understanding your its input, I can't do better ). These are important changes if your file is not entirely comprised of date strings. Additionally, I am matching at least 8 characters, as modern date strings are going to be more like 10 characters long. Finally, I added a command p to sed. This tells it to print the matching lines, which is necessary because I specifically said not to print the nonmatching lines.
The next bit is the xargs iteslf. The sed will write a date string out to xargs's standard input. I set only a few settings for xargs. By default it will add the standard input to the end of the command, separated by a space. I didn't want a space, so I used -I to specify a replacement string. % doesn't have a special meaning; its just a placeholder that gets replaced with the input. I used % because its not a special character but rarely is used in commands. Finally, I added -n 1 to make sure only 1 input was used per execution of date. ( xargs can also add many inputs together, as in my ls example above).
The end result? Sed matches lines that consist, exclusively, of 8 or more numeric values, outputting the matching lines. The pipe then sends this output to xargs, which takes each line separately (-n 1) and, replacing the placeholder (-I %) with each match, then executes the date command.
This is a shell pattern I really like, and use every day, and with some clever tweaks, can be very powerful. I encourage anyone who uses linux shell to get to know xargs right away.
There is another option for GNU sed users. While the BSD land folks were pretty true to their old BSD unix roots, the GNU folks, who wrote their userspace from scratch, added many wonderful enhancements to the standards. GNU Sed can apparently run a subshell command for you and then do the replacement for you, which would be dramatically easier. Since you are using the bsd style date invocation, I'm going to assume you don't have gnu sed at your disposal.
Using sed: tested with macOs only
There is a slight difference with the command date that should use the flag (-r) instead of (-d) exclusive to macOS
echo 12687278 | sed "s/\([0-9]\{8,\}\)/$(date -r \1)/g"
Results:
Thu Jan 1 09:00:01 JST 1970
Centos / Linux
Bash
I have a log file, which has lots of text in and epoch numbers all over the place. I want to replace all epochs whereever they are into readable date/time.
I've been wanting to this via sed, as that seems the tool for the job. I can't seem to get the replacement part of sed to actually parse the variable(epoch) to it for conversion.
Sample of what I'm working with...
echo "Some stuff 1346474454 And not working" \
| sed 's/1[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]/'"`bpdbm -ctime \&`"'/g'
Some stuff 0 = Thu Jan 1 01:00:00 1970 And not working
The bpdbm part will convert a supplied epoch variable into useful date. Like this..
bpdbm -ctime 1346474454
1346474454 = Sat Sep 1 05:40:54 2012
So how do i get the "found" item to be parsed into a command. As i don't seem to be able to get it to work.
Any help would be lovely. If there is another way, that would be cool...but i suspect sed will be quickest.
Thanks for your time!
that seems the tool for the job
No, it is not. sed can use & only itself, there is no way how to make it an argument to a command. You need something more powerful, e.g. Perl:
perl -pe 'if ( ($t) = /(1[0-9]+)/ ) { s/$t/localtime($t)/e }'
You can do it with GNU sed, the input:
infile
Some stuff 1346474454 And not working
GNU sed supports /e parameter which allows for piping command output into pattern space, one way to take advantage of this with bpdbm:
sed 's/(.*)(1[0-9]{9})(.*)/echo \1 $(bpdbm -ctime \2) \3/e' infile
Or with coreutils date:
sed 's/(.*)(1[0-9]{9})(.*)/echo \1 $(date -d #\2) \3/e' infile
output with date
Some stuff Sat Sep 1 06:40:54 CEST 2012 And not working
To get the same output as with bpdbm:
sed 's/(.*)(1[0-9]{9})(.*)/echo "\1$(date -d #\2 +\"%a %b %_d %T %Y\")\3"/e' infile
output
Some stuff Sat Sep 1 06:40:54 2012 And not working
Note, this only replaces the last epoch found on a line. Re-run if there are more.