I am responsible for deploying an application and as part of the deployment process, I need to get the latest committer timestamp in strict ISO 8601 format and place it in the database. For recent versions of git, it's fairly straight forward:
$ git log --pretty=format:'%cI' -n1
2019-05-29T10:24:58+04:00
But unfortunately, I have a few older instances where the switch %cI does not exist in the installed git. So to get close to the ISO 8601, I use %ci instead, resulting in:
$ git log --pretty=format:%ci -n1
2019-05-29 10:24:58 +0400
We will eventually upgrade the machines to a later version of operating system, and consequently a more modern version of git. Newer versions of git has this in the man page:
• %ci: committer date, ISO 8601-like format
• %cI: committer date, strict ISO 8601 format
But in the meantime, I want to massage this ISO 8601-like string on the older machines to the strict ISO 8601 format above. I wrote a bad looking one-liner to do this and it seems to work:
$ git log --pretty=format:'%ci' -n1 | sed -e 's/ /T/1' -e 's/ //1' -e 's/.\{22\}/&:/1'
2019-05-29T10:24:58+04:00
Simply, it replaces the first space with a T, the second space with nothing and inserts a colon : at the 22nd character position. How do I do this better? I tried looking up ways to do this with the date command; no luck.
you mean something like this?
date -d "$(git log --pretty=format:%ci -n1)" --iso-8601=seconds
Using sed:
$ echo '2019-05-29 10:24:58 +0400' | sed -e 's/ /T/' -e 's/ //' -e 's/..$/:\0/'
2019-05-29T10:24:58+04:00
Replace the first space with T, remove the second space, replace the last 2 characters with a colon and those 2 chars.
Related
I have this crazy long command-turned-(bash)script that outputs a tiny table with and object and some other data relate to it. I want to single the name of the object out to use it as a variable to further run more commands.
It outputs something like this:
ID PATH NAME VERSION UPGRADE STATUS
vm-13034 /abc/def somethingelse vX.X.X-XXX-XXXXXXX Up to date
The value I'm interested in is ID, which always is vm-#####. With an online tool I came up with the regex ^vm-\d{5,}$ (, bc the number is incrementing). So I figured vosxls | grep -E ^vm-\d{5,}$ (vosxls is the script's name for the long command) would work but it returns nothing. I tried it with -w, -e, grep -e "vm-\d{5,}" <(vosxls) and a few more adding and remove the enclosing ^ & $ characters when it would throw an error, enclosing the string in soft- and hard quotes or nothing at all, whatever would work.
Everything else that didn't error, returned nothing.
I read some examples with Perl and stuff similar to that but I stayed away and generally have stayed as basic as possible so I can run it in another system without issues. The most I've diverted from the most basic stuff (which are all I know anyway) was egrep which is I believe is the same as grep -e if I'm not mistaken.
How can the string be printed out? Is it because I'm already using a script it somehow affects how grep works? The script has no variables, it is merely a shorthand to avoid typing a lot of difficult to remember syntax+values.
Thanks.
Could be as simple as
vosxls | grep -Eo "^vm-\d{5,}"
Specifying -o allows you to return whatever matches the regex only.
-o, --only-matching
Prints only the matching part of the lines.
Perhaps something like vosxlx | sed -n 's/\(vm.*\ \)\(\/.*\)/\1/p'?
Or maybe vosxlx | awk '/vm/ {print $1}'
foo | perl -ne 'print $1 if /^(vm-\d+)/'
Your version of grep might not have extended regex support:
$:/mnt/c/Users/sukuj$ cat /tmp/s.txt
ID PATH NAME VERSION UPGRADE STATUS
vm-13034 /abc/def somethingelse vX.X.X-XXX-XXXXXXX Up to date
vm-23034 /abc/def somethingelse vX.X.X-XXX-XXXXXXX Up to date
$:/mnt/c/Users/sukuj$ grep -oE 'vm-[0-9]+' /tmp/s.txt
vm-13034
vm-23034
$:/mnt/c/Users/sukuj$ grep -oE 'vm-[0-9]{5,}' /tmp/s.txt
vm-13034
vm-23034
$:/mnt/c/Users/sukuj$ grep -V
grep (GNU grep) 3.4
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Written by Mike Haertel and others; see
<https://git.sv.gnu.org/cgit/grep.git/tree/AUTHORS>.
suku#DESKTOP-GID8MQV:/mnt/c/Users/sukuj$
You can't use \d with the -E option of grep to denote a digit. -P, respectively -Po, will work. But if you already know that you need the first field of the last line of the output, you don't need grep. A
vosxlx | tail -n 1 | cut -d ' ' -f 1
would do as well.
I wanna change unix epoch to normal date
i'm trying:
sed < file.json -e 's/\([0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]/`date -r \1`/g'
any hint?
With the lack of information from your post, I can not give you a better answer than this but it is possible to execute commands using sed!
You have different ways to do it you can use
directly sed e instruction followed by the command to be
executed, if you do not pass a command to e then it will treat the content of the pattern buffer as external command.
use a simple substitute command with sed and pipe the output to sh
Example 1:
echo 12687278 | sed "s/\([0-9]\{8,\}\)/date -d #\1/;e"
Example 2:
echo 12687278 | sed "s/\([0-9]\{8,\}\)/date -d #\1/" |sh
Test 1 (with Japanese locale LC_TIME=ja_JP.UTF-8):
Test 2 (with Japanese locale LC_TIME=ja_JP.UTF-8):
Remarks:
I will let you adapt the date command accordingly to your system specifications
Since modern dates are longer than 8 characters, the sed command uses an
open ended length specifier of at least 8, rather than exactly 8.
Allan has a nice way to tackle dynamic arguments: write a script dynamically and pipe it to a shell! It works. It tends to be a bit more insecure because you could potentially pipe unintentional shell components to sh - for example if rm -f some-important-file was in the file along with the numbers , the sed pipeline wouldn't change that line, and it would also be passed to sh along with the date commands. Obviously, this is only a concern if you don't control the input. But mistakes can happen.
A similar method I much prefer is with xargs. It's a bit of a head trip for new users, but very powerful. The idea behind xargs is that it takes its input from its standard in, then adds it to the command comprised of its own non-option arguments and runs the command(s). For instance,
$ echo -e "/tmp\n/usr/lib" | xargs ls -d
/tmp /usr/lib
Its a trivial example of course, but you can see more exactly how this works by adding an echo:
echo -e "/tmp\n/usr/lib" | xargs echo ls -d
ls -d /tmp /usr/lib
The input to xargs becomes the additional arguments to the command specified in xargs's own arguments. Read that twice if necessary, or better yet, fiddle with this powerful tool, and the light bulb should come on.
Here's how I would approach what you're doing. Of course I'm not sure if this is actually a logical thing to do in your case, but given the detail you went into in your question, it's the best I can do.
$ cat dates.txt
Dates:
1517363346
I can run a command like this:
$ sed -ne '/^[0-9]\{8,\}$/ p' < dates.txt | xargs -I % -n 1 date -d #%
Tue Jan 30 19:49:06 CST 2018
Makes sense, because I used the commnad echo -e "Dates:\ndate +%s" > dates.txt to make the file a few minutes before I wrote this post! Let's go through it together and I'll break down what I'm doing here.
For one thing, I'm running sed with -n. This tells it not to print the lines by default. That makes this script work if not every line has an 8+ digit "date" in it. I also added anchors to the start (^) and end ($) of the regex so the line had only the approprate digits ( I realize this may not be perfect for you, but without understanding your its input, I can't do better ). These are important changes if your file is not entirely comprised of date strings. Additionally, I am matching at least 8 characters, as modern date strings are going to be more like 10 characters long. Finally, I added a command p to sed. This tells it to print the matching lines, which is necessary because I specifically said not to print the nonmatching lines.
The next bit is the xargs iteslf. The sed will write a date string out to xargs's standard input. I set only a few settings for xargs. By default it will add the standard input to the end of the command, separated by a space. I didn't want a space, so I used -I to specify a replacement string. % doesn't have a special meaning; its just a placeholder that gets replaced with the input. I used % because its not a special character but rarely is used in commands. Finally, I added -n 1 to make sure only 1 input was used per execution of date. ( xargs can also add many inputs together, as in my ls example above).
The end result? Sed matches lines that consist, exclusively, of 8 or more numeric values, outputting the matching lines. The pipe then sends this output to xargs, which takes each line separately (-n 1) and, replacing the placeholder (-I %) with each match, then executes the date command.
This is a shell pattern I really like, and use every day, and with some clever tweaks, can be very powerful. I encourage anyone who uses linux shell to get to know xargs right away.
There is another option for GNU sed users. While the BSD land folks were pretty true to their old BSD unix roots, the GNU folks, who wrote their userspace from scratch, added many wonderful enhancements to the standards. GNU Sed can apparently run a subshell command for you and then do the replacement for you, which would be dramatically easier. Since you are using the bsd style date invocation, I'm going to assume you don't have gnu sed at your disposal.
Using sed: tested with macOs only
There is a slight difference with the command date that should use the flag (-r) instead of (-d) exclusive to macOS
echo 12687278 | sed "s/\([0-9]\{8,\}\)/$(date -r \1)/g"
Results:
Thu Jan 1 09:00:01 JST 1970
I'm trying to put together a bash/sh script that gets the UTC time of the last commit from a svn repo (the other VCSs are easy)
I understand that i can just do svn propget --revprop -rHEAD svn:date and get it rather easily, but there is no guarantee that the svn checkout will be online, so I'd prefer an offline version, if possible.
Maybe something to do with getting the UTC time from svn info? (by screwing with the timezones)
Summary: How can i get the UTC time of a svn commit, while not having access to the server?
Thanks
You can use svn log -r COMMITTED and extract date info from it. This is valid for offline copies.
svn log -r COMMITTED | sed -nE 's/^r.*\| ([0-9]{4}-[0-9]{2}-[0-9]{2} \S+ \S+).*/\1/p' | xargs -i -- date -ud '{}' '+%s'
The -u option makes date show UTC time instead.
Actually we don't need to use xargs:
date -ud "$(exec svn log -r COMMITTED | sed -nE 's/^r.*\| ([0-9]{4}-[0-9]{2}-[0-9]{2} \S+ \S+).*/\1/p')" '+%s'
UPDATE: I got the wrong command. The command above won't work offline. Here's the right one:
date -ud "$(svn info | sed -nE 's/^Last Changed Date: (\S+ \S+ \S+).*/\1/p')" '+%s'
I'm silly. As soon as i actually realised i just need to convert one timezone to another, it was obvious:
date -u +[format] -d $(svn info | <some grepping and cutting here>)
In my case, this is:
date -u +"%Y%m%d-%H%M" -d "$(svn info | grep 'Date' | cut -d' ' -f4-6)"
Of course, my solution probably isn't optimal, and if someone knows a better way, that'd be much appreciated :)
It turns out that the xml output of "svn info" has a zulu timestamp:
$ svn info --xml | grep date
<date>2015-04-30T15:38:49.371762Z</date>
So your bash command might be:
svn info --xml | grep -oP '(?<=<date>).*?(?=</date>)'
I just stumbled on this post. Ended up figuring out that svn uses env variable TZ, so for example:
TZ= svn log
will log dates in UTC
I'm trying to use SED to extract text from a log file. I can do a search-and-replace without too much trouble:
sed 's/foo/bar/' mylog.txt
However, I want to make the search case-insensitive. From what I've googled, it looks like appending i to the end of the command should work:
sed 's/foo/bar/i' mylog.txt
However, this gives me an error message:
sed: 1: "s/foo/bar/i": bad flag in substitute command: 'i'
What's going wrong here, and how do I fix it?
Update: Starting with macOS Big Sur (11.0), sed now does support the I flag for case-insensitive matching, so the command in the question should now work (BSD sed doesn't reporting its version, but you can go by the date at the bottom of the man page, which should be March 27, 2017 or more recent); a simple example:
# BSD sed on macOS Big Sur and above (and GNU sed, the default on Linux)
$ sed 's/ö/#/I' <<<'FÖO'
F#O # `I` matched the uppercase Ö correctly against its lowercase counterpart
Note: I (uppercase) is the documented form of the flag, but i works as well.
Similarly, starting with macOS Big Sur (11.0) awk now is locale-aware (awk --version should report 20200816 or more recent):
# BSD awk on macOS Big Sur and above (and GNU awk, the default on Linux)
$ awk 'tolower($0)' <<<'FÖO'
föo # non-ASCII character Ö was properly lowercased
The following applies to macOS up to Catalina (10.15):
To be clear: On macOS, sed - which is the BSD implementation - does NOT support case-insensitive matching - hard to believe, but true. The formerly accepted answer, which itself shows a GNU sed command, gained that status because of the perl-based solution mentioned in the comments.
To make that Perl solution work with foreign characters as well, via UTF-8, use something like:
perl -C -Mutf8 -pe 's/öœ/oo/i' <<< "FÖŒ" # -> "Foo"
-C turns on UTF-8 support for streams and files, assuming the current locale is UTF-8-based.
-Mutf8 tells Perl to interpret the source code as UTF-8 (in this case, the string passed to -pe) - this is the shorter equivalent of the more verbose -e 'use utf8;'.Thanks, Mark Reed
(Note that using awk is not an option either, as awk on macOS (i.e., BWK awk and BSD awk) appears to be completely unaware of locales altogether - its tolower() and toupper() functions ignore foreign characters (and sub() / gsub() don't have case-insensitivity flags to begin with).)
A note on the relationship of sed and awk to the POSIX standard:
BSD sed and awk limit their functionality mostly to what the POSIX sed and
POSIX awk specs mandate, whereas their GNU counterparts implement many more extensions.
Editor's note: This solution doesn't work on macOS (out of the box), because it only applies to GNU sed, whereas macOS comes with BSD sed.
Capitalize the 'I'.
sed 's/foo/bar/I' file
Another work-around for sed on Mac OS X is to install gsedfrom MacPorts or HomeBrew and then create the alias sed='gsed'.
If you are doing pattern matching first, e.g.,
/pattern/s/xx/yy/g
then you want to put the I after the pattern:
/pattern/Is/xx/yy/g
Example:
echo Fred | sed '/fred/Is//willma/g'
returns willma; without the I, it returns the string untouched (Fred).
The sed FAQ addresses the closely related case-insensitive search. It points out that a) many versions of sed support a flag for it and b) it's awkward to do in sed, you should rather use awk or Perl.
But to do it in POSIX sed, they suggest three options (adapted for substitution here):
Convert to uppercase and store original line in hold space; this won't work for substitutions, though, as the original content will be restored before printing, so it's only good for insert or adding lines based on a case-insensitive match.
Maybe the possibilities are limited to FOO, Foo and foo. These can be covered by
s/FOO/bar/;s/[Ff]oo/bar/
To search for all possible matches, one can use bracket expressions for each character:
s/[Ff][Oo][Oo]/bar/
The Mac version of sed seems a bit limited. One way to work around this is to use a linux container (via Docker) which has a useable version of sed:
cat your_file.txt | docker run -i busybox /bin/sed -r 's/[0-9]{4}/****/Ig'
Use following to replace all occurrences:
sed 's/foo/bar/gI' mylog.txt
I had a similar need, and came up with this:
this command to simply find all the files:
grep -i -l -r foo ./*
this one to exclude this_shell.sh (in case you put the command in a script called this_shell.sh), tee the output to the console to see what happened, and then use sed on each file name found to replace the text foo with bar:
grep -i -l -r --exclude "this_shell.sh" foo ./* | tee /dev/fd/2 | while read -r x; do sed -b -i 's/foo/bar/gi' "$x"; done
I chose this method, as I didn't like having all the timestamps changed for files not modified. feeding the grep result allows only the files with target text to be looked at (thus likely may improve performance / speed as well)
be sure to backup your files & test before using. May not work in some environments for files with embedded spaces. (?)
Following should be fine:
sed -i 's/foo/bar/gi' mylog.txt
C:\crp\cnp>sed -V
GNU sed version 3.02
Copyright (C) 1998 Free Software Foundation, Inc.......
C:\crp\cnp>type f.f
a a a
a a a
Trying to replace 'a' with spaces.
C:\crp\cnp>type f.f | sed -e s/a/\d032/g
d032 d032 d032
d032 d032 d032
why isn't it working?
I don't mind whether i'm finding or replacing spaces or new lines.. I just want to be able to specify them. It doesn't seem to be working and I don't know why.
(Replacing spaces or a space, with f, doesn't work)
C:\crp\cnp>echo a a | sed s/\d32/f/
a a
Note- it seems it might work in 4.2 , But i'm interested in 3.02 'cos that's the version bundled with unxutils http://unxutils.sourceforge.net/
Update to question-
thanks for paxdiablo's tip.. about gnu32win, I am now using that instead of unxutils. It is more up to date. I can now specify spaces. And tip of ghostdog, and paxdiablo, I see about the double quotes. I am fine specifying spaces with \d(since using 4.2) or with a space.
But, I still can't remove new lines
C:\crp>type f.f | sed -e "s/\r\n/f/g"
a aa
b bb
c cc
C:\crp>type f.f | sed -e "s/\d013\d010/f/g"
a aa
b bb
c cc
C:\crp>type f.f | sed -e "s/\x0D\x0A/f/g"
a aa
b bb
c cc
Note: This question was from 2010. Now it's 2020. Gnuwin32 is out of date(like the last time its Gnuwin32 sed was updated was 2010, with Sed 4.2.1 which was from 2009), Unxutils is even more out of date. So Gnuwin32 as of writing is a decade out of date, and Unxutils is more like 2 decades out of date, as of 2020. Cygwin is still kept up to date and as of writing is on Sed v 4.4 which is from 2017.
Why aren't you just using a space character itself rather than some funny encoding? As in:
sed -e 's/a/ /g'
For what it's worth, the command you gave also fails to work in 4.2.1 but, if you add in the quotes, it does work. So I suggest you change it to:
sed -e 's/a/\d032/g'
Apologies, I've just noticed you're running Windows so you've probably got CygWin or GnuWin32 (or equivalent).
Quotes work differently under Windows so you should try two things. The first is to use " instead of ' quotes:
sed -e "s/a/ /g"
Otherwise, the escape character in Windows is ^ so something like this should be able to escape the space:
sed -e s/a/^ /g
As an aside, I'd be looking to switch to GnuWin32, if possible, which has more recent versions of sed (for example). It doesn't look like UnxUtils has had an update since 2003 based on that web page you link to. You can get individual packages from here. You're looking for coreutils which contains the bulk of the UNIX text processing toolkit.
But, if you're stuck with UnxUtils, I'd just use the actual space rather than a decimal code, and then I'd use tr to get rid of new lines:
tr -d "\n"
assuming of course that the tr in textutils can handle that syntax :-)
I stuck with the same problem on Win XP and double quotes didn't work when trying to print new line "\n".
The solution was to use the new UnxUpdates.zip from http://unxutils.sourceforge.net/
It works correct with "\n".
Sed version states:
GNU sed version 4.0.7
On windows, use double quotes
sed "s/a/\d032/g" file
or just
sed "s/a/ /g" file