I am trying to figure out how to add a comma between two line outputs of application name and application version.
Here is my present code and output:
ordepmod-mbp:~ ordepmod$ mdls -name kMDItemFSName /Applications/*VMware*.app -name kMDItemVersion | sed 's/[^"]*"\([^"]*\)".*/\1/' | xargs -L 2
VMware Fusion.app 8.1.1
VMware Horizon Client.app 4.0.1
I cannot figure out the syntax to use that would keep the present command as simple as possible, while replacing ".app " with ".app, ". So the intended output would look like (manually edited):
VMware Fusion.app, 8.1.1
VMware Horizon Client.app, 4.0.1
I don't have VMware to test with, so I'll use iTunes instead.
Simplest Method
Basically, I am specifying raw output from mdls to get rid of all the extraneous stuff, then using tr to transliterate the NUL between the two fields into a comma. This is simple, but doesn't give you the space afterwards, but may be good enough.
mdls -name kMDItemFSName -name kMDItemVersion -raw /Applications/iTunes.app | tr '\0' ','
iTunes.app,12.4.1
Slightly Harder, and more accurate
Replacing NULs with tr is ok, but you cannot get both the comma and the space because tr only does "one-for-one" replacements. sed can replace the single NUL with a comma and a space, but it is not very easy with the non-GNU version of sed on OSX, so I would go with perl instead.
mdls -name kMDItemFSName -name kMDItemVersion -raw /Applications/Textedit.app | perl -pe 's/\x0/, /'
TextEdit.app, 1.11
Wildcard Version
Or, if you want to use wildcards, I would go with awk
mdls -name kMDItemFSName -name kMDItemVersion /Applications/*.app | awk -F'"' '/kMDItemFSName/{n=$2} /kMDItemVersion/{print n, $2}' OFS=', '
Output
App Store.app, 2.1
Automator.app, 2.6
Calculator.app, 10.8
Calendar.app, 8.0
Carbon Copy Cloner.app, 4.1.9
Chess.app, 3.13
Contacts.app, 9.0
...
Sonos.app, 6.2.2
Stickies.app, 10.0
System Preferences.app, 14.0
TextEdit.app, 1.11
TextWrangler.app, 5.0.2
Time Machine.app, 1.3
VirtualBox.app, 5.0.24
Xcode.app, 7.3
iBooks.app, 1.5
iMovie.app, 10.1.2
iTunes.app, 12.4.1
The awk command is basically setting the input field separator to ", and then every time it sees kMDItemFSName it saves the second field (which is the app name) as n. Then, every time it sees kMDItemVersion, it outputs the saved name and the second field (which is the version). The OFS (output field separator) is set to a comma and a space.
Related
I'm trying to use command line tools to edit some CSV I have in the following format for several year folders:
dataset
year_1 (i.e. 1929)
csv_filename_1.csv
csv_filename_2.csv
csv_filename_3.csv
...
year_2
...
I'm trying to append the file name to its content, creating a new column called filename with ./year_1/csv_filename_1.csv to all columns in it. After that, I would gzip it.
Due to the number of year folders (almost 100) and the CSVs quantities in each (totaling 100k+), I plan to use gnu parallel to run it, and
I was trying to use sed doing something like
fname="1929/csv_filename_1.csv" && \ # to simulate parallel's parameterization
sed -E -e '1s/$/,filename/' \ # append ",filename" to CSV header
-e '2,\$s/$/,${fname}/' ${fname} \ # append the filename string to the content
But I can't get the sed to work with the second expression because I either get "${fname}" written as-is to the file, or the sed error "sed: -e expression #1, char 6: unknown command: '\'" complaining about a comma or the slash. I also have tried to group the expressions like -e '1{s/$/,filename/};2,\${s/$/,${fname}/}' for no avail.
Currently, I gave up sed and started trying with awk, but not knowing why it didn't work is bothering me, so I came to ask why and how to make it work.
Just one more piece of info regarding how I intend to run this thing. It would be something like
find ~/dataset -iname "*csv" -print0 | parallel -0 -j0 '<the whole command here (sed + gz)>'
How could I do this? What am I forgetting? Thanks, folks!
PS: I just got it with awk
awk -v d="csv_filename_1.csv" -F"," 'FNR==1{a="filename"} FNR>1{a=d} {print $0","a}' csv_filename_1.csv | less
This might work for you (GNU parallel and sed):
find . -type f -name '*.csv' | parallel sed -i \''1s/$/,filename/;1!s#$#,{}#'\' {}
Use find to deliver the filename to the parallel command.
Use sed to append ,filename to the heading of each file and the file name present in {} to each line in the file.
N.B. The use of alternative delimiters s#...#...# in the second sed command to allow for the filename slashes. Also the find should be executed in the dataset directory.
I have this crazy long command-turned-(bash)script that outputs a tiny table with and object and some other data relate to it. I want to single the name of the object out to use it as a variable to further run more commands.
It outputs something like this:
ID PATH NAME VERSION UPGRADE STATUS
vm-13034 /abc/def somethingelse vX.X.X-XXX-XXXXXXX Up to date
The value I'm interested in is ID, which always is vm-#####. With an online tool I came up with the regex ^vm-\d{5,}$ (, bc the number is incrementing). So I figured vosxls | grep -E ^vm-\d{5,}$ (vosxls is the script's name for the long command) would work but it returns nothing. I tried it with -w, -e, grep -e "vm-\d{5,}" <(vosxls) and a few more adding and remove the enclosing ^ & $ characters when it would throw an error, enclosing the string in soft- and hard quotes or nothing at all, whatever would work.
Everything else that didn't error, returned nothing.
I read some examples with Perl and stuff similar to that but I stayed away and generally have stayed as basic as possible so I can run it in another system without issues. The most I've diverted from the most basic stuff (which are all I know anyway) was egrep which is I believe is the same as grep -e if I'm not mistaken.
How can the string be printed out? Is it because I'm already using a script it somehow affects how grep works? The script has no variables, it is merely a shorthand to avoid typing a lot of difficult to remember syntax+values.
Thanks.
Could be as simple as
vosxls | grep -Eo "^vm-\d{5,}"
Specifying -o allows you to return whatever matches the regex only.
-o, --only-matching
Prints only the matching part of the lines.
Perhaps something like vosxlx | sed -n 's/\(vm.*\ \)\(\/.*\)/\1/p'?
Or maybe vosxlx | awk '/vm/ {print $1}'
foo | perl -ne 'print $1 if /^(vm-\d+)/'
Your version of grep might not have extended regex support:
$:/mnt/c/Users/sukuj$ cat /tmp/s.txt
ID PATH NAME VERSION UPGRADE STATUS
vm-13034 /abc/def somethingelse vX.X.X-XXX-XXXXXXX Up to date
vm-23034 /abc/def somethingelse vX.X.X-XXX-XXXXXXX Up to date
$:/mnt/c/Users/sukuj$ grep -oE 'vm-[0-9]+' /tmp/s.txt
vm-13034
vm-23034
$:/mnt/c/Users/sukuj$ grep -oE 'vm-[0-9]{5,}' /tmp/s.txt
vm-13034
vm-23034
$:/mnt/c/Users/sukuj$ grep -V
grep (GNU grep) 3.4
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Written by Mike Haertel and others; see
<https://git.sv.gnu.org/cgit/grep.git/tree/AUTHORS>.
suku#DESKTOP-GID8MQV:/mnt/c/Users/sukuj$
You can't use \d with the -E option of grep to denote a digit. -P, respectively -Po, will work. But if you already know that you need the first field of the last line of the output, you don't need grep. A
vosxlx | tail -n 1 | cut -d ' ' -f 1
would do as well.
How can I convert a Windows dir path (say c:/libs/Qt-static) to the correct POSIX dir path (/c/libs/Qt-static) by means of standard msys features? And vice versa?
Cygwin, Git Bash, and MSYS2 have a readymade utility called cygpath.exe just for doing that.
Output type options:
-d, --dos print DOS (short) form of NAMEs (C:\PROGRA~1\)
-m, --mixed like --windows, but with regular slashes (C:/WINNT)
-M, --mode report on mode of file (binmode or textmode)
-u, --unix (default) print Unix form of NAMEs (/cygdrive/c/winnt)
-w, --windows print Windows form of NAMEs (C:\WINNT)
-t, --type TYPE print TYPE form: 'dos', 'mixed', 'unix', or 'windows'
I don't know msys, but a quick google search showed me that it includes the sed utility. So, assuming it works similar in msys than it does on native Linux, here's one way how to do it:
From Windows to POSIX
You'll have to replace all backslashes with slashes, remove the first colon after the drive letter, and add a slash at the beginning:
echo "/$pth" | sed 's/\\/\//g' | sed 's/://'
or, as noted by xaizek,
echo "/$pth" | sed -e 's/\\/\//g' -e 's/://'
From POSIX to Windows
You'll have to add a semi-colon, remove the first slash and replace all slashes with backslashes:
echo "$pth" | sed 's/^\///' | sed 's/\//\\/g' | sed 's/^./\0:/'
or more efficiently,
echo "$pth" | sed -e 's/^\///' -e 's/\//\\/g' -e 's/^./\0:/'
where $pth is a variable storing the Windows or POSIX path, respectively.
Just use cygpath:
$ cygpath -w "/c/foo/bar"
-> C:\foo\bar
$ cygpath -u "C:\foo\bar"
-> /c/foo/bar
You may wonder: "Do I have cygpath installed?" Well,
If you're using the git-bash shell, then yes.
If you're in cygwin or MSYS2, then yes.
If you're in another shell, but you have installed git-bash before, then cygpath can be found at git-bash-install-folder\usr\bin\cygpath.exe.
Else: maybe not, but I'm sure you can find a way to installed it.
The "correct" way in MSYS is:
$ MSYS_NO_PATHCONV=1 taskkill /F /T /IM ssh-agent.exe
This avoids having to manually translate slashes. It simply de-activates the path conversion.
Here is my implementation (tested on git bash).
From POSIX to Windows
sed '
\,/$, !s,$,/,
\,^/, s,/,:/,2
s,^/,,
s,/,\\,g
' <<< "$#"
Works for:
/c/git
relative/dir
c:/git
~
.
..
/c
/c/
./relative/dir
/sd0/some/dir/
except
/
<path with space>
Explanation:
\,^/, s,/,:/,2 (converts /drive/dir/ to /drive:/dir/) is the heart of it and inserts : before the 2nd /. I use , for delim instead of / for readability. If starting with / (\,^/,), then replace / with :/ for the 2nd occurrence. I do not want to assume drive letter length of 1 so this works for /sd0/some/dir.
s,^/,, removes the leading / and s,/,\\,g converts all / to \.
\,/$, !s,$,/, is to handle the corner case of /c and ensure 2nd / (/c/) for the next command to work.
Note:
If here string <<< does not work in your shell then you can echo and pipe as
echo "$#" | sed ...
Errata
Here e script
just FYI - at least for my git version 2.26.2.windows.1
e.g. if I have a path like C:\dev\work_setup\msk, I can go directly to Git Bash and type
cd "C:\dev\work_setup\msk"
this will result in current folder being changed to /c/dev/work_setup/msk - so this type of conversion seems to be done automatically, as long as I put the Windows path inside double quotes. Unfortunately I don't have references to original documentation that would back that up.
My solution works with a list of folders/files and it's done in 2 steps.
Suppose you would like to replace a path from D:\example to /example for a list of file where this Windows path has been repetead.
The first step it changes the backlashes into slashes
grep -lr "D:\\\\example" /parent-folder | xargs -d'\n' sed -i 's+\\+\/+g'
Note that parent-folder could be root (/) or whatever you like and -d'\n' parameter is necessary if you have filenames or folder names with white spaces.
Second step it substitutes the D:/example into /example:
grep -lr "D:/example" /parent-folder | xargs -d'\n' sed -i 's+D:+/example+g'
I wanted to share this solution since it tooks me some time to make this 2 lines but it has been really helpfull job (I'm migrating a Windows App to a Linux Server with tons of Windows paths inside').
The answer of #hello_earth is misleading, due to Windows path must be double backslashed like:
cd "e:\\dir\\subdir\\path"
otherwise the shell will find escape-sequences.
I'm trying to use SED to extract text from a log file. I can do a search-and-replace without too much trouble:
sed 's/foo/bar/' mylog.txt
However, I want to make the search case-insensitive. From what I've googled, it looks like appending i to the end of the command should work:
sed 's/foo/bar/i' mylog.txt
However, this gives me an error message:
sed: 1: "s/foo/bar/i": bad flag in substitute command: 'i'
What's going wrong here, and how do I fix it?
Update: Starting with macOS Big Sur (11.0), sed now does support the I flag for case-insensitive matching, so the command in the question should now work (BSD sed doesn't reporting its version, but you can go by the date at the bottom of the man page, which should be March 27, 2017 or more recent); a simple example:
# BSD sed on macOS Big Sur and above (and GNU sed, the default on Linux)
$ sed 's/ö/#/I' <<<'FÖO'
F#O # `I` matched the uppercase Ö correctly against its lowercase counterpart
Note: I (uppercase) is the documented form of the flag, but i works as well.
Similarly, starting with macOS Big Sur (11.0) awk now is locale-aware (awk --version should report 20200816 or more recent):
# BSD awk on macOS Big Sur and above (and GNU awk, the default on Linux)
$ awk 'tolower($0)' <<<'FÖO'
föo # non-ASCII character Ö was properly lowercased
The following applies to macOS up to Catalina (10.15):
To be clear: On macOS, sed - which is the BSD implementation - does NOT support case-insensitive matching - hard to believe, but true. The formerly accepted answer, which itself shows a GNU sed command, gained that status because of the perl-based solution mentioned in the comments.
To make that Perl solution work with foreign characters as well, via UTF-8, use something like:
perl -C -Mutf8 -pe 's/öœ/oo/i' <<< "FÖŒ" # -> "Foo"
-C turns on UTF-8 support for streams and files, assuming the current locale is UTF-8-based.
-Mutf8 tells Perl to interpret the source code as UTF-8 (in this case, the string passed to -pe) - this is the shorter equivalent of the more verbose -e 'use utf8;'.Thanks, Mark Reed
(Note that using awk is not an option either, as awk on macOS (i.e., BWK awk and BSD awk) appears to be completely unaware of locales altogether - its tolower() and toupper() functions ignore foreign characters (and sub() / gsub() don't have case-insensitivity flags to begin with).)
A note on the relationship of sed and awk to the POSIX standard:
BSD sed and awk limit their functionality mostly to what the POSIX sed and
POSIX awk specs mandate, whereas their GNU counterparts implement many more extensions.
Editor's note: This solution doesn't work on macOS (out of the box), because it only applies to GNU sed, whereas macOS comes with BSD sed.
Capitalize the 'I'.
sed 's/foo/bar/I' file
Another work-around for sed on Mac OS X is to install gsedfrom MacPorts or HomeBrew and then create the alias sed='gsed'.
If you are doing pattern matching first, e.g.,
/pattern/s/xx/yy/g
then you want to put the I after the pattern:
/pattern/Is/xx/yy/g
Example:
echo Fred | sed '/fred/Is//willma/g'
returns willma; without the I, it returns the string untouched (Fred).
The sed FAQ addresses the closely related case-insensitive search. It points out that a) many versions of sed support a flag for it and b) it's awkward to do in sed, you should rather use awk or Perl.
But to do it in POSIX sed, they suggest three options (adapted for substitution here):
Convert to uppercase and store original line in hold space; this won't work for substitutions, though, as the original content will be restored before printing, so it's only good for insert or adding lines based on a case-insensitive match.
Maybe the possibilities are limited to FOO, Foo and foo. These can be covered by
s/FOO/bar/;s/[Ff]oo/bar/
To search for all possible matches, one can use bracket expressions for each character:
s/[Ff][Oo][Oo]/bar/
The Mac version of sed seems a bit limited. One way to work around this is to use a linux container (via Docker) which has a useable version of sed:
cat your_file.txt | docker run -i busybox /bin/sed -r 's/[0-9]{4}/****/Ig'
Use following to replace all occurrences:
sed 's/foo/bar/gI' mylog.txt
I had a similar need, and came up with this:
this command to simply find all the files:
grep -i -l -r foo ./*
this one to exclude this_shell.sh (in case you put the command in a script called this_shell.sh), tee the output to the console to see what happened, and then use sed on each file name found to replace the text foo with bar:
grep -i -l -r --exclude "this_shell.sh" foo ./* | tee /dev/fd/2 | while read -r x; do sed -b -i 's/foo/bar/gi' "$x"; done
I chose this method, as I didn't like having all the timestamps changed for files not modified. feeding the grep result allows only the files with target text to be looked at (thus likely may improve performance / speed as well)
be sure to backup your files & test before using. May not work in some environments for files with embedded spaces. (?)
Following should be fine:
sed -i 's/foo/bar/gi' mylog.txt
C:\crp\cnp>sed -V
GNU sed version 3.02
Copyright (C) 1998 Free Software Foundation, Inc.......
C:\crp\cnp>type f.f
a a a
a a a
Trying to replace 'a' with spaces.
C:\crp\cnp>type f.f | sed -e s/a/\d032/g
d032 d032 d032
d032 d032 d032
why isn't it working?
I don't mind whether i'm finding or replacing spaces or new lines.. I just want to be able to specify them. It doesn't seem to be working and I don't know why.
(Replacing spaces or a space, with f, doesn't work)
C:\crp\cnp>echo a a | sed s/\d32/f/
a a
Note- it seems it might work in 4.2 , But i'm interested in 3.02 'cos that's the version bundled with unxutils http://unxutils.sourceforge.net/
Update to question-
thanks for paxdiablo's tip.. about gnu32win, I am now using that instead of unxutils. It is more up to date. I can now specify spaces. And tip of ghostdog, and paxdiablo, I see about the double quotes. I am fine specifying spaces with \d(since using 4.2) or with a space.
But, I still can't remove new lines
C:\crp>type f.f | sed -e "s/\r\n/f/g"
a aa
b bb
c cc
C:\crp>type f.f | sed -e "s/\d013\d010/f/g"
a aa
b bb
c cc
C:\crp>type f.f | sed -e "s/\x0D\x0A/f/g"
a aa
b bb
c cc
Note: This question was from 2010. Now it's 2020. Gnuwin32 is out of date(like the last time its Gnuwin32 sed was updated was 2010, with Sed 4.2.1 which was from 2009), Unxutils is even more out of date. So Gnuwin32 as of writing is a decade out of date, and Unxutils is more like 2 decades out of date, as of 2020. Cygwin is still kept up to date and as of writing is on Sed v 4.4 which is from 2017.
Why aren't you just using a space character itself rather than some funny encoding? As in:
sed -e 's/a/ /g'
For what it's worth, the command you gave also fails to work in 4.2.1 but, if you add in the quotes, it does work. So I suggest you change it to:
sed -e 's/a/\d032/g'
Apologies, I've just noticed you're running Windows so you've probably got CygWin or GnuWin32 (or equivalent).
Quotes work differently under Windows so you should try two things. The first is to use " instead of ' quotes:
sed -e "s/a/ /g"
Otherwise, the escape character in Windows is ^ so something like this should be able to escape the space:
sed -e s/a/^ /g
As an aside, I'd be looking to switch to GnuWin32, if possible, which has more recent versions of sed (for example). It doesn't look like UnxUtils has had an update since 2003 based on that web page you link to. You can get individual packages from here. You're looking for coreutils which contains the bulk of the UNIX text processing toolkit.
But, if you're stuck with UnxUtils, I'd just use the actual space rather than a decimal code, and then I'd use tr to get rid of new lines:
tr -d "\n"
assuming of course that the tr in textutils can handle that syntax :-)
I stuck with the same problem on Win XP and double quotes didn't work when trying to print new line "\n".
The solution was to use the new UnxUpdates.zip from http://unxutils.sourceforge.net/
It works correct with "\n".
Sed version states:
GNU sed version 4.0.7
On windows, use double quotes
sed "s/a/\d032/g" file
or just
sed "s/a/ /g" file