remove first column from hexdump output - bash

I have a hexdump output that looks like this
0101f10 64534 64943 00568 00262 01077 00721 00297 00140
0101f20 00748 00288 02211 01124 02533 01271 02451 00997
0101f30 03056 01248 02894 01026 02397 00696 00646 65114
0101f40 00943 64707 01113 64179 01135 64179 00805 64109
0101f50 00514 64045 64654 63037 63026 62014 62173 61625
I want to remove the first column, but I don't know what delimiter has been used by the hexdump command. I tried with awk and cut, but cant figure it out. Any help is appreciated.
Output I want is
64534 64943 00568 00262 01077 00721 00297 00140
00748 00288 02211 01124 02533 01271 02451 00997
03056 01248 02894 01026 02397 00696 00646 65114
00943 64707 01113 64179 01135 64179 00805 64109
00514 64045 64654 63037 63026 62014 62173 61625

With sed
sed 's/[^[:blank:]]*[[:blank:]]*//' infile
With gnu sed
sed 's/\S*\s*//' infile

input... | sed -E $'s/ +/\t/g' | cut -f2-
(Assuming Bash for $'\t', but GNU sed supports \t directly anyway.)

hexdump /path/to/file | awk '{sub(/[^ ]+ /, ""); print $0}'
This will do the job.

If the the delimiter is really a bunch of space, use tr to squeeze-repeats (-s) of psace to a tab and use cut for getting rid of the first column:
$ cat file | tr -s ' ' '\t' | cut -f 2-
64534 64943 00568 00262 01077 00721 00297 00140
00748 00288 02211 01124 02533 01271 02451 00997
03056 01248 02894 01026 02397 00696 00646 65114
00943 64707 01113 64179 01135 64179 00805 64109
00514 64045 64654 63037 63026 62014 62173 61625

All solution above works fine, Just adding an awk solution.
So, you only need to omit first column, but get the rest of it, you can try this :
awk '{$1=""; print $0}' /path/to/hexfile
It works perfectly, except that it leaves a space at beginning of each line. If that bothers you, there is a workaround using the substr() function in awk itself.
awk '{$1=""; print substr($0,2)}' /path/to/hexfile
To see more possible ways to do it, follow this link

Related

Remove duplicates from the same line in a file

How do I remove below duplicates from the same line in a file? I need the duplicates removed including semicolon.
For example from the below output of a file I need only "dg01.server.wmq.host=jms1001-01-ri5.ri5.dc2.responsys.com" similarly other lines of the file.
dg01.server.wmq.host=jms1001-01-ri5.ri5.dc2.responsys.com;jms1001-02-ri5.ri5.dc2.responsys.com dg02.server.wmq.host=jms1002-01-ri5.ri5.dc2.responsys.com;jms1002-02-ri5.ri5.dc2.responsys.com dg03.server.wmq.host=jms1003-01-ri5.ri5.dc2.responsys.com;jms1003-02-ri5.ri5.dc2.responsys.com dg04.server.wmq.host=jms1004-01-ri5.ri5.dc2.responsys.com;jms1004-02-ri5.ri5.dc2.responsys.com dg05.server.wmq.host=jms1005-01-ri5.ri5.dc2.responsys.com;jms1005-02-ri5.ri5.dc2.responsys.com dg06.server.wmq.host=jms1006-01-ri5.ri5.dc2.responsys.com;jms1006-02-ri5.ri5.dc2.responsys.com dg07.server.wmq.host=jms1007-01-ri5.ri5.dc2.responsys.com;jms1007-02-ri5.ri5.dc2.responsys.com dg08.server.wmq.host=jms1008-01-ri5.ri5.dc2.responsys.com;jms1008-02-ri5.ri5.dc2.responsys.com dg09.server.wmq.host=jms1009-01-ri5.ri5.dc2.responsys.com;jms1009-02-ri5.ri5.dc2.responsys.com dg10.server.wmq.host=jms1010-01-ri5.ri5.dc2.responsys.com;jms1010-02-ri5.ri5.dc2.responsys.com dg11.server.wmq.host=jms1011-01-ri5.ri5.dc2.responsys.com;jms1011-02-ri5.ri5.dc2.responsys.com dg12.server.wmq.host=jms1012-01-ri5.ri5.dc2.responsys.com;jms1012-02-ri5.ri5.dc2.responsys.com dg13.server.wmq.host=jms1013-01-ri5.ri5.dc2.responsys.com;jms1013-02-ri5.ri5.dc2.responsys.com dg14.server.wmq.host=jms1014-01-ri5.ri5.dc2.responsys.com;jms1014-02-ri5.ri5.dc2.responsys.com dg15.server.wmq.host=jms1015-01-ri5.ri5.dc2.responsys.com;jms1015-02-ri5.ri5.dc2.responsys.com dg16.server.wmq.host=jms1001-01-ri5.ri5.dc2.responsys.com;jms1001-02-ri5.ri5.dc2.responsys.com dg17.server.wmq.host=jms1002-01-ri5.ri5.dc2.responsys.com;jms1002-02-ri5.ri5.dc2.responsys.com dg18.server.wmq.host=jms1003-01-ri5.ri5.dc2.responsys.com;jms1003-02-ri5.ri5.dc2.responsys.com dg19.server.wmq.host=jms1004-01-ri5.ri5.dc2.responsys.com;jms1004-02-ri5.ri5.dc2.responsys.com dg20.server.wmq.host=jms1005-01-ri5.ri5.dc2.responsys.com;jms1005-02-ri5.ri5.dc2.responsys.com dg21.server.wmq.host=jms1006-01-ri5.ri5.dc2.responsys.com;jms1006-02-ri5.ri5.dc2.responsys.com dg22.server.wmq.host=jms1007-01-ri5.ri5.dc2.responsys.com;jms1007-02-ri5.ri5.dc2.responsys.com dg23.server.wmq.host=jms1008-01-ri5.ri5.dc2.responsys.com;jms1008-02-ri5.ri5.dc2.responsys.com dg24.server.wmq.host=jms1009-01-ri5.ri5.dc2.responsys.com;jms1009-02-ri5.ri5.dc2.responsys.com dg25.server.wmq.host=jms1010-01-ri5.ri5.dc2.responsys.com;jms1010-02-ri5.ri5.dc2.responsys.com dg26.server.wmq.host=jms1011-01-ri5.ri5.dc2.responsys.com;jms1011-02-ri5.ri5.dc2.responsys.com dg27.server.wmq.host=jms1012-01-ri5.ri5.dc2.responsys.com;jms1012-02-ri5.ri5.dc2.responsys.com dg28.server.wmq.host=jms1013-01-ri5.ri5.dc2.responsys.com;jms1013-02-ri5.ri5.dc2.responsys.com dg29.server.wmq.host=jms1014-01-ri5.ri5.dc2.responsys.com;jms1014-02-ri5.ri5.dc2.responsys.com dg30.server.wmq.host=jms1015-01-ri5.ri5.dc2.responsys.com;jms1015-02-ri5.ri5.dc2.responsys.com dg31.server.wmq.host=jms1001-01-ri5.ri5.dc2.responsys.com;jms1001-02-ri5.ri5.dc2.responsys.com dg32.server.wmq.host=jms1002-01-ri5.ri5.dc2.responsys.com;jms1002-02-ri5.ri5.dc2.responsys.com dg33.server.wmq.host=jms1003-01-ri5.ri5.dc2.responsys.com;jms1003-02-ri5.ri5.dc2.responsys.com dg34.server.wmq.host=jms1004-01-ri5.ri5.dc2.responsys.com;jms1004-02-ri5.ri5.dc2.responsys.com dg35.server.wmq.host=jms1009-01-ri5.ri5.dc2.responsys.com;jms1009-02-ri5.ri5.dc2.responsys.com dg36.server.wmq.host=jms1010-01-ri5.ri5.dc2.responsys.com;jms1010-02-ri5.ri5.dc2.responsys.com dg37.server.wmq.host=jms1011-01-ri5.ri5.dc2.responsys.com;jms1011-02-ri5.ri5.dc2.responsys.com dg38.server.wmq.host=jms1012-01-ri5.ri5.dc2.responsys.com;jms1012-02-ri5.ri5.dc2.responsys.com dg39.server.wmq.host=jms1007-01-ri5.ri5.dc2.responsys.com;jms1007-02-ri5.ri5.dc2.responsys.com dg40.server.wmq.host=jms1008-01-ri5.ri5.dc2.responsys.com;jms1008-02-ri5.ri5.dc2.responsys.com
Assuming dg01.server.wmq.host=jms1001-01-ri5.ri5.dc2.responsys.com;jms1001-02-ri5.ri5.dc2.responsys.com is a line in your input file and you're only interested in the dg01.server.wmq.host=jms1001-01-ri5.ri5.dc2.responsys.com part (up to, but not including, the semicolumn) you can obtain the desired output by running:
cat inputfile | awk -F ';' {'print $1'}
Another way to obtain the same output, as pointed out by #Shawn, would be:
cut -d ';' -f1 inputfile

Grep and awk use

i try one day but dont fixed. I dont know this method.
content query --uri content://com.android.contacts/contacts | grep "+9053158888" | awk -F'[,,= ]' '{cmd="content delete --uri content://com.android.contacts/contacts/"$(NF-3);system(cmd)}'
but not finding
My string
Row: 9991 last_time_contacted=0, phonetic_name=NULL, custom_ringtone=NULL, contact_status_ts=NULL, pinned=0, photo_id=NULL, photo_file_id=NULL, contact_status_res_package=NULL, contact_chat_capability=NULL, contact_status_icon=NULL, display_name_alt=+90532555688, sort_key_alt=+90532555688, in_visible_group=1, starred=0, contact_status_label=NULL, phonebook_label=#, is_user_profile=0, has_phone_number=1, display_name_source=40, phonetic_name_style=0, send_to_voicemail=0, lookup=0r10070-24121C1814241820221C1A14.3789r10071-24121C1814241820221C1A14.0r10072-24121C1814241820221C1A14.0r10073-24121C1814241820221C1A14.0r10074-24121C1814241820221C1A14.0r10075-24121C1814241820221C1A14.0r10078-24121C1814241820221C1A14.0r10082-24121C1814241820221C1A14.0r10083-24121C1814241820221C1A14.0r10084-24121C1814241820221C1A14.0r10085-24121C1814241820221C1A14.0r10086-24121C1814241820221C1A14.0r10087-24121C1814241820221C1A14.0r10092-24121C1814241820221C1A14.0r10094-24121C1814241820221C1A14.0r10097-24121C1814241820221C1A14, phonebook_label_alt=#, contact_last_updated_timestamp=1612984348874, photo_uri=NULL, phonebook_bucket=213, contact_status=NULL, display_name=+90532555688, sort_key=+90532555688, photo_thumb_uri=NULL, contact_presence=NULL, in_default_directory=1, times_contacted=0, _id=10097, name_raw_contact_id=10070, phonebook_bucket_alt=213
i need string " _id=10097 "
You may use this grep to find word _id followed by a = and 1+ digits:
... | grep -Eo '\b_id=[0-9]+'
_id=10097
To get all occurrences of if try following, written and tested with shown samples in GNU grep. Where str is your shell variable have your shown sample input in it.
echo "$str" | grep -oP ', \K_id=\d+'
OR try with awk:
echo "$str" |
awk 'match($0,/, _id=[0-9]+/){print substr($0,RSTART+2,RLENGTH-2)}'
Above will output as:
_id=10097

Separating joined columns with awk

I have a data file which looks like the following:
0.00000-130250.92921 28880.20200-159131.13121 301.58706
0.05000-130250.73120 28156.69202-158407.42322 294.03167
0.10000-130250.79137 28237.16138-158487.95275 294.87198
0.15000-130250.81209 28168.63042-158419.44250 294.15634
0.20000-130250.82418 28149.57611-158400.40029 293.95736
0.25000-130250.88438 28069.57135-158320.45573 293.12189
0.30000-130251.06059 28071.30576-158322.36635 293.14000
0.35000-130250.96639 28084.46351-158335.42990 293.27741
as you can see some of the columns which start with "-" sign are
joined to the previous one, for instance: 0.35000-130250.96639
this should be 0.35000 and -130250.96639. I can separate the
columns with VIM but I wanted to know if it is possible to do that
with AWK.
Thanks.
You can use sed: replace each - with a space and -:
sed -e 's/-/ -/g' input > output
The /g means globally, i.e. it replaces all occurrences on each line, not just the first one.
Using just awk
awk '{ gsub("-"," -") ; print }'

Bash command to extract characters in a string

I want to write a small script to generate the location of a file in an NGINX cache directory.
The format of the path is:
/path/to/nginx/cache/d8/40/32/13febd65d65112badd0aa90a15d84032
Note the last 6 characters: d8 40 32, are represented in the path.
As an input I give the md5 hash (13febd65d65112badd0aa90a15d84032) and I want to generate the output: d8/40/32/13febd65d65112badd0aa90a15d84032
I'm sure sed or awk will be handy, but I don't know yet how...
This awk can make it:
awk 'BEGIN{FS=""; OFS="/"}{print $(NF-5)$(NF-4), $(NF-3)$(NF-2), $(NF-1)$NF, $0}'
Explanation
BEGIN{FS=""; OFS="/"}. FS="" sets the input field separator to be "", so that every char will be a different field. OFS="/" sets the output field separator as /, for print matters.
print ... $(NF-1)$NF, $0 prints the penultimate field and the last one all together; then, the whole string. The comma is "filled" with the OFS, which is /.
Test
$ awk 'BEGIN{FS=""; OFS="/"}{print $(NF-5)$(NF-4), $(NF-3)$(NF-2), $(NF-1)$NF, $0}' <<< "13febd65d65112badd0aa90a15d84032"
d8/40/32/13febd65d65112badd0aa90a15d84032
Or with a file:
$ cat a
13febd65d65112badd0aa90a15d84032
13febd65d65112badd0aa90a15f1f2f3
$ awk 'BEGIN{FS=""; OFS="/"}{print $(NF-5)$(NF-4), $(NF-3)$(NF-2), $(NF-1)$NF, $0}' a
d8/40/32/13febd65d65112badd0aa90a15d84032
f1/f2/f3/13febd65d65112badd0aa90a15f1f2f3
With sed:
echo '13febd65d65112badd0aa90a15d84032' | \
sed -n 's/\(.*\([0-9a-f]\{2\}\)\([0-9a-f]\{2\}\)\([0-9a-f]\{2\}\)\)$/\2\/\3\/\4\/\1/p;'
Having GNU sed you can even simplify the pattern using the -r option. Now you won't need to escape {} and () any more. Using ~ as the regex delimiter allows to use the path separator / without need to escape it:
sed -nr 's~(.*([0-9a-f]{2})([0-9a-f]{2})([0-9a-f]{2}))$~\2/\3/\4/\1~p;'
Output:
d8/40/32/13febd65d65112badd0aa90a15d84032
Explained simple the pattern does the following: It matches:
(all (n-5 - n-4) (n-3 - n-2) (n-1 - n-0))
and replaces it by
/$1/$2/$3/$0
You can use a regular expression to separate each of the last 3 bytes from the rest of the hash.
hash=13febd65d65112badd0aa90a15d84032
[[ $hash =~ (..)(..)(..)$ ]]
new_path="/path/to/nginx/cache/${BASH_REMATCH[1]}/${BASH_REMATCH[2]}/${BASH_REMATCH[3]}/$hash"
Base="/path/to/nginx/cache/"
echo '13febd65d65112badd0aa90a15d84032' | \
sed "s|\(.*\(..\)\(..\)\(..\)\)|${Base}\2/\3/\4/\1|"
# or
# sed sed 's|.*\(..\)\(..\)\(..\)$|${Base}\1/\2/\3/&|'
Assuming info is a correct MD5 (and only) string
First of all - thanks to all of the responders - this was extremely quick!
I also did my own scripting meantime, and came up with this solution:
Run this script with a parameter of the URL you're looking for (www.example.com/article/76232?q=hello for example)
#!/bin/bash
path=$1
md5=$(echo -n "$path" | md5sum | cut -f1 -d' ')
p3=$(echo "${md5:0-2:2}")
p2=$(echo "${md5:0-4:2}")
p1=$(echo "${md5:0-6:2}")
echo "/path/to/nginx/cache/$p1/$p2/$p3/$md5"
This assumes the NGINX cache has a key structure of 2:2:2.

sed edit of text with variables and special characters

I'm on OS X and writing a bash script to edit text in a file which includes some known text with special characters. There will be a variable too which needs to be retained and some text entered or replaced.
Here is the input file contents:
user_pref("intl.charsetmenu.browser.cache", "UTF-8");
user_pref("network.automatic-ntlm-auth.trusted-uris", "search.co.za");
user_pref("network.cookie.prefsMigrated", true);
I currently have this code:
existingTrusts=`more ~/prefs.js | grep "network.automatic-ntlm-auth.trusted-uris" | awk '{print $2}' | sed 's/);//g' | sed 's/"//g'`
trustSites="company.com,organisation.co.uk,$existingTrusts"
replacementValue='"user_pref("network.automatic-ntlm-auth.trusted-uris", "$trustSites");"'
sed -i 's/^user_pref("network.automatic-ntlm-auth.trusted-uris/$replacementValue/' ~/prefs.js > ~/newPrefs.js
Any help appreciated.
You are using too many pipes to set your existingTrusts variable. Set your variables like this:
existingTrusts=$(awk '/network.automatic-ntlm-auth.trusted-uris/ {gsub(/"|\);/, "", $2); print $2}' ~/prefs.js)
trustSites="company.com,organisation.co.uk,$existingTrusts"
replacementValue='user_pref("network.automatic-ntlm-auth.trusted-uris", "'$trustSites'");'
# and now finally your sed command
sed 's/^user_pref("network.automatic-ntlm-auth.trusted-uris".*$/'"$replacementValue"'/' ~/prefs.js > ~/newPrefs.js
Why so complicated?
trustedSites='company.com,organisation.co.uk,'
sed -i '' -e '/network.automatic-ntlm-auth.trusted-uris/s/, "\([^"]*\)/, "'"${trustedSites}"'\1/' prefs.js
This is imperfect because
It uses unescaped . in a pattern where a literal . is presumed
It presumes , " will appear exactly as that exactly where expected
These things could be fixed.

Resources