My MySQL was installed on a remote host, I am interactive with Mysql by PHP PDO. I want to check the SQL statements I have executed. So I search on google and use tcpdump. here are the codes what get on web.
tcpdump -i eth0 -s 0 -l -w - dst port 3306 | strings | perl -e '
while(<>) { chomp; next if /^[^ ]+[ ]*$/;
if(/^(SELECT|UPDATE|DELETE|INSERT|SET|COMMIT|ROLLBACK|CREATE|DROP|ALTER|CALL)/i) {
if (defined $q) { print "$q\n"; }
$q=$_;
} else {
$_ =~ s/^[ \t]+//; $q.=" $_";
}
}'
but I found it can show the utf8 words.
This post Make sure your terminal supports outputting UTF-8 notice me that I should check if my terminal supports outputting UTF-8.
but My terminal does support utf8.
I alse try the recommaned solution above, but It still does not work.
tcpdump -lnpi lo tcp port 80 -s 16000 -w - | tr -t '[^[:print:]]' ''
tcpdump -lnpi lo tcp port 80 -s 16000 -w - | strings -e S -n 1
so I wonder How to display utf8 characters captured by tcpdump?
Related
I have the sample file which is in not having standard format and need to generate CSV file with the following conditions, i am trying to figure out the solution past few days for this
ab.db.contact-points=10.135.64.46,10.135.2.6,10.135.8.4
ab.db.port=9042
ab.token.uri=10.135.83.42,10.135.83.41
ab.db.port=9042#9042
ab.token.uri=10.135.83.42
deeplyiourl=https://deeply-internal-npe.example.com/testing/deep/v1/events/
TEST.URL=http://testing.unix.great20000.org:8115/CYBER/SOURCE
ABCDSURL=https://testing-tposs.data.dx-pk1.cf.example.com:443/saveResume/updateDecision?decisionXML={decisionXML}
simple.abcd02.uri=https://abcd02.pro.api.great.example.com/v1/author/testing?grant_type=abcd_credentials
abcd02.defo2-url=tcp://10.158.150.25:7222,tcp://10.158.150.99:72
findingdata.sentry.url=http://create.test.data.com:8555/FirstData/Payment
tstign.endpoint=http://create.test.ext.example.com:6002/ECL1/GatewayV3Proxy/ChargeSale
basic.endpoint=http://Validating/author/testing/v4/internal/test
TEST.URL=http://tesing.great.com
Conditions:
Consider only URL’s and Host name , IP address like endpoints with .com or .org or an IP should be considered.
Remove http:// and Https://
URL’s ending with .com or .org
remove anything after .com or .org
if there is port info in the URL use that port info
I have tried the below script but not getting the expected output
grep -P '((?<=[^0-9.]|^)[1-9][0-9]{0,2}(\.([0-9]{0,3})){3}(?=[^0-9.]|$)|(http|ftp|https|ftps|sftp)://([\w_-]+(?:(?:\.[\w_-]+)+))([\w.,#?^=%&:/+#-]*[\w#?^=%&/+#-])?|\.port|\.host|contact-points|\.uri|\.endpoint)' FASOfflineReviewAction.properties|grep '^[^#]'|awk '{split($0,a,"#"); print a[1]}'|awk '{split($0,a,"="); print a[1],a[2]}'|sed '/.com\|.org\|10.\|17./!d'|sed 's/^\|#/,/g'|awk '/http:\/\// {split($1,a,":");if (a[3] == "") print 80; else print a[3]}
/https:\/\// {split($1,a,":");if (a[3] == "") print 443; else print a[3]}
/Points/ {print $2,"9042"}
/host/ {h=$2}
/port/ {print h,$2; h=""}'|sed 's/com.*/com/'|sed 's/org.*/org/'|awk -F'[, ]' '{for(i=1;i<NF;i++){print $i,$NF}}'|awk 'BEGIN{OFS=","} {$1=$1} 1'|sed '/^[0-9]*$/d'|awk -F, '$1 != $2' |sed -E 's_^https?://__'
Desired output
hostname port
10.135.64.46 9042
10.135.2.6 9042
10.135.8.4 9042
10.135.83.42 9042
10.135.83.41 9042
10.135.83.42 9042
deeply-internal-npe.example.com 443
testing.unix.great20000.org 8115
testing-tposs.data.dx-pk1.cf.example.com 443
10.158.150.25 7222
10.158.150.99 72
create.test.data.com 8555
create.test.ext.example.com 6002
tesing.great.com 80
This might work for you (GNU grep and sed):
grep -Eio '([0-9]{1,3}\.){3}[0-9]{1,3}(:[0-9]{1,4})?|https?://[^/]*\.(com|org)(:[0-9]{1,4})?' file |
sed -E '1{x;s/^/hostname port/p;x};s#^https?://##I;/:/!s/$/:9042/;y/:/ /'
The solution is split into two parts:
Extract the IP's and/or URL's using grep
Add header and add/split ports using sed
I am trying to encode some hex values to base64 in shell script.
nmurshed#ugster05:~$ echo -n "1906 1d8b fb01 3e78 5c21 85db 58a7 0bf9 a6bf 1e42 cb59 95cd 99be 66f7 8758 cf46 315f 1607 66f7 6793 e5b3 61f9 fa03 952d 9101 b129 7180 6f1d ca93 3494 55e0 0e2e" | xxd -r -p | base64
GQYdi/sBPnhcIYXbWKcL+aa/HkLLWZXNmb5m94dYz0YxXxYHZvdnk+WzYfn6A5UtkQGxKXGAbx3K
kzSUVeAOLg==
I get a automatic new line after 76 charecters, Is there a way to avoid that ?
Online i found, use "-n" to ignore new lines...Can anyone suggest something ?
echo -n doesn't actually matter here: It controls whether there's a newline on the output from echo, but whether echo emits a newline has no bearing on whether xxd or base64 emit newlines.
Because xxd ignores any trailing newline in the input, echo or echo -n will behave precisely the same here; whether there's a newline by echo makes no difference, because that newline (if it exists) will be consumed by xxd when reading its input. Rather, what you ultimately care about is the output of base64, which is what is generating your final result.
Assuming you have the GNU version of base64, add -w 0 to disable line wrapping in its output. Thus:
printf '%s' "1906 1d8b fb01 3e78 5c21 85db 58a7 0bf9 a6bf 1e42 cb59 95cd 99be 66f7 8758 cf46 315f 1607 66f7 6793 e5b3 61f9 fa03 952d 9101 b129 7180 6f1d ca93 3494 55e0 0e2e" \
| xxd -r -p \
| base64 -w 0
I had a similar problem where
var1=$(echo -n "$USER:$PASSWORD" | base64)
was resulting in an erroneous base64 encoded value which was unusable in my next step of the script, used printf & it worked fine. Here is my code:
var1=$(printf "%s" "${USER}:${PASSWORD}" | base64)
I have app A and B and they are communicating using unix socket. What I need is to tap this socket and listen/send the communication for evaluation?
socat -t100 -x -v UNIX-LISTEN:/tmp/.sock,mode=777,reuseaddr,fork ,UNIX-CONNECT:/tmp/.sock_original
Works fine for dumping it into console, but how can I add like UDP-SENDTO?
And can I?
Thanks.
Ok, I found the way. It's not optimal but working
socat -t100 -x -v UNIX-LISTEN:/tmp/.sock,mode=777,reuseaddr,fork ,UNIX-CONNECT:/tmp/.sock_original |
awk '{ if (lines > 0) { print; --lines; }} /^>/ { lines = 1}' | while read -r line; do echo $line > /dev/udp/localhost/6543; done
It's filtering out the first line and sending out just packets going in one direction (the /^>/).
I've trace log generated by strace command like on running PHP by:
sudo strace -e sendto -fp $(pgrep -n php) -o strace.log
And the output looks like:
11208 sendto(4, "set 29170397297_-cache-schema 85 0 127240\r\n\257\202\v\0?\0\0\0\2\27\10stdClass\24\7\21\3cid\21\6schema\21\4d\37ata\25\n\247\21\5block\24\6\21\6fields\24\f\21\3bid\24\2\5\21\4type 0\37erial\21\10not null\5\21\6module\24\4\16\7\21\7va\37rchar\21\6length\6#\16\t\5\21\7default\r\21\5de\2lta#\5\16\v\16\f\6 \35\7\16\r\21\0010\21\5t \207C\30#6\2\16\r\r n\4tatus#0\4\21\3int/\7\6\0\21\4size \222\finy\21\6weight\24\3 ;\0\22\300 \6\6region#8\340\5P\5custom\27\300,\17\16\23\16\24\21\nvisibility\340\t\34\7\5pages\24\2 \205\3\4tex#\206 \261\1it \365\0\5\240\0\377y\10\r\21\ftransl!N\2ble %\1ca!a\340\3Q\0\1n\31\vprimary key\24\1\6\0\16\6\21\vunique#\21\ts\24\1\21\3tmd\24\3 \31\0\20 2\v\n\6\2\16\16\21\7index \210\10\1\21\4list\24\5\240\36\0\21 \36\10\26\6\3\16\25\6\4\16\n \1\6\4\21\4name \7\0\na\317\2_ro\252\0\5!$\0\n \3\341\2\23\0\16\340\0\16A\214\2\21\3r!\354# \v\22\21\10unsigned\5#\332\0\36\213\0\n \213\0\16 l\6%\16!\24\1\16%\271\0%#p\5\16#\16$\21\f\200l\241b#n\2\4\16\6M\2\10\16&#E\4\21\4bod\201_\5\32\16\t\4\16\23B\\\2g\16\34 \30\3info .\0\7a\255\0\200#q!L\5\6forma\201\332B/!d\2\4\16\37 y\0*y\0 \225a;\240\201\2'\21\van\0_\207\200\2\5\16\1\340\0U =#U\1\16\3#\222 \212\2lob#O\n\23\16)\21\6expire#\30\342\0\26\7\21\7create\241\17< \25\0\n\203\1\"\177\0dY\0\22 \305\5\5small\240!a\32\0.\230\0.\240\240\0\1\240\240\3,\21\vb S\2kpo\"\313\2s\24\6!\220\2\t\21\2\241q\0\10 ?\4\21\tno \213\6ort\5\21\fm\";\3ine_A\313\232\241\3\2\5\16#\340\4\16!\345\340\0U\223\340\0'AC\4sourc\202\202\340\3\27\0\v\200\27\0_C\326\340\0074\1\16\21_\240\363\2\1\16\25\340\3\16\r\0\21\vmultipliers\31\0- \223\1\21\t\341\0\30B-\0\1!\10\0003a\253\0005\v\0005ac \327Dz\"\364 \20\0\10 \6\0 #\333\r\0165\16\36\0163\21\nidenti$x\nr\0166\21\vadmin_ce\10\21\5label\21\f\244H\6 hook\21\23\240\r\0_\340\1\375\fs\21\3api\24\4\21\5own F\0062\16C\16B\21\17 H\5imum_v \260$\25\7\6\1\21\17curr m\340\1\22!\242\0002\"\305\0022\21\20\340\1N\5_groupa\247\2\6\0163\352\0\10 \352\2\0164\5 \325C%\341\0P\341\5\220\1\0162aQA\26\4\16:\5\21\17\201\321\1 c\"$\5back\21#\340\7b\0_\200!\340\3\311\1\16\7C\340\0a!\312\1\no \300#\240!&}\241\237\0\0\242e\341\4n\5\16;\24\10\16< \7\2=\21\35\340\1m\0\320\0 \342\3XAz\v\16>\16G\16?\16#\16A\21\30\341\tT\201\5\1\21\22\200\243\0 B0\6 string#o\4toolsbD\1\16C \260\0D!D\4C\16L\16E!P\0F \3\201T\16G\21\21ckeditor_set%\266\0gE\323\0\5%Q\0# 4#\345!)\"w#\372\1\21\10\340\0!\0\1 \31\0\32\240\334\4#\16\n\21\10\300D \r\2O\21\25\300\r\6_input_\244+\340\16V\1\16+ \31\340\4h X\0\2!;\0# \245\0+ \247\0Q T\7R\21\26comme#/\0_%\266\2cko W\3pane ;\4\5\24\10\21\7#\v\0_\243\257\301\231\1\21\4F\35 !\340\1\22F\323\0021\21\10\"\311'B\0e#\223A\254&f`\346\"~\6\vcollap&q%\227\340\6\35\2\0\21\t\240\35\344\1a\3009\0\0#\212\300.\0001\200L$\247\1enFl\344\0\216\300,\0\1G\5\3view\340\0002\300\177 \372\0\1 K\0T!"..., 8196, MSG_NOSIGNAL|MSG_MORE, NULL, 0) = 8196
It sounds like these are represented by ordinary C escape codes.
I've tried to decode them in shell by printf like:
while read line; do printf "%s" "$line"; done < <(cat strace.log | head -n2)
but it failed (looks like it doesn't make any sense):
11208 sendto(4, "set 29170397297_-cache-schema 85 0 127240rn257202v0?00022710stdClass247213cid216schema214d37ata25n247215block246216fields24f213bid2425214type 037erial2110not null5216module244167217va37rchar216length6#16t5217defaultr215de2lta#516v16f6 35716r210010215t 207C30#6216rr n4tatus#04213int/760214size 222finy216weight243 ;022300 66region#83405P5custom27300,171623162421nvisibility340t3475pages242 20534tex#206 2611it 365052400377y10r21ftransl!N2ble %1ca!a3403Q01n31vprimary key2416016621vunique#21ts241213tmd243 31020 2vn621616217index 210101214list24524036021 3610266316256416n 164214name 70na3172_ro25205!$0n 3341223016340016A2142213r!354# v222110unsigned5#3320362130n 213016 l6%16!24116%2710%#p516#16$21f200l241b#n24166M21016&#E4214bod201_53216t41623B\2g1634 303info .07a2550200#q!L56forma201332B/!d241637 y0*y0 225a;2402012'21van0_207200251613400U =#U1163#222 2122lob#On2316)216expire#303420267217create24117< 250n2031"1770dY022 30555small240!a320.`2300.240240012402403,21vb S2kpo"3132s246!2202t212241q010...
Is there any better way to parse the output of strace command to see plain strings passed to recvfrom/sendto?
Ideally it is possible to print printable characters including new lines (\r\n), but cut-off NULLs and other non-printable characters?
The problem why read doesn't work, because shell is already escaping the characters, so the string is doubled escaped, therefore \r\n is printed as rn.
To ignore escaping of characters by shell, you can use read -r which allow backslashes to escape any characters (so they're treated literally). Here is example:
while read -r line; do printf "%b\n" "$line"; done < strace.log | strings
Since it's a binary data, above example also includes strings command to display only printable strings.
Strace also support printing all strings in hex when -x is specified, but it'll work the same.
Here is the version to parse strace output in real-time:
while read -r line;
do printf "%b\n" "$line" | strings
done < <(sudo strace -e recvfrom,sendto -s 1000 -fp $(pgrep -n php) 2>/dev/stdout)
Further more strings, can be replaced by more specific filter using grep, to get only what is inside double quotes:
grep -o '".\+[^"]"' | grep -o '[^"]\+[^"]'
however this may still print binary formats.
To avoid that, lets simplify the whole process, so lets define the following formatter alias:
alias format-strace='grep --line-buffered -o '\''".\+[^"]"'\'' | grep --line-buffered -o '\''[^"]*[^"]'\'' | while read -r line; do printf "%b" $line; done | tr "\r\n" "\275\276" | tr -d "[:cntrl:]" | tr "\275\276" "\r\n"'
where:
grep -o '".\+[^"]"' - select double-quoted string with quotes
grep -o '[^"]*[^"]' - select text within the double quotes
while read -r line - store each line into $line and do some action (help read)
printf "%b" $line - print line by expanding backslash escape sequences
tr "\r\n" "\275\276" - temporarily replace \r\n into \275\276
tr -d "[:cntrl:]" - remove all control characters
tr "\275\276" "\r\n" - restore new line endings
then the complete example to trace some command (e.g. php) can look like:
strace -e trace=read,write,recvfrom,sendto -s 1000 -fp $(pgrep -n php) 2>&1 | format-strace
Check for similar example: How to view the output of a running process in another bash session? at Unix.SE
This is driving me crazy, im trying to do some MD5 calculation based on the fritzbox SPEC for logging in. Basically you have to convert a challenge and the password into UTF-16LE and then hash it by md5, then concat challenge-md5(uft-16le(challenge-password))
To do so i'm using iconv and md5 from mac OSX in a script
echo -n "challenge-password1234" | iconv -f ISO8859-1 -t UTF-16LE | md5
Which outputs to 2f42ad272c7aec4c94f0d9525080e6de
Doing the exact thing by just pasting it in the shell outputs to 1722e126192656712a1d352e550f1317
The latter one is correct (accepted by fritzbox) the first one is wrong.
Calling the script with bash script.sh results in the proper hash, calling it with sh script.sh results in the wrong hash, which leads to the new question: How come the output is any different between sh and bash?
Different versions of echo behave in very different ways. Some take command options (like -n) that modify their behavior (including -n suppressing the trailing linefeed), and some don't. Some interpret escape sequences in the string itself (including \c at the end of the string suppressing the trailing linefeed)... and some don't. Some do both. It appears the version of echo (/bin/echo) on your system doesn't take options, and therefore treats -n as a string to be printed. If you're using bash, its builtin version overrides /bin/echo, and does interpret flags.
Basically, echo is a mess of inconsistency and portability traps. So don't use it, use printf instead. It's a little more complicated because you have to specify a format string, then the actual stuff you want printed, but it can save a ton of headaches.
$ printf "%s" "challenge-password1234" | iconv -f ISO8859-1 -t UTF-16LE | md5
1722e126192656712a1d352e550f1317
And by the way, here's what the echo command was actually printing:
$ printf "%s\n" "-n challenge-password1234" | iconv -f ISO8859-1 -t UTF-16LE | md5
2f42ad272c7aec4c94f0d9525080e6de
You are specifically asking for the difference between script and command line result. Please note that there can be other cases in which the script result will not be useable for your Fritzbox.
The sample code for the session handling in the AVM Fritzbox documentation is written in C# see
https://avm.de/fileadmin/user_upload/Global/Service/Schnittstellen/AVM_Technical_Note_-_Session_ID.pdf
from that code I derived https://github.com/WolfgangFahl/fritz-csharp-api
and added some tests:
https://github.com/WolfgangFahl/fritz-csharp-api/blob/master/fritzsimpletest.cs
which where inspired by the tests in:
https://github.com/WolfgangFahl/fritzbox-java-api/blob/master/src/test/java/com/github/kaklakariada/fritzbox/Md5ServiceTest.java
Basically there were 5 examples:
"" -> "d41d8cd98f00b204e9800998ecf8427e"
"secret", "09433e1853385270b51511571e35eeca"
"test", "c8059e2ec7419f590e79d7f1b774bfe6"
"1234567z-äbc", "9e224a41eeefa284df7bb0f26c2913e2";
"!\"§$%&/()=?ßüäöÜÄÖ-.,;:_`´+*#'<>≤|" -> "ad44a7cb10a95cb0c4d7ae90b0ff118a"
and yours is now example number 6:
"challenge-password1234" -> "1722e126192656712a1d352e550f1317"
and these behave the same in the Java and C# implementation. Now trying out these with the bash script below which has
echo -n "$l_s" | iconv --from-code ISO8859-1 --to-code UTF-16LE | md5sum -b | gawk '{print substr($0,1,32)}'
which is e.g. discussed in https://www.ip-phone-forum.de/threads/fritzbox-challenge-response-in-sh.264639/
as it's getmd5 function gives different results for the Umlaut cases e.g. in my bash on Mac OS Sierra.
There is already some debug output added.
The encoding given for 1234567z-äbc has the byte sequence 2d c3 a4 62 63 while e.g. the java implementation has 2d e4 62 63.
So beware of umlauts in your password - the fritzbox access might fail using this script solution. I am looking for a workaround and will post it here when i find it.
bash script
#!/bin/bash
# WF 2017-10-30
# Fritzbox handling
#
# get the property with the given name
# params
# 1: the property name e.g. fritzbox.url, fritzbox.username, fritzbox.password
#
getprop() {
local l_prop="$1"
cat $HOME/.fritzbox/application.properties | grep "$l_prop" | cut -f2 -d=
}
#
# get a value from the fritzbox login_sid.lua
#
getboxval() {
local l_node="$1"
local l_response="$2"
if [ "$l_response" != "" ]
then
l_data="&response=$l_response"
fi
fxml=/tmp/fxml$$
curl --insecure -s "${box_url}/login_sid.lua?username=${username}$l_response" > $fxml
cat $fxml |
gawk -v node=$l_node 'match($0,"<"node">([0-9a-f]+)</"node">",m) { print m[1] }'
cat $fxml
rm $fxml
}
#
# get the md5 for the given string
#
# see https://avm.de/fileadmin/user_upload/Global/Service/Schnittstellen/AVM_Technical_Note_-_Session_ID.pdf
#
# param
# 1: s - the string
#
# return
# md5
#
getmd5() {
local l_s="$1"
echo -n "$l_s" | iconv -f ISO8859-1 -t UTF-16LE | od -x
echo -n "$l_s" | iconv --from-code ISO8859-1 --to-code UTF-16LE | md5sum -b | gawk '{print substr($0,1,32)}'
}
# get global settings from application properties
box_url=$(getprop fritzbox.url)
username=$(getprop fritzbox.username)
password=$(getprop fritzbox.password)
# uncomment to test
getmd5 ""
# should be d41d8cd98f00b204e9800998ecf8427e
getmd5 secret
# should be 09433e1853385270b51511571e35eeca
getmd5 test
# should be c8059e2ec7419f590e79d7f1b774bfe6
getmd5 1234567z-äbc
# should be 9e224a41eeefa284df7bb0f26c2913e2
getmd5 "!\"§$%&/()=?ßüäöÜÄÖ-.,;:_\`´+*#'<>≤|"
# should be ad44a7cb10a95cb0c4d7ae90b0ff118a
exit
# Login and get SID
challenge=$(getboxval Challenge "")
echo "challenge=$challenge"
md5=$(getmd5 "${challenge}-${password}")
echo "md5=$md5"
response="${challenge}-${md5}"
echo "response=$response"
getboxval SID "$response"