I can't find the reasons why my case statement doesn't work when looking tail output.
tail -F -n1 /var/log/pihole.log |
while read input; do
echo "$input" | hexdump -C # just to physically compare the output
case $input in
cached|blacklisted|blocked)
echo "We have a match!";;
*)
echo "No match!"
esac
done
This always returns No match!, even if the strings are in the $input.
:~ $ ./pihole_test.sh
00000000 4a 61 6e 20 31 20 31 31 3a 35 35 3a 35 38 20 64 |Jan 1 11:55:58 d|
00000010 6e 73 6d 61 73 71 5b 36 39 36 5d 3a 20 65 78 61 |nsmasq[696]: exa|
00000020 63 74 6c 79 20 62 6c 61 63 6b 6c 69 73 74 65 64 |ctly blacklisted|
00000030 20 70 6c 61 79 2e 67 6f 6f 67 6c 65 2e 63 6f 6d | play.google.com|
00000040 20 69 73 20 30 2e 30 2e 30 2e 30 0a | is 0.0.0.0.|
0000004c
No match!
Replace
cached|blacklisted|blocked)
with
*cached*|*blacklisted*|*blocked*)
to match substrings.
Related
The hexadecimal file is from the charles proxy's hex format, and it may contains invisible characters, the example format content is:
00000000 7b 22 73 75 70 70 6f 72 74 73 5f 69 6d 70 6c 69 {"supports_impli
00000010 63 69 74 5f 73 64 6b 5f 6c 6f 67 67 69 6e 67 22 cit_sdk_logging"
00000020 3a 74 72 75 65 2c 22 67 64 70 76 34 5f 6e 75 78 :true,"gdpv4_nux
00000030 5f 65 6e 61 62 6c 65 64 22 3a 66 61 6c 73 65 2c _enabled":false,
00000040 22 61 6e 64 72 6f 69 64 5f 73 64 6b 5f 65 72 72 "android_sdk_err
00000050 6f 72 5f 63 61 74 65 67 6f 72 69 65 73 22 3a 5b or_categories":[
00000060 7b 22 6e 61 6d 65 22 3a 22 6c 6f 67 69 6e 5f 72 {"name":"login_r
00000070 65 63 6f 76 65 72 61 62 6c 65 22 2c 22 69 74 65 ecoverable","ite
00000080 6d 73 22 3a 5b 7b 22 63 6f 64 65 22 3a 31 30 32 ms":[{"code":102
00000090 7d 2c 7b 22 63 6f 64 65 22 3a 31 39 30 7d 5d 2c },{"code":190}],
000000a0 22 72 65 63 6f 76 65 72 79 5f 6d 65 73 73 61 67 "recovery_messag
000000b0 65 22 3a 22 5c 75 38 62 66 37 5c 75 39 31 63 64 e":"\u8bf7\u91cd
000000c0 5c 75 36 35 62 30 5c 75 37 36 37 62 5c 75 35 66 \u65b0\u767b\u5f
000000d0 35 35 5c 75 35 65 39 34 5c 75 37 35 32 38 5c 75 55\u5e94\u7528\u
000000e0 37 61 30 62 5c 75 35 65 38 66 5c 75 66 66 30 63 7a0b\u5e8f\uff0c
000000f0 5c 75 35 31 38 64 5c 75 36 62 32 31 5c 75 38 66 \u518d\u6b21\u8f
00000100 64 65 5c 75 36 33 61 35 20 46 61 63 65 62 6f 6f de\u63a5 Faceboo
00000110 6b 20 5c 75 35 65 31 30 5c 75 36 32 33 37 5c 75 k \u5e10\u6237\u
00000120 33 30 30 32 22 7d 5d 2c 22 61 70 70 5f 65 76 65 3002"}],"app_eve
00000130 6e 74 73 5f 73 65 73 73 69 6f 6e 5f 74 69 6d 65 nts_session_time
00000140 6f 75 74 22 3a 36 30 2c 22 61 70 70 5f 65 76 65 out":60,"app_eve
00000150 6e 74 73 5f 66 65 61 74 75 72 65 5f 62 69 74 6d nts_feature_bitm
00000160 61 73 6b 22 3a 36 35 35 35 39 2c 22 73 65 61 6d ask":65559,"seam
00000170 6c 65 73 73 5f 6c 6f 67 69 6e 22 3a 31 2c 22 73 less_login":1,"s
00000180 6d 61 72 74 5f 6c 6f 67 69 6e 5f 62 6f 6f 6b 6d mart_login_bookm
00000190 61 72 6b 5f 69 63 6f 6e 5f 75 72 6c 22 3a 22 68 ark_icon_url":"h
000001a0 74 74 70 73 3a 5c 2f 5c 2f 73 74 61 74 69 63 2e ttps:\/\/static.
000001b0 78 78 2e 66 62 63 64 6e 2e 6e 65 74 5c 2f 72 73 xx.fbcdn.net\/rs
000001c0 72 63 2e 70 68 70 5c 2f 76 33 5c 2f 79 73 5c 2f rc.php\/v3\/ys\/
000001d0 72 5c 2f 43 36 5a 75 74 59 44 53 61 61 56 2e 70 r\/C6ZutYDSaaV.p
000001e0 6e 67 22 2c 22 73 6d 61 72 74 5f 6c 6f 67 69 6e ng","smart_login
000001f0 5f 6d 65 6e 75 5f 69 63 6f 6e 5f 75 72 6c 22 3a _menu_icon_url":
00000200 22 68 74 74 70 73 3a 5c 2f 5c 2f 73 74 61 74 69 "https:\/\/stati
00000210 63 2e 78 78 2e 66 62 63 64 6e 2e 6e 65 74 5c 2f c.xx.fbcdn.net\/
00000220 72 73 72 63 2e 70 68 70 5c 2f 76 33 5c 2f 79 73 rsrc.php\/v3\/ys
00000230 5c 2f 72 5c 2f 30 69 61 72 70 6e 77 64 6d 45 78 \/r\/0iarpnwdmEx
00000240 2e 70 6e 67 22 2c 22 72 65 73 74 72 69 63 74 69 .png","restricti
00000250 76 65 5f 64 61 74 61 5f 66 69 6c 74 65 72 5f 70 ve_data_filter_p
00000260 61 72 61 6d 73 22 3a 22 7b 7d 22 2c 22 61 61 6d arams":"{}","aam
00000270 5f 72 75 6c 65 73 22 3a 22 7b 7d 22 2c 22 73 75 _rules":"{}","su
00000280 67 67 65 73 74 65 64 5f 65 76 65 6e 74 73 5f 73 ggested_events_s
00000290 65 74 74 69 6e 67 22 3a 22 7b 5c 22 70 72 6f 64 etting":"{\"prod
000002a0 75 63 74 69 6f 6e 5f 65 76 65 6e 74 73 5c 22 3a uction_events\":
000002b0 5b 5d 2c 5c 22 65 6c 69 67 69 62 6c 65 5f 66 6f [],\"eligible_fo
000002c0 72 5f 70 72 65 64 69 63 74 69 6f 6e 5f 65 76 65 r_prediction_eve
000002d0 6e 74 73 5c 22 3a 5b 5c 22 66 62 5f 6d 6f 62 69 nts\":[\"fb_mobi
000002e0 6c 65 5f 61 64 64 5f 74 6f 5f 63 61 72 74 5c 22 le_add_to_cart\"
000002f0 2c 5c 22 66 62 5f 6d 6f 62 69 6c 65 5f 70 75 72 ,\"fb_mobile_pur
00000300 63 68 61 73 65 5c 22 2c 5c 22 66 62 5f 6d 6f 62 chase\",\"fb_mob
00000310 69 6c 65 5f 63 6f 6d 70 6c 65 74 65 5f 72 65 67 ile_complete_reg
00000320 69 73 74 72 61 74 69 6f 6e 5c 22 2c 5c 22 66 62 istration\",\"fb
00000330 5f 6d 6f 62 69 6c 65 5f 69 6e 69 74 69 61 74 65 _mobile_initiate
00000340 64 5f 63 68 65 63 6b 6f 75 74 5c 22 5d 7d 22 2c d_checkout\"]}",
00000350 22 69 64 22 3a 22 31 36 33 35 34 33 35 31 34 39 "id":"1635435149
00000360 30 39 30 34 35 22 7d 09045"}
Read the file and convert the hexadecimal content to golang slice []byte, is there any convenient method? how to split the lines in the file?
I have solved it by using fixed length of the hexadecimal content. the offset length of each line is 8, so the hexadecimal content begins at offset 8, and the max length of the hexadecimal is 57. The code is as follows:
const (
beginOffset = 8 // the offset of the hexadecimal content begins
endLength = 57 // the length of the hexadecimal content ends
)
func ReadHexData(src []byte) ([]byte, error) {
lines := strings.Split(string(src), "\n")
var data string
for _, line := range lines {
for i := beginOffset; i < endLength; i++ {
if line[i] != ' ' {
data = data + string(line[i])
}
}
}
return hex.DecodeString(data)
}
I have a csv file with text and numbers.
If a number is bigger than 1000, formatted like this: 1 000,
so it has a space as thousand separator, but it is not space. I tried to sed it, and it worked where real space was, but not in this format.
It is also not TAB, I removed all the TABs with "expand -t 1".
The following is a line that demonstrates the issue:
x17_Provident_GDN_REMARKETING_provident.hu_listák;Display_Hálózat;Szeged;2021-03-09;Kedd;Mobil;HUF;1 736;9;130.83;0.00
In penultimate row, in column 8: 1 736
is the problem.
And running this: grep -E -m 1 -e '[;]1[^;]+736[;]' <yourfile.csv | hexdump -C
gives:
00000000 78 31 37 5f 50 72 6f 76 69 64 65 6e 74 5f 47 44 |x17_Provident_GD|
00000010 4e 5f 52 45 4d 41 52 4b 45 54 49 4e 47 5f 70 72 |N_REMARKETING_pr|
00000020 6f 76 69 64 65 6e 74 2e 68 75 5f 6c 69 73 74 c3 |ovident.hu_list.|
00000030 a1 6b 3b 44 69 73 70 6c 61 79 5f 48 c3 a1 6c c3 |.k;Display_H..l.|
00000040 b3 7a 61 74 3b 53 7a 65 67 65 64 3b 32 30 32 31 |.zat;Szeged;2021|
00000050 2d 30 33 2d 30 39 3b 4b 65 64 64 3b 4d 6f 62 69 |-03-09;Kedd;Mobi|
00000060 6c 3b 48 55 46 3b 31 c2 a0 37 33 36 3b 39 3b 31 |l;HUF;1..736;9;1|
00000070 33 30 2e 38 33 3b 30 2e 30 30 0a |30.83;0.00.|
0000007b
It's a 2 byte, UTF-8 encoded non breaking space - c2 a0.
You can use perl to safely remove it.
perl -pe 's/\xc2\xa0//g' dirty.csv > clean.csv
After we know it is No break space, I simply sed it on mac with entry method:
opt+space
cat test4.csv | sed 's/ //g'
Similar to perl, you can use GNU sed with LC_ALL=C:
LC_ALL=C sed 's/\xc2\xa0//g'
This question already has answers here:
How to convert DOS/Windows newline (CRLF) to Unix newline (LF)
(23 answers)
Are shell scripts sensitive to encoding and line endings?
(14 answers)
Closed 5 years ago.
I have this code but doesn't work.
line="20170425"
anycopia=${line:0:4}
mescopia=${line:4:2}
diacopia=${line:6:2}
echo $anycopia
echo $mescopia
echo $diacopia
DATE=$(date +%Y%m%d)
any=${DATE:0:4}
mes=${DATE:4:2}
dia=${DATE:6:2}
echo $any
echo $mes
echo $dia
if [ $anycopia == $any ]; then
echo "equals"
else
echo "not equals"
fi
Error:
syntax error near unexpected token fi
I've tried changing "then" but it doesn't matter, just like this:
if [ $anycopia == $any ]
then
echo "equals"
else
echo "not equals"
fi
And same error going on all over the time.
PD: Other answers in Stack Overflow with same question didn't work for me.
Edit:
I did this command:
hexdump -C script.sh
This is the output:
00000000 6c 69 6e 65 3d 22 32 30 31 37 30 34 32 35 22 0d |line="20170425".|
00000010 0a 61 6e 79 63 6f 70 69 61 3d 24 7b 6c 69 6e 65 |.anycopia=${line|
00000020 3a 30 3a 34 7d 0d 0a 6d 65 73 63 6f 70 69 61 3d |:0:4}..mescopia=|
00000030 24 7b 6c 69 6e 65 3a 34 3a 32 7d 0d 0a 64 69 61 |${line:4:2}..dia|
00000040 63 6f 70 69 61 3d 24 7b 6c 69 6e 65 3a 36 3a 32 |copia=${line:6:2|
00000050 7d 0d 0a 65 63 68 6f 20 24 61 6e 79 63 6f 70 69 |}..echo $anycopi|
00000060 61 0d 0a 65 63 68 6f 20 24 6d 65 73 63 6f 70 69 |a..echo $mescopi|
00000070 61 0d 0a 65 63 68 6f 20 24 64 69 61 63 6f 70 69 |a..echo $diacopi|
00000080 61 0d 0a 44 41 54 45 3d 24 28 64 61 74 65 20 2b |a..DATE=$(date +|
00000090 25 59 25 6d 25 64 29 0d 0a 61 6e 79 3d 24 7b 44 |%Y%m%d)..any=${D|
000000a0 41 54 45 3a 30 3a 34 7d 0d 0a 6d 65 73 3d 24 7b |ATE:0:4}..mes=${|
000000b0 44 41 54 45 3a 34 3a 32 7d 0d 0a 64 69 61 3d 24 |DATE:4:2}..dia=$|
000000c0 7b 44 41 54 45 3a 36 3a 32 7d 0d 0a 65 63 68 6f |{DATE:6:2}..echo|
000000d0 20 24 61 6e 79 0d 0a 65 63 68 6f 20 24 6d 65 73 | $any..echo $mes|
000000e0 0d 0a 65 63 68 6f 20 24 64 69 61 0d 0a 69 66 20 |..echo $dia..if |
000000f0 5b 20 24 61 6e 79 63 6f 70 69 61 20 3d 3d 20 24 |[ $anycopia == $|
00000100 61 6e 79 20 5d 3b 20 74 68 65 6e 0d 0a 20 20 20 |any ]; then.. |
00000110 20 65 63 68 6f 20 22 68 6f 6c 61 22 0d 0a 65 6c | echo "hola"..el|
00000120 73 65 0d 0a 20 20 20 20 65 63 68 6f 20 22 61 64 |se.. echo "ad|
00000130 65 75 22 0d 0a 66 69 0d 0a |eu"..fi..|
00000139
PDD: I'm running this with Bash on Ubuntu on Windows.
Edit2:
user#DESKTOP-UO9KRO4:/mnt/d$ cat -v script.sh
line="20170425"^M
anycopia=${line:0:4}^M
mescopia=${line:4:2}^M
diacopia=${line:6:2}^M
echo $anycopia^M
echo $mescopia^M
echo $diacopia^M
DATE=$(date +%Y%m%d)^M
any=${DATE:0:4}^M
mes=${DATE:4:2}^M
dia=${DATE:6:2}^M
echo $any^M
echo $mes^M
echo $dia^M
if [ $anycopia == $any ]; then^M
echo "hola"^M
else^M
echo "adeu"^M
fi^M
^M is a carriage return, and is commonly seen when files are copied from Windows. Run dos2unix to clean up those meta-characters.
dos2unix script.sh
Also as a safe coding practice,
Always double-quote your variables to not let them split when they contains spaces or any shell meta characters
Define a proper she-bang i.e. the interpreter using which the script should run. (Most cases if bash is available #!/usr/bin/env bash or #!/bin/bash)
I am using Net::Telnet to connect to a HP ProCurve Switch to login and backup the config. However I ran into issues where waitfor returns VT100/ANSI control chars:
< 0x00000: ff fd 18 ff fd 1f ff fb 01 1b 5b 32 4a 1b 5b 3f ..........[2J.[?
< 0x00010: 37 6c 1b 5b 33 3b 32 33 72 1b 5b 3f 36 6c 1b 5b 7l.[3;23r.[?6l.[
< 0x00020: 31 3b 31 48 1b 5b 3f 32 35 6c 1b 5b 31 3b 31 48 1;1H.[?25l.[1;1H
< 0x00030: 48 50 20 4a 39 37 32 38 41 20 32 39 32 30 2d 34 HP J9728A 2920-4
< 0x00040: 38 47 20 53 77 69 74 63 68 0d 0d 0a 53 6f 66 74 8G Switch...Soft
< 0x00050: 77 61 72 65 20 72 65 76 69 73 69 6f 6e 20 57 42 ware revision WB
< 0x00060: 2e 31 35 2e 31 32 2e 30 30 31 35 0d 0d 0a 0d 0d .15.12.0015.....
< 0x00070: 0a 43 6f 70 79 72 69 67 68 74 20 28 43 29 20 31 .Copyright (C) 1
< 0x00080: 39 39 31 2d 32 30 31 34 20 48 65 77 6c 65 74 74 991-2014 Hewlett
< 0x00090: 2d 50 61 63 6b 61 72 64 20 44 65 76 65 6c 6f 70 -Packard Develop
< 0x000a0: 6d 65 6e 74 20 43 6f 6d 70 61 6e 79 2c 20 4c 2e ment Company, L.
< 0x000b0: 50 2e 0d 0a 0d 0a 20 20 20 20 20 20 20 20 20 20 P.....
Unfortunately, this screws up waitfor because if I try to waitfor(/^password:/i) it will return a string with those control chars in it, or wait forever since the regex is never matched.
Is there any way to have Net::Telnet automatically remove those control characters? Is there any way to have waitfor only care about ASCII printable characters?
I'm implementing a small web server into a wifi micro. To aid in development and test, I have ported it to Windows console program.
I use chunked transfer processing. The following is what shows up on the browser:
0059
Hello World
0
The 59 is the hex size of the chunk and the 0 is the chunked terminating size
This is the data captured via wireshark:
This is the first message I send which are the headers
0000 48 54 54 50 2f 31 2e 31 20 32 30 30 20 4f 4b 0d HTTP/1.1 200 OK.
0010 0a 53 65 72 76 65 72 3a 20 54 72 61 6e 73 66 65 .Server: Transfe
0020 72 2d 45 6e 63 6f 64 69 6e 67 3a 20 63 68 75 6e r-Encoding: chun
0030 6b 65 64 0d 0a 43 6f 6e 74 65 6e 74 2d 54 79 70 ked..Content-Typ
0040 65 3a 20 74 65 78 74 2f 68 74 6d 6c 0d 0a 43 61 e: text/html..Ca
0050 63 68 65 2d 43 6f 6e 74 72 6f 6c 3a 20 6d 61 78 che-Control: max
0060 2d 61 67 65 3d 33 36 30 30 2c 20 6d 75 73 74 2d -age=3600, must-
0070 72 65 76 61 6c 69 64 61 74 65 0d 0a 0d 0a revalidate....
The next block is the chunked data
0000 30 30 35 39 0d 0a 3c 68 74 6d 6c 3e 0a 3c 68 65 0059..<html>.<he
0010 61 64 3e 3c 74 69 74 6c 65 3e 57 65 62 20 53 65 ad><title>Web Se
0020 72 76 65 72 3c 2f 74 69 74 6c 65 3e 0a 3c 2f 68 rver</title>.</h
0030 65 61 64 3e 0a 3c 62 6f 64 79 3e 0a 3c 68 31 3e ead>.<body>.<h1>
0040 48 65 6c 6c 6f 20 57 6f 72 6c 64 3c 2f 68 31 3e Hello World</h1>
0050 0a 3c 2f 62 6f 64 79 3e 3c 2f 68 74 6d 6c 3e 0d .</body></html>.
0060 0a 30 0d 0a 0d 0a .0....
The chunked values are being displayed on both Chrome and IE.
Can anyone see an issue with my data that would cause the issue.
Thanks
Solved:
I mistakenly remove the server name so now the browser is taking the transfer encoding as the server name and does not understand the chunked message size -- it thinks its just data to display.