I have file like this:
[some_lines_unimportant]
#Field List (hex, dec, bin, char)
00 00 1e 14 0 0 30 20 999909999 000090000 0001-1110
(...)
00 00 3e 01 0 0 62 01 888888888 000088000 0100-1000
[some_lines_unimportant]
I would like to be able to extract f.e 3 lines that are under the #Field list ... line. How can I do that in bash.
EDITED:
this is the file:
# ------------------------------------------------------------------------------------
# ======= Pkt to be transmitted (hex) ================================================
01 00 fe 03 4d 00 00 00 00 02 fe 00 aa 00 00 00 00 00 00 00 04 b8 EOP
# Initiating transfer ...
# ======= Received pkt (hex) =========================================================
02 fe 03 0d 00 fe 00 aa 00 00 00 04 84 00 01 00 10 cc EOP
# ========= Reply pkt fields (hex) ===================================================
Reply SpW Addr:
02
# Initiator Log Addr (1 byte), Proto ID (1 byte), Instr (1 byte), Status (1 byte):
fe 03 0d 00
# Target Log Addr (1 byte), Trans ID (2 bytes MS LS), Reserved (1 byte):
fe 00 aa 00
# Data Length (3 bytes MS to LS), Header CRC (1 byte):
00 00 04 84
# Field List (hex, dec, bin, char, [dec line count]):
00 01 00 10 0 1 0 16 0000-0000 0000-0001 0000-0000 0001-0000 . . . . [ 1]
# Data CRC (1 byte):
cc
# End of Pkt character:
EOP
# ------------------------------------------------------------------------------------
# ######### Transaction time: 0.225 ms
######### End of online test #########################################################
Use grep like this, where -A specifies how many lines AFTER the match:
grep -A 3 "^#Field List" file
#Field List (hex, dec, bin, char)
00 00 1e 14 0 0 30 20 999909999 000090000 0001-1110
(...)
00 00 3e 01 0 0 62 01 888888888 000088000 0100-1000
Or, if you don't want the first line that matches
grep -A 3 "^#Field List" file | sed 1d
00 00 1e 14 0 0 30 20 999909999 000090000 0001-1110
(...)
00 00 3e 01 0 0 62 01 888888888 000088000 0100-1000
The circumflex (or hat, ^) matches the start of the line, i.e. the word #Field must be at the very start of the line for it to be matched.
Use grep with the --after-context/-A argument:
grep -A3 '#Field List'
Related
I have an image with following IFD entries
01 00 00 03 00 00 00 01 09 A0 00 00
01 01 00 03 00 00 00 01 0C B0 00 00
.
.
.
01 0D 00 02 00 00 00 0E 00 00 DF A2
.
.
Here
Tag ID - 010D (which is document name)
Tag type - 0002 (ASCII)
Count -0000000E (14)
Offset address - 0000 DFA2
Similarly, what does last 4 bytes indicates (Value or offset) in IFD entry of 0100 (Image width) and 0101 (Image length)?
This is HaraldK’s comment, which seems to have solved OP’s problem:
You will find this explained in the TIFF 6.0 specification. For the Width/Length cases, the value is a short (0x0003), and the count is 1 (2 * 1 < 4), thus the value is written "in-line". Your image is 0x9a0 * 0xcb0, or 2464 * 3248 pixels. For the DocumentName tag, the value is ASCII, the count is 14, and as this is > 4 bytes (1 * 14 > 4), the following value is an offset, and the real tag value can be found at that offset (0xdfa2).
I have a text file which contains NUL characters in random rows. I want to find first NUL character and delete entire row from that NUL character as in below:
Input:
1 2 3 4 20170821NUL20170821NULNULNULNUL 123 NULNULNUL
1 2 3 4 20170821 20170821 6 7 10 123 10 11 13
1 2 3 4 20170821NUL20170821NULNULNULNUL 123 NULNULNUL
1 2 3 4 20170821NUL20170821NULNULNULNUL 123 NULNULNUL
Output:
1 2 3 4 20170821
1 2 3 4 20170821 20170821 6 7 10 123 10 11 13
1 2 3 4 20170821
1 2 3 4 20170821
I have the following to read text file data to a variable and loop through the data and replace NUL:
sInfile = WScript.Arguments(1)
'Create file system object
Set oFSO = CreateObject("Scripting.FileSystemObject")
Set oFS = oFSO.OpenTextFile(sInfile)
sData = oFS.ReadAll
oFS.Close
Set oFS = Nothing
MsgBox("File Read Completed")
'Remove Rest of the line from NULL
Do While InStr(sData, "\00.*") > 0
sData = Replace(sData, "\00.*", "")
Loop
'Cleanup and end
Set oFS = Nothing
WScript.Quit
The script went passed without any errors but I can't see any changes to the data.
EDIT 1:
Updated code:
Const ForReading = 1
Const ForWriting = 2
Const TriStateUseDefault = -2
If (WScript.Arguments.Count > 0) Then
sInfile = WScript.Arguments(0)
Else
WScript.Echo "No filename specified."
WScript.Quit
End If
If (WScript.Arguments.Count > 1) Then
sOutfile = WScript.Arguments(1)
Else
sOutfile = sInfile
End If
'Get the text file from cmd file
sInfile = Wscript.Arguments(1)
' Create file system object
Set oFSO = CreateObject("Scripting.FileSystemObject")
Set oFS = oFSO.OpenTextFile(sInfile)
sData = oFS.ReadAll
oFS.Close
Set oFS = Nothing
' Remove Rest of the line from NULL
Set re = New RegExp
re.Pattern = Chr(0) & ".*"
re.Global = True
sData = re.Replace(sData, "")
Set oOutfile = oFSO.OpenTextFile(sOutfile, ForWriting, True)
oOutfile.Write(sData)
oOutfile.Close
Set oOutfile = Nothing
' Cleanup and end
Set oFS = Nothing
WScript.Quit
Here is the sample input I am giving:
I would like to see the output as below:
But I got the below output:
ਊਊਊਊਊਊਊਊਊਊ
EDIT 2:
I am not aware of hex editors. Here is the sample input of HextDump:
FF FE 4A 00 42 00 43 00 09 00 31 00 32 00 33 00 34 00 38 00 36 00 37 00 38
00 09 00 38 00 37 00 09 00 30 00 09 00 30 00 09 00 31 00 32 00 33 00 09 00
32 00 30 00 31 00 37 00 09 00 31 00 32 00 33 00 34 00 09 00 31 00 33 00 34
00 32 00 30 00 09 00 32 00 30 00 31 00 37 00 30 00 38 00 30 00 39 00 09 00
35 00 31 00 30 00 33 00 09 00 09 00 09 00 09 00 33 00 34 00 31 00 34 00 38
00 38 00 09 00 32 00 09 00 32 00 30 00 31 00 37 00 09 00 38 00 09 00 31 00
09 00 37 00 09 00 2D 00 32 00 36 00 34 00 30 00 09 00 2D 00 33 00 39 00 33
00 2E 00 31 00 36 00 31 00 33 00 37 00 35 00 09 00 2D 00 33 00 33 00 32 00
2E 00 34 00 36 00 38 00 35 00 37 00 39 00 09 00 41 00 30 00 31 00 31 00 32
00 35 00 38 00 39 00 2F 00 33 00 34 00 31 00 34 00 38 00 38 00 2F 00 09 00
09 00 09 00 09 00 09 00 09 00 09 00 09 00 32 00 09 00 09 00 09 00 32 00 31
00 37 00 38 00 31 00 09 00 58 00 59 00 5A 00 09 00 58 00 59 00 5A 00 09 00
58 00 59 00 5A 00 09 00 31 00 32 00 33 00 09 00 31 00 32 00 33 00 09 00 2D
00 32 00 36 00 34 00 09 00 58 00 59 00 5A 00 09 00 31 00 09 00 31 00 09 00
31 00 32 00 33 00 09 00 09 00 09 00 32 00 31 00 37 00 38 00 32 00 31 00 0D
00 0A 00 41 00 42 00 43 00 09 00 31 00 32 00 33 00 34 00 38 00 36 00 37 00
and the HexDump of output which I got FF FE 4A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A 0A
You are trying to specify regex pattern for Replace() function, that won't work. Generally, you don't need to use regex at all.
Here is non-regex code:
With CreateObject("Scripting.FileSystemObject").OpenTextFile(WScript.Arguments(1), 1, False, 0)
sData = ""
If Not .AtEndOfStream Then sData = .ReadAll
.Close
End With
a = Split(sData, vbCrLf)
For i = 0 To UBound(a)
q = Instr(a(i), Chr(0))
If q > 0 Then a(i) = Mid(a(i), 1, q - 1)
Next
sData = Join(a, vbCrLf)
And here is regex version:
With CreateObject("Scripting.FileSystemObject").OpenTextFile(WScript.Arguments(1), 1, False, 0)
sData = ""
If Not .AtEndOfStream Then sData = .ReadAll
.Close
End With
With CreateObject("VBScript.RegExp")
.Pattern = "^(.*?)\x00.*$"
.Global = True
.Multiline = True
sData = .Replace(sData, "$1")
End With
The Replace function doesn't do regular expression replacements, and VBScript also doesn't recognize \0 as the character NUL. For the former you need the Replace method of a regular expression object, for the latter you need the Chr function. Also, you don't need a loop, since you read the content of the file as a single string anyway.
However, your file is apparently UTF-16 LE encoded, which means that each character is represented by 2 bytes, one of which is zero for ANSI characters. If you read such files as ANSI files your replacement would remove everything after the first byte. You need to set the 4th parameter of the OpenTextFile method to -1 in order to handle the file as a UTF-16 (vulgo Unicode) file.
Change this:
Set oFS = oFSO.OpenTextFile(sInfile)
sData = oFS.ReadAll
oFS.Close
Set oFS = Nothing
...
Do While InStr(sData, "\00.*") > 0
sData = Replace(sData, "\00.*", "")
Loop
...
Set oOutfile = oFSO.OpenTextFile(sOutfile, ForWriting, True)
oOutfile.Write(sData)
oOutfile.Close
Set oOutfile = Nothing
into this:
sData = oFSO.OpenTextFile(sInfile, 1, False, -1).ReadAll
Set re = New RegExp
re.Pattern = Chr(0) & "[^\r\n]*"
re.Global = True
sData = re.Replace(sData, "")
oFSO.OpenTextFile(sOutfile, 2, True, -1).Write sData
and the problem will disappear.
The pattern [^\r\n]* (any number of characters that are neither carriage-return nor line-feed) is used to keep Windows line breaks intact. Those consist of the two characters carriage-return and line-feed (CR-LF). The regular expression meta-character . does not match line-feeds, but it does match carriage-return, so those would be removed when using the pattern .*.
For clarity: the above code will remove a NUL character and the remainder of the line from each line containing a NUL character. Lines not containing NUL characters will not be affected.
If you want the entire text after a NUL character removed (including subsequent lines) you could do it like this:
Set re = New RegExp
re.Pattern = Chr(0) & "[\s\S]*"
sData = re.Replace(sData, "")
I've just now installed Visual Leak Detector (2.3) on Windows 8. I tested it with blank CRT program (in Visual Studio 2012) that does nothing.
#include <vld.h>
int main(int argc, char** argv)
{
return 0;
}
When I run it VLD reports strange leaks in vc++ crt module:
Visual Leak Detector Version 2.3 installed.
WARNING: Visual Leak Detector detected memory leaks!
---------- Block 31 at 0x0000000052C07530: 70 bytes ----------
Call Stack:
f:\dd\vctools\crt_bld\self_64_amd64\crt\src\stdenvp.c (127): my_application.exe!_setenvp + 0x27 bytes
f:\dd\vctools\crt_bld\self_64_amd64\crt\src\crt0.c (223): my_application.exe!__tmainCRTStartup + 0x5 bytes
f:\dd\vctools\crt_bld\self_64_amd64\crt\src\crt0.c (164): my_application.exe!mainCRTStartup
0x00000000FAF8167E (File and line number not available): KERNEL32.DLL!BaseThreadInitThunk + 0x1A bytes
0x00000000FD8CC3F1 (File and line number not available): ntdll.dll!RtlUserThreadStart + 0x21 bytes
Data:
20 B5 C0 52 50 00 00 00 50 92 C0 52 50 00 00 00 ...RP... P..RP...
20 91 DD E1 F6 07 00 00 7E 00 00 00 02 00 00 00 ........ ~.......
12 00 00 00 00 00 00 00 1F 00 00 00 FD FD FD FD ........ ........
50 52 4F 43 45 53 53 4F 52 5F 4C 45 56 45 4C 3D PROCESSO R_LEVEL=
36 00 FD FD FD FD 6....... ........
---------- Block 40 at 0x0000000052C075D0: 72 bytes ----------
Call Stack:
f:\dd\vctools\crt_bld\self_64_amd64\crt\src\stdenvp.c (127): my_application.exe!_setenvp + 0x27 bytes
f:\dd\vctools\crt_bld\self_64_amd64\crt\src\crt0.c (223): my_application.exe!__tmainCRTStartup + 0x5 bytes
f:\dd\vctools\crt_bld\self_64_amd64\crt\src\crt0.c (164): my_application.exe!mainCRTStartup
0x00000000FAF8167E (File and line number not available): KERNEL32.DLL!BaseThreadInitThunk + 0x1A bytes
0x00000000FD8CC3F1 (File and line number not available): ntdll.dll!RtlUserThreadStart + 0x21 bytes
Data:
F0 94 C0 52 50 00 00 00 20 76 C0 52 50 00 00 00 ...RP... .v.RP...
20 91 DD E1 F6 07 00 00 7E 00 00 00 02 00 00 00 ........ ~.......
14 00 00 00 00 00 00 00 28 00 00 00 FD FD FD FD ........ (.......
53 45 53 53 49 4F 4E 4E 41 4D 45 3D 43 6F 6E 73 SESSIONN AME=Cons
6F 6C 65 00 FD FD FD FD ole..... ........
---------- Block 41 at 0x0000000052C07620: 67 bytes ----------
Call Stack:
f:\dd\vctools\crt_bld\self_64_amd64\crt\src\stdenvp.c (127): my_application.exe!_setenvp + 0x27 bytes
f:\dd\vctools\crt_bld\self_64_amd64\crt\src\crt0.c (223): my_application.exe!__tmainCRTStartup + 0x5 bytes
f:\dd\vctools\crt_bld\self_64_amd64\crt\src\crt0.c (164): my_application.exe!mainCRTStartup
0x00000000FAF8167E (File and line number not available): KERNEL32.DLL!BaseThreadInitThunk + 0x1A bytes
0x00000000FD8CC3F1 (File and line number not available): ntdll.dll!RtlUserThreadStart + 0x21 bytes
Data:
D0 75 C0 52 50 00 00 00 D0 96 C0 52 50 00 00 00 .u.RP... ...RP...
20 91 DD E1 F6 07 00 00 7E 00 00 00 02 00 00 00 ........ ~.......
0F 00 00 00 00 00 00 00 29 00 00 00 FD FD FD FD ........ ).......
53 79 73 74 65 6D 44 72 69 76 65 3D 43 3A 00 FD SystemDr ive=C:..
FD FD FD ........ ........
---------- Block 43 at 0x0000000052C07670: 65 bytes ----------
Call Stack:
f:\dd\vctools\crt_bld\self_64_amd64\crt\src\stdenvp.c (127): my_application.exe!_setenvp + 0x27 bytes
f:\dd\vctools\crt_bld\self_64_amd64\crt\src\crt0.c (223): my_application.exe!__tmainCRTStartup + 0x5 bytes
f:\dd\vctools\crt_bld\self_64_amd64\crt\src\crt0.c (164): my_application.exe!mainCRTStartup
0x00000000FAF8167E (File and line number not available): KERNEL32.DLL!BaseThreadInitThunk + 0x1A bytes
0x00000000FD8CC3F1 (File and line number not available): ntdll.dll!RtlUserThreadStart + 0x21 bytes
Data:
D0 96 C0 52 50 00 00 00 C0 76 C0 52 50 00 00 00 ...RP... .v.RP...
20 91 DD E1 F6 07 00 00 7E 00 00 00 02 00 00 00 ........ ~.......
0D 00 00 00 00 00 00 00 2B 00 00 00 FD FD FD FD ........ +.......
54 45 4D 50 3D 46 3A 5C 54 45 4D 50 00 FD FD FD TEMP=F:\ TEMP....
FD ........ ........
Visual Leak Detector detected 48 memory leaks (6044 bytes).
Largest number used: 15094 bytes.
Total allocations: 25276 bytes.
Visual Leak Detector is now exiting.
There are not much details about this on net however in this msdn forum a comment says:
That's not really a 'leak' so much as 'preparing your environment for
execution'. It's making a writable copy of the process's environment
for programs which expect it, and they will be released when the
process exits. You can safely ignore the report.
However, I want to suppress these lines from appearing it to report (If they genuinely are not leaks)
Has anyone experienced this and know how to sort out?
Easiest solution apparently is to add StartDisabled=yes in the vld.ini file and then explicitly enable it from the first line of main() . Sure, you'll also miss "memory leaks" from your global objects but that's usually equally harmless.
I found this was bug in VLD 2.3. Details of bug are here. I just downloaded v2.4rc2 (vld-2.4rc2-setup.exe) and this issue not seems appearing anymore. (I had downloaded v2.3 because it has been marked stable)
This is the hexdump of a black 1x1 PNG made in Gimp and exported with minimal information:
89 50 4E 47 0D 0A 1A 0A 00 00 00 0D 49 48 44 52
00 00 00 01 00 00 00 01 08 02 00 00 00 90 77 53
DE 00 00 00 0C 49 44 41 54 08 D7 63 60 60 60 00
00 00 04 00 01 27 34 27 0A 00 00 00 00 49 45 4E
44 AE 42 60 82
Now after reading the specification I am quite sure what most of them mean, except for bytes 30-34 between the IHDR and IDAT chunk: 90 77 53 DE
Can someone enlighten me?
Those numbers are the CRC checksum for the previous chunk. See in the official specification: 5 Datastream structure for a general overview, and in particular 5.3 Chunk layout.
A CRC is calculated for, and appended to each separate chunk:
A four-byte CRC (Cyclic Redundancy Code) calculated on the preceding bytes in the chunk, including the chunk type field and chunk data fields, but not including the length field. The CRC can be used to check for corruption of the data. The CRC is always present, even for chunks containing no data.
Here is your 1x1 pixel image, annotated byte for byte. Right after the data of each of the chunks IHDR, IDAT, and IEND is a CRC for the preceding data.
File: test.png
89 50 4E 47 0D 0A 1A 0A
Header 0x89 "PNG" CR LF ^Z LF checks out okay
===========
00 00 00 0D
49 48 44 52
00 00 00 01 00 00 00 01 08 02 00 00 00
90 77 53 DE
block: "IHDR", 13 bytes [49484452]
Width: 1
Height: 1
Bit depth: 8
Color type: 2 = Color
(Bits per pixel: 8)
(Bytes per pixel: 3)
Compression method: 0
Filter method: 0
Interlace method: 0 (none)
CRC: 907753DE
===========
00 00 00 0C
49 44 41 54
08 D7 63 60 60 60 00 00 00 04 00 01
27 34 27 0A
block: "IDAT", 12 bytes [49444154]
expanded result: 4 (as expected)
(Row 0 Filter:0)
decompresses into
00 00 00 00
CRC: 2734270A
===========
00 00 00 00
49 45 4E 44
AE 42 60 82
block: "IEND", 0 bytes [49454E44]
CRC: AE426082
The IDAT data decompresses into four 0's: the first one is the row filter (0, meaning 'none') and the next 3 bytes are Red, Green, Blue values for the one single pixel.
I have problems with writing data to serial port with boost.
I use slightly modified minicom_client class from here:
https://code.google.com/p/vkbrd/source/browse/trunk/src/Dg5Emulator/MiniComClient.h
https://code.google.com/p/vkbrd/source/browse/trunk/src/Dg5Emulator/MiniComClient.cpp
First 17 bytes in message are sent good, but rest is substituted by:
FD FD FD FD AB AB AB AB AB AB AB AB EE FE EE FE 00 00 00 etc
Message length is correct.
When I call
for (int i = 0; i < 128; ++i)
miniComClient.do_write(i);
I receive this on the second side:
00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F FD FD FD FD AB AB AB AB
AB AB AB AB EE FE EE FE 00 00 00 00 00 00 00 00 2E F1 5C 42 BD 8B 07 18
40 6D C1 03 E8 9E 40 02 00 00 00 00 00 00 00 00 04 00 00 00 01 00 00 00
F4 62 00 00 FD FD FD FD 40 54 40 02 FD FD FD FD AB AB AB AB AB AB AB AB
00 00 00 00 00 00 00 00 2F F1 5C 43 BF 8B 07 18 A0 A3 3B 02 38 A5 3B 02
00 00 00 00 00 00 00 00 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F
FD FD FD FD AB AB AB AB AB AB AB AB EE FE EE FE 00 00 00 00 00 00 00 00
2E F1 5C 42 BD 8B 07 18 40 6D C1 03 E8 9E 40 02 00 00 00 00 00 00 00 00
04 00 00 00 01 00 00 00 F4 62 00 00 FD FD FD FD 40 54 40 02 FD FD FD FD
AB AB AB AB AB AB AB AB 00 00 00 00 00 00 00 00 2F F1 5C 43 BF 8B 07 18
A0 A3 3B 02 38 A5 3B 02 00 00 00 00 00 00 00 00
I have triple checked baud rate, stop bits and parity.
Is there some way to change send buffer size? Maybe this is some other problem.
Values like 0xfdfdfdfd, 0xabababab and 0xfeeefeee are magic numbers.
In other words, you are looking at uninitialized memory allocated from the heap. It is a pretty standard bug in serial port programming to assume that you get the entire message with one read. That only ever happens when you debug your program, slowing it down enough. Never when it runs at full speed, serial ports are very slow devices. You must pay attention to bytes_transferred in the read_complete() function and not interpret received data until you received all the bytes. This requires keeping track of the total, buffering partial data and having a good idea when the full response was received.
The byte value where the data turns into garbage is a magic number as well. 0x10 is the ASCII code for the Line Feed control character. Which is very often used as the End-Of-Message indicator. Be sure to configure whatever software you use on the other end of the wire to not check for a line terminator and only display the bytes it actually receives.
The reason of this problem is my "slight" modification to original code.
I wanted to make writing whole byte array simpler (instead of calling do_write() in loop), so I've added method:
void miniComClient::do_write_buffer( QByteArray msg )
bool write_in_progress = !writeMsgs.empty(); // is there anything currently being written?
for(int i = 0; i < msg.size(); ++i)
writeMsgs.push_back(msg[i]); // store in write buffer
if (!write_in_progress) // if nothing is currently being written, then start
write_start();
}
But the worst thing was my try to optimize things, so I've modified write_start() to something like this (send everything in buffer instead of just next byte):
void miniComClient::write_start( void )
{
// Start an asynchronous write and call write_complete when it completes or fails
boost::asio::async_write(serialPort,
boost::asio::buffer(&writeMsgs.front(), writeMsgs.size()),
boost::bind(&miniComClient::write_complete,
this,
boost::asio::placeholders::error));
}
This was a terrible mistake, because writeMsgs is of type std:deque so it is not stored continuously in memory, so it sends some bytes from the point I wanted, and garbage after that.