Append slice in go routine - go

I am trying to teach myself Go. I have written a simple client / server app that has some encryption and very simple packet structure.
I have a go routine for listening and then sending data to each client connected. In my function that sends data to each client i'm appending a message to a header but its doing some strange behavior.
func ClientSender(client *Client) {
for {
input := <-client.Outgoing //TODO add cleanup quit bool
for _, clientz := range RoomList[client.Room].Clients { //TODO rename to client connections = ClientList
temp := input
dbgMsg.Printf("RAW SENDER: % x", input)
dbgMsg.Printf("INPUT END1: % x", input)
dbgMsg.Printf("AES KEY % x\n", clientz.AES_Key)
dbgMsg.Printf("INPUT END2: % x", input)
dbgMsg.Printf("pre ecnryp: % x\n", input[30:])
dbgMsg.Printf("INPUT END3: % x", input)
encPayload := input[30:]
dbgMsg.Printf("INPUT END4: % x", input)
header := input[:30]
dbgMsg.Printf("INPUT END5: % x", input)
e,_ := barrenoid.Encrypt(clientz.AES_Key, encPayload)
dbgMsg.Printf("INPUT END6: % x", input)
dbgMsg.Printf("header: % x\n", input[:30])
dbgMsg.Printf("payload: % x\n", input[30:])
dbgMsg.Printf("encrypt: % x\n", e)
dbgMsg.Printf("TEMP: % x\n", temp)
asdf := append(header, e...)
dbgMsg.Printf("SENDING: % x", asdf)
//_, err := clientz.Conn.Write(payload)
//chkError(err)
input = temp
dbgMsg.Printf("INPUT END7: % x", input)
}
}
}
The value of "input" get changed and i cant figure out why. here is output from the above code:
INFO: 2016/02/22 10:47:38 RAW SENDER: 0d ae 00 00 00 00 56 cb 57 ca 41 6e 6f 6e 79 6d 6f 75 73 00 00 00 00 00 00 00 00 00 00 00 0a
INFO: 2016/02/22 10:47:38 INPUT END1: 0d ae 00 00 00 00 56 cb 57 ca 41 6e 6f 6e 79 6d 6f 75 73 00 00 00 00 00 00 00 00 00 00 00 0a
INFO: 2016/02/22 10:47:38 AES KEY 06 89 c9 d7 ad ec 4a d0 33 bf fa ab 6e 05 cd 51 87 8b f0 ad 60 a8 36 47 ca 8f 7a f8 b8 6f 1c ce
INFO: 2016/02/22 10:47:38 INPUT END2: 0d ae 00 00 00 00 56 cb 57 ca 41 6e 6f 6e 79 6d 6f 75 73 00 00 00 00 00 00 00 00 00 00 00 0a
INFO: 2016/02/22 10:47:38 pre ecnryp: 0a
INFO: 2016/02/22 10:47:38 INPUT END3: 0d ae 00 00 00 00 56 cb 57 ca 41 6e 6f 6e 79 6d 6f 75 73 00 00 00 00 00 00 00 00 00 00 00 0a
INFO: 2016/02/22 10:47:38 INPUT END4: 0d ae 00 00 00 00 56 cb 57 ca 41 6e 6f 6e 79 6d 6f 75 73 00 00 00 00 00 00 00 00 00 00 00 0a
INFO: 2016/02/22 10:47:38 INPUT END5: 0d ae 00 00 00 00 56 cb 57 ca 41 6e 6f 6e 79 6d 6f 75 73 00 00 00 00 00 00 00 00 00 00 00 0a
INFO: 2016/02/22 10:47:38 INPUT END6: 0d ae 00 00 00 00 56 cb 57 ca 41 6e 6f 6e 79 6d 6f 75 73 00 00 00 00 00 00 00 00 00 00 00 0a
INFO: 2016/02/22 10:47:38 header: 0d ae 00 00 00 00 56 cb 57 ca 41 6e 6f 6e 79 6d 6f 75 73 00 00 00 00 00 00 00 00 00 00 00
INFO: 2016/02/22 10:47:38 payload: 0a
INFO: 2016/02/22 10:47:38 encrypt: ***c8*** 7e ff f9 f5 c3 ce 1e 1d 44 91 b7 fb 09 5d e0 7e
INFO: 2016/02/22 10:47:38 TEMP: 0d ae 00 00 00 00 56 cb 57 ca 41 6e 6f 6e 79 6d 6f 75 73 00 00 00 00 00 00 00 00 00 00 00 0a
INFO: 2016/02/22 10:47:38 SENDING: 0d ae 00 00 00 00 56 cb 57 ca 41 6e 6f 6e 79 6d 6f 75 73 00 00 00 00 00 00 00 00 00 00 00 c8 7e ff f9 f5 c3 ce 1e 1d 44 91 b7 fb 09 5d e0 7e
INFO: 2016/02/22 10:47:38 INPUT END7: 0d ae 00 00 00 00 56 cb 57 ca 41 6e 6f 6e 79 6d 6f 75 73 00 00 00 00 00 00 00 00 00 00 00 ***c8***
I cant figure out why the line containing "INPUT END7" is not equal the the "input" value.
The last byte is ALWAYS equal to the first byte in "encrypted"output...
Here is code that sends slice to channel:
func ClientReader(client *Client) {
//Main Read loop
for {
bytesRead, buffer := client.Read()
if bytesRead < HEADERSIZE {
//client.Outgoing <- protocolPacker(0x0D, 0xAE, []byte(""), []byte("Minimum header not recieved."))
client.Close()
break // Connection to host is broken
}
//dbgMsg.Printf("RAW RECIEVED % x", buffer)
cmdBit, encryptionByte, ts, payload := protocolParser(buffer)
dbgMsg.Printf("CMDBIT: % x, ENCBIT: % x, TS: % d, PAYLOAD: % x", cmdBit, encryptionByte, ts, payload)
if encryptionByte == 0xAE {
payload, _ = barrenoid.Decrypt(client.AES_Key, payload)
dbgMsg.Printf("Decrypted payload % x\n", payload)
} else if encryptionByte == 0x00 {
// no need to decrypt
} else {
//bad packet reject
}
if cmdBit == 0x0D{
//payload, _ = barrenoid.Encrypt(client.AES_Key, payload)
client.Outgoing <- protocolPacker(0x0D, 0xAE, []byte(client.Name), payload)
} else if cmdBit == 0x1C {
client.Name = string(payload)
} else {
//bad packet reject
//client.Outgoing <- protocolPacker(0x0D, 0xAE, []byte(client.Name), []byte("Unknown command bit."))
}
}


The slices temp and input share the same backing array. Modifications through one slice are visible through the other. The line containing "INPUT END7" is not the same as the line with "INPUT END1" because the backing array of the slices is modified on this line:
asdf := append(header, e...)
You can copy the backing array using this line of code:
temp := append([]byte(nil), input...)

Related

Outlook calendar item clipboard format documentation?

Short Version
Is there any documentation on the Outlook RenPrivateAppointment clipboard format used to transfer appointments?
Long version
As a reminder, for anything on the clipboard, the source application can present you the data in a number of different formats. The receiver can go through the list, in order, and decide which format it understands the best.
In the case of my Outlook appointment, the formats are:
0: "RenPrivateSourceFolder" (IStream)
1: "RenPrivateMessages" (IStream)
2: "RenPrivateItem" (HGlobal)
3: "FileGroupDescriptor" (HGlobal)
4: CFSTR_FILEDESCRIPTOR (HGlobal)
5: CFSTR_FILENAME (File)
6: CFSTR_FILECONTENTS (IStream, IStorage)
7: "Object Descriptor" (HGlobal)
8: "RenPrivateAppointment" (IStream)
9: CF_TEXT (HGlobal)
10: CF_UNICODETEXT (HGlobal)
Looking at the content of the various formats, the most promising looks like the RenPrivateAppointment format:
01 00 00 00 C0 C8 1E 0D 60 CE 1E 0D 01 00 00 00 ....ÀÈ.`Î......
6A CB 1E 0D 79 CB 1E 0D 41 00 00 00 41 73 6B 20 jË..yË..A...Ask
71 75 65 73 74 69 6F 6E 20 61 62 6F 75 74 20 61 question about a
70 70 6F 69 6E 74 6D 65 6E 74 20 63 6C 69 70 62 ppointment clipb
6F 61 72 64 20 66 6F 72 6D 61 74 20 6F 6E 20 53 oard format on S
74 61 63 6B 6F 76 65 72 66 6C 6F 77 00 02 00 00 tackoverflow...
00 02 00 00 00 18 00 00 00 00 00 00 00 BC B9 6E ............¼¹n
9C 12 F8 D3 43 AC B7 74 81 5E F0 3D FC 04 D2 97 œ.øÓC¬·t.^ð=ü.Ò—
00 00 00 00 00 00 00 00 00 02 00 00 00 00 00 00 ...............
00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00 FF 92 81 02 41 00 73 00 6B 00 20 00 71 00 75 .ÿ’.A.s.k. .q.u
00 65 00 73 00 74 00 69 00 6F 00 6E 00 20 00 61 .e.s.t.i.o.n. .a
00 62 00 6F 00 75 00 74 00 20 00 61 00 70 00 70 .b.o.u.t. .a.p.p
00 6F 00 69 00 6E 00 74 00 6D 00 65 00 6E 00 74 .o.i.n.t.m.e.n.t
00 20 00 63 00 6C 00 69 00 70 00 62 00 6F 00 61 . .c.l.i.p.b.o.a
00 72 00 64 00 20 00 66 00 6F 00 72 00 6D 00 61 .r.d. .f.o.r.m.a
00 74 00 20 00 6F 00 6E 00 20 00 53 00 74 00 61 .t. .o.n. .S.t.a
00 63 00 6B 00 6F 00 76 00 65 00 72 00 66 00 6C .c.k.o.v.e.r.f.l
00 6F 00 77 00 00 00 01 00 00 00 00 00 FF FF FF .o.w.........ÿÿÿ
FF ÿ
Some of this can be interpreted:
Clipboard format "RenPrivateAppointment"
01 00 00 00 ; always 0x00000001 (Version 1?)
C0 C8 1E 0D ; Start day of appt. minutes from 1/1/1601 0x0D1EC8C0 = 220,121,280 minutes = 7/11/2019 12:00 am
60 CE 1E 0D ; End day of appt. minutes from 1/1/1601 0x0D1ECE60 = 220,122,720 minutes = 7/12/2019 12:00 am
01 00 00 00 ; 0x00000001 (fixed)
6A CB 1E 0D ; Start of appt. minutes from 1/1/1601 0x0D1ECB6A = 220,121,962 minutes = 7/11/2019 11:22 am
79 CB 1E 0D ; End of appt. minutes from 1/1/1601 0x0D1ECB79 = 220,121,977 minutes = 7/11/2019 11:37 am
; "Ask question about appointment clipboard format on Stackoverflow.\0"
41 00 00 00 ; String length prefix, including null terminator (0x00000041 = 65 characters)
41 73 6B 20 71 75 65 73 Ask ques
74 69 6F 6E 20 61 62 6F tion abo
75 74 20 61 70 70 6F 69 ut appoi
6E 74 6D 65 6E 74 20 63 ntment c
6C 69 70 62 6F 61 72 64 lipboard
20 66 6F 72 6D 61 74 20 format
6F 6E 20 53 74 61 63 6B on Stack
6F 76 65 72 66 6C 6F 77 overflow
00 .
02 00 00 00 ; 0x0000002 = 2
02 00 00 00 ; 0x0000002 = 2
18 00 00 00 ; 0x00000018 = 24
00 00 00 00 ; 0x00000000 = 0
BC B9 6E 9C 12 F8 D3 43 ; always
AC B7 74 81 5E F0 3D FC ; always
04 D2 97 00 ; varies (~32 ticks per day) 0x0097D204 = 9,949,700
00 00 00 00
00 00 00 00
02 00 00 00 ; 0x00000002 = 2
00 00 00 00
01 00 00 00 ; 0x00000001 = 1
00 00 00 00
00 00 00 00
00 00 00 00
FF 92 81 02 ; always 0x028192FF
; N"Ask question about appointment clipboard format on Stackoverflow\0"
41 00 73 00 6B 00 20 00 71 00 75 00 65 00 73 00 A.s.k. .q.u.e.s.
74 00 69 00 6F 00 6E 00 20 00 61 00 62 00 6F 00 t.i.o.n. .a.b.o.
75 00 74 00 20 00 61 00 70 00 70 00 6F 00 69 00 u.t. .a.p.p.o.i.
6E 00 74 00 6D 00 65 00 6E 00 74 00 20 00 63 00 n.t.m.e.n.t. .c.
6C 00 69 00 70 00 62 00 6F 00 61 00 72 00 64 00 l.i.p.b.o.a.r.d.
20 00 66 00 6F 00 72 00 6D 00 61 00 74 00 20 00 .f.o.r.m.a.t. .
6F 00 6E 00 20 00 53 00 74 00 61 00 63 00 6B 00 o.n. .S.t.a.c.k.
6F 00 76 00 65 00 72 00 66 00 6C 00 6F 00 77 00 o.v.e.r.f.l.o.w.
00 00 ..
01 00 ; padding to DWORD
00 00 00 00
FF FF FF FF ; footer
Is there any documentation on RenPrivateAppointment, or any other the other formats that would allow rich interactions by the user?
Note: This is not automating Outlook. This is handling the IDataObject placed on the clipboard by Outlook. I want to retrieve:
start time
end time
description
See also
C# parse outlook calendar item (i'm not in C#)
microsoft.public.win32.programmer.ole: Identify correctly outlook items in Drag and Drop.

There is a project on GitHub that parses the RenPrivateAppointment clipboard format: https://github.com/yasoonOfficial/outlook-dndprotocol

The RenPrivateAppointment format isn't documented. You may read about that on the DragDrop Event in Outlook Calendar thread which has an official comment from a VSTO team member. Also, you may take a look at the Drag and Drop with Outlook page.

Download the metadata from peers failed by bep_0009 [golang]

When I send an extension request message to peers, but no response from peers.
I got Handshake and Extended Bitfield and Have Piece ok, but no extension data message.
This is my Golang code.
package main
import (
"bytes"
"crypto/rand"
"encoding/binary"
"fmt"
"github.com/IncSW/go-bencode"
"io"
"log"
"net"
"time"
)
func readBuffer(conn *net.TCPConn, size uint32) ([]byte, error) {
//temp := make([]byte, size)
//_, err := conn.Read(temp)
//if err != nil {
// return nil, fmt.Errorf("read %d bytes message failed: %v", size, err)
//}
//return temp, nil
buffer := bytes.NewBuffer(nil)
conn.SetReadDeadline(time.Now().Add(time.Second * 120))
_, err := io.CopyN(buffer, conn, int64(size))
if err != nil {
return nil, fmt.Errorf("read %d bytes message failed: %v", size, err)
}
return buffer.Bytes(), nil
}
func next(conn *net.TCPConn) ([]byte, error) {
buffer, err := readBuffer(conn, 4)
if nil != err {
return nil, err
}
buffer, err = readBuffer(conn, binary.BigEndian.Uint32(buffer))
if nil != err {
return nil, err
}
return buffer, nil
}
func getHandshake(infoHash []byte, nodeId []byte) []byte {
packet := []byte{19, 66, 105, 116, 84, 111, 114, 114, 101, 110, 116, 32, 112, 114, 111, 116, 111, 99, 111, 108, 0, 0, 0, 0, 0, 16, 0, 1}
packet = append(packet, infoHash...)
packet = append(packet, nodeId...)
return packet
}
// http://www.bittorrent.org/beps/bep_0009.html
// http://www.bittorrent.org/beps/bep_0010.html
func peerWire(addr string, infoHash []byte, nodeId []byte) {
dial, err := net.DialTimeout("tcp", addr, 10*time.Second)
if nil != err {
return
}
conn := dial.(*net.TCPConn)
conn.SetLinger(0)
defer conn.Close()
// handshake
handshake := getHandshake(infoHash, nodeId)
conn.Write(handshake)
handshakeRes := make([]byte, 68)
_, err = conn.Read(handshakeRes)
if !(bytes.Equal(handshake[:20], handshakeRes[:20]) && handshakeRes[25]&0x10 != 0) {
log.Println("invalid handshake response")
return
}
// extended handshake
// [length prefix][BitTorrent message ID][extended message ID]
extendedHandshake, _ := bencode.Marshal(map[string]interface{}{
"m": map[string]interface{}{"ut_metadata": 1},
})
extendedHandshake = append([]byte{20, 0}, extendedHandshake...)
size := make([]byte, 4)
extendedHandshake = append(size, extendedHandshake...)
conn.Write(extendedHandshake)
buffer, err := next(conn)
if nil != err {
log.Println(err.Error())
return
}
if 0 < len(buffer) {
msgUT, err := bencode.Unmarshal(buffer[2:])
if nil != err {
log.Println("error")
return
}
metadataSize, ok := msgUT.(map[string]interface{})["metadata_size"].(int64)
if !ok {
return
}
m, ok := msgUT.(map[string]interface{})["m"].(map[string]interface{})
if !ok {
return
}
utMetadata, ok := m["ut_metadata"].(int64)
if !ok {
return
}
numberOfPieces := metadataSize / 16384
if metadataSize%16384 != 0 {
numberOfPieces++
}
for i := 0; i < int(numberOfPieces); i++ {
packet, err := bencode.Marshal(map[string]interface{}{"msg_type":0, "piece":int(i)})
if nil != err {
log.Println(err)
}
packet = append([]byte{20, byte(utMetadata)}, packet...)
size := make([]byte, 4)
binary.BigEndian.PutUint32(size, uint32(len(packet)))
packet = append(size, packet...)
conn.Write(packet)
break
}
piece := make([]byte, 0)
for {
buffer, err := next(conn)
if nil != err {
log.Println(err.Error())
break
}
log.Println(buffer)
if 20 != buffer[0] {
continue
}
log.Println("buffer: ", buffer)
piece = append(piece, buffer...)
}
log.Println("index :", bytes.Index(piece, []byte("ee")))
}
}
func main() {
nodeId := make([]byte, 20)
rand.Read(nodeId)
infoHash := []byte{231, 130, 244, 163, 244, 122, 203, 232, 78, 218, 29, 116, 240, 232, 146, 236, 199, 72, 132, 254}
addr := "203.81.67.114:55254"
peerWire(addr, infoHash, nodeId)
}
I think the extension request message "[]byte{0, 0, 0, 27, 20, 2, 100, 53, 58, 112, 105, 101, 99, 101, 105, 48, 101, 56, 58, 109, 115, 103, 95, 116, 121, 112, 101, 105, 48, 101, 101}" format is correct.
WireShark Dump Data

From the pcap:
00000044 00 00 00 00 14 00 64 31 3a 6d 64 31 31 3a 75 74 ......d1 :md11:ut
00000054 5f 6d 65 74 61 64 61 74 61 69 31 65 65 65 _metadat ai1eee
You're sending 00 00 00 00 as the length for the extension handshake. That will be interpreted as a keepalive message and the rest as a message with length 14 00 64 31 and type 3a i.e. complete nonsense.
Complete pcap:
00000000 13 42 69 74 54 6f 72 72 65 6e 74 20 70 72 6f 74 .BitTorr ent prot
00000010 6f 63 6f 6c 00 00 00 00 00 10 00 01 e7 82 f4 a3 ocol.... ........
00000020 f4 7a cb e8 4e da 1d 74 f0 e8 92 ec c7 48 84 fe .z..N..t .....H..
00000030 0d 3a 6c 0f 88 e8 41 45 b9 96 b4 52 62 eb 33 cc .:l...AE ...Rb.3.
00000040 ca 73 8b bf .s..
00000000 13 42 69 74 54 6f 72 72 65 6e 74 20 70 72 6f 74 .BitTorr ent prot
00000010 6f 63 6f 6c 00 00 00 00 00 10 00 05 e7 82 f4 a3 ocol.... ........
00000020 f4 7a cb e8 4e da 1d 74 f0 e8 92 ec c7 48 84 fe .z..N..t .....H..
00000030 2d 55 54 33 35 34 53 2d 58 ae 73 ad 79 d9 8e d6 -UT354S- X.s.y...
00000040 bd be b0 5f 00 00 00 e6 14 00 64 31 3a 65 69 30 ..._.... ..d1:ei0
00000050 65 34 3a 69 70 76 34 34 3a cb 51 43 72 31 32 3a e4:ipv44 :.QCr12:
00000060 63 6f 6d 70 6c 65 74 65 5f 61 67 6f 69 complete _agoi
00000044 00 00 00 00 14 00 64 31 3a 6d 64 31 31 3a 75 74 ......d1 :md11:ut
00000054 5f 6d 65 74 61 64 61 74 61 69 31 65 65 65 _metadat ai1eee
0000006D 33 65 31 3a 6d 64 31 31 3a 75 70 6c 6f 61 64 5f 3e1:md11 :upload_
0000007D 6f 6e 6c 79 69 33 65 31 31 3a 6c 74 5f 64 6f 6e onlyi3e1 1:lt_don
0000008D 74 68 61 76 65 69 37 65 31 32 3a 75 74 5f 68 6f thavei7e 12:ut_ho
0000009D 6c 65 70 75 6e 63 68 69 34 65 31 31 3a 75 74 5f lepunchi 4e11:ut_
000000AD 6d 65 74 61 64 61 74 61 69 32 65 36 3a 75 74 5f metadata i2e6:ut_
000000BD 70 65 78 69 31 65 31 30 3a 75 74 5f 63 6f 6d 6d pexi1e10 :ut_comm
000000CD 65 6e 74 69 36 65 65 31 33 3a 6d 65 74 61 64 61 enti6ee1 3:metada
000000DD 74 61 5f 73 69 7a 65 69 38 38 38 32 65 31 3a 70 ta_sizei 8882e1:p
000000ED 69 35 35 32 35 34 65 34 3a 72 65 71 71 69 32 35 i55254e4 :reqqi25
000000FD 35 65 31 3a 76 31 35 3a ce bc 54 6f 72 72 65 6e 5e1:v15: ..Torren
0000010D 74 20 33 2e 35 2e 34 32 3a 79 70 69 33 31 34 30 t 3.5.42 :ypi3140
0000011D 38 65 36 3a 79 6f 75 72 69 70 34 3a 75 8b d0 19 8e6:your ip4:u...
0000012D 65 00 00 00 38 05 ff ef ff df f7 f7 ff ff ff ff e...8... ........
0000013D ff df ff ff df ff ff ff ef ff ff ee fe ff ff ff ........ ........
0000014D 6f ff ff 99 ff ff fd ff ff ff ff ff ff ff ef ff o....... ........
0000015D ff fe f7 ff ff fc ff bf ff ff bf ff c0 00 00 00 ........ ........
0000016D 05 04 00 00 00 5a 00 00 00 05 04 00 00 00 0b 00 .....Z.. ........
0000017D 00 00 05 04 00 00 00 e9 00 00 00 05 04 00 00 00 ........ ........
0000018D ea 00 00 00 05 04 00 00 01 64 00 00 00 05 04 00 ........ .d......
0000019D 00 01 5f 00 00 00 05 04 00 00 00 d3 00 00 00 05 .._..... ........
000001AD 04 00 00 01 7f 00 00 00 05 04 00 00 00 1a 00 00 ........ ........
000001BD 00 05 04 00 00 00 24 00 00 00 05 04 00 00 01 89 ......$. ........
000001CD 00 00 00 05 04 00 00 00 ee 00 00 00 05 04 00 00 ........ ........
000001DD 01 06 00 00 00 05 04 00 00 00 ed 00 00 00 05 04 ........ ........
000001ED 00 00 00 2c 00 00 00 05 04 00 00 00 93 00 00 00 ...,.... ........
000001FD 05 04 00 00 00 ab 00 00 00 05 04 00 00 00 d0 00 ........ ........
0000020D 00 00 05 04 00 00 00 b7 00 00 00 05 04 00 00 01 ........ ........
0000021D a1 00 00 00 05 04 00 00 00 72 00 00 00 05 04 00 ........ .r......
0000022D 00 01 43 00 00 00 05 04 00 00 01 7e 00 00 00 05 ..C..... ...~....
0000023D 04 00 00 00 af .....
00000062 00 00 00 1b 14 02 64 38 3a 6d 73 67 5f 74 79 70 ......d8 :msg_typ
00000072 65 69 30 65 35 3a 70 69 65 63 65 69 30 65 65 ei0e5:pi ecei0ee
00000242 13 42 69 74 54 6f 72 72 65 6e 74 20 70 72 6f 74 .BitTorr ent prot
00000252 6f 63 6f 6c 00 00 00 00 00 10 00 05 e7 82 f4 a3 ocol.... ........
00000262 f4 7a cb e8 4e da 1d 74 f0 e8 92 ec c7 48 84 fe .z..N..t .....H..
00000272 2d 55 54 33 35 34 53 2d 58 ae 73 ad 79 d9 8e d6 -UT354S- X.s.y...
00000282 bd be b0 5f 00 00 00 e6 14 00 64 31 3a 65 69 30 ..._.... ..d1:ei0
00000292 65 34 3a 69 70 76 34 34 3a cb 51 43 72 31 32 3a e4:ipv44 :.QCr12:
000002A2 63 6f 6d 70 6c 65 74 65 5f 61 67 6f 69 33 65 31 complete _agoi3e1
000002B2 3a 6d 64 31 31 3a 75 70 6c 6f 61 64 5f 6f 6e 6c :md11:up load_onl
000002C2 79 69 33 65 31 31 3a 6c 74 5f 64 6f 6e 74 68 61 yi3e11:l t_dontha
000002D2 76 65 69 37 65 31 32 3a 75 74 5f 68 6f 6c 65 70 vei7e12: ut_holep
000002E2 75 6e 63 68 69 34 65 31 31 3a 75 74 5f 6d 65 74 unchi4e1 1:ut_met
000002F2 61 64 61 74 61 69 32 65 36 3a 75 74 5f 70 65 78 adatai2e 6:ut_pex
00000302 69 31 65 31 30 3a 75 74 5f 63 6f 6d 6d 65 6e 74 i1e10:ut _comment
00000312 69 36 65 65 31 33 3a 6d 65 74 61 64 61 74 61 5f i6ee13:m etadata_
00000322 73 69 7a 65 69 38 38 38 32 65 31 3a 70 69 35 35 sizei888 2e1:pi55
00000332 32 35 34 65 34 3a 72 65 71 71 69 32 35 35 65 31 254e4:re qqi255e1
00000342 3a 76 31 35 3a ce bc 54 6f 72 72 65 6e 74 20 33 :v15:..T orrent 3
00000352 2e 35 2e 34 32 3a 79 70 69 33 31 34 30 38 65 36 .5.42:yp i31408e6
00000362 3a 79 6f 75 72 69 70 34 3a 75 8b d0 19 65 00 00 :yourip4 :u...e..
00000372 00 38 05 ff ef ff df f7 f7 ff ff ff ff ff df ff .8...... ........
00000382 ff df ff ff ff ef ff ff ee fe ff ff ff 6f ff ff ........ .....o..
00000392 99 ff ff fd ff ff ff ff ff ff ff ef ff ff fe f7 ........ ........
000003A2 ff ff fc ff bf ff ff bf ff c0 00 00 00 05 04 00 ........ ........
000003B2 00 00 5a 00 00 00 05 04 00 00 00 0b 00 00 00 05 ..Z..... ........
000003C2 04 00 00 00 e9 00 00 00 05 04 00 00 00 ea 00 00 ........ ........
000003D2 00 05 04 00 00 01 64 00 00 00 05 04 00 00 01 5f ......d. ......._
000003E2 00 00 00 05 04 00 00 00 d3 00 00 00 05 04 00 00 ........ ........
000003F2 01 7f 00 00 00 05 04 00 00 00 1a 00 00 00 05 04 ........ ........
00000402 00 00 00 24 00 00 00 05 04 00 00 01 89 00 00 00 ...$.... ........
00000412 05 04 00 00 00 ee 00 00 00 05 04 00 00 01 06 00 ........ ........
00000422 00 00 05 04 00 00 00 ed 00 00 00 05 04 00 00 00 ........ ........
00000432 2c 00 00 00 05 04 00 00 00 93 00 00 00 05 04 00 ,....... ........
00000442 00 00 ab 00 00 00 05 04 00 00 00 d0 00 00 00 05 ........ ........
00000452 04 00 00 00 b7 00 00 00 05 04 00 00 01 a1 00 00 ........ ........
00000462 00 05 04 00 00 00 72 00 00 00 05 04 00 00 01 43 ......r. .......C
00000472 00 00 00 05 04 00 00 01 7e 00 00 00 05 04 00 00 ........ ~.......
00000482 00 af 00 00 00 03 09 d7 d6 ........ .

Parsing an ASCII text file using GAWK

I have been trying to parse an ASCII text file of the following format --
0 0 0x2de0 [0x98]: PERF_RECORD_MMAP -1/0: [0xffffffffc06ae000(0x5000) # 0]: x /lib/modules/4.4.0-83-generic/kernel/net/ipv4/netfilter/nf_reject_ipv4.ko
0x2e78 [0x90]: event: 1
.
. ... raw event: size 144 bytes
. 0000: 01 00 00 00 01 00 90 00 ff ff ff ff 00 00 00 00 ................
. 0010: 00 30 6b c0 ff ff ff ff 00 50 00 00 00 00 00 00 .0k......P......
. 0020: 00 00 00 00 00 00 00 00 2f 6c 69 62 2f 6d 6f 64 ......../lib/mod
. 0030: 75 6c 65 73 2f 34 2e 34 2e 30 2d 38 33 2d 67 65 ules/4.4.0-83-ge
. 0040: 6e 65 72 69 63 2f 6b 65 72 6e 65 6c 2f 6e 65 74 neric/kernel/net
. 0050: 2f 69 70 76 34 2f 6e 65 74 66 69 6c 74 65 72 2f /ipv4/netfilter/
. 0060: 69 70 74 5f 52 45 4a 45 43 54 2e 6b 6f 00 2e 6b ipt_REJECT.ko..k
. 0070: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
. 0080: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0 0 0x2e78 [0x90]: PERF_RECORD_MMAP -1/0: [0xffffffffc06b3000(0x5000) # 0]: x /lib/modules/4.4.0-83-generic/kernel/net/ipv4/netfilter/ipt_REJECT.ko
0x2f08 [0x88]: event: 1
.
. ... raw event: size 136 bytes
. 0000: 01 00 00 00 01 00 88 00 ff ff ff ff 00 00 00 00 ................
. 0010: 00 80 6b c0 ff ff ff ff 00 50 00 00 00 00 00 00 ..k......P......
. 0020: 00 00 00 00 00 00 00 00 2f 6c 69 62 2f 6d 6f 64 ......../lib/mod
. 0030: 75 6c 65 73 2f 34 2e 34 2e 30 2d 38 33 2d 67 65 ules/4.4.0-83-ge
. 0040: 6e 65 72 69 63 2f 6b 65 72 6e 65 6c 2f 6e 65 74 neric/kernel/net
. 0050: 2f 6e 65 74 66 69 6c 74 65 72 2f 78 74 5f 74 63 /netfilter/xt_tc
. 0060: 70 75 64 70 2e 6b 6f 00 00 00 00 00 00 00 00 00 pudp.ko.........
. 0070: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
. 0080: 00 00 00 00 00 00 00 00
........[some other data]........
0x11590 [0x30]: PERF_RECORD_AUXTRACE size: 0x2002a0 offset: 0 ref: 0x2d44e6441a3c2 idx: 0 tid: -1 cpu: 0
.
. ... Intel Processor Trace data: size 2097824 bytes
. 00000000: 02 82 02 82 02 82 02 82 02 82 02 82 02 82 02 82 PSB
. 00000010: 00 00 00 PAD
. 00000013: 99 20 MODE.TSX TXAbort:0 InTX:0
. 00000015: 99 01 MODE.Exec 64
. 00000017: 7d 08 45 06 81 ff ff 00 FUP 0xffff81064508
. 0000001f: 00 00 00 00 00 00 00 PAD
. 00000026: 02 43 00 76 49 1f 00 00 PIP 0xfa4bb00 (NR=0)
. 0000002e: 00 00 00 00 00 00 00 00 PAD
--- continued ---
The file will have several headers - as you can see in my snippet here.
PERF_RECORD_MMAP and PERF_RECORD_AUXTRACE
There will be other headers in the file as well.
What I want is that all the headers having PERF_RECORD_AUXTRACE in my text file should only be considered. All the data following the PERF_RECORD_AUXTRACE in my file should only be collected (i.e. all of the data starting with Intel Processor Trace Data). The PERF_RECORD_AUXTRACE header also has a size field with the use of which I can specify how much of data is there to be collected within the PERF_RECORD_AUXTRACE header.
Edit #1 :
So basically, given the above input file snippet, I want the output to be of the following form (all the lines after record containing PERF_RECORD_AUXTRACE)...
.
. ... Intel Processor Trace data: size 2097824 bytes
. 00000000: 02 82 02 82 02 82 02 82 02 82 02 82 02 82 02 82 PSB
. 00000010: 00 00 00 PAD
. 00000013: 99 20 MODE.TSX TXAbort:0 InTX:0
. 00000015: 99 01 MODE.Exec 64
. 00000017: 7d 08 45 06 81 ff ff 00 FUP 0xffff81064508
. 0000001f: 00 00 00 00 00 00 00 PAD
. 00000026: 02 43 00 76 49 1f 00 00 PIP 0xfa4bb00 (NR=0)
. 0000002e: 00 00 00 00 00 00 00 00 PAD
--- continued ---
EDIT #2 : This is another requirement that I have --
If I have an input snippet like below --
0 0 0x230 [0x60]: PERF_RECORD_MMAP -1/0: [0xffffffff81000000(0x3f000000) # 0xffffffff81000000]: x [kernel.kallsyms]_text
0x290 [0x88]: event: 1
.
. ... raw event: size 136 bytes
. 0000: 01 00 00 00 01 00 88 00 ff ff ff ff 00 00 00 00 ................
. 0010: 00 00 00 c0 ff ff ff ff 00 90 00 00 00 00 00 00 ................
. 0020: 00 00 00 00 00 00 00 00 2f 6c 69 62 2f 6d 6f 64 ......../lib/mod
. 0030: 75 6c 65 73 2f 34 2e 34 2e 30 2d 38 33 2d 67 65 ules/4.4.0-83-ge
. 0040: 6e 65 72 69 63 2f 6b 65 72 6e 65 6c 2f 64 72 69 neric/kernel/dri
. 0050: 76 65 72 73 2f 61 74 61 2f 6c 69 62 61 68 63 69 vers/ata/libahci
. 0060: 2e 6b 6f 00 00 00 00 00 00 00 00 00 00 00 00 00 .ko.............
. 0070: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
. 0080: 00 00 00 00 00 00 00 00 ........
0x11590 [0x30]: PERF_RECORD_AUXTRACE size: 0x2002a0 offset: 0 ref: 0x2d44e6441a3c2 idx: 0 tid: -1 cpu: 0
.
. ... Intel Processor Trace data: size 2097824 bytes
. 00000000: 02 82 02 82 02 82 02 82 02 82 02 82 02 82 02 82 PSB
. 00000010: 00 00 00 PAD
. 00000013: 99 20 MODE.TSX TXAbort:0 InTX:0
. 00000015: 99 01 MODE.Exec 64
. 00000017: 7d 08 45 06 81 ff ff 00 FUP 0xffff81064508
. 0000001f: 00 00 00 00 00 00 00 PAD
. 00000026: 02 43 00 76 49 1f 00 00 PIP 0xfa4bb00 (NR=0)
. 0000002e: 00 00 00 00 00 00 00 00 PAD
. 00000036: 02 c8 c2 3a 7c 00 00 00 VMCS 0x7c3ac2
0 0 0x290 [0x88]: PERF_RECORD_MMAP -1/0: [0xffffffffc0000000(0x9000) # 0]: x /lib/modules/4.4.0-83-generic/kernel/drivers/ata/libahci.ko
0x318 [0x98]: event: 1
.
. ... raw event: size 152 bytes
. 0000: 01 00 00 00 01 00 98 00 ff ff ff ff 00 00 00 00 ................
. 0010: 00 90 00 c0 ff ff ff ff 00 50 00 00 00 00 00 00 .........P......
. 0020: 00 00 00 00 00 00 00 00 2f 6c 69 62 2f 6d 6f 64 ......../lib/mod
. 0030: 75 6c 65 73 2f 34 2e 34 2e 30 2d 38 33 2d 67 65 ules/4.4.0-83-ge
. 0040: 6e 65 72 69 63 2f 6b 65 72 6e 65 6c 2f 64 72 69 neric/kernel/dri
. 0050: 76 65 72 73 2f 76 69 64 65 6f 2f 66 62 64 65 76 vers/video/fbdev
. 0060: 2f 63 6f 72 65 2f 66 62 5f 73 79 73 5f 66 6f 70 /core/fb_sys_fop
. 0070: 73 2e 6b 6f 00 00 00 00 00 00 00 00 00 00 00 00 s.ko............
. 0080: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
. 0090: 00 00 00 00 00 00 00 00 ........
0x11590 [0x30]: PERF_RECORD_AUXTRACE size: 0x2002a0 offset: 0 ref: 0x2d44e6441a3c2 idx: 0 tid: -1 cpu: 0
.
. ... Intel Processor Trace data: size 2097824 bytes
. 00000000: 02 82 02 82 02 82 02 82 02 82 02 82 02 82 02 82 PSB
. 00000010: 00 00 00 PAD
. 00000013: 99 20 MODE.TSX TXAbort:0 InTX:0
. 00000015: 99 01 MODE.Exec 64
. 00000017: 7d 08 45 06 81 ff ff 00 FUP 0xffff81064508
. 0000001f: 00 00 00 00 00 00 00 PAD
. 00000026: 02 43 00 76 49 1f 00 00 PIP 0xfa4bb00 (NR=0)
. 0000002e: 00 00 00 00 00 00 00 00 PAD
. 00000036: 02 c8 c2 3a 7c 00 00 00 VMCS 0x7c3ac2
I only would need the data under the records containing PERF_RECORD_AUXTRACE just like this. It would be great if the first line that contains
Intel Processor Trace Data : size 2097824 bytes
can also be avoided from my output.
. 00000000: 02 82 02 82 02 82 02 82 02 82 02 82 02 82 02 82 PSB
. 00000010: 00 00 00 PAD
. 00000013: 99 20 MODE.TSX TXAbort:0 InTX:0
. 00000015: 99 01 MODE.Exec 64
. 00000017: 7d 08 45 06 81 ff ff 00 FUP 0xffff81064508
. 0000001f: 00 00 00 00 00 00 00 PAD
. 00000026: 02 43 00 76 49 1f 00 00 PIP 0xfa4bb00 (NR=0)
. 0000002e: 00 00 00 00 00 00 00 00 PAD
. 00000000: 02 82 02 82 02 82 02 82 02 82 02 82 02 82 02 82 PSB
. 00000010: 00 00 00 PAD
. 00000013: 99 20 MODE.TSX TXAbort:0 InTX:0
. 00000015: 99 01 MODE.Exec 64
. 00000017: 7d 08 45 06 81 ff ff 00 FUP 0xffff81064508
. 0000001f: 00 00 00 00 00 00 00 PAD
. 00000026: 02 43 00 76 49 1f 00 00 PIP 0xfa4bb00 (NR=0)
. 0000002e: 00 00 00 00 00 00 00 00 PAD
Edit #3 : This is what I initially tried to do.. but which obviously does not work!
cat "$file" | gawk -F' ' -- '
/PERF_RECORD_AUXTRACE / {
offset = strtonum($1)
hsize = strtonum(substr($2, 2))
size = strtonum($5)
idx = strtonum($11)
ext = ""
ofile = sprintf("raw-pt.txt")
begin = offset + hsize
cmd = sprintf("dd if=%s of=%s conv=notrunc oflag=append ibs=1 " \
"count=%d status=none", file, ofile, size)
#!cmd = sprintf("sed p")
if (dry_run != 0) {
print cmd
}
else {
system(cmd)
}
}
I am not quite sure how can I properly parse this file to exactly get what I want. I also am not sure if using Python would help.
How to resolve this ?

To get the output you say you want from the input you posted is just:
awk 'f; /PERF_RECORD_AUXTRACE/{f=1}' file
If that's not actually all you want then edit your question to clarify your requirements and provide different sample input/output that more truly demonstrates your problem if necessary.

Overwrite/Throw error on duplicate key in Go LevelDB

When using the following code below I expect to have only one entry in the db.
I use the syndtr/goleveldb LevelDB implementation for Go.
for count := 0; count < 5; count++ {
err := db.Put([]byte("key"), []byte("value"))
if err != nil {
t.Error(err)
}
}
Instead, a hexdump reveals that there are 5 entries:
00000000 a8 ef d2 d4 17 00 01 01 00 00 00 00 00 00 00 01 |................|
00000010 00 00 00 01 03 6b 65 79 05 76 61 6c 75 65 10 23 |.....key.value.#|
00000020 44 b5 17 00 01 02 00 00 00 00 00 00 00 01 00 00 |D...............|
00000030 00 01 03 6b 65 79 05 76 61 6c 75 65 77 be 34 95 |...key.valuew.4.|
00000040 17 00 01 03 00 00 00 00 00 00 00 01 00 00 00 01 |................|
00000050 03 6b 65 79 05 76 61 6c 75 65 08 35 86 60 17 00 |.key.value.5.`..|
00000060 01 04 00 00 00 00 00 00 00 01 00 00 00 01 03 6b |...............k|
00000070 65 79 05 76 61 6c 75 65 6f 8c f6 00 17 00 01 05 |ey.valueo.......|
00000080 00 00 00 00 00 00 00 01 00 00 00 01 03 6b 65 79 |.............key|
00000090 05 76 61 6c 75 65 |.value|
00000096
From my understanding LevelDB should by default overwrite any duplicate key or at least throw an error.
How can I control when the value gets overwritten or how can I get an error (other than checking db.Has(key) every time that I db.Put(key)?
Does compaction have anything to do with this?

LevelDB is a log-structured merge-tree and has transaction logs (the ones you were inspecting with hexdump). With this code you can validate that only one value is present for your key:
package main
import ( "fmt"
"github.com/syndtr/goleveldb/leveldb"
)
func main () {
db, err := leveldb.OpenFile("mydb.test", nil)
if err != nil { fmt.Println(err) }
for count := 0; count < 5; count++ {
err := db.Put([]byte("keys"), []byte("values"), nil)
if err != nil {
fmt.Println(err)
}
}
iter := db.NewIterator(nil, nil)
for iter.Next() {
key := iter.Key()
value := iter.Value()
fmt.Println(string(key), string(value))
}
iter.Release()
defer db.Close()
}
How can I control when the value gets overwritten?
You can't, if the key is present, its value gets overwritten

Conflicting answers when I try to find out endian-ness of my Macbook Pro

I have a Macbook Pro and am getting contradictory answers when I try to determine its endian-ness.
Method 1
python -c "import sys;print sys.byteorder" tells me I am on a little endian system
Method 2
I have a text file. I used iconv to convert it into UTF16. Its supposed to detect the endianness of the computer and convert it into that format. So here I go:
iconv -f utf-8 -t utf-16 file.txt > utf16.txt
file utf16.txt
utf16.txt: Big-endian UTF-16 Unicode English text
vi utf16.txt works and hexdump -C utf16.txt shows:
00000000 fe ff 00 33 00 39 00 38 00 31 00 36 00 30 00 38 |...3.9.8.1.6.0.8|
00000010 00 09 00 54 00 69 00 61 00 20 00 4a 00 75 00 61 |...T.i.a. .J.u.a|
00000020 00 6e 00 61 00 20 00 52 00 69 00 76 00 65 00 72 |.n.a. .R.i.v.e.r|
00000030 00 09 00 54 00 69 00 61 00 20 00 4a 00 75 00 61 |...T.i.a. .J.u.a|
00000040 00 6e 00 61 00 20 00 52 00 69 00 76 00 65 00 72 |.n.a. .R.i.v.e.r|
00000050 00 09 00 52 00 69 00 6f 00 20 00 54 00 69 00 61 |...R.i.o. .T.i.a|
00000060 00 6a 00 75 00 61 00 6e 00 61 00 2c 00 52 00 69 |.j.u.a.n.a.,.R.i|
00000070 00 6f 00 20 00 54 00 69 00 6a 00 75 00 61 00 6e |.o. .T.i.j.u.a.n|
00000080 00 61 00 2c 00 52 00 ed 00 6f 00 20 00 54 00 69 |.a.,.R...o. .T.i|
00000090 00 6a 00 75 00 61 00 6e 00 61 00 2c 00 54 00 69 |.j.u.a.n.a.,.T.i|
if I convert it to little-endian and manually insert a BOM like this:
( printf "\xff\xfe" ; iconv -f utf-8 -t utf-16le file.txt ) > UTF16LEBOM.txt
file UTF16LEBOM.txt
UTF16LEBOM.txt: Little-endian UTF-16 Unicode English text
vi UTF16LEBOM.txt works
and hexdump -C UTF16LEBOM.txt shows
00000000 ff fe 33 00 39 00 38 00 31 00 36 00 30 00 38 00 |..3.9.8.1.6.0.8.|
00000010 09 00 54 00 69 00 61 00 20 00 4a 00 75 00 61 00 |..T.i.a. .J.u.a.|
00000020 6e 00 61 00 20 00 52 00 69 00 76 00 65 00 72 00 |n.a. .R.i.v.e.r.|
00000030 09 00 54 00 69 00 61 00 20 00 4a 00 75 00 61 00 |..T.i.a. .J.u.a.|
00000040 6e 00 61 00 20 00 52 00 69 00 76 00 65 00 72 00 |n.a. .R.i.v.e.r.|
00000050 09 00 52 00 69 00 6f 00 20 00 54 00 69 00 61 00 |..R.i.o. .T.i.a.|
00000060 6a 00 75 00 61 00 6e 00 61 00 2c 00 52 00 69 00 |j.u.a.n.a.,.R.i.|
00000070 6f 00 20 00 54 00 69 00 6a 00 75 00 61 00 6e 00 |o. .T.i.j.u.a.n.|
00000080 61 00 2c 00 52 00 ed 00 6f 00 20 00 54 00 69 00 |a.,.R...o. .T.i.|
00000090 6a 00 75 00 61 00 6e 00 61 00 2c 00 54 00 69 00 |j.u.a.n.a.,.T.i.|
From this link:
The other approach is to include a magic number, such as 0xFEFF,
before every piece of data. If you read the magic number and it is
0xFEFF, it means the data is in the same format as your machine, and
all is well.
If you read the magic number and it is 0xFFFE (it is backwards), it
means the data was written in a format different from your own. You'll
have to translate it.
Who is right and why am I getting contradictory answers?

"Endian-ness of my Macbook Pro" means nothing. You need to be more specific; different applications will have different impressions. As you've just seen, you can arbitrarily encode bytes in a file. In the end, a series of bytes is just that, and files are ultimately just a series of bytes that can be read in either fashion. In the context of programming (Stack Overflow) what's important is knowing a) Whether the input you are getting is in Big Endian or Little Endian, and b) Whether the output you send should be in Big Endian or Little Endian.
If your question is the conventional reading of files, the answer is usually Little Endian. But, for example, network data tends to be Big Endian.

Resources