I'm trying to write a Go program, to download data from aws kinesis data stream. I read that kinesis data stream encode the data with base64, so I need first decode with base64. However, I can't figure out what encoding was used on the data as it is passed, from cloudwatch logs to kinesis data stream.
I'm trying the different decoding method but none works. My unprocessed byte array downloaded from kinesis data stream is as the following:
[31 139 8 0 0 0 0 0 0 0 53 206 65 11 130 64 16 134 225 191 178 204 89 130 178 34 246 22 97 30 178 130 12 58 68 196 166 147 14 233 174 236 140 69 68 255 61 204 58 190 204 7 243 188 160 70 102 83 224 254 217 32 104 88 108 55 251 221 54 57 175 163 52 157 199 17 4 224 30 22 125 119 169 92 155 63 140 100 101 226 10 134 0 42 87 196 222 181 13 104 232 43 21 143 166 238 147 219 11 103 158 26 33 103 151 84 9 122 6 125 60 125 119 209 29 173 116 249 2 202 251 185 80 141 44 166 110 64 15 167 227 201 48 28 79 166 225 108 20 6 127 94 7 56 36 234 199 83 63 158 86 139 18 179 27 217 66 149 104 42 41 149 187 170 28 89 200 154 238 179 90 145 69 38 86 252 165 13 224 125 122 127 0 234 141 66 79 242 0 0 0]
Can someone give me some tips how to process this piece of data?
You can use a subscription filter with Kinesis, Lambda, or Kinesis Data Firehose. Logs that are sent to a receiving service through a subscription filter are base64 encoded and compressed with the gzip format.
https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/SubscriptionFilters.html
Related
I have a list of more than 100000 records.
per example the values from 21 to 84 are continuous, then it will be 21-84 but if it is not continuous as the case 84 87, then it need to be 84,87 separated by ,
at beginning of each line will be the value 11111.
The values from the list will be in the column range of 21 to 80 with, at last.
The length of each row need to be maximum 80.
here is the input file.
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
87
85
86
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
108
111
109
112
110
113
115
114
117
116
118
124
125
120
122
123
126
132
127
133
128
130
131
135
136
137
138
139
140
141
142
143
144
145
146
148
147
149
150
151
152
153
154
155
156
158
157
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
184
183
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
214
here in the output file desired.
111111 21-84,87,85-86,88-106,108,111,109,112,110,113,115,114,117,
111111 116,118,124-125,120,122-123,126,132,127,133,128,130-131,
111111 135-146,148,147,149-156,158,157,159-182,184,183,185-212,214,
thanks in advance
Presented without explanation: check the man pages for the commands used and come back with questions:
awk '
function printrange() { print start (start == last ? "" : "-" last) }
NR == 1 {start=last=$1; next}
$1 == last+1 {last=$1; next}
{printrange(); start=last=$1}
END {printrange()}
' file | paste -sd" " | fold -sw 60 | tr ' ' ',' | sed 's/^/111111 /'
111111 21-84,87,85-86,88-106,108,111,109,112,110,113,115,114,117,
111111 116,118,124-125,120,122-123,126,132,127,133,128,130-131,
111111 135-146,148,147,149-156,158,157,159-182,184,183,185-212,214
I'm trying to use a variable range with ruby, but my code does not work;
ruby -e ' input2=145..170 ; input3= input2.to_s.gsub(/(.*?)\.\.(.*?)/) { 5.upto($2.to_i) { |i| print i, " " } }; print input3' > zzmf
But I obtained 5170
This part fails:
5.upto($2.to_i) { |i| print i, " " }
I expected:
5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 5170
I don't think gsub is what you need, try the match example below. [2] gets the second match from the regex /(\d+)..(\d+)/ applied to "147..170"
5.upto("147..170".match(/(\d+)\.\.(\d+)/)[2].to_i) { |i| print i, " "}
gsub is intended for string substitution.
https://ruby-doc.org/core-2.1.4/String.html#method-i-gsub
I see my code and I confuse in regular expression
I use this .*?
and the correct is this .*
(.*)/
ruby -e ' input2=145..170 ; input3= input2.to_s.gsub(/(.*?)\.\.(.*)/) { 5.upto($2.to_i) { |i| print i, " " } }; print input3' > zzmf
thanks for your responses
Below is a full section of bytes stream I got from Exchange2013 server after sending activesync command 'itemoperations' from iPhone.
31 139 8 0 0 0 0 0 4 0 237 189 7 96 28 73 150 37 38 47 109 202 123 127 74 245 74 215 224 116 161 8 128 96 19 36 216 144 64 16 236 193 136 205 230 146 236 29 105 71 35 41 171 42 129 202 101 86 101 93 102 22 64 204 237 157 188 247 222 123 239 189 247 222 123 239 189 247 186 59 157 78 39 247 223 255 63 92 102 100 1 108 246 206 74 218 201 158 33 128 170 200 31 63 126 124 31 63 34 126 237 95 243 167 127 141 95 227 183 58 253 226 215 222 253 53 126 205 23 207 248 199 175 241 155 255 196 175 125 255 119 191 151 61 220 167 127 240 247 111 245 123 253 26 191 249 119 127 237 54 127 215 222 93 149 89 177 196 71 207 127 237 215 95 61 124 253 229 79 31 191 251 226 233 233 15 126 234 167 143 175 190 248 233 183 87 95 60 61 198 255 119 190 124 243 19 59 47 62 255 226 7 191 207 155 159 250 233 23 139 223 231 222 139 167 211 221 23 63 248 226 250 167 126 250 247 217 161 191 247 126 234 233 239 115 255 197 226 171 189 159 250 233 239 44 94 252 244 23 244 115 122 253 226 167 207 246 190 120 67 127 63 125 187 67 237 174 95 44 78 247 94 188 249 125 126 240 226 7 63 177 75 144 232 253 179 123 47 126 154 127 18 220 47 222 125 177 248 234 7 218 159 254 255 212 251 221 252 127 26 126 255 6 248 30 95 253 212 27 243 25 193 249 193 219 31 124 241 211 223 41 191 248 193 233 15 8 151 125 134 251 70 254 254 242 233 23 215 95 60 125 50 255 125 246 126 159 61 250 219 194 122 241 148 218 253 128 218 253 244 23 192 237 7 47 8 159 23 63 125 122 77 227 218 37 24 10 231 39 8 206 23 87 244 61 189 251 157 183 47 126 250 43 140 135 218 31 83 59 124 126 76 99 195 223 103 123 47 208 15 141 235 139 31 224 221 51 130 71 125 124 206 116 217 121 241 131 175 238 125 241 131 87 229 139 207 127 31 162 171 224 255 133 197 95 255 14 199 255 238 133 163 129 155 31 249 158 254 254 170 67 163 83 31 30 254 255 217 175 241 107 226 249 127 0 225 220 243 27 28 2 0 0
I don't know why it is not normal and expect it sholud start like this:
3 1 106 0 0 20 69 77 3 49 0 1 78 70 77 3 49 0 1 0 17 81 3 82 103 65 65 65 65 65 117 50 72 120 85 112 65 ..........
I think the response must have been encrypted, but what's the encryption algorithm?
Will be very appreciated for any ideas.
I have got the answer, the stream 31 139 8 0 0 0 0 0 4 0 237 189 7 96 28 73 150 37 ...... has been compressed by Gzip
This question continues the discussion started here. I found out that the HTTP response body can't be unmarshaled into JSON object because of deflate compression of the latter. Now I wonder how can I perform decompression with Golang. I will appreciate anyone who can show the errors in my code.
Input data
I've dumped the HTTP response body into the 'test' file. Here is it:
$ cat test
x��PAN�0�
;��NtJ�FӮdU�|"oVR�C%�f�����Z.�^Hs�dW뮑�'��DH�S�SFVC����r)G,�����<���z}�x_g�+�2��sl�r/�Oy>��J3\�G�9���N���#[5M�^v/�2Ҕ��|�h��[�~7�_崛<D*���/��i
Let's make sure that this file can be decompressed and even contains valid JSON:
$ zlib-flate -uncompress < test
{"timestamp":{"tv_sec":1428488670,"tv_usec":197041},"string_timestamp":"2015-04-08 10:24:30.197041","monitor_status":"enabled","commands":{"REVERSE_LOOKUP":{"cache":{"outside":{"successes":0,"failures":0,"size":0,"time":0},"internal":{"successes":0,"failures":0,"size":0,"time":0}},"disk":{"outside":{"successes":0,"failures":0,"size":0,"time":0},"internal":{"successes":13366,"failures":0,"size":0,"time":501808}},"total":{"storage":{"successes":0,"failures":0},"proxy":{"successes":13366,"failures":0}}},"clients":{}}}
$ zlib-flate -uncompress < test | python -m json.tool
{
"commands": {
"REVERSE_LOOKUP": {
"cache": {
....
Source code
package main
import (
"bytes"
"compress/flate"
"fmt"
"io/ioutil"
)
func main() {
fname := "./test"
content, err := ioutil.ReadFile(fname)
if err != nil {
panic(err)
}
fmt.Println("File content:\n", content)
enflated, err := ioutil.ReadAll(flate.NewReader(bytes.NewReader(content)))
if err != nil {
panic(err)
}
fmt.Println("Enflated:\n", enflated)
}
Error
$ go run uncompress.go
File content:
[120 156 181 80 65 78 195 48 16 252 10 242 57 69 118 226 166 38 247 156 64 42 42 130 107 100 156 165 88 196 118 149 93 35 160 234 223 89 183 61 112 42 226 192 109 118 118 102 103 180 123 65 62 0 146 13 59 209 237 5 189 15 8 78 116 74 215 70 27 211 174 100 85 184 124 34 111 86 82 171 67 37 144 102 31 183 195 15 167 168 165 90 46 164 94 72 115 165 100 87 235 174 145 215 39 189 168 68 72 209 83 154 7 22 83 70 86 67 180 207 19 140 188 114 41 4 27 71 44 225 155 254 169 223 60 244 195 221 122 125 251 120 95 24 103 221 43 20 144 50 161 31 143 16 179 115 128 8 108 225 114 47 214 79 121 62 15 232 191 224 8 74 51 6 92 213 71 130 57 218 233 175 78 182 142 30 223 254 35 91 53 77 219 94 118 47 165 50 210 148 18 148 232 124 128 31 104 183 151 91 176 126 55 167 143 207 95 3 15 229 180 155 60 68 42 159 231 241 27 47 165 167 25]
panic: flate: corrupt input before offset 5
goroutine 1 [running]:
runtime.panic(0x4a7180, 0x5)
/usr/lib/go/src/pkg/runtime/panic.c:266 +0xb6
main.main()
/home/isaev/side-projects/elliptics-manager/uncompress.go:20 +0x2a3
exit status 2
PS Ubuntu 14.10, Go 1.2.1
Your input is not a simple deflated block, it's a zlib stream.
According to the ZLIB Compressed Data Format Specification 3.3 the first 2 bytes are:
-------------
| CMF | FLG |
-------------
The Compression Method and flags. Your input starts with [120, 156] which is 78 9C in hexa. This is the Default Compression. Also no dictionary follows, so the subsequent data is the compressed data.
Bits 0 to 3 are CM Compression Method and bits 4 to 7 are CINFO Compression Info. In this case CINFO=7 indicates a 32K window size, CM=8 denotes the "deflate" compression method. FLG bit 5 tells if a dictionary is preset, which is in this case. Details of the FLG are also in the linked RFC 1950.
So your input basically tells the rest of the data was constructed using default compression, but the go flate package does not decode this.
Change your decompression to omit the first 2 bytes like this and it will work:
enflated, err := ioutil.ReadAll(flate.NewReader(bytes.NewReader(content[2:])))
Try it on the Go Playground. But...
Use Proper ZLib decompression!
We got lucky this time because the compression level is the default and dictionary was preset. If not, you won't be able to decode it using the flate package. Since the input is a zlib stream, you should use the compress/zlib package to properly decode it and not rely on luck:
r, err := zlib.NewReader(bytes.NewReader(content))
if err != nil {
panic(err)
}
enflated, err := ioutil.ReadAll(r)
if err != nil {
panic(err)
}
fmt.Println(string(enflated))
Try the zlib variant on the Go Playground.
I am looking at all possible paths through a graph. I have written a DFS algorithm that finds all these paths. I want to make sure that my algorithm works correctly and that no two paths are identical. My algorithm returns a list that looks as follows:
....
[[2770]]
[1] 1 2 3 52 53 54 55 56 57 58 59 60 12 11 10 9 8 78 79 80 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129
[38] 130 131 132 133 134 137 138 139 140 141 142 143 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166
[[2771]]
[1] 1 2 3 52 53 54 55 56 57 58 59 60 12 11 10 9 8 78 79 80 113 114 115 143 144 145 146 147 148 149 150 151 152 153 154 155 156
[38] 157 158 159 160 161 162 163 164 165 166
[[2772]]
[1] 1 2 3 52 53 54 55 56 57 58 59 60 12 11 10 9 8 78 79 80 113 114 115 143 150 151 152 153 154 155 156 157 158 159 160 161 162
[38] 163 164 165 166
As you can see, the list is 2772 elements long. This means there are 2,772 paths through this graph. How can I easily compare all the list elements to make sure there are no duplicates. Just to be clear, the same set of numbers but in a different ordering represents a different path and is not a duplicate!
Thank you for your help!
maybe something like
test<-list(1:2,3:4,5:7,1:10,3:4,4:3)
dups<-duplicated(test)
idups<-seq_along(test)[dups]