Convert Redigo Pipeline result to strings - go

I managed to Pipeline multiple HGETALL commands, but I can't manage to convert them to strings.
My sample code is this:
// Initialize Redis (Redigo) client on port 6379
// and default address 127.0.0.1/localhost
client, err := redis.Dial("tcp", ":6379")
if err != nil {
panic(err)
}
defer client.Close()
// Initialize Pipeline
client.Send("MULTI")
// Send writes the command to the connection's output buffer
client.Send("HGETALL", "post:1") // Where "post:1" contains " title 'hi' "
client.Send("HGETALL", "post:2") // Where "post:1" contains " title 'hello' "
// Execute the Pipeline
pipe_prox, err := client.Do("EXEC")
if err != nil {
panic(err)
}
log.Println(pipe_prox)
It is fine as long as you're comfortable showing non-string results.. What I'm getting is this:
[[[116 105 116 108 101] [104 105]] [[116 105 116 108 101] [104 101 108 108 111]]]
But what I need is:
"title" "hi" "title" "hello"
I've tried the following and other combinations as well:
result, _ := redis.Strings(pipe_prox, err)
log.Println(pipe_prox)
But all I get is: []
I should note that it works with multiple HGET key value commands, but that's not what I need.
What am I doing wrong? How should I do it to convert the "numerical map" to strings?
Thanks for any help

Each HGETALL returns it's own series of values, which need to be converted to strings, and the pipeline is returning a series of those. Use the generic redis.Values to break down this outer structure first then you can parse the inner slices.
// Execute the Pipeline
pipe_prox, err := redis.Values(client.Do("EXEC"))
if err != nil {
panic(err)
}
for _, v := range pipe_prox {
s, err := redis.Strings(v, nil)
if err != nil {
fmt.Println("Not a bulk strings repsonse", err)
}
fmt.Println(s)
}
prints:
[title hi]
[title hello]

you can do it like this:
pipe_prox, err := redis.Values(client.Do("EXEC"))
for _, v := range pipe_prox.([]interface{}) {
fmt.Println(v)
}

Related

Event data from Smart Contract parsing issue with go-ethereum

I am unable to get the full data emitted from SC event by using types.Log channel. Is there any way so that I can have all the data from an event emitted?
The event I'm trying to parse:
PairCreated(address indexed,address indexed,address,uint)
My code:
for {
select {
case err := <-sub.Err():
log.Fatal(err)
case vLog := <-logs:
fmt.Printf("Log Block Number: %d\n", vLog.BlockNumber)
fmt.Printf("Log Index: %d\n", vLog.Index)
event := make(map[string]interface{})
err := contractAbi.UnpackIntoMap(event, "PairCreated", vLog.Data)
if err != nil {
log.Fatal(err)
}
fmt.Println(event)
}
}
I could only parse the last two arguments of the event.
I understood what was wrong here.
If an argument is declared as indexed that argument goes to Topics instead of Data. And there can be at most 3 topics. So, I tried to unpack the topics but failed. And succeeded with the following way:
token1 := common.HexToAddress(vLog.Topics[1].Hex())
token2 := common.HexToAddress(vLog.Topics[2].Hex())
And pair was in Data
So, the final code is:
for {
select {
case err := <-sub.Err():
log.Fatal(err)
case vLog := <-logs:
fmt.Printf("Log Block Number: %d\n", vLog.BlockNumber)
fmt.Printf("Log Index: %d\n", vLog.Index)
event := make(map[string]interface{})
err := contractAbi.UnpackIntoMap(event, "PairCreated", vLog.Data)
if err != nil {
log.Fatal(err)
}
fmt.Println(event)
token1 := common.HexToAddress(vLog.Topics[1].Hex())
token2 := common.HexToAddress(vLog.Topics[2].Hex())
}
}

How to count cells with data

I'm getting values from range that I specified. But is there a way to count those values?
I'm using this method to get values from the sheets.
spreadsheetId :=""
range2 :=""
valueRenderOption := "FORMATTED_VALUE"
resp, err :=sheetsService.Spreadsheets.Values.Get(spreadsheetId, range2).ValueRenderOption(valueRenderOption).Do()
if err != nil {
log.Fatal(err)
}
Modification points:
For sheetsService.Spreadsheets.Values.Get(spreadsheetId, range2).ValueRenderOption(valueRenderOption).Do(), when A1:C1 gives to range2, the returned value is {"majorDimension":"ROWS","range":"A1:C1","values":[["###","###","###"]]}. In this case, resp.Values is 2 dimensional array. By this, len(resp.Values) returns 1. I think that this is the reason of your issue.
When you want to retrieve the total number of cells from sheetsService.Spreadsheets.Values.Get(spreadsheetId, range2).ValueRenderOption(valueRenderOption).Do(), how about the following modification?
Modified script:
resp, err := sheetsService.Spreadsheets.Values.Get(spreadsheetId, range2).ValueRenderOption(valueRenderOption).Do()
if err != nil {
log.Fatal(err)
}
count := 0
for _, e := range resp.Values {
count += len(e)
}
fmt.Println(count)
In this case, when A1:C1 is used for range2, 3 is returned.
Note:
For example, when you want to retrieve the total number of cells which are not empty, you can also use the following script.
resp, err := sheetsService.Spreadsheets.Values.Get(spreadsheetId, range2).ValueRenderOption(valueRenderOption).Do()
if err != nil {
log.Fatal(err)
}
count := 0
for _, e := range resp.Values {
for _, f := range e {
if f != "" {
count++
}
}
}
fmt.Println(count)
In this modified script, it supposes that you have already been able to retrieve the values from Google Spreadsheet using Sheets API. Please be careful this.
Reference:
Method: spreadsheets.values.get

Golang Serialize go-radix Tree to file?

I've been working on serializing a radix tree (used for indexing) to a file in golang. The radix tree nodes are storing 6-bit roaring bitmaps (see https://github.com/RoaringBitmap/roaring). The following code is what I am using, and the output I am getting when trying to load it back into memory:
serializedTree := i.index.ToMap()
encodeFile, err := os.Create(fmt.Sprintf("./serialized/%s/%s", appindex.name, i.field))
if err != nil {
panic(err)
}
e := gob.NewEncoder(encodeFile)
err = e.Encode(serializedTree)
encodeFile.Close()
// Turn it back for testing
decodeFile, err := os.Open(fmt.Sprintf("./serialized/%s/%s", appindex.name, i.field))
defer decodeFile.Close()
d := gob.NewDecoder(decodeFile)
decoded := make(map[string]interface{})
err = d.Decode(&decoded)
fmt.Println("before decode", serializedTree)
fmt.Println("after decode", decoded)
if err != nil {
fmt.Println("!!! Error serializing", err)
panic(err)
}
Output:
before decode map[dan:{1822509180252590512} dan1:{6238704462486574203} goodman:{1822509180252590512,6238704462486574203}]
after decode map[]
!!! Error serializing EOF
panic: EOF
goroutine 1 [running]:
main.(*appIndexes).SerializeIndex(0xc000098240)
(I understand the decode is empty because the gob package doesn't modify on EOF error)
I've noticed that when trying with bytes directly, only 15 bytes are being stored on disk (which is way too few). Trying with the encoding/json package with json.Marshall() and json.Unmarshall() and I see 33 bytes stored, but they are loading in empty (the roaring bitmaps are gone):
post encode map[dan:map[] dan1:map[] goodman:map[]]
I feel like this has something to do with the fact that I am trying to serialize a map[string]interface{} rather than something like a map[string]int, but I am still fairly green with golang.
See https://repl.it/#danthegoodman/SelfishMoralCharactermapping#main.go for an example and my testing.
I believe I fixed it by converting the map[string]interface{} into a map[string]*roaring64.Bitmap before writing to disk, then decoding it back into a map[string]*roaring64.Bitmap then converting it back to a map[string]interface{}
m2 := make(map[string]*roaring64.Bitmap)
// Convert m1 to m2
for key, value := range m1 {
m2[key] = value.(*roaring64.Bitmap)
}
fmt.Println("m1", m1)
fmt.Println("m2", m2)
encodeFile, err := os.Create("./test")
if err != nil {
panic(err)
}
e := gob.NewEncoder(encodeFile)
err = e.Encode(m2)
encodeFile.Close()
// Turn it back for testing
decodeFile, err := os.Open("./test")
defer decodeFile.Close()
d := gob.NewDecoder(decodeFile)
decoded := make(map[string]*roaring64.Bitmap)
err = d.Decode(&decoded)
fmt.Println("before decode", m2)
fmt.Println("after decode", decoded)
if err != nil {
fmt.Println("!!! Error serializing", err)
panic(err)
}
m3 := make(map[string]interface{})
// Convert m2 to m3
for key, value := range m2 {
m3[key] = value
}
afterDecTree := radix.NewFromMap(m3)
See https://repl.it/#danthegoodman/VictoriousUtterMention#main.go for a working example

How to send multiple commands in one sesssion but save outputs separately

My code is supposed to SSH to a remote-host (let’s say Routers) and run multiple commands on the remote-host and return the outputs.
The code attached is simplified and has three parts:
Main function: Reads list of commands and then by using the ExecCommands function dials/ssh to a remote-host to execute the commands.
ExecCommands function takes the remote-host IP, list of commands and SSH ClientConfig that is used for SSH. Then it dials to the IP and run the commands one-by-one. At the end, returns the output of all commands in only one string
InsecureClientConfig function that actually doesn’t do much except creating a SSH ClientConfig which is used for ExecCommands function
This program works well when I just want to apply some commands or config and save the wholes result. I mean ExecCommands takes the bunch of commands, push all of them to the remote-host and returns (or saves) the whole output of applied commands in one string as output.
Problem:
I cannot process the output of each command individually. For example, assume that I apply CMD1, CMD2, CMD3, … to the remote-host#1 by using ExecCommands function. Since it gives me back the whole output in one string, it is hard to find which output belongs to which CMD
Goal:
Modify or re-design ExecCommands function to the way that it provides separate output for each command it applies. It means if for remote-host#1 it applies 10 commands, I should have 10 separate strings as output.
Conditions/Restrictions:
I can not create any extra session for commands and must apply all commands in the first SSH session I created, i.e. cannot create multiple Sessions and use Run, Shell, Output, Start function in SSH package
No re-authentication is allowed. For example, I have only a single one-time-password that can be used for all remote-hosts.
Remote hosts don't support "echo" like commands similar to what you have in Linux
The remote-hosts dont’s support any type of APIs
Points:
Main focus is the function ExecCommands. I put a simplified version of the whole code to give an idea
I am using stdout, err := session.StdoutPipe() to run multiple commands which means -as pipe - it's Reader only is possible to be read when the job is done.
An option is to use Session.Stdout and Session.Stdin inside of the for loop in ExecCommands function. Tried but was not successful.
Code:
package main
import (
"errors"
"fmt"
"io/ioutil"
"log"
"time"
"golang.org/x/crypto/ssh"
)
func main() {
// List of the commands should be sent to the devices
listCMDs := []string{
"set cli op-command-xml-output on",
"test routing fib-lookup virtual-router default ip 1.1.1.1",
"test routing fib-lookup virtual-router default ip 2.2.2.2",
"show interface ethernet1/1",
"show interface ethernet1/2",
"test security-policy-match protocol 6 source 1.1.1.1 destination 2.2.2.2 destination-port 443 from ZONE1 to ZONE2",
"test security-policy-match protocol 6 source 10.0.0.1 destination 10.0.2.1 destination-port 443 from ZONE1 to ZONE2",
"exit",
}
sshconfig := InsecureClientConfig("admin", "admin")
s, err := ExecCommands("192.168.1.250", listCMDs, sshconfig)
fmt.Println(s, err)
}
// ExecCommands ...
func ExecCommands(ipAddr string, commands []string, sshconfig *ssh.ClientConfig) (string, error) {
// Gets IP, credentials and config/commands, SSH Config (Timeout, Ciphers, ...) and returns
// output of the device as "string" and an error. If error == nil, means program was able to SSH with no issue
// Creating outerr as Output Error.
outerr := errors.New("nil")
outerr = nil
// Creating Output as String
var outputStr string
// Dial to the remote-host
client, err := ssh.Dial("tcp", ipAddr+":22", sshconfig)
if err != nil {
log.Fatal(err)
}
defer client.Close()
// Create sesssion
session, err := client.NewSession()
if err != nil {
log.Fatal(err)
}
defer session.Close()
// StdinPipee() returns a pipe that will be connected to the remote command's standard input when the command starts.
// StdoutPipe() returns a pipe that will be connected to the remote command's standard output when the command starts.
stdin, err := session.StdinPipe()
if err != nil {
log.Fatal(err)
}
stdout, err := session.StdoutPipe()
if err != nil {
log.Fatal(err)
}
// Start remote shell
err = session.Shell()
if err != nil {
log.Fatal(err)
}
// Send the commands to the remotehost one by one.
for _, cmd := range commands {
_, err := stdin.Write([]byte(cmd + "\n"))
if err != nil {
log.Fatal(err)
}
}
// Wait for session to finish
err = session.Wait()
if err != nil {
log.Fatal(err)
}
strByte, _ := ioutil.ReadAll(stdout)
outputStr = string(strByte)
return outputStr, outerr
}
// InsecureClientConfig ...
func InsecureClientConfig(userStr, passStr string) *ssh.ClientConfig {
SSHconfig := &ssh.ClientConfig{
User: userStr,
Timeout: 5 * time.Second,
Auth: []ssh.AuthMethod{ssh.Password(passStr)},
HostKeyCallback: ssh.InsecureIgnoreHostKey(),
Config: ssh.Config{
Ciphers: []string{"aes128-ctr", "aes192-ctr", "aes256-ctr", "aes128-cbc", "aes192-cbc",
"aes256-cbc", "3des-cbc", "des-cbc"},
KeyExchanges: []string{"diffie-hellman-group1-sha1",
"diffie-hellman-group-exchange-sha1",
"diffie-hellman-group14-sha1"},
},
}
return SSHconfig
}
This works properly:
package main
import (
"bufio"
"errors"
"fmt"
"log"
"time"
"golang.org/x/crypto/ssh"
)
func main() {
// List of the commands should be sent to the devices
listCMDs := []string{
"set cli op-command-xml-output on\n",
"test routing fib-lookup virtual-router default ip 1.1.1.1\n",
"test routing fib-lookup virtual-router default ip 2.2.2.2\n",
"show interface ethernet1/1\n",
"show interface ethernet1/2\n",
"test security-policy-match protocol 6 source 1.1.1.1 destination 2.2.2.2 destination-port 443 from ZONE1 to ZONE2\n",
"test security-policy-match protocol 6 source 10.0.0.1 destination 10.0.2.1 destination-port 443 from ZONE1 to ZONE2\n",
"exit",
}
sshconfig := InsecureClientConfig("admin", "Ghazanfar1!")
s, _ := ExecCommands("192.168.1.249", listCMDs, sshconfig)
for _, item := range s {
fmt.Println(item)
fmt.Println("-------------------------------")
}
}
// ExecCommands ...
func ExecCommands(ipAddr string, commands []string, sshconfig *ssh.ClientConfig) ([]string, error) {
// Gets IP, credentials and config/commands, SSH Config (Timeout, Ciphers, ...) and returns
// output of the device as "string" and an error. If error == nil, means program was able to SSH with no issue
// Creating outerr as Output Error.
outerr := errors.New("nil")
outerr = nil
// Creating Output as String
var outputStr []string
var strTmp string
// Dial to the remote-host
client, err := ssh.Dial("tcp", ipAddr+":22", sshconfig)
if err != nil {
log.Fatal(err)
}
defer client.Close()
// Create sesssion
session, err := client.NewSession()
if err != nil {
log.Fatal(err)
}
defer session.Close()
// StdinPipee() returns a pipe that will be connected to the remote command's standard input when the command starts.
// StdoutPipe() returns a pipe that will be connected to the remote command's standard output when the command starts.
stdin, err := session.StdinPipe()
if err != nil {
log.Fatal(err)
}
stdout, err := session.StdoutPipe()
if err != nil {
log.Fatal(err)
}
// Start remote shell
err = session.Shell()
if err != nil {
log.Fatal(err)
}
stdinLines := make(chan string)
go func() {
scanner := bufio.NewScanner(stdout)
for scanner.Scan() {
stdinLines <- scanner.Text()
}
if err := scanner.Err(); err != nil {
log.Printf("scanner failed: %v", err)
}
close(stdinLines)
}()
// Send the commands to the remotehost one by one.
for i, cmd := range commands {
_, err := stdin.Write([]byte(cmd + "\n"))
if err != nil {
log.Fatal(err)
}
if i == len(commands)-1 {
_ = stdin.Close() // send eof
}
// wait for command to complete
// we'll assume the moment we've gone 1 secs w/o any output that our command is done
timer := time.NewTimer(0)
InputLoop:
for {
timer.Reset(time.Second)
select {
case line, ok := <-stdinLines:
if !ok {
log.Println("Finished processing")
break InputLoop
}
strTmp += line
strTmp += "\n"
case <-timer.C:
break InputLoop
}
}
outputStr = append(outputStr, strTmp)
//log.Printf("Finished processing %v\n", cmd)
strTmp = ""
}
// Wait for session to finish
err = session.Wait()
if err != nil {
log.Fatal(err)
}
return outputStr, outerr
}
// InsecureClientConfig ...
func InsecureClientConfig(userStr, passStr string) *ssh.ClientConfig {
SSHconfig := &ssh.ClientConfig{
User: userStr,
Timeout: 5 * time.Second,
Auth: []ssh.AuthMethod{ssh.Password(passStr)},
HostKeyCallback: ssh.InsecureIgnoreHostKey(),
Config: ssh.Config{
Ciphers: []string{"aes128-ctr", "aes192-ctr", "aes256-ctr", "aes128-cbc", "aes192-cbc",
"aes256-cbc", "3des-cbc", "des-cbc"},
KeyExchanges: []string{"diffie-hellman-group1-sha1",
"diffie-hellman-group-exchange-sha1",
"diffie-hellman-group14-sha1"},
},
}
return SSHconfig
}
Since you have limited number of commands to run on special hardwares and you know the pattern of the each command's output, you may use strings.Split or regexp to split the output.
And if you do not have echo command, but know any command with fast response with unique output pattern, then you may replace it with echo command in the following example (number 2).
Since a session only accepts one call to Run, Start, Shell, Output, or CombinedOutput, and you do not want to start a new session per command:
The key is to use a strings.Builder and empty it using sb.Reset() befor sending the command, and using io.Copy to copy concurrently the session's stdout into strings.Builder (assuming you do not need session's stderr):
sb := new(strings.Builder)
go io.Copy(sb, stdout)
This works if you know how much to wait for each command (tested):
sb := new(strings.Builder)
go io.Copy(sb, stdout)
commands := []string{"uname -a", "sleep 1", "pwd", "whoami", "exit"}
wait := []time.Duration{10, 1200, 20, 10, 10} // * time.Millisecond
ans := []string{}
time.Sleep(10 * time.Millisecond) // wait for the ssh greetings
// Send the commands to the remotehost one by one.
for i, cmd := range commands {
sb.Reset()
fmt.Println("*** command:\t", cmd)
_, err := stdin.Write([]byte(cmd + "\n"))
if err != nil {
log.Fatal(err)
}
time.Sleep(wait[i] * time.Millisecond) // wait for the command to finish
s := sb.String()
fmt.Println("*** response:\t", s)
ans = append(ans, s)
}
Using string delimiter and strings.Split (Note: You may replace echo with any fast command with known output pattern):
sb := new(strings.Builder)
go io.Copy(sb, stdout)
commands := []string{"uname -a", "sleep 1", "pwd", "whoami"}
delim := "********--------========12345678"
for _, cmd := range commands {
_, err = stdin.Write([]byte("echo " + delim + "\n"))
if err != nil {
log.Fatal(err)
}
_, err := stdin.Write([]byte(cmd + "\n"))
if err != nil {
log.Fatal(err)
}
}
_, err = stdin.Write([]byte("exit\n"))
if err != nil {
log.Fatal(err)
}
err = session.Wait() // Wait for session to exit
if err != nil {
log.Fatal(err)
}
ans := strings.Split(sb.String(), delim)
ans = ans[1:] // remove ssh greetings
Check this out: https://github.com/yahoo/vssh
You can set sessions to how many commands you need to run concurrently then send each command to remote host through run method and get the result individually!

golang.org/x/sys/windows/registry (k Key)ReadSubKeyNames() returns empty slice

I'm trying to read a list of registry names under the given key, but i get an empty slice of strings, instead of actual registry data. Anything i'm doing incorrectly here?
Expected result: []string{"ApplicationBase", "RunTimeVersion", ...}
Actual result printed by CMD: "PS Sub Key Names: []"
k, err := registry.OpenKey(registry.LOCAL_MACHINE,`SOFTWARE\Microsoft\PowerShell\1\PowerShellEngine`, registry.QUERY_VALUE|registry.ENUMERATE_SUB_KEYS)
if err != nil {
log.Fatal(err)
}
defer k.Close()
sn, err := k.ReadSubKeyNames(-1)
if err != nil {
log.Fatal(err)
}
fmt.Printf("PS Sub Key Names: %q\n", sn)
I figured it out - should have used (k Key)ReadValueNames to get keys of values.

Resources