etcd fatal error: failed to reserve page summary memory - go

when i try to run etcd from user account (Total RAM: 8GB, virtual memory of 1GB) in a 64 bit Linux environment (limited resources), I am getting the following error:
fatal error: failed to reserve page summary memory
runtime stack:
runtime.throw(
{0x108df50?, 0x7fffd8aa6850?}
)
/usr/local/go/src/runtime/panic.go:1047 +0x5d fp=0x7fffd8aa6800 sp=0x7fffd8aa67d0 pc=0x437edd
runtime.(*pageAlloc).sysInit(0x194f2f0)
/usr/local/go/src/runtime/mpagealloc_64bit.go:82 +0x195 fp=0x7fffd8aa6888 sp=0x7fffd8aa6800 pc=0x42c9b5
runtime.(*pageAlloc).init(0x194f2f0, 0x194f2e0, 0x0?)
/usr/local/go/src/runtime/mpagealloc.go:324 +0x70 fp=0x7fffd8aa68b0 sp=0x7fffd8aa6888 pc=0x42a610
runtime.(*mheap).init(0x194f2e0)
/usr/local/go/src/runtime/mheap.go:721 +0x13f fp=0x7fffd8aa68e8 sp=0x7fffd8aa68b0 pc=0x427a3f
runtime.mallocinit()
/usr/local/go/src/runtime/malloc.go:407 +0xb2 fp=0x7fffd8aa6910 sp=0x7fffd8aa68e8 pc=0x40c4f2
runtime.schedinit()
/usr/local/go/src/runtime/proc.go:693 +0xab fp=0x7fffd8aa6970 sp=0x7fffd8aa6910 pc=0x43b8ab
runtime.rt0_go()
/usr/local/go/src/runtime/asm_amd64.s:345 +0x11c fp=0x7fffd8aa6978 sp=0x7fffd8aa6970 pc=0x469c3c
But the same works fine from root account.
Can someone please support?
I was expecting this should give same output from root and user account. This seems like a virtual memory issue to me.

As your request isn't very clear, I'm going to share with you a working piece of code that can successfully interact with etcd:
package main
import (
"context"
"fmt"
"time"
clientv3 "go.etcd.io/etcd/client/v3"
)
func main() {
cli, err := clientv3.New(clientv3.Config{
Endpoints: []string{"localhost:2379"},
DialTimeout: 5 * time.Second,
})
if err != nil {
panic(err)
}
defer cli.Close()
// insert values
res, err := cli.Put(context.Background(), "key1", "val1")
if err != nil {
panic(err)
}
fmt.Printf("revision: %q\n", res.Header.Revision)
// delete all keys
defer cli.Delete(context.Background(), "key", clientv3.WithPrefix())
// get object from etcd
key, err := cli.Get(context.Background(), "key1")
if err != nil {
panic(err)
}
fmt.Printf("key: %q\tvalue: %q\n", key.Kvs[0].Key, key.Kvs[0].Value)
}
This code assumes that you've got a running instance of ETCD that listens on port 2379 for incoming requests. I run ETCD with Docker with the following script:
docker run -d -p 2379:2379 -p 2380:2380 --env ALLOW_NONE_AUTHENTICATION=yes --env ETCD_ADVERTISE_CLIENT_URLS=http://etcd-server:2379 bitnami/etcd:latest
Hope this helps in clarifying a little bit!

Related

Tendermint access leveldb via node

I have a sample go application running with tendermint. I would like to access the data Tendermint's blockchain stores in leveldb while both the application and the node are running.
I have tried to run a basic go script that iterates over the tx_index.db which stores the tx.hash and the corresponding result of every transaction. But when I try to run the script the program panics with an error message.
Here's the go script to iterate over the tx_index.db
package main
import (
"fmt"
"github.com/syndtr/goleveldb/leveldb"
)
func main() {
db, err := leveldb.OpenFile("tx_index.db", nil)
if err != nil {
panic(err)
}
iter := db.NewIterator(nil, nil)
for iter.Next() {
key := iter.Key()
h_key := fmt.Sprintf("%X", key)
value := iter.Value()
h_value := fmt.Sprintf("%X", value)
fmt.Printf("Key: %v \nValue: %v \n", h_key, h_value)
}
iter.Release()
err = iter.Error()
defer db.Close()
}
Upon running the above script while the application and the tendermint node are running. I get the following error message!
panic: resource temporarily unavailable
goroutine 1 [running]:
main.main()
$HOME/tendermint/data/read_db.go:11 +0x375
exit status 2
I am interested in accessing the tx_index.db while the application and tendermint node are running to get the transaction hash of the transactions performed.
PS: If it's feasible I'd like to know if there's a way to get tx.hash by accessing the tx_index.db by interacting with the tendermint node.

Using Go-IPFS Programmatically

I would love to be able to use Go-IPFS within my Go program, however it is totally undocumented. This is where my reasearch lead me:
package main
import (
"context"
"fmt"
"io/ioutil"
"log"
"os"
"path/filepath"
"gx/ipfs/QmSP88ryZkHSRn1fnngAaV2Vcn63WUJzAavnRM9CVdU1Ky/go-ipfs-cmdkit/files"
"github.com/ipfs/go-ipfs/core"
"github.com/ipfs/go-ipfs/core/coreunix"
)
func main() {
ctx := context.Background()
node, err := core.NewNode(ctx, &core.BuildCfg{})
if err != nil {
log.Fatalf("Failed to start IPFS node: %v", err)
}
reader, err := coreunix.Cat(ctx, node, "QmPZ9gcCEpqKTo6aq61g2nXGUhM4iCL3ewB6LDXZCtioEB")
if err != nil {
log.Fatalf("Failed to look up IPFS welcome page: %v", err)
}
blob, err := ioutil.ReadAll(reader)
if err != nil {
log.Fatalf("Failed to retrieve IPFS welcome page: %v", err)
}
fmt.Println(string(blob))
}
However I am not sure about the difference of
context.Background() Vs context.TODO() Vs context.WithCancel(context.Background()).
And most importantly, how can choose where IPFS will put the IPFS repo and make sure it also initializes it?
How can I enable and use Pubsub along with its commands subscribe and publish?
How can I add and pin a file to IPFS with the possibility to also input a stream for big files?
Is coreunix.Cat suitable to read files with a stream as well?
How can I keep the node "listening" like when you run the ipfs daemon from the CLI and have everything runnng on all the ports like webui, swarm, etc.?
How about this to add files? Does this use streams or reads the entire file into memory? How can this be improved?
func addFile(ctx *context.Context, node *core.IpfsNode, path *string) error {
file, err := os.Open(*path)
if err != nil {
return err
}
adder, err := coreunix.NewAdder(*ctx, node.Pinning, node.Blockstore, node.DAG)
if err != nil {
return err
}
filename := filepath.Base(*path)
fileReader := files.NewReaderFile(filename, filename, file, nil)
adder.AddFile(fileReader)
adder.PinRoot()
return nil
}
You may want to breakdown your question into smaller pieces, I have been playing with a source code of go-ipfs for a while and here's the general instruction I could give you:
Most of the data structures like, context, DAG, IPFSNode and so on are defined in form of go structures, and you should be able to find them in gx/.../... directory where also you should be able to see detailed information about each variable used ( simple string search through the directory should get you to your needed source file)
All the methods are defined in folder github.com/../.. directory
Clear the concept of pointers as they are using pointers most of the time to pass parameters to functions

Exclude SSH Command Execution Status from Output

I am trying to develop a SSH client using Go language. I've written the below code to get commands from user, execute them on the remote server and print the response.
There's a small problem with the below code. Printing output on the screen prints the command execution status(nil/error status) at the end of response. How do I exclude this from my output?
SSH.go
package main
import "fmt"
import "io"
import "bufio"
import "os"
import "net"
import "golang.org/x/crypto/ssh"
func main(){
sshConfig := &ssh.ClientConfig{
User: "[USERNAME]",
Auth: []ssh.AuthMethod{
ssh.Password("[PASSWORD]"),
},
HostKeyCallback: func(hostname string, remote net.Addr, key ssh.PublicKey) error {
return nil
},
}
connection,err := ssh.Dial("tcp", "[IP]:[PORT]", sshConfig)
if err != nil {
fmt.Println("Failed to connect: %s", err)
}
reader := bufio.NewReader(os.Stdin)
for{
input, _ := reader.ReadString('\n')
session,err := connection.NewSession()
if err != nil {
fmt.Println("Failed to create session: %s", err)
}
stdout,err := session.StdoutPipe()
if err != nil {
fmt.Println("Failed to get stdout: %s", err)
}
go io.Copy(os.Stdout, stdout)
output := session.Run(input);
fmt.Println(output)
}
}
Current Result
hello
Process exited with status 127
df -hP /tmp
Filesystem Size Used Avail Use% Mounted on
/dev/sda6 198G 13G 176G 7% /
<nil>
Expected Result
hello
df -hP /tmp
Filesystem Size Used Avail Use% Mounted on
/dev/sda6 198G 13G 176G 7% /
The <nil> pinted above is the output of session.Run, which returns an error (as per docs).
The output of the command is already being displayed on your console by the following goroutine:
go io.Copy(os.Stdout, stdout)
You don't need to print the output, which is nil since no error occurs when session.Run is call complete. A better thing to do would be to check the error:
// ...
if err := session.Run(input); err != nil {
// handle error
}
// ...

How to connect to a remote socket in docker/engine-api?

I want to retrieve all docker images of a remote machine, so I am using docker/engine-api: https://github.com/docker/engine-api
I was successful in returning the docker images of my local machine
with the following code:
package main
import (
"fmt"
"github.com/docker/engine-api/client"
"github.com/docker/engine-api/types"
"golang.org/x/net/context"
)
func main() {
defaultHeaders := map[string]string{"User-Agent": "engine-api-cli-1.0"}
cli, err := client.NewClient("unix:///var/run/docker.sock", "v1.22", nil, defaultHeaders)
if err != nil {
panic(err)
}
options := types.ContainerListOptions{All: true}
containers, err := cli.ContainerList(context.Background(), options)
if err != nil {
panic(err)
}
for _, c := range containers {
fmt.Println(c.ID)
}
}
But now does anybody know how can I retrieve the docker images of a remote machine given its address,username, and password
That kind of Unix socket is only accessible through proccesses in the same machine.
To access your docker from a remote machine you need to run it with a special configuration to run over ip.
This configuration is DOCKER_OPTS="-H <ip_address>:<port>" (or -H 0.0.0.0:<port> if you whant it to listen on all interfaces), and it depends the version you are running of docker where you must configure it.
Here you can find more information on where to configure DOCKER_OPTS depending on the operation system version.
Hope it helps!

Linux Network namespaces unexpected behavior

So I've been playing around with Network namespaces recently.
I put together a simple code, built it and noticed something very weird happening.
The code is as follows:
package main
import (
"fmt"
"log"
"net"
"os"
"path"
"syscall"
)
const (
NsRunDir = "/var/run/netns"
SelfNetNs = "/proc/self/ns/net"
)
func main() {
netNsPath := path.Join(NsRunDir, "myns")
os.Mkdir(NsRunDir, 0755)
if err := syscall.Mount(NsRunDir, NsRunDir, "none", syscall.MS_BIND, ""); err != nil {
log.Fatalf("Could not create Network namespace: %s", err)
}
fd, err := syscall.Open(netNsPath, syscall.O_RDONLY|syscall.O_CREAT|syscall.O_EXCL, 0)
if err != nil {
log.Fatalf("Could not create Network namespace: %s", err)
}
syscall.Close(fd)
if err := syscall.Unshare(syscall.CLONE_NEWNET); err != nil {
log.Fatalf("Could not clone new Network namespace: %s", err)
}
if err := syscall.Mount(SelfNetNs, netNsPath, "none", syscall.MS_BIND, ""); err != nil {
log.Fatalf("Could not Mount Network namespace: %s", err)
}
if err := syscall.Unmount(netNsPath, syscall.MNT_DETACH); err != nil {
log.Fatalf("Could not Unmount new Network namespace: %s", err)
}
if err := syscall.Unlink(netNsPath); err != nil {
log.Fatalf("Could not Unlink new Network namespace: %s", err)
}
ifcs, _ := net.Interfaces()
for _, ifc := range ifcs {
fmt.Printf("%#v\n", ifc)
}
}
Now, when you run this code on Trusty 14.04, you will see something weird happening.
This happens when you run the binary several times in a row.
Sometimes it prints out all Host's interfaces, sometimes it simply prints out just a loopback interface which means that the range loop at the end of the program seems to be executing once when the namespace is still attached and sometimes when it's already been detached.
I'm totally confused why this is happening but I'm thinking it's either my code or maybe I'm just missing out something in terms of the program execution or some kernel stuff.
Any help would be massively appreciated.
Thanks
Update1:
So it seems the "strange" behaviour has to do with how golang schedules goroutines across OS Threads. So you need to make sure that you handle the runtime well. What I mean by that is, that if you lock the code execution to one OS Thread you will get consistent results. You can do this by adding the following runtime package statement:
runtime.LockOSThread()
However this still does not solve my problem, but now I think it all comes down to understanding of Namespaces. I need to look more into that.
Update2:
To give you a little bit more context into why you should use the above OS Thread Lock when running bunch of syscalls and experience the similar "strangeness" of correct behavior, give ththis blogpost a read. It describes runtime and go schedulers. It was written for go 1.1 but it still gives very good overview.

Resources