Hello I'm about to port my two almost working simple fuse filesystems from bazillion fuse to go-fuse. go-fuse api seems more complex.
The question is:
In NewServer(), which RawFileSystem to use?
How to implement callbacks for read, readdir etc.?
Where to use WaitMount()?
What are DeleteNotify(), EntryNotify()?
ok i found the solutions
1.
make a struct that contains nodefs.Node:
type my_root struct {nodefs.Node}
initialize it
my = &my_root{Node: nodefs.NewDefaultNode()}
make a connection and a raw filesystem
con := nodefs.NewFileSystemConnector(my, nil)
raw := fuse.NewRawFileSystem(con.RawFS())
finally, fire up the fuse fs
server, err := fuse.NewServer(raw, f.dir, optz)
like this:
func (my_root) OpenDir(context *fuse.Context) ([]fuse.DirEntry, fuse.Status) {}
func (my_root) Lookup(out *fuse.Attr, name string, context *fuse.Context) (node *nodefs.Inode, code fuse.Status)
after step 1, like this:
server.WaitMount()
i didn't need this.
Related
I'm rather new to Golang and not sure yet, how to use certain language constructs. Currently I have following code (with test debug outputs), which does not provide expected result:
json, _ := json.Marshal(struct)
fmt.Println(json)
f,_ := os.Create(fmt.Sprintf("/tmp/%s.json", "asd"))
i,_ := f.Write(json)
fmt.Println(i)
b, err := ioutil.ReadAll(f)
fmt.Print(b)
I expect the following behaviour:
translating the struct to a byte array
creating a new file
append the byte array to the file
However, the file is always empty when I run the code in my environment (AWS Lambda), as well as using it in the Golang Playground.
The output of above code looks like this:
[123 34 ... <hug array of bytes>]
1384
[]
which leads me to believe I'm using f.Write() not correctly, although I followed the package documentation. All other outputs indicate expected behavior, so what is my mistake? I'm somewhat restricted to using the File interface, otherwise I'd have gone with ioutil.WriteFile(). My assumption is a misunderstanding of pointer/values at some point, but the compiler prevented a usage of &f.
After f.Write(), your current position in the file is at the end of it, so ioutil.ReadAll() will read from that position and return nothing.
You need to call f.Sync() to make sure that the data is persistently saved to the disk, and then f.Seek(0, 0) to rewind to the beginning of the file first.
Update: from comments, it seems that you only need to serialize the JSON and pass it forward as io.Reader, for that you don't really need a file, thanks to bytes.Buffer:
data, _ := json.Marshal(s)
buf := bytes.NewBuffer(data)
b, _ := ioutil.ReadAll(buf)
fmt.Print(string(b))
I am trying to persist a given stream of data to an S3 compatible storage.
The size is not known before the stream ends and can vary from 5MB to ~500GB.
I tried different possibilities but did not find a better solution than to implement sharding myself. My best guess is to make a buffer of a fixed size fill it with my stream and write it to the S3.
Is there a better solution? Maybe a way where this is transparent to me, without writing the whole stream to memory?
The aws-sdk-go readme has an example programm that takes data from stdin and writes it to S3: https://github.com/aws/aws-sdk-go#using-the-go-sdk
When I try to pipe data in with a pipe | I get the following error:
failed to upload object, SerializationError: failed to compute request body size
caused by: seek /dev/stdin: illegal seek
Am I doing something wrong or is the example not working as I expect it to?
I although tried minio-go, with PutObject() or client.PutObjectStreaming().
This is functional but consumes as much memory as the data to store.
Is there a better solution?
Is there a small example program that can pipe arbitrary data into S3?
You can use the sdk's Uploader to handle uploads of unknown size but you'll need to make the os.Stdin "unseekable" by wrapping it into an io.Reader. This is because the Uploader, while it requires only an io.Reader as the input body, under the hood it does a check to see whether the input body is also a Seeker and if it is, it does call Seek on it. And since os.Stdin is just an *os.File which implements the Seeker interface, by default, you would get the same error you got from PutObjectWithContext.
The Uploader also allows you to upload the data in chunks whose size you can configure and you can also configure how many of those chunks should be uploaded concurrently.
Here's a modified version of the linked example, stripped off of code that can remain unchanged.
package main
import (
// ...
"io"
"github.com/aws/aws-sdk-go/service/s3/s3manager"
)
type reader struct {
r io.Reader
}
func (r *reader) Read(p []byte) (int, error) {
return r.r.Read(p)
}
func main() {
// ... parse flags
sess := session.Must(session.NewSession())
uploader := s3manager.NewUploader(sess, func(u *s3manager.Uploader) {
u.PartSize = 20 << 20 // 20MB
// ... more configuration
})
// ... context stuff
_, err := uploader.UploadWithContext(ctx, &s3manager.UploadInput{
Bucket: aws.String(bucket),
Key: aws.String(key),
Body: &reader{os.Stdin},
})
// ... handle error
}
As to whether this is a better solution than minio-go I do not know, you'll have to test that yourself.
I'm using streadway/amqp to do a tie in from rabbitmq to our alert system. I need a method that can return a list of all the currently declared queues (exchanges would be nice too!) so that I can go through and get all the message counts.
I'm digging through the api documentation here...
http://godoc.org/github.com/streadway/amqp#Queue
...but I don't seem to be finding what I'm looking for. We're currently using a bash call to 'rabbitmqctl list_queues' but that's a kludge way to get this information, requires a custom sudo setting, and fires off hundreds of log entries a day to the secure log.
edit: method meaning, 'a way to get this piece of information' as opposed to an actual call, though a call would be great I don't believe it exists.
Answered my own question. There isn't a way! The amqp spec doesn't have a standard way of finding this out which seems like a glaring oversight to me. However, since my backend is rabbitmq with the management plugin, I can make a call to that to get this information.
from https://stackoverflow.com/a/21286370/5076297 (in python, I'll just have to translate this and probably also figure out the call to get vhosts):
import requests
def rest_queue_list(user='guest', password='guest', host='localhost', port=15672, virtual_host=None):
url = 'http://%s:%s/api/queues/%s' % (host, port, virtual_host or '')
response = requests.get(url, auth=(user, password))
queues = [q['name'] for q in response.json()]
return queues
edit: In golang (this was a headache to figure out as I haven't done anything with structures in years)
package main
import (
"fmt"
"net/http"
"encoding/json"
)
func main() {
type Queue struct {
Name string `json:name`
VHost string `json:vhost`
}
manager := "http://127.0.0.1:15672/api/queues/"
client := &http.Client{}
req, _ := http.NewRequest("GET", manager, nil)
req.SetBasicAuth("guest", "guest")
resp, _ := client.Do(req)
value := make([]Queue, 0)
json.NewDecoder(resp.Body).Decode(&value)
fmt.Println(value)
}
Output looks like this (I have two queues named hello and test)
[{hello /} {test /}]
I want to implement a webdav-server with Go and found a new "x" package here:
But I don't know how to use this package to get it done.
Can someone help me with this issue?
I tried this:
func main(){
fs := new(webdav.FileSystem)
ls := new(webdav.LockSystem)
h := new(webdav.Handler)
h.FileSystem = *fs
h.LockSystem = *ls
//then use the Handler.ServeHTTP Method as the http.HandleFunc
http.HandleFunc("/", h.ServeHTTP)
http.ListenAndServe(":5555", nil)
}
If I try to connect to the server, I get an internal server error.
What am I doing wrong?
Thanks for your help.
The x/net/webdav is still in early phase of development. Many critical parts are still being implemented, and it can not be used as such at this moment. Taking a look at the source code over half of the necessary structures and functions are still completely missing.
Unfortunately there are no Go based webdav server implementations at this moment. (In case someone can correct me, please feel free to do so!)
func main(){
fs := new(webdav.FileSystem)
ls := new(webdav.LockSystem)
h := new(webdav.Handler)
h.FileSystem = fs
h.LockSystem = ls
//then use the Handler.ServeHTTP Method as the http.HandleFunc
http.HandleFunc("/", h.ServeHTTP)
http.ListenAndServe(":5555", nil)
}
try to remove the * before "fs" and "ls" because they are already pointers.
NB : if you have to assign pointer use & and not *
Create a webdav server on http://localhost:8080/ which mounts the folder C:\myfiles.
package main
import (
"net/http"
"golang.org/x/net/webdav"
)
func main() {
handler := &webdav.Handler{
FileSystem: webdav.Dir(`C:\myfiles`),
LockSystem: webdav.NewMemLS(),
}
http.ListenAndServe("localhost:8080", handler)
}
Mount to Letter E: in windows:
net use e: http://localhost:8080/
Open mounted drive in explorer
explorer.exe e:
I'm working on my first real Go project and have been searching for some tools to handle the configuration.
Finally, I've found this tool: https://github.com/spf13/viper which is really nice but I have some issues when I try to handle some more complex configurations such as the following config.yaml example:
app:
name: "project-name"
version 1
models:
modelA:
varA: "foo"
varB: "bar"
modelB:
varA: "baz"
varB: "qux"
varC: "norf"
I don't know how to get the values from modelB for example. While looking at the lib code, I've found the followings but I don't really understand how to use it:
// Marshals the config into a Struct
func Marshal(rawVal interface{}) error {...}
func AllSettings() map[string]interface{} {...}
What I want is to be able, from everywhere in my package, to do something like:
modelsConf := viper.Get("models")
fmt.Println(modelsConf["modelA"]["varA"])
Could someone explain me the best way to achieve this?
Since the "models" block is a map, it's a bit easier to call
m := viper.GetStringMap("models")
m will be a map[string]interface {}
Then, you get the value of m[key], which is an interface {}, so you cast it to map[interface {}]interface {} :
m := v.GetStringMap("models")
mm := m["modelA"].(map[interface{}]interface{})
Now you can access "varA" key passing the key as an interface {} :
mmm := mm[string("varA")]
mmm is foo
You can simply use:
m := viper.Get("models.modelA")
or
newViperForModelA := viper.Sub("models").Sub("modelA")