I'm trying to use the unstructured.UnstructuredList to reuse come logic for configmap and secret.
However, after adding the ListAndDeployReferredObject, I started to see tons of trace as Starting reflector *unstructured.Unstructured was added to my log file.
Am I doing something odd or I'm missing some setting for using the unstructured.Unstructured?
Thanks in advance.
func (r *ReconcileSubscription) ListAndDeployReferredObject(instance *appv1alpha1.Subscription, gvk schema.GroupVersionKind, refObj referredObject) error {
insName := instance.GetName()
insNs := instance.GetNamespace()
uObjList := &unstructured.UnstructuredList{}
uObjList.SetGroupVersionKind(gvk)
opts := &client.ListOptions{Namespace: insNs}
err := r.Client.List(context.TODO(), uObjList, opts)
if err != nil && !errors.IsNotFound(err) {
klog.Errorf("Failed to list referred objects with error %v ", err)
return err
}
// other logics...
}
I0326 23:05:58.955589 95169 reflector.go:120] Starting reflector *unstructured.Unstructured (10m0s) from pkg/mod/k8s.io/client-go#v0.0.0-20191016111102-bec269661e48/tools/cache/reflector.go:96
...
I0326 23:15:18.718932 95169 reflector.go:158] Listing and watching *unstructured.Unstructured from pkg/mod/k8s.io/client-go#v0.0.0-20191016111102-bec269661e48/tools/cache/reflector.go:96
I figured out these prints are normal, since we are using the dynamic client on our controller for caches
Related
I am trying to figure out how to get a simple bq load command to work with https://godoc.org/cloud.google.com/go/bigquery#Table.LoaderFrom
Running it manually it looks like this:
bq load --source_format=AVRO --ignore_unknown_values --replace=true mydataset.mytable gs://mybucket/table/*
And running it in my golang with exec.Command() successfully looks like this:
exec.Command("bq", "load", "--source_format=AVRO", "--ignore_unknown_values",
"--replace=true", "mydataset.mytable",
"gs://mybucket/table/*")
However, I cannot get this program to run without a segmentation fault when trying to get the load and job.wait to run successfully it seems to be getting a segmentation violation at the job.wait line of the program
package main
import (
"context"
"log"
"cloud.google.com/go/bigquery"
)
func main(){
ctx := context.Background()
client, err := bigquery.NewClient(ctx, "my-project-id")
if err != nil {
// TODO: Handle error.
}
gcsRef := bigquery.NewGCSReference("gs://mybucket/table/*")
gcsRef.SourceFormat = "AVRO"
gcsRef.IgnoreUnknownValues = true
// TODO: set other options on the GCSReference.
ds := client.Dataset("mydataset")
loader := ds.Table("mytable").LoaderFrom(gcsRef)
// TODO: set other options on the Loader.
job, err := loader.Run(ctx)
if err != nil {
// TODO: Handle error.
}
status, err := job.Wait(ctx) //seg faults right here
if err != nil {
// TODO: Handle error.
}
if status.Err() != nil {
// TODO: Handle error.
}
}
The panic is probably coming from a nil pointer reference to the job variable.
I would suggest including a log.Fatal(err)
In all of your err!= nil blocks.
This will help get you closer to why job is not being assigned correctly.
When you're writing one off scripts like this one in go log.Fatal is a great way to exit the program and print exactly what the issue is.
With go you're always trying to bubble errors up the stack to determine if the code should continue to execute, if things can be recovered, or if it's just a fatal thing and you should end the program.
For more info on the logging package checkout here: https://golang.org/pkg/log/
If you're just starting out learning go here are some awesome resources that can help give you ideas on how different types of programs can be designed.
https://github.com/dashpradeep99/https-github.com-miguellgt-books/tree/master/go
Best,
Christopher
My question is this package-specific: https://github.com/knq/chromedp/
This is just a glimpse of a log when running chromedp Actions. It is not event an Action-related piece. What is the correct way to make the logging silent or less verbose?
The solution is simple:
Code taken from their sample:
// create chrome instance
c, err := cdp.New(ctxt, cdp.WithLog(log.Printf))
if err != nil {
log.Fatal(err)
}
change to
// create chrome instance
c, err := cdp.New(ctxt) // no WithLog() option
if err != nil {
log.Fatal(err)
}
Didn't tested
I have no expirience with that package, but accourding the usage example, cbp initialized with the log function:
// create chrome instance
c, err := cdp.New(ctxt, cdp.WithLog(log.Printf)) <- in this case log.Printf
if err != nil {
log.Fatal(err)
}
I assume you can set log level for the log.Printf with log.SetFlags(0)
// create chrome instance
log.SetFlags(0)
c, err := cdp.New(ctxt, cdp.WithLog(log.Printf))
if err != nil {
log.Fatal(err)
}
or pass an empty function(or initialize without a log function).
There is also Discard to disable logging at all, log.SetOutput(ioutil.Discard)
I have a slightly funky issue in golang. Essentially I have a slice of strings which represent file paths. I then run a cat against those filepaths to combine the files before sorting, deduping, etc.
here is the section of code (where 'applicableReductions' is the string slice):
applicableReductions := []string{}
for _, fqFromListName := range fqFromListNames {
filePath := GetFilePath()
//BROKE CODE GOES HERE
}
applicableReductions = append(applicableReductions, filePath)
fileOut, err := os.Create(toListWriteTmpFilePath)
if err != nil {
return err
}
cat := exec.Command("cat", applicableReductions...)
catStdOut, err := cat.StdoutPipe()
if err != nil {
return err
}
go func(cat *exec.Cmd) error {
if err := cat.Start(); err != nil {
return fmt.Errorf("File reduction error (cat) : %s", err)
}
return nil
}(cat)
// Init Writer & write file
writer := bufio.NewWriter(fileOut)
defer writer.Flush()
_, err = io.Copy(writer, catStdOut)
if err != nil {
return err
}
if err = cat.Wait(); err != nil {
return err
}
fDiff.StandardiseData(fileOut, toListUpdateFolderPath, list.Name)
The above works fine. The problem comes when I try to append a new ele to the array. I have a seperate function which creates a new file from db content which is then added to the applicableReductions slice.
func RetrieveDomainsFromDB(collection *Collection, listName, outputPath string) error {
domains, err := domainReviews.GetDomainsForList(listName)
if err != nil {
return err
}
if len(domains) < 1 {
return ErrNoDomainReviewsForList
}
fh, err := os.OpenFile(outputPath, os.O_RDWR, 0774)
if err != nil {
fh, err = os.Create(outputPath)
if err != nil {
return err
}
}
defer fh.Close()
_, err = fh.WriteString(strings.Join(domains, "\n"))
if err != nil {
return err
}
return nil
}
If I call the above function and append the filePath to the applicableReduction slice, it is in there but doesnt get called by cat.
To clarify, when I put the following where it says BROKE CODE GOES HERE:
if dbSource {
err = r.RetrieveDomainsFromDB(collection, ToListName, filePath)
if err != nil {
return err
continue
}
}
The filepath can be seen when doing fmt.Println(applicableReductions) but the content of the files contents are not seen in the cat output file.
I thought perhaps a delay in the file being written so i tried adding a time.wait, tis didnt help. However the solution I found was to sort the slice, e.g this code above the call to exec cat solves the problem but I dont know why:
sort.Strings(applicableReductions)
I have confirmed all files present on both successful and unsucessful runs the only difference is without the sort, the content of the final appended file is missing
An explanation from a go-pro out there would be very much appreciated, let me know if you need more info, debug - happy to oblige to understand
UPDATE
It has been suggested that this is the same issue as here: Golang append an item to a slice, I think I understand the issue there and I'm not saying this isnt the same but I cannot see the same thing happenning - the slice in question is not touched from outside the main function (e.g. no editing of the slice in RetrieveDomainsFromDB function), I create the slice before a loop, append to it within a loop and then use it after the loop - Ive added an example at the top to show how the slice is built - please could someone clarify where this slice is being copied if this is the case
UPDATE AND CLOSE
Please close question - the issue was unrelated to the use of a string slice. Turns out that I was reading from the final output file before bufio-writer had been flushed (at end of function before defer flush kicked in on function return)
I think the sorting was just re-arranging the problem so I didnt notice it persisted or possibly giving some time for the buffer to flush. Either way sorted now with a manual call to flush.
Thanks for all help provided
So I've been playing around with Network namespaces recently.
I put together a simple code, built it and noticed something very weird happening.
The code is as follows:
package main
import (
"fmt"
"log"
"net"
"os"
"path"
"syscall"
)
const (
NsRunDir = "/var/run/netns"
SelfNetNs = "/proc/self/ns/net"
)
func main() {
netNsPath := path.Join(NsRunDir, "myns")
os.Mkdir(NsRunDir, 0755)
if err := syscall.Mount(NsRunDir, NsRunDir, "none", syscall.MS_BIND, ""); err != nil {
log.Fatalf("Could not create Network namespace: %s", err)
}
fd, err := syscall.Open(netNsPath, syscall.O_RDONLY|syscall.O_CREAT|syscall.O_EXCL, 0)
if err != nil {
log.Fatalf("Could not create Network namespace: %s", err)
}
syscall.Close(fd)
if err := syscall.Unshare(syscall.CLONE_NEWNET); err != nil {
log.Fatalf("Could not clone new Network namespace: %s", err)
}
if err := syscall.Mount(SelfNetNs, netNsPath, "none", syscall.MS_BIND, ""); err != nil {
log.Fatalf("Could not Mount Network namespace: %s", err)
}
if err := syscall.Unmount(netNsPath, syscall.MNT_DETACH); err != nil {
log.Fatalf("Could not Unmount new Network namespace: %s", err)
}
if err := syscall.Unlink(netNsPath); err != nil {
log.Fatalf("Could not Unlink new Network namespace: %s", err)
}
ifcs, _ := net.Interfaces()
for _, ifc := range ifcs {
fmt.Printf("%#v\n", ifc)
}
}
Now, when you run this code on Trusty 14.04, you will see something weird happening.
This happens when you run the binary several times in a row.
Sometimes it prints out all Host's interfaces, sometimes it simply prints out just a loopback interface which means that the range loop at the end of the program seems to be executing once when the namespace is still attached and sometimes when it's already been detached.
I'm totally confused why this is happening but I'm thinking it's either my code or maybe I'm just missing out something in terms of the program execution or some kernel stuff.
Any help would be massively appreciated.
Thanks
Update1:
So it seems the "strange" behaviour has to do with how golang schedules goroutines across OS Threads. So you need to make sure that you handle the runtime well. What I mean by that is, that if you lock the code execution to one OS Thread you will get consistent results. You can do this by adding the following runtime package statement:
runtime.LockOSThread()
However this still does not solve my problem, but now I think it all comes down to understanding of Namespaces. I need to look more into that.
Update2:
To give you a little bit more context into why you should use the above OS Thread Lock when running bunch of syscalls and experience the similar "strangeness" of correct behavior, give ththis blogpost a read. It describes runtime and go schedulers. It was written for go 1.1 but it still gives very good overview.
If I do
s, err := os.Stat(path)
and err != nil I need to know if the file doesn't exist vs I don't have permission to access it, etc. How do I get the underlying error code? Reading the os package docs it seems to suggest that I read the text of the error string - surely not?
What FUZxxl says.
From the os.Stat documentation:
Stat returns a FileInfo describing the named file. If there is an error, it will be of type *PathError.
PathError is documented on the same page, stating that it holds the operation that caused the error, the path to the file that caused it and the underlying system's error. In case the file was not found when calling os.Stat, the returned error would be something like this:
&PathError{"stat", "/your/file", syscall.Errno(2)}
Since the underlying error is inherently depending on the OS you use, the only thing that you can do is to
understand PathError.Err. For UNIX systems the syscall package has the Errno error type returned by syscalls like syscall.Stat. You can compare this value with the constants in the syscall package and handle the error (Click to play):
stat, err := os.Stat(file)
if perr, ok := err.(*os.PathError); ok {
switch perr.Err.(syscall.Errno) {
case syscall.ENOENT: fmt.Println("No such file or directory.")
default: panic("Unknown error")
}
}
The shorter way of doing this is to use os.IsNotExist which does pretty much the above
and is, most importantly, platform independent:
stat, err := os.Stat(file)
if err != nil && os.IsNotExist(err) {
// ...
}
The other answer is great, but I wanted add a note about this suggestion:
stat, err := os.Stat(file)
if err != nil && os.IsNotExist(err) {
// ...
}
I found that in many cases, I was needing to take different actions depending on
each test, so in reality you have three branches here. Here is code I use for
that:
stat, err := os.Stat(file)
if os.IsNotExist(err) {
// branch one
} else if err != nil {
// branch two
}
// branch three