I'm using elasticsearch, kibana on docker, and Go.
time="2019-09-17T09:52:02+08:00" level=panic msg="no active connection found: no Elasticsearch node available"
panic: (*logrus.Entry) (0x736fe0,0xc000136150)
import (
"github.com/olivere/elastic/v7"
"github.com/sirupsen/logrus"
"gopkg.in/sohlich/elogrus.v7"
)
func main() {
log := logrus.New()
client, err := elastic.NewClient(elastic.SetURL("http://localhost:9200"))
if err != nil {
log.Panic(err)
}
hook, err := elogrus.NewAsyncElasticHook(client, "localhost", logrus.DebugLevel, "mylog")
if err != nil {
log.Panic(err)
}
log.Hooks.Add(hook)
log.WithFields(logrus.Fields{
"name": "joe",
"age": 42,
}).Error("Hello world!")
}
This is the package I'm using
https://github.com/sohlich/elogrus
You have to authenticate, if you didn't change default password you might use something like this:
Also set off the sniff. More details about it: https://github.com/olivere/elastic/wiki/Sniffing
client, err := elastic.NewClient(
elastic.SetBasicAuth("elastic", "changeme"),
elastic.SetURL("http://localhost:9200"),
elastic.SetSniff(false),
)
Related
I was looking for ways to set or update instance Labels on GCP Compute Instance and labelFingerprint confused me.
then I figure it out and I'm putting the code in the answer section.
I used this simple code to add new labels to GCP instances.
package main
import (
"context"
"log"
"os"
"golang.org/x/oauth2/google"
"google.golang.org/api/compute/v1"
)
func main() {
addLabelToGCPInstances()
}
func addLabelToGCPInstances() error {
// You can pass these as args
project := "Your GCP Project ID"
zone := "europe-west2-a"
instance := "milad-test-instance"
prodLablesMap := map[string]string{
"production": "true",
"environment": "production",
}
ctx := context.Background()
os.Setenv("GOOGLE_APPLICATION_CREDENTIALS", "gke.json")
c, err := google.DefaultClient(ctx, compute.CloudPlatformScope)
if err != nil {
return err
}
computeService, err := compute.New(c)
if err != nil {
return err
}
respInstance, err := computeService.Instances.Get(project, zone, instance).Context(ctx).Do()
if err != nil {
log.Fatal(err)
}
rb := &compute.InstancesSetLabelsRequest{
Labels: prodLablesMap,
LabelFingerprint: respInstance.LabelFingerprint,
}
respLabels, err := computeService.Instances.SetLabels(project, zone, instance, rb).Context(ctx).Do()
if err != nil {
log.Fatal(err)
}
_ = respLabels
return err
}
This is just an example you can work around and do more error handling and etc.
I have a Vertex AI model deployed on an endpoint and want to do some prediction from my app in Golang.
To do this I create code inspired by this example : https://cloud.google.com/go/docs/reference/cloud.google.com/go/aiplatform/latest/apiv1?hl=en
const file = "MY_BASE64_IMAGE"
func main() {
ctx := context.Background()
c, err := aiplatform.NewPredictionClient(cox)
if err != nil {
log.Printf("QueryVertex NewPredictionClient - Err:%s", err)
}
defer c.Close()
parameters, err := structpb.NewValue(map[string]interface{}{
"confidenceThreshold": 0.2,
"maxPredictions": 5,
})
if err != nil {
log.Printf("QueryVertex structpb.NewValue parameters - Err:%s", err)
}
instance, err := structpb.NewValue(map[string]interface{}{
"content": file,
})
if err != nil {
log.Printf("QueryVertex structpb.NewValue instance - Err:%s", err)
}
reqP := &aiplatformpb.PredictRequest{
Endpoint: "projects/PROJECT_ID/locations/LOCATION_ID/endpoints/ENDPOINT_ID",
Instances: []*structpb.Value{instance},
Parameters: parameters,
}
resp, err := c.Predict(cox, reqP)
if err != nil {
log.Printf("QueryVertex Predict - Err:%s", err)
}
log.Printf("QueryVertex Res:%+v", resp)
}
I put the path to my service account JSON file on GOOGLE_APPLICATION_CREDENTIALS environment variable.
But when I run my test app I obtain this error message:
QueryVertex Predict - Err:rpc error: code = Unimplemented desc = unexpected HTTP status code received from server: 404 (Not Found); transport: received unexpected content-type "text/html; charset=UTF-8"
QueryVertex Res:<nil>
As #DazWilkin suggested, configure the client option to specify the specific regional endpoint with a port 443:
option.WithEndpoint("<region>-aiplatform.googleapis.com:443")
Try like below:
func main() {
ctx := context.Background()
c, err := aiplatform.NewPredictionClient(
ctx,
option.WithEndpoint("<region>-aiplatform.googleapis.com:443"),
)
if err != nil {
log.Printf("QueryVertex NewPredictionClient - Err:%s", err)
}
defer c.Close()
.
.
I'm unfamiliar with Google's (Vertex?) AI Platform and unable to test this hypothesis but it appears that the API uses location-specific endpoints.
Can you try configuring the client's ClientOption to specify the specific regional endpoint, i.e.:
url := fmt.Sprintf("https://%s-aiplatform.googleapis.com", location)
opts := []option.ClientOption{
option.WithEndpoint(url),
}
And:
package main
import (
"context"
"fmt"
"log"
"os"
aiplatform "cloud.google.com/go/aiplatform/apiv1"
"google.golang.org/api/option"
aiplatformpb "google.golang.org/genproto/googleapis/cloud/aiplatform/v1"
"google.golang.org/protobuf/types/known/structpb"
)
const file = "MY_BASE64_IMAGE"
func main() {
// Values from the environment
project := os.Getenv("PROJECT")
location := os.Getenv("LOCATION")
endpoint := os.Getenv("ENDPOINT")
ctx := context.Background()
// Configure the client with a region-specific endpoint
url := fmt.Sprintf("https://%s-aiplatform.googleapis.com", location)
opts := []option.ClientOption{
option.WithEndpoint(url),
}
c, err := aiplatform.NewPredictionClient(ctx, opts...)
if err != nil {
log.Fatal(err)
}
defer c.Close()
parameters, err := structpb.NewValue(map[string]interface{}{
"confidenceThreshold": 0.2,
"maxPredictions": 5,
})
if err != nil {
log.Fatal(err)
}
instance, err := structpb.NewValue(map[string]interface{}{
"content": file,
})
if err != nil {
log.Printf("QueryVertex structpb.NewValue instance - Err:%s", err)
}
rqst := &aiplatformpb.PredictRequest{
Endpoint: fmt.Sprintf("projects/%s/locations/%s/endpoints/%s",
project,
location,
endpoint,
),
Instances: []*structpb.Value{
instance,
},
Parameters: parameters,
}
resp, err := c.Predict(ctx, rqst)
if err != nil {
log.Fatal(err)
}
log.Printf("QueryVertex Res:%+v", resp)
}
Try to do something like this
[...]
url := fmt.Sprintf("%s-aiplatform.googleapis.com:443", location)
[..]
I am trying to use Google Translate API on Windows(my own computer). I have an issue with default credentials.
Error: **translate.NewClient: dialing: google: could not find default credentials.
I have enough balance in google cloud.
I started Translate API. (API Enabled)
I added $env:GOOGLE_APPLICATION_CREDENTIALS="KEY_PATH"
package main
import (
"context"
"fmt"
"log"
"cloud.google.com/go/storage"
"cloud.google.com/go/translate"
"golang.org/x/text/language"
"google.golang.org/api/iterator"
"google.golang.org/api/option"
)
func translateTextWithModel(targetLanguage, text, model string) (string, error) {
// targetLanguage := "ja"
// text := "The Go Gopher is cute"
// model := "nmt"
ctx := context.Background()
lang, err := language.Parse(targetLanguage)
if err != nil {
return "", fmt.Errorf("language.Parse: %v", err)
}
client, err := translate.NewClient(ctx)
if err != nil {
return "", fmt.Errorf("translate.NewClient: %v", err)
}
defer client.Close()
resp, err := client.Translate(ctx, []string{text}, lang, &translate.Options{
Model: model, // Either "nmt" or "base".
})
if err != nil {
return "", fmt.Errorf("Translate: %v", err)
}
if len(resp) == 0 {
return "", nil
}
return resp[0].Text, nil
}
func main() {
Json_path := "C:/Users/Mels/Documents/GoogleTools/cred-9dfos6bb49f.json"
ProjectID := "cred"
fmt.Println("RUNNING...")
explicit(Json_path, ProjectID)
fmt.Println(translateTextWithModel("ja", "Hello World", "nmt"))
}
// explicit reads credentials from the specified path.
func explicit(jsonPath, projectID string) {
ctx := context.Background()
client, err := storage.NewClient(ctx, option.WithCredentialsFile(jsonPath))
if err != nil {
log.Fatal(err)
}
defer client.Close()
fmt.Println("Buckets:")
it := client.Buckets(ctx, projectID)
for {
battrs, err := it.Next()
if err == iterator.Done {
break
}
if err != nil {
log.Fatal(err)
}
fmt.Println(battrs.Name)
}
}
JSON File
{
"type": "service_account",
"project_id": "xxxxxxx",
"private_key_id": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"private_key": "-----BEGIN PRIVATE KEY-----XXXXXXXX-----END PRIVATE KEY-----\n",
"client_email": "xxxxxx#xxxxxx.iam.gserviceaccount.com",
"client_id": "11111111111",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"api_key": "xxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"token_uri": "https://oauth2.googleapis.com/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/xxxxxxx%xxxxxxx.iam.gserviceaccount.com"
}
If the library can't find the default credentials, then you can try to create a client with the credentials path.
Even if this might not be the best option for you (although I prefer it to the env variable), it'll help you diagnose the issue a little better.
In order to create the client with a path to the credentials file, you need to import google.golang.org/api/option, and create the client with the WithCredentialsFile option. Note in the docs that the client can be created with additional options:
func NewClient(ctx context.Context, opts ...option.ClientOption) (*Client, error)
A somewhat more complete example on how to create a client with options would be the following (apply the required parts to your current code as needed):
package main
import (
"cloud.google.com/go/translate"
"context"
"google.golang.org/api/option"
)
func main() {
ctx := context.Background()
client, err := translate.NewClient(ctx, option.WithCredentialsFile("/path/to/your/file"))
if err != nil {
// TODO: handle error.
}
// Use the client.
// Close the client when finished.
if err := client.Close(); err != nil {
// TODO: handle error.
}
}
(This is just a copy of the example in the docs with the additional option included.)
I solved the issue. I downloaded "google cloud shell SDK" and I used "gcloud auth application-default login" code. SDK provides a JSON file and I replaced it with new JSON file.
I do not recommend cloud.google.com/translate/docs/setup instructions. Direct use Google cloud SDK.
I am trying to connect to a remote EC2-server through Go code using a PEM key provided by AWS. I am able to log in to the server through the command line using the PEM key.
I have done the following so far.
package main
import (
"io"
"io/ioutil"
"os"
"time"
"golang.org/x/crypto/ssh"
)
func publicKey(path string) ssh.AuthMethod {
key, err := ioutil.ReadFile(path)
if err != nil {
panic(err)
}
signer, err := ssh.ParsePrivateKey(key)
if err != nil {
panic(err)
}
return ssh.PublicKeys(signer)
}
func runCommand(cmd string, conn *ssh.Client) {
sess, err := conn.NewSession()
if err != nil {
panic(err)
}
defer sess.Close()
sessStdOut, err := sess.StdoutPipe()
if err != nil {
panic(err)
}
go io.Copy(os.Stdout, sessStdOut)
sessStderr, err := sess.StderrPipe()
if err != nil {
panic(err)
}
go io.Copy(os.Stderr, sessStderr)
err = sess.Run(cmd) // eg., /usr/bin/whoami
if err != nil {
panic(err)
}
}
func main() {
config := &ssh.ClientConfig{
User: "ec2-user",
Auth: []ssh.AuthMethod{
publicKey("mykey"),
},
Timeout: 15 * time.Second,
HostKeyCallback: ssh.InsecureIgnoreHostKey(),
}
conn, err := ssh.Dial("tcp", "remote-server:22", config)
if err != nil {
panic(err.Error())
}
defer conn.Close()
runCommand("whoami", conn)
}
I keep getting the following error. What am I missing?
panic: ssh: handshake failed: read tcp "localhost"->"remotehost:22": read: operation timed out
In the above message I have replaced the IP addresses of the actual machines with bogus names.
I was able to resolve this since I was using an incorrect port to connect to :). I should be using port 22 for SSH, but was using some other service port.
Thanks.
// datastore1
package main
import (
"fmt"
"io/ioutil"
"log"
"time"
"golang.org/x/net/context"
"golang.org/x/oauth2/google"
"google.golang.org/cloud"
"google.golang.org/cloud/datastore"
)
const (
// ScopeDatastore grants permissions to view and/or manage datastore entities
copeDatastore = "https://www.googleapis.com/auth/datastore"
// ScopeUserEmail grants permission to view the user's email address.
// It is required to access the datastore.
ScopeUserEmail = "https://www.googleapis.com/auth/userinfo.email"
)
type ehrEntity struct {
email *datastore.Key
firstname string
lastname string
address string
age int8
dateofbirth time.Time
sex bool
}
func getCtx() *datastore.Client {
// Initialize an authorized transport with Google Developers Console
// JSON key. Read the google package examples to learn more about
// different authorization flows you can use.
// http://godoc.org/golang.org/x/oauth2/google
jsonKey, err := ioutil.ReadFile("filename.json")
opts, err := google.JWTConfigFromJSON(
jsonKey,
datastore.ScopeDatastore,
datastore.ScopeUserEmail,
)
if err != nil {
log.Fatal(err)
}
ctx := context.Background()
client, err := datastore.NewClient(ctx, "xxxx", cloud.WithTokenSource(opts.TokenSource(ctx)))
if err != nil {
log.Fatal(err)
}
// Use the context (see other examples)
return client
}
func ExampleGet() {
ctx := context.Background()
client, err := datastore.NewClient(ctx, "xxxx")
if err != nil {
log.Fatal(err)
}
key := datastore.NewKey(ctx, "User", "tluu#abc.com", 0, nil)
ehr := ehrEntity{
nil,
"tri",
"luu",
"addr1",
20,
time.Date(2009, time.January, 10, 23, 0, 0, 0, time.UTC),
false}
if err := client.Get(ctx, key, ehr); err != nil {
log.Fatal(err)
}
}
func main() {
getCtx()
fmt.Println("Pass authentication")
ExampleGet()
}
When I run go file, It return error follow:
Pass authentication (Pass getCtx() function).
Error in ExampleGet()
May be at
ctx := context.Background()
client, err := datastore.NewClient(ctx, "xxxx")
if err != nil {
log.Fatal(err)
}
Error:
2016/01/09 22:08:43 Post https://www.googleapis.com/datastore/v1beta2/datasets/xxxx/lookup: oauth2: cannot fetch token: 400 Bad Request
Response: {
"error" : "invalid_grant"
}
How to resolve this error?
This appears to have worked for me.
If you are on linux try the following:
1. apt-get update
2. apt-get install ntp
3. /etc/init.d/ntp restart