APM Go Agent isn't Sending Data to the APM Server - go

Question

I have an Elastic APM-Server up and running and it has successfully established connection with Elasticsearch.
Then I installed an Elastic APM Go agent:
go get -u go.elastic.co/apm
It returned the following:
finding go.elastic.co/apm v1.8.0
finding github.com/stretchr/testify v1.4.0
finding github.com/prometheus/procfs v0.0.3
finding github.com/google/go-cmp v0.3.1
finding github.com/armon/go-radix v1.0.0
finding github.com/santhosh-tekuri/jsonschema v1.2.4
finding github.com/cucumber/godog v0.8.1
finding golang.org/x/sys v0.0.0-20191204072324-ce4227a45e2e
finding go.elastic.co/fastjson v1.0.0
finding github.com/google/go-cmp v0.5.1
finding github.com/prometheus/procfs v0.1.3
finding golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e
finding golang.org/x/sys latest
finding github.com/elastic/go-sysinfo v1.1.1
finding golang.org/x/sync latest
finding golang.org/x/sys v0.0.0-20200106162015-b016eb3dc98e
finding github.com/stretchr/testify v1.6.1
finding go.elastic.co/fastjson v1.1.0
finding github.com/cucumber/godog v0.10.0
finding github.com/stretchr/objx v0.3.0
finding github.com/elastic/go-sysinfo v1.4.0
finding gopkg.in/yaml.v2 v2.2.2
finding github.com/joeshaw/multierror v0.0.0-20140124173710-69b34d4ec901
finding golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543
finding github.com/prometheus/procfs v0.0.0-20190425082905-87a4384529e0
finding github.com/elastic/go-windows v1.0.0
finding golang.org/x/sys v0.0.0-20191025021431-6c3a3bfe00ae
finding golang.org/x/xerrors latest
finding gopkg.in/yaml.v2 v2.3.0
finding golang.org/x/tools v0.0.0-20200509030707-2212a7e161a5
finding github.com/yuin/goldmark v1.1.27
finding github.com/cucumber/messages-go/v10 v10.0.3
finding gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c
finding howett.net/plist v0.0.0-20181124034731-591f970eefbb
finding golang.org/x/net v0.0.0-20200226121028-0de0cce0169b
finding gopkg.in/check.v1 latest
finding golang.org/x/mod v0.2.0
finding golang.org/x/tools latest
finding github.com/hashicorp/go-memdb v1.2.1
finding golang.org/x/net latest
finding github.com/cucumber/gherkin-go/v11 v11.0.0
finding github.com/joeshaw/multierror latest
finding golang.org/x/crypto latest
finding github.com/yuin/goldmark v1.1.32
finding github.com/elastic/go-windows v1.0.1
finding golang.org/x/mod v0.3.0
finding golang.org/x/text v0.3.3
finding howett.net/plist latest
finding golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e
finding golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550
finding golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd
finding golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898
finding golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9
finding golang.org/x/net v0.0.0-20200625001655-4c5254603344
finding github.com/kr/pretty v0.2.1
finding github.com/jessevdk/go-flags v1.4.0
finding github.com/kr/text v0.2.0
finding github.com/kr/pty v1.1.8
finding github.com/hashicorp/go-immutable-radix v1.2.0
finding github.com/hashicorp/golang-lru v0.5.4
finding github.com/yuin/goldmark v1.2.1
finding github.com/aslakhellesoy/gox v1.0.100
finding gopkg.in/yaml.v3 latest
finding gopkg.in/yaml.v2 v2.2.8
finding golang.org/x/sys v0.0.0-20190813064441-fde4db37ae7a
finding github.com/gogo/protobuf v1.3.1
finding github.com/gofrs/uuid v3.2.0+incompatible
finding github.com/stretchr/testify v1.5.1
finding github.com/creack/pty v1.1.7
finding github.com/creack/pty v1.1.9
finding github.com/cucumber/messages-go/v10 v10.0.1
finding github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e
finding github.com/hashicorp/go-uuid v1.0.0
finding github.com/hashicorp/go-version v1.0.0
finding github.com/mitchellh/iochan v1.0.0
finding gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15
finding github.com/kr/pretty v0.2.0
finding github.com/niemeyer/pretty latest
finding github.com/hashicorp/go-uuid v1.0.2
finding github.com/hashicorp/go-version v1.2.1
finding github.com/creack/pty v1.1.11
finding github.com/kisielk/errcheck v1.2.0
finding github.com/gofrs/uuid v3.3.0+incompatible
finding github.com/kisielk/errcheck v1.4.0
finding golang.org/x/tools v0.0.0-20181030221726-6c7e314b6563
finding golang.org/x/tools v0.0.0-20200619180055-7c47624df98f
downloading go.elastic.co/apm v1.8.0
extracting go.elastic.co/apm v1.8.0
downloading github.com/elastic/go-sysinfo v1.4.0
downloading golang.org/x/sys v0.0.0-20200819035508-9a32b3aa38f5
downloading go.elastic.co/fastjson v1.1.0
downloading github.com/armon/go-radix v1.0.0
extracting go.elastic.co/fastjson v1.1.0
extracting github.com/armon/go-radix v1.0.0
extracting github.com/elastic/go-sysinfo v1.4.0
downloading howett.net/plist v0.0.0-20200419221736-3b63eb3a43b5
downloading github.com/joeshaw/multierror v0.0.0-20140124173710-69b34d4ec901
downloading github.com/prometheus/procfs v0.1.3
extracting github.com/joeshaw/multierror v0.0.0-20140124173710-69b34d4ec901
extracting howett.net/plist v0.0.0-20200419221736-3b63eb3a43b5
extracting github.com/prometheus/procfs v0.1.3
extracting golang.org/x/sys v0.0.0-20200819035508-9a32b3aa38f5
Then I setup the ELASTIC_APM_SERVER_URL and ELASTIC_APM_SERVICE_NAME:
export ELASTIC_APM_SERVER_URL=http://my-apm-server-url
export ELASTIC_APM_SERVICE_NAME=agent-name
However, I don't see the agent getting registered in the APM dashboard.
It isn't sending any data to the APM Server.
How do I make sure that the agent is running? How do I check the agent log as to why it isn't able to connect to the APM server?

Answer 1

Since you didn't mention it above: did you instrument a Go application? The Elastic APM Go "Agent" is a package which you use to instrument your application source code. It is not an independent process, but runs within your application.
So, first (if you haven't already) instrument your application. See https://www.elastic.co/guide/en/apm/agent/go/current/getting-started.html#instrumenting-source
Here's an example web server using Echo, and the apmechov4 instrumentation module:
package main
import (
"fmt"
"net/http"
echo "github.com/labstack/echo/v4"
"go.elastic.co/apm/module/apmechov4"
)
func main() {
e := echo.New()
e.Use(apmechov4.Middleware())
e.GET("/hello/:name", func(c echo.Context) error {
fmt.Println(c.Param("name"))
return nil
})
http.ListenAndServe(":8080", e)
}
If you run that and send some requests to http://localhost:8080/hello/world, you should soon see requests in the APM app in Kibana.
If you still don't see anything in Kibana, you can follow https://www.elastic.co/guide/en/apm/agent/go/current/troubleshooting.html#agent-logging to enable logging. Here's what you can expect to see if the agent is able to successfully send data to the server:
$ ELASTIC_APM_LOG_FILE=stderr ELASTIC_APM_LOG_LEVEL=debug go run main.go
{"level":"debug","time":"2020-08-19T13:33:28+08:00","message":"sent request with 3 transactions, 0 spans, 0 errors, 0 metricsets"}
{"level":"debug","time":"2020-08-19T13:33:46+08:00","message":"gathering metrics"}
{"level":"debug","time":"2020-08-19T13:33:56+08:00","message":"sent request with 0 transactions, 0 spans, 0 errors, 3 metricsets"}
If on the other hand the server is inaccessible, you would see something like this:
{"level":"error","time":"2020-08-19T13:38:01+08:00","message":"config request failed: sending config request failed: Get \"http://localhost:8200/config/v1/agents?service.name=main\": dial tcp 127.0.0.1:8200: connect: connection refused"}