I wrote a Go program which doesn't need to retrieve external http calls at all by default. I tried to deploy it on Google Cloud Run and received the following error:
The user-provided container failed to start and listen on the port
defined provided by the PORT=8080 environment variable. Logs for this
revision might contain more information.
I understand it happens because my code doesn't provide a port. As this answer states:
container must listen for incoming HTTP requests on the port
that is defined by Cloud Run and provided in the $PORT environment
variable
My question is what can I do if wouldn't like define any ports and just want to run the same code I run locally? Is there an alternate solution to deploy my code without it, or I must add it anyway if I want run the code from Cloud Run?
For containers that do not require an HTTP listener (HTTP server), use Cloud Run Jobs.
Cloud Run Jobs is in preview.
Your Go program must exit with exit code 0 for success and non-zero for failure.
Your container should not listen on a port or start a web server.
Environment variables are different from Cloud Run.
Container instances run until the container instance exits, until the task timeout is reached, or until the container crashes. Task timeout default is 10 minutes, max is one hour.
Cloud Run - Create jobs
Related
I've a distributed JMeter setup with 1 client (controller) and 2 servers (generators).
Now while a test is executing on the setup if a generator crashes in between, the controller gets hanged even after the test duration ends.
Is there a way to reconnect the controller with the generator after the generator comes up again during the same test execution?
No there is no such configuration option and it is advised to restart servers.
Usually this is due to connectivity issues between server and controller like port not open.
For reference properties have a look at:
https://jmeter.apache.org/usermanual/properties_reference.html#remote
For 1/ you can add this to user.properties:
client.continue_on_fail=true
server.exitaftertest=true
INFO [main] org.apache.nifi.bootstrap.Command Apache NiFi is currently running, listening to Bootstrap on port 20740, PID=31405
That means NiFi is running, and I can see its log in logs/nifi-app.log.
The UI is available at https://servername:9443/nifi. It successfully initiated communication with Bootstrap.
But i have this error, and the UI fails to appear. Do you have any solutions?
Failed to invoke #OnEnabled method of JettyWebSocketServer[id=01591009-1d2b-177f-e304-a7cc87d735ce] due to java.net.BindException: Address already in use
"java.net.BindException: Address already in use" means a port is already in use by something else on your system.
In this case it shows that it is coming from the JettyWebSocketServer controller service which is trying to bind to port 9998.
You can use "netstat -lntp" to see what is using port 9998. It could also be possible that you have more than one instance of the controller service with both of them configured with port 9998 and one of them is binging to it and the other fails.
Seems like a Controller Service that fails... Try browsing your flow.xml.gz and check for any ControllerServices named JettyWebSocketServer. Change their <state></state> from <state>ENABLED</state> to <state>DISABLED</state> and try running NiFi again.
You can tell it is a controller service by the #enabledannotation which invokes a method that needs to run when the controller service is enabled.
NiFi shouldn't fail to start because of a failing controller service but it seems to be the problem(probably a bug).
After it start back up, you can configure the controller service to run on a different port.
I also suggest, that if it is the case, that you open an issue to Apache about this since it seems like a pretty major bug.
I'm trying to create a simple consumer-provider contract with Pact and Pact Broker.
The consumer is working and it's publishing the interactions to the broker as expected, but the provider is failing to when running the tests.
The error is:
--- FAIL: TestPact (10.14s)
pact.go:270: Error: Expected server to start < 10s. Timed out waiting for Daemon on port 0 - are you
sure it's running?
Here is the repo with the provider
https://github.com/tutabeier/pact-consumer-provider/tree/master/provider
Most specifically, here is the test:
https://github.com/tutabeier/pact-consumer-provider/blob/master/provider/cmd/provider/main_test.go
I have no idea what I'm doing wrong.
«on port 0» — this is flat out weird: no service is ever listening on port 0, but this number is commonly used to bind a service to a port, and in this case it means the OS is asked to pick any free port and assign it to that service. One then is expected to ask the bound socket to tell it what port it ended up listening on.
Still, the code in main_test.go uses hard-coded port numbers, and neither of them is 0.
So I have no idea but at least tried to keep your efforts going.
I have spun up a bosh cluster on AWS, running a concourse deployment. For this I used a tool called concourse-up. I spun up a windows worker outside the created VPC of the cluster, and I am trying to register the worker through the atc but this step fails with an error. I have opened up all ports and both a web VM and the Worker VM. I have tried several things but there are two specific errors that I get:
When I connect the worker without a --peer-ip, the worker registers so I can see it through the fly cli, but I get this error in the log(snippet below), and jobs will fail with this error:
Put /volumes/47c1c26c-274b-4f04-4dea-01d476ed949e/stream-in?path=.: read tcp 10.0.0.7:59478->10.0.0.7:39198: read: connection reset by peer
{"timestamp":"1513510128.917933226","source":"worker","message":"worker.setup.no-assets","log_level":1,"data":{"session":"1"}}
{"timestamp":"1513510128.920933962","source":"worker","message":"worker.garden.started","log_level":1,"data":{"session":"2"}}
{"timestamp":"1513510128.921934128","source":"baggageclaim","message":"baggageclaim.listening","log_level":1,"data":{"addr":"127.0.0.1:7788"}}
{"timestamp":"1513510130.645173311","source":"tsa","message":"tsa.connection.channel.forward-worker.register.start","log_level":1,"data":{"remote":"34.242.192.32:56803","session":"12.1.1.5","worker-address":"10.0.0.7:38380","worker-platform":"windows","worker-tags":""}}
{"timestamp":"1513510130.649989367","source":"tsa","message":"tsa.connection.channel.forward-worker.register.reached-worker","log_level":0,"data":{"baggageclaim-took":"2.251829ms","garden-took":"2.492218ms","remote":"34.242.192.32:56803","session":"12.1.1.5"}}
{"timestamp":"1513510128.960758924","source":"baggageclaim","message":"baggageclaim.repository.get-volume.volume-not-found","log_level":1,"data":{"session":"1.2","volume":"resource-certs"}}
{"timestamp":"1513510128.960758924","source":"baggageclaim","message":"baggageclaim.api.volume-server.get-volume.volume-not-found","log_level":1,"data":{"session":"2.1.2","volume":"resource-certs"}}
{"timestamp":"1513510128.963933945","source":"baggageclaim","message":"baggageclaim.repository.create-volume.failed-to-materialize-strategy","log_level":2,"data":{"error":"mkdir C:\\Users\\Administrator\\workspace\\concourse-workspace\\volumes\\init\\resource-certs: Cannot create a file when that file already exists.","handle":"resource-certs","session":"1.3"}}
supposedly this is not the right way to do it if I follow the official docs: https://concourse-ci.org/clusters-with-bosh.html#configuring-bosh-tsa
The other thing I tried is using the --peer-ip which is recommended is the official docs if you are outside a cluster and have no connection to any of the resources other than the atc. But this will not even register the worker, it fails with this error in the log:
{"timestamp":"1513510497.727977514","source":"tsa","message":"tsa.connection.channel.register-worker.register.start","log_level":1,"data":{"remote":"34.242.192.32:56828","session":"13.1.1.8","worker-address":"34.242.40.26:7777","worker-platform":"windows","worker-tags":""}}
{"timestamp":"1513510497.728802919","source":"tsa","message":"tsa.connection.channel.register-worker.register.failed-to-fetch-containers","log_level":2,"data":{"error":"Get http://api/containers: dial tcp 34.242.40.26:7777: getsockopt: connection refused","remote":"34.242.192.32:56828","session":"13.1.1.8"}}
{"timestamp":"1513510497.729249239","source":"tsa","message":"tsa.connection.channel.register-worker.register.failed-to-list-volumes","log_level":2,"data":{"error":"Get http://34.242.40.26:7788/volumes: dial tcp 34.242.40.26:7788: getsockopt: connection refused","remote":"34.242.192.32:56828","session":"13.1.1.8"}}
{"timestamp":"1513510497.729336023","source":"tsa","message":"tsa.connection.channel.register-worker.register.failed-to-reach-worker","log_level":1,"data":{"baggageclaim-took":"469.89µs","garden-took":"666.118µs","remote":"34.242.192.32:56828","session":"13.1.1.8"}}
{"timestamp":"1513510497.729401112","source":"tsa","message":"tsa.connection.channel.register-worker.register.done","log_level":1,"data":{"remote":"34.242.192.32:56828","session":"13.1.1.8","worker-address":"34.242.40.26:7777","worker-platform":"windows","worker-tags":""}}
I have used this guide to configure the worker: http://www.chrisumbel.com/article/windows_worker_to_bosh_deployed_concourse and the official docs
This issue was never resolved, but I believe that it is no longer relevant. Since this answer was posted concourse have added BOSH orchestrated windows workers :
https://github.com/pivotal-cf-experimental/concourse-windows-worker-release
They have also updated their documentation official documentation: https://concourse-ci.org/worker-pools.html
For those of you running Go backends in production:
What is your stack / configuration for running a Go web application?
I haven't seen much on this topic besides people using the standard library net/http package to keep a server running. I read using Nginx to pass requests to a Go server - nginx with Go
This seems a little fragile to me. For instance, the server would not automatically restart if the machine was restarted (without additional configuration scripts).
Is there a more solid production setup?
An aside about my intent - I'm planning out a Go powered REST backend server for my next project and want to make sure Go is going to be viable for launching the project live before I invest too much into it.
Go programs can listen on port 80 and serve HTTP requests directly. Instead, you may want to use a reverse proxy in front of your Go program, so that it listens on port 80 and and connects to your program on port, say, 4000. There are many reason for doing the latter: not having to run your Go program as root, serving other websites/services on the same host, SSL termination, load balancing, logging, etc.
I use HAProxy in front. Any reverse proxy could work. Nginx is also a great option (much more popular than HAProxy and capable of doing more).
HAProxy is very easy to configure if you read its documentation (HTML version). My whole haproxy.cfg file for one of my Go projects follows, in case you need a starting pont.
global
log 127.0.0.1 local0
maxconn 10000
user haproxy
group haproxy
daemon
defaults
log global
mode http
option httplog
option dontlognull
retries 3
timeout connect 5000
timeout client 50000
timeout server 50000
frontend http
bind :80
acl is_stats hdr(host) -i hastats.myapp.com
use_backend stats if is_stats
default_backend myapp
capture request header Host len 20
capture request header Referer len 50
backend myapp
server main 127.0.0.1:4000
backend stats
mode http
stats enable
stats scope http
stats scope myapp
stats realm Haproxy\ Statistics
stats uri /
stats auth username:password
Nginx is even easier.
Regarding service control, I run my Go program as a system service. I think everybody does that. My server runs Ubuntu, so it uses Upstart. I have put this at /etc/init/myapp.conf for Upstart to control my program:
start on runlevel [2345]
stop on runlevel [!2345]
chdir /home/myapp/myapp
setgid myapp
setuid myapp
exec ./myapp start 1>>_logs/stdout.log 2>>_logs/stderr.log
Another aspect is deployment. One option is to deploy by just sending binary file of the program and necessary assets. This is a pretty great solution IMO. I use the other option: compiling on server. (I’ll switch to deploying with binary files when I set up a so-called “Continuous Integration/Deployment” system.)
I have a small shell script on the server that pulls code for my project from a remote Git repository, builds it with Go, copies the binaries and other assets to ~/myapp/, and restarts the service.
Overall, the whole thing is not very different from any other server setup: you have to have a way to run your code and have it serve HTTP requests. In practice, Go has proved to be very stable for this stuff.
nginx for:
Reverse HTTP proxy to my Go application
Static file handling
SSL termination
HTTP headers (Cache-Control, et. al)
Access logs (and therefore leveraging system log rotation)
Rewrites (naked to www, http:// to https://, etc.)
nginx makes this very easy, and although you can serve directly from Go thanks to net/http, there's a lot of "re-inventing the wheel" and stuff like global HTTP headers involves some boilerplate you can probably avoid.
supervisord for managing my Go binary. Ubuntu's Upstart (as mentioned by Mostafa) is also good, but I like supervisord as it's relatively distro-agnostic and is well documented.
Supervisord, for me:
Runs my Go binary as needed
Brings it up after a crash
Holds my environmental variables (session auth keys, etc.) as part of a single config.
Runs my DB (to make sure my Go binary isn't running without it)
For those who want simple go app running as a daemon, use systemd (Supported by many linux distros) instead of Upstart.
Create a service file at
touch /etc/systemd/system/my-go-daemon.service
Enter
[Unit]
Description=My Go App
[Service]
Type=simple
WorkingDirectory=/my/go/app/directory
ExecStart=/usr/lib/go run main.go
[Install]
WantedBy=multi-user.target
Then enable and start the service
systemctl enable my-go-daemon
systemctl start my-go-daemon
systemctl status my-go-daemon
systemd has a separate journaling system that will let you tail logs for easy trouble-shooting.
You can bind your binary to a socket to Internet domain privileged ports (port numbers less than 1024) using setcap
setcap 'cap_net_bind_service=+ep' /path/to/binary
This command needs to be escalated. sudo as necessary
Every new version of your program will result in a new binary that will need to be reauthorized by setcap
setcap documentation
cap_net_bind_service documentation