What version of Go are you using?
go version go1.13 linux/amd64
What OS and processor architecture are you using?
OS: CentOS-7 x86_64 GNU/Linux
What did I do?
Span 100 threads using goroutine reach thread read data from
redis,data is in json which contains key csv_file path.Each csv
contains 1000 tokens. Each thread pop data from redis and read csv and
spawn 1000 threads, how much tokens that much thread and call APNS
push. During the push getting "EOF". 90% calls failed with this error.
I have set my OS ulimit is 500000
what did you expect to see? It should get processed 10M tokens in 1 minute
What did I see instead?
I am getting following error when call APNS sevice with load testing
on dev certificate time="2020-02-25T08:54:44-05:00" level=info
msg="Push Error:%!(EXTRA *url.Error=Post
https://api.sandbox.push.apple.com/3/device/eoQtFQtlL4s:APA91bGrV0HqQH4qbxe
ZCJrX-XMHj63: EOF)" 90% calls failed with this error. each thread with
1000 tokens publishe taken 2s with EOF error which is extermelly slow.
Further informations:
Aim:
My aim is publish 10M tokens in 1 minute
Where I run the code:
I am running in golang code in aws EC2 instance in Virginia us-east-1
My question:
When this error came and how can I fix?
It will be great if I can get help.
Related
I have a blocking issue installing openwhisk with docker
I typed make quick-start right after a git pull of the project incubator-openwhisk-devtools. My OS is Fedora 29, docker version is 18.09.0, docker-compose version is 1.22.0. JDk 8 Oracle.
I get the following error:
[...]
adding the function to whisk ...
ok: created action hello
invoking the function ...
error: Unable to invoke action 'hello': The server is currently unavailable (because it is overloaded or down for maintenance). (code ciOZDS8VySDyVuETF14n8QqB9wifUboT)
[...]
[ERROR] [#tid_sid_unknown] [Invoker] failed to ping the controller: org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for health-0: 30069 ms has passed since batch creation plus linger time
[ERROR] [#tid_sid_unknown] [KafkaProducerConnector] sending message on topic 'health' failed: Expiring 1 record(s) for health-0: 30009 ms has passed since batch creation plus linger time
Please note that controller-local-logs.log is never created.
If I issue a touch controller-local-logs.log in the right directory the log file is always empty after I try to issue make quick-start again.
http://localhost:8888/ping gives me the right answer: pong.
http://localhost:9222 is not reacheable.
Where am I wrong?
Thank you in advance
I have a Go program that uses aws-sdk-go to talk to dynamodb. Dependencies are vendored. Go version 1.7.1. aws-sdk-go version 1.6.24. The program works as expected in all the following environments:
dev box from shell (Arch Linux)
docker container running on my dev box (Docker 1.13.1)
Ec2 instance from shell (Ubuntu 16.04)
When I run the docker container on kubernetes (same one I tested on my dev box), I get the following error:
2017/03/02 22:30:13 DEBUG ERROR: Request dynamodb/GetItem:
---[ REQUEST DUMP ERROR ]-----------------------------
net/http: invalid header field value "AWS4-HMAC-SHA256 Credential=hidden\n/20170302/us-east-1/dynamodb/aws4_request, SignedHeaders=accept-encoding;content-length;content-type;host;x-amz-date;x-amz-target, Signature=483f56dd0b17d8945d3c2f2044b7f97e531190602f132a4d5f828264b3a2cff2" for key Authorization
-----------------------------------------------------
2017/03/02 22:30:13 DEBUG: Response dynamodb/GetItem Details:
---[ RESPONSE ]--------------------------------------
HTTP/0.0 000 status code 0
Content-Length: 0
Based on:
https://golang.org/src/net/http/transport.go
https://godoc.org/golang.org/x/net/lex/httplex#ValidHeaderFieldValue
It looks like the problem is with the header value validation, yet I am at a loss to understand why it works everywhere except on my k8s cluster. The cluster is composed of Ec2 instances running the latest CoreOS stable ami (CoreOS stable 1235.8.0)
The docker image that works on my dev machine is scratch based. To troubleshoot I created an image based on Ubuntu latest with a separate go program that just does a simple get item from dynamodb. When this image is run on my k8s cluster and the program run from an interactive shell, I get the same errors. I have confirmed I can ping the dynamodb endpoints from this env.
I am having a hard time troubleshooting this issue: am I missing something stupid here? Can someone point me in the right direction or have an idea of what is going on?
remember the "-n" when you do this:
echo -n key | base64
The \n after hidden is certainly invalid. Not sure if it is actually there or somehow got inserted when you were cleansing for posting.
Consider:
package main
import (
"fmt"
"golang.org/x/net/lex/httplex"
)
func main() {
fmt.Println("Is valid (without new line)", httplex.ValidHeaderFieldValue("AWS4-HMAC-SHA256 Credential=hidden/20170302/us-east-1/dynamodb/aws4_request, SignedHeaders=accept-encoding;content-length;content-type;host;x-amz-date;x-amz-target, Signature=483f56dd0b17d8945d3c2f2044b7f97e531190602f132a4d5f828264b3a2cff2"))
fmt.Println("Is valid (with new line)", httplex.ValidHeaderFieldValue("AWS4-HMAC-SHA256 Credential=hidden\n/20170302/us-east-1/dynamodb/aws4_request, SignedHeaders=accept-encoding;content-length;content-type;host;x-amz-date;x-amz-target, Signature=483f56dd0b17d8945d3c2f2044b7f97e531190602f132a4d5f828264b3a2cff2"))
}
One guess would be wherever the real hidden value is getting pulled from (config file etc) mistakenly has the \n in there and it's happily getting pulled into your header, but only in this case.
I'm experiencing an unresponsive socket in with my Puma setup after random time. Up to this point I don't have a clue what's causing the issue. I was hoping somebody over here can help we with some answers or point me in the right direction. I'm having the following setup:
I'm using the official docker ruby-2.2.3-slim image together with the latest puma release 2.15.3, I've also installed Nginx as a reverse proxy. But I'm already sure Nginx isn't the problem over here because and I've tried to verify if the socket was working using this script. And the socket wasn't working, I got a timeout over there as well so I could ignore Nginx.
This is a testing environment so the server isn't experiencing any extreme load, I've also check memory consumption it has still several GB's of free space so that couldn't be the issue either.
What triggered me to look at the puma socket was the error message I got in my Nginx error logging:
upstream timed out (110: Connection timed out) while reading response header from upstream
Also I couldn't find anything in the logs of puma indicating what is going wrong, over here are my puma setup:
threads 0, 16
app_dir = ENV.fetch('APP_HOME')
environment ENV['RAILS_ENV']
daemonize
bind "unix://#{app_dir}/sockets/puma.sock"
stdout_redirect "#{app_dir}/log/puma.stdout.log", "#{app_dir}/log/puma.stderr.log", true
pidfile "#{app_dir}/pids/puma.pid"
state_path "#{app_dir}/pids/puma.state"
activate_control_app
on_worker_boot do
require 'active_record'
ActiveRecord::Base.connection.disconnect! rescue ActiveRecord::ConnectionNotEstablished
ActiveRecord::Base.establish_connection(YAML.load_file("#{app_dir}/config/database.yml")[ENV['RAILS_ENV']])
end
And this it the output in my puma state file:
---
pid: 43
config: !ruby/object:Puma::Configuration
cli_options:
conf:
options:
:min_threads: 0
:max_threads: 16
:quiet: false
:debug: false
:binds:
- unix:///APP/sockets/puma.sock
:workers: 1
:daemon: true
:mode: :http
:before_fork: []
:worker_timeout: 60
:worker_boot_timeout: 60
:worker_shutdown_timeout: 30
:environment: staging
:redirect_stdout: "/APP/log/puma.stdout.log"
:redirect_stderr: "/APP/log/puma.stderr.log"
:redirect_append: true
:pidfile: "/APP/pids/puma.pid"
:state: "/APP/pids/puma.state"
:control_url: unix:///tmp/puma-status-1449260516541-37
:config_file: config/puma.rb
:control_url_temp: "/tmp/puma-status-1449260516541-37"
:control_auth_token: cda8879717be7a645ea323d931b88d4b
:tag: APP
The application itself is a Rails app on the latest version 4.2.5, it's deployed on GCE (Google Container Engine).
If somebody could give me some pointer's on how to debug this any further would be very much appreciated. Because now I don't see any output anywhere which could help me any further.
EDIT
I replaced the unix socket with tcp connection to Puma with the same result, still hangs after x time
I'd start with:
How many requests get processed successfully per instance of puma?
Make sure you log the beginning and end of each request with the thread id of the thread executing it, what do you see?
Not knowing more about your application, I'd say it's likely the threads get stuck doing some long/blocking calls without timeouts or spinning on some computation until the whole thread pool gets depleted.
We'll see.
I finally found out why my application was behaving the way it was.
After trying to use a tcp connection and switching to Unicorn I start looking into other possible sources.
That's when I thought maybe my connection to Google Cloud SQL could be the problem. Once I read the faq of Cloud SQL, they mentioned that you have to tweak you Compute instances to ensure they keep open your DB connection. So I performed the next steps they recommend and that solved the problem for me, I added them just in case:
# Display the current tcp_keepalive_time value.
$ cat /proc/sys/net/ipv4/tcp_keepalive_time
# Set tcp_keepalive_time to 60 seconds and make it permanent across reboots.
$ echo 'net.ipv4.tcp_keepalive_time = 60' | sudo tee -a /etc/sysctl.conf
# Apply the change.
$ sudo /sbin/sysctl --load=/etc/sysctl.conf
# Display the tcp_keepalive_time value to verify the change was applied.
$ cat /proc/sys/net/ipv4/tcp_keepalive_time
when i run from windows-7 with cygwin to connect CFEngine bersion 3.4.2
cf-agent -Bs 217.64.173.210
Challenge response from server 217.64.173.210/217.64.173.210 was incorrect!
I: Made in version 'not specified' of '/var/cfengine/inputs/update.cf' near line 47
!! Authentication dialogue with 217.64.173.210 failed
Challenge response from server 217.64.173.210/217.64.173.210 was incorrect!
I: Made in version 'not specified' of '/var/cfengine/inputs/update.cf' near line
and in /var/cfengine/inputs/update.cf on line 47 is
47 : perms => m("600"),
on cgwin in folder keys
/var/cfengine/ppkeys
localhost.pub
localhost.priv
root-MD5=b8825ba0a0e7017e34b15766d3b3ac58 (which is also at CFEngine Server Side shared ky)
on Cf-Engine Server Side
/var/cfengine/ppkeys/
localhost.priv
localhost.pub
root-MD5=b8825ba0a0e7017e34b15766d3b3ac58
With Regards
Sandeep
Did you also get the server to trust the client's key? like so:
cf-key -t root-MD5=b8825ba0a0e7017e34b15766d3b3ac58
(on the server)
Also, try restarting cf-serverd in verbose mode with the -v switch on the server, and watch what error messages you get on that end.
I downloaded gsoap 2.8 and went into the samples folder and ran a make. Everything seems to have built fine. I then navigated into the "ssl" folder and ran the sllserver in one xterm and ran sslclient in a second xterm window. (I am running RHEL 6) The server seems to run fine, it says "Bind successful: socket = 4". But when I run the client I receive the following message:
Error -1 fault: SOAP-ENV:Client [no subcode]
"End of file or no input: Operation interrupted or timed out (30 s receive delay) (30 s send delay)"
Detail: [no detail]
I have not modified any of the sample code, so it seems like it should just work. Can anyone please give me some advice as to what I should look at? I am trying to learn how to set up a soap server that uses ssl. (I have a gsoap server running already) I searched all day for an example on the web and as usual, there is not one.
Thank you so much for any help.
You could rebuild this example with compiler switch -DDEBUG to enable message logging (make 'sslclient_CFLAGS = -DWITH_OPENSSL -DWITH_GZIP -DDEBUG'). The TEST.log will tell what went wrong. I suspect it is a network issue with the server address/port that is set by default to "https://localhost:18081".
You could set the timeout parameter: soap.recv_timeout = 60 (for 60 seconds)