K6 script failing in azure devops - performance

Setup
We are using K6 tool for Load testing our API.
Locally scripts work perfectly fine with virtual users up to 50
problem statement
While running the same script in Azure devops using K6 task and virtual user 15 for duration 60s most of the request (almost 90%) is failing because of timeout and following is the error :
level=warning msg="Request Failed" error="Get "https://xxxxx": dial tcp xxxx: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond."
Not sure
a) What is causing the timeout?
b) What are the other options?
PS: Api is behind Api Management Service in Azure and load test works locally fine even on 10 times higher load

Related

Error : 14 UNAVAILABLE: DNS resolution failed on BloomRPC golang GRPC

I'm figuring out how I can test my golang project using Bloomrpc. I've turned on the backend server, client server and db server but an error pop up like this on Bloomrpc:

org.apache.jorphan.util.JMeterError: Could not return sample

I am running Jmeter test in distributed mode. I have set up SSH tunneling for slaves since they are not in same region. I have executed 2700 users which ran fine. When we try to run 5200 users, users are going into finished status although steady state is 1 hours. I am using ultimate thread group.
4500 users are running fine and 600 Users are going into finished state out of 5200 usesr.
Seeing below error in Jmeter Server logs:
ERROR o.a.j.t.JMeterThread: Test failed!
org.apache.jorphan.util.JMeterError: Could not return sample
at org.apache.jmeter.samplers.StandardSampleSender.sampleOccurred(StandardSampleSender.java:70) ~[ApacheJMeter_core.jar:3.3 r1808647]
Caused by: java.rmi.ConnectException: Connection refused to host: 127.0.0.1; nested exception is:
java.net.ConnectException: Connection refused: connect
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(Unknown Source) ~[?:1.8.0_151]
Any idea? What is causing this?
It seems from logs that you tunnel is broken:
Caused by: java.rmi.ConnectException: Connection refused to host: 127.0.0.1;
See this:
https://superuser.com/questions/37738/how-to-reliably-keep-an-ssh-tunnel-open
The error is about JMeter slave being unable to establish connection to deliver results to the master so the problem is on SSH server side, you can look into its logs to see what went wrong.
The options are in:
SSH stands for "Secure Shell" so all the traffic gets encrypted. It also gets compressed in order to decrease footprint. When it comes to high loads it might be the case SSH server and/or client consumes a lot of resources and i.e. being killed by OOM Killer or whatever. You can try configuring your SSH server to disable compression and use weaker encryption algorithm, for example arcfour
It might be easier to move "master" to the "other region" than passing a lot of test results over SSH
Another option could be using VPN with no encryption, this way all JMeter nodes will be in the same subnet.
And finally you can disable automatic sending of test results from slaves to the master:
set mode=DiskStore property on JMeter Slaves
when your test is finished collect results from slaves and copy them over to master
use Merge Results Tool in order to combine multiple results into single .jtl file. Merge Results Tool can be installed using JMeter Plugins Manager

Concourse external windows worker failed register through atc

I have spun up a bosh cluster on AWS, running a concourse deployment. For this I used a tool called concourse-up. I spun up a windows worker outside the created VPC of the cluster, and I am trying to register the worker through the atc but this step fails with an error. I have opened up all ports and both a web VM and the Worker VM. I have tried several things but there are two specific errors that I get:
When I connect the worker without a --peer-ip, the worker registers so I can see it through the fly cli, but I get this error in the log(snippet below), and jobs will fail with this error:
Put /volumes/47c1c26c-274b-4f04-4dea-01d476ed949e/stream-in?path=.: read tcp 10.0.0.7:59478->10.0.0.7:39198: read: connection reset by peer
{"timestamp":"1513510128.917933226","source":"worker","message":"worker.setup.no-assets","log_level":1,"data":{"session":"1"}}
{"timestamp":"1513510128.920933962","source":"worker","message":"worker.garden.started","log_level":1,"data":{"session":"2"}}
{"timestamp":"1513510128.921934128","source":"baggageclaim","message":"baggageclaim.listening","log_level":1,"data":{"addr":"127.0.0.1:7788"}}
{"timestamp":"1513510130.645173311","source":"tsa","message":"tsa.connection.channel.forward-worker.register.start","log_level":1,"data":{"remote":"34.242.192.32:56803","session":"12.1.1.5","worker-address":"10.0.0.7:38380","worker-platform":"windows","worker-tags":""}}
{"timestamp":"1513510130.649989367","source":"tsa","message":"tsa.connection.channel.forward-worker.register.reached-worker","log_level":0,"data":{"baggageclaim-took":"2.251829ms","garden-took":"2.492218ms","remote":"34.242.192.32:56803","session":"12.1.1.5"}}
{"timestamp":"1513510128.960758924","source":"baggageclaim","message":"baggageclaim.repository.get-volume.volume-not-found","log_level":1,"data":{"session":"1.2","volume":"resource-certs"}}
{"timestamp":"1513510128.960758924","source":"baggageclaim","message":"baggageclaim.api.volume-server.get-volume.volume-not-found","log_level":1,"data":{"session":"2.1.2","volume":"resource-certs"}}
{"timestamp":"1513510128.963933945","source":"baggageclaim","message":"baggageclaim.repository.create-volume.failed-to-materialize-strategy","log_level":2,"data":{"error":"mkdir C:\\Users\\Administrator\\workspace\\concourse-workspace\\volumes\\init\\resource-certs: Cannot create a file when that file already exists.","handle":"resource-certs","session":"1.3"}}
supposedly this is not the right way to do it if I follow the official docs: https://concourse-ci.org/clusters-with-bosh.html#configuring-bosh-tsa
The other thing I tried is using the --peer-ip which is recommended is the official docs if you are outside a cluster and have no connection to any of the resources other than the atc. But this will not even register the worker, it fails with this error in the log:
{"timestamp":"1513510497.727977514","source":"tsa","message":"tsa.connection.channel.register-worker.register.start","log_level":1,"data":{"remote":"34.242.192.32:56828","session":"13.1.1.8","worker-address":"34.242.40.26:7777","worker-platform":"windows","worker-tags":""}}
{"timestamp":"1513510497.728802919","source":"tsa","message":"tsa.connection.channel.register-worker.register.failed-to-fetch-containers","log_level":2,"data":{"error":"Get http://api/containers: dial tcp 34.242.40.26:7777: getsockopt: connection refused","remote":"34.242.192.32:56828","session":"13.1.1.8"}}
{"timestamp":"1513510497.729249239","source":"tsa","message":"tsa.connection.channel.register-worker.register.failed-to-list-volumes","log_level":2,"data":{"error":"Get http://34.242.40.26:7788/volumes: dial tcp 34.242.40.26:7788: getsockopt: connection refused","remote":"34.242.192.32:56828","session":"13.1.1.8"}}
{"timestamp":"1513510497.729336023","source":"tsa","message":"tsa.connection.channel.register-worker.register.failed-to-reach-worker","log_level":1,"data":{"baggageclaim-took":"469.89µs","garden-took":"666.118µs","remote":"34.242.192.32:56828","session":"13.1.1.8"}}
{"timestamp":"1513510497.729401112","source":"tsa","message":"tsa.connection.channel.register-worker.register.done","log_level":1,"data":{"remote":"34.242.192.32:56828","session":"13.1.1.8","worker-address":"34.242.40.26:7777","worker-platform":"windows","worker-tags":""}}
I have used this guide to configure the worker: http://www.chrisumbel.com/article/windows_worker_to_bosh_deployed_concourse and the official docs
This issue was never resolved, but I believe that it is no longer relevant. Since this answer was posted concourse have added BOSH orchestrated windows workers :
https://github.com/pivotal-cf-experimental/concourse-windows-worker-release
They have also updated their documentation official documentation: https://concourse-ci.org/worker-pools.html

Unable to connect to feedback.sandbox.apple.com via tcp client in pushsharp

Unable to connect to the feedback.sandbox.push.apple.com with port number 2196.
Opened the ports (5223, 2915, 2916, 443) in windows firewall with TCP/SSL.
Tried connecting via telnet too, but did not help.
Getting the below exception
Service Exception: PushSharp.Apple.ApplePushService -> System.Net.Sockets.Socket
Exception (0x80004005): A connection attempt failed because the connected party
did not properly respond after a period of`enter code here` time, or established connection faile
d because connected host has failed to respond 17.172.232.45:2196
at System.Net.Sockets.TcpClient..ctor(String hostname, Int32 port)
at PushSharp.Apple.FeedbackService.Run(ApplePushChannelSettings settings, Can
cellationToken cancelToken) in d:\Sample Apps\Push Notification\PushSharp-master
\PushSharp-master\PushSharp.Apple\FeedbackService.cs:line 60
at PushSharp.Apple.ApplePushService.<>c__DisplayClass4.<.ctor>b__1(Object sta
te) in d:\Sample Apps\Push Notification\PushSharp-master\PushSharp-master\PushSh
arp.Apple\ApplePushService.cs:line 51
Failure: PushSharp.Apple.ApplePushService -> The maximum number of Send attempts
to send the notification was reached! -> {"aps":{"alert":"Hello World!","badge"
:7,"sound":"sound.caf"}}
Also, this does not need certificate to connect to sandbox, so why this exception?
You do need a development certificate from the Apple Developer portal to connect to sandbox. But I don't know if you need to use it to connect to feedback.sandbox.apple.
I can't seem to find a straight answer on this whether feedback services still works. I think APNS Feedback services are now deprecated. This could be why you're getting this error.
Does APNS Feedback service no longer exist as per new APIs?
APNS Feedback Service - is it still alive?

How to diagnostic establish connection slow

Recently I meet a weird problem, sometimes the browser will need about 20seconds to establish connecting to the IIS web server
1. this cannot be repro each time, when successful access the site, the following connecting will be very fast. After restart Application pool and Web Site, the first time connect will be slow.<br>
2. When success gernerate connection success, after minutes, the establsh connection will getting slow again.<br>
3. this is not relate to authentication, both Windows Authentication and Anonmymous will have the same problem.<br>
4. Installed HTTPWatch on my client machine, it shows that 20 seconds is connecting time before wating when this problem happen.<br>
5. localhost cannot repro, each time establish connection is fast
From the log of IIS, then first request is get after
how to debug this? seems this is relate to the network, porxy, domain, is there some tool can help to diagnostic this problem, and can tell me what the server is doing during the time slot of establishing connection

Resources