OpenDaylight: No OpenFlow connection from Open vSwitch to Controller - opendaylight

No connection between the controller and open vswitch.
opendaylight-user#root>info
Karaf
Karaf version 4.0.10
Karaf home /opt/odl
Karaf base /opt/odl
OSGi Framework org.eclipse.osgi-3.10.101.v20150820-1432
JVM
Java Virtual Machine OpenJDK 64-Bit Server VM version 25.144-b01
Version 1.8.0_144
Vendor Oracle Corporation
Pid 4312
Uptime 19 hours 16 minutes
Total compile time 3 minutes
Threads
Live threads 218
Daemon threads 99
Peak 221
Total started 8170
Memory
Current heap size 621,509 kbytes
Maximum heap size 1,864,192 kbytes
Committed heap size 1,030,144 kbytes
Pending objects 0
Garbage collector Name = 'PS Scavenge', Collections = 533, Time = 12.404 seconds
Garbage collector Name = 'PS MarkSweep', Collections = 24, Time = 15.571 seconds
Classes
Current classes loaded 23,639
Total classes loaded 23,827
Total classes unloaded 188
Operating system
Name Linux version 3.10.0-514.21.1.el7.x86_64
Architecture amd64
Processors 4
opendaylight-user#root>
The OpenFlow Session stats show. The 10.10.10.10 is the IP address of the switch:
opendaylight-user#root>ofp:show-session-stats
SESSION : /10.10.10.10:35616
CONNECTION_DISCONNECTED_BY_DEVICE : 1
SESSION : /10.10.10.10:35592
CONNECTION_DISCONNECTED_BY_DEVICE : 1
SESSION : /10.10.10.10:35608
CONNECTION_DISCONNECTED_BY_DEVICE : 1
SESSION : /10.10.10.10:51110
CONNECTION_DISCONNECTED_BY_DEVICE : 1
SESSION : /10.10.10.10:35610
CONNECTION_DISCONNECTED_BY_DEVICE : 1
SESSION : /10.10.10.10:35612
CONNECTION_DISCONNECTED_BY_DEVICE : 1
SESSION : /10.10.10.10:35614
CONNECTION_DISCONNECTED_BY_DEVICE : 1
opendaylight-user#root>
The logs show the following lines:
2018-02-26 11:54:08,732 | DEBUG | pool-58-thread-1 | LLDPSpeaker | 296 - org.opendaylight.openflowplugin.applications.lldp-speaker - 0.5.1 | Sending LLDP frames to 0 ports...
2018-02-26 11:54:10,092 | DEBUG | ntLoopGroup-15-7 | nflowProtocolListenerInitialImpl | 305 - org.opendaylight.openflowplugin.impl - 0.5.1 | echo request received: 0
2018-02-26 11:54:13,732 | DEBUG | pool-58-thread-1 | LLDPSpeaker | 296 - org.opendaylight.openflowplugin.applications.lldp-speaker - 0.5.1 | Sending LLDP frames to 0 ports...
2018-02-26 11:54:15,092 | DEBUG | ntLoopGroup-15-7 | nflowProtocolListenerInitialImpl | 305 - org.opendaylight.openflowplugin.impl - 0.5.1 | echo request received: 0
2018-02-26 11:54:16,073 | DEBUG | pool-75-thread-1 | MessageIntelligenceAgencyImpl | 305 - org.opendaylight.openflowplugin.impl - 0.5.1 | FROM_SWITCH: no activity detected
2018-02-26 11:54:16,073 | DEBUG | pool-75-thread-1 | MessageIntelligenceAgencyImpl | 305 - org.opendaylight.openflowplugin.impl - 0.5.1 | FROM_SWITCH_TRANSLATE_IN_SUCCESS: no activity detected
2018-02-26 11:54:16,073 | DEBUG | pool-75-thread-1 | MessageIntelligenceAgencyImpl | 305 - org.opendaylight.openflowplugin.impl - 0.5.1 | FROM_SWITCH_TRANSLATE_OUT_SUCCESS: no activity detected
2018-02-26 11:54:16,073 | DEBUG | pool-75-thread-1 | MessageIntelligenceAgencyImpl | 305 - org.opendaylight.openflowplugin.impl - 0.5.1 | FROM_SWITCH_TRANSLATE_SRC_FAILURE: no activity detected
2018-02-26 11:54:16,073 | DEBUG | pool-75-thread-1 | MessageIntelligenceAgencyImpl | 305 - org.opendaylight.openflowplugin.impl - 0.5.1 | FROM_SWITCH_PACKET_IN_LIMIT_REACHED_AND_DROPPED: no activity detected
2018-02-26 11:54:16,073 | DEBUG | pool-75-thread-1 | MessageIntelligenceAgencyImpl | 305 - org.opendaylight.openflowplugin.impl - 0.5.1 | FROM_SWITCH_NOTIFICATION_REJECTED: no activity detected
2018-02-26 11:54:16,073 | DEBUG | pool-75-thread-1 | MessageIntelligenceAgencyImpl | 305 - org.opendaylight.openflowplugin.impl - 0.5.1 | FROM_SWITCH_PUBLISHED_SUCCESS: no activity detected
2018-02-26 11:54:16,073 | DEBUG | pool-75-thread-1 | MessageIntelligenceAgencyImpl | 305 - org.opendaylight.openflowplugin.impl - 0.5.1 | FROM_SWITCH_PUBLISHED_FAILURE: no activity detected
2018-02-26 11:54:16,073 | DEBUG | pool-75-thread-1 | MessageIntelligenceAgencyImpl | 305 - org.opendaylight.openflowplugin.impl - 0.5.1 | TO_SWITCH_ENTERED: no activity detected
2018-02-26 11:54:16,073 | DEBUG | pool-75-thread-1 | MessageIntelligenceAgencyImpl | 305 - org.opendaylight.openflowplugin.impl - 0.5.1 | TO_SWITCH_DISREGARDED: no activity detected
2018-02-26 11:54:16,073 | DEBUG | pool-75-thread-1 | MessageIntelligenceAgencyImpl | 305 - org.opendaylight.openflowplugin.impl - 0.5.1 | TO_SWITCH_RESERVATION_REJECTED: no activity detected
2018-02-26 11:54:16,073 | DEBUG | pool-75-thread-1 | MessageIntelligenceAgencyImpl | 305 - org.opendaylight.openflowplugin.impl - 0.5.1 | TO_SWITCH_READY_FOR_SUBMIT: no activity detected
2018-02-26 11:54:16,073 | DEBUG | pool-75-thread-1 | MessageIntelligenceAgencyImpl | 305 - org.opendaylight.openflowplugin.impl - 0.5.1 | TO_SWITCH_SUBMIT_SUCCESS: no activity detected
2018-02-26 11:54:16,073 | DEBUG | pool-75-thread-1 | MessageIntelligenceAgencyImpl | 305 - org.opendaylight.openflowplugin.impl - 0.5.1 | TO_SWITCH_SUBMIT_SUCCESS_NO_RESPONSE: no activity detected
2018-02-26 11:54:16,073 | DEBUG | pool-75-thread-1 | MessageIntelligenceAgencyImpl | 305 - org.opendaylight.openflowplugin.impl - 0.5.1 | TO_SWITCH_SUBMIT_FAILURE: no activity detected
2018-02-26 11:54:16,073 | DEBUG | pool-75-thread-1 | MessageIntelligenceAgencyImpl | 305 - org.opendaylight.openflowplugin.impl - 0.5.1 | TO_SWITCH_SUBMIT_ERROR: no activity detected
2018-02-26 11:54:16,073 | DEBUG | pool-75-thread-1 | MessageIntelligenceAgencyImpl | 305 - org.opendaylight.openflowplugin.impl - 0.5.1 | REQUEST_STACK_FREED: no activity detected
2018-02-26 11:54:16,073 | DEBUG | pool-75-thread-1 | MessageIntelligenceAgencyImpl | 305 - org.opendaylight.openflowplugin.impl - 0.5.1 | OFJ_BACKPRESSURE_ON: no activity detected
2018-02-26 11:54:16,073 | DEBUG | pool-75-thread-1 | MessageIntelligenceAgencyImpl | 305 - org.opendaylight.openflowplugin.impl - 0.5.1 | OFJ_BACKPRESSURE_OFF: no activity detected
No nodes in OpenFlow topology:
opendaylight-user#root>openflow:getallnodes
No node is connected yet
opendaylight-user#root>
On the wire, I see OpenFlow packets between the controller and the switch:
Frame 1: 74 bytes on wire (592 bits), 74 bytes captured (592 bits) on interface 0
Internet Protocol Version 4, Src: 10.10.10.10 (10.10.10.10), Dst: 10.10.10.20 (10.10.10.20)
Transmission Control Protocol, Src Port: 35338 (35338), Dst Port: openflow (6653), Seq: 1, Ack: 1, Len: 8
Openflow Protocol
Header
Version: 0x04
Type: Echo request (SM) - OFPT_ECHO_REQUEST (2)
Length: 8
Transaction ID: 0
Frame 2: 74 bytes on wire (592 bits), 74 bytes captured (592 bits) on interface 0
Internet Protocol Version 4, Src: 10.10.10.20 (10.10.10.20), Dst: 10.10.10.10 (10.10.10.10)
Transmission Control Protocol, Src Port: openflow (6653), Dst Port: 35338 (35338), Seq: 1, Ack: 9, Len: 8
Openflow Protocol
Header
Version: 0x04
Type: Echo reply (SM) - OFPT_ECHO_REPLY (3)
Length: 8
Transaction ID: 0
Frame 3: 66 bytes on wire (528 bits), 66 bytes captured (528 bits) on interface 0
Internet Protocol Version 4, Src: 10.10.10.10 (10.10.10.10), Dst: 10.10.10.20 (10.10.10.20)
Transmission Control Protocol, Src Port: 35338 (35338), Dst Port: openflow (6653), Seq: 9, Ack: 9, Len: 0
Flags: 0x010 (ACK)
000. .... .... = Reserved: Not set
...0 .... .... = Nonce: Not set
.... 0... .... = Congestion Window Reduced (CWR): Not set
.... .0.. .... = ECN-Echo: Not set
.... ..0. .... = Urgent: Not set
.... ...1 .... = Acknowledgment: Set
.... .... 0... = Push: Not set
.... .... .0.. = Reset: Not set
.... .... ..0. = Syn: Not set
.... .... ...0 = Fin: Not set
Options: (12 bytes), No-Operation (NOP), No-Operation (NOP), Timestamps
No-Operation (NOP)
Type: 1
0... .... = Copy on fragmentation: No
.00. .... = Class: Control (0)
...0 0001 = Number: No-Operation (NOP) (1)
Timestamps: TSval 140811003, TSecr 165720510
Kind: Timestamp (8)
Length: 10
Timestamp value: 140811003
Timestamp echo reply: 165720510
Now, on Open vSwitch side:
# ovs-vsctl show
934472aa-72ca-4f61-834c-a86bd41a9b27
Manager "tcp:10.10.10.20:6640"
is_connected: true
Bridge "vsw0"
Controller "tcp:10.10.10.20:6653"
is_connected: true
Port "339bfa2f6b7b4_l"
Interface "339bfa2f6b7b4_l"
Port "0de326ec0ace4_l"
Interface "0de326ec0ace4_l"
Port "vsw0"
Interface "vsw0"
type: internal
ovs_version: "2.9.0"
The controller goes from ACTIVE to IDLE:
# ovs-vsctl list controller
_uuid : ac3f5c94-624a-47ce-90c3-782060e4ec3c
connection_mode : []
controller_burst_limit: []
controller_rate_limit: []
enable_async_messages: []
external_ids : {}
inactivity_probe : []
is_connected : true
local_gateway : []
local_ip : []
local_netmask : []
max_backoff : []
other_config : {}
role : other
status : {sec_since_connect="42353", state=ACTIVE}
target : "tcp:10.10.10.20:6653"
The logs show the following.
2018-02-26T16:59:55.091Z|90890|vconn|DBG|tcp:10.10.10.20:6653: sent (Success): OFPT_ECHO_REQUEST (OF1.3) (xid=0x0): 0 bytes of payload
2018-02-26T16:59:55.092Z|90891|vconn|DBG|tcp:10.10.10.20:6653: received: OFPT_ECHO_REPLY (OF1.3) (xid=0x0): 0 bytes of payload
2018-02-26T16:59:55.092Z|90892|rconn|DBG|vsw0<->tcp:10.10.10.20:6653: entering ACTIVE
2018-02-26T17:00:00.091Z|90893|rconn|DBG|vsw0<->tcp:10.10.10.20:6653: idle 5 seconds, sending inactivity probe
2018-02-26T17:00:00.091Z|90894|rconn|DBG|vsw0<->tcp:10.10.10.20:6653: entering IDLE
Any ideas why the OF connection is not being formed?

The issue was that the bridge was not set with OpenFlow 1.3 protocol.
ovs-vsctl set bridge vsw0 protocols=OpenFlow13
Right after that, the session came up:
opendaylight-user#root>ofp:show-session-stats
SESSION : openflow:231865136161864
CONNECTION_CREATED : 1
opendaylight-user#root>
The controller is now "master":
# ovs-vsctl list controller
_uuid : ac3f5c94-624a-47ce-90c3-782060e4ec3c
connection_mode : []
controller_burst_limit: []
controller_rate_limit: []
enable_async_messages: []
external_ids : {}
inactivity_probe : []
is_connected : true
local_gateway : []
local_ip : []
local_netmask : []
max_backoff : []
other_config : {}
role : master
status : {sec_since_connect="116", state=ACTIVE}
target : "tcp:10.10.10.20:6653"

Related

USRP - daughterboard installation issue

I'm trying to receive a signal whith a center frequency equal to 2e8Hz and a bandwith slightly smaller than 2e5Hz with my USRP X310.
I use the rx_sampletofile.cpp func of uhd 3.10.1 as shown :
./rx_sampletofile --file test.bin --duration --rate 4e8 --nsamps 4e8 1 --freq 2e8 --type float --bw 8e7 --skip-lo
When I look at the terminal, everything is executed without error or warnings but I see that the actual rx frequency isn't changed afterall ...
Setting RX Rate: 20.000000 Msps...
Actual RX Rate: 20.000000 Msps...
Setting RX Freq: 200.000000 MHz...
Setting RX LO Offset: 0.000000 MHz...
Actual RX Freq: 0.000000 MHz...
Setting RX Bandwidth: 0.250000 MHz...
Actual RX Bandwidth: 0.250000 MHz...
I tried to change RX Freq to lower freq and also--lo-offset but it always stays at RX Freq = 0Hz so if you have any idea I'll take it.
Thanks
It appears that my issue is finally related to the daughterboards because the drivers don't detect them well.
Daughterboard Issue :
_____________________________________________________
| | /
| | | RX Dboard: A
| | | ID: Unknown (0x0095)
| | | Serial: 31F94F3
| | | _____________________________________________________
| | | /
| | | | RX Frontend: 0
| | | | Name: Unknown (0x0095) - 0
| | | | Antennas:
| | | | Sensors:
| | | | Freq range: 0.000 to 0.000 MHz
| | | | Gain Elements: None
| | | | Bandwidth range: 0.0 to 0.0 step 0.0 Hz
| | | | Connection Type: IQ
| | | | Uses LO offset: No
| | | _____________________________________________________
| | | /
| | | | RX Codec: A
| | | | Name: ads62p48
| | | | Gain range digital: 0.0 to 6.0 step 0.5 dB
| | _____________________________________________________
| | /
| | | RX Dboard: B
| | | _____________________________________________________
| | | /
| | | | RX Frontend: 0
| | | | Name: Unknown (0xffff) - 0
| | | | Antennas:
| | | | Sensors:
| | | | Freq range: 0.000 to 0.000 MHz
| | | | Gain Elements: None
| | | | Bandwidth range: 0.0 to 0.0 step 0.0 Hz
| | | | Connection Type: IQ
| | | | Uses LO offset: No
| | | _____________________________________________________
| | | /
| | | | RX Codec: B
| | | | Name: ads62p48
| | | | Gain range digital: 0.0 to 6.0 step 0.5 dB
| | _____________________________________________________
Your UHD is too old for your hardware revision of the TwinRX daughterboard.
The only solution is to use a more modern version of UHD. This will also require you to load a more modern version of the FPGA image.

How can I access the current ZonedDateTime (or Date) in a "route" when processing with a "timer" loop?

Background:
The "configure()" method, in the sample code below, loops at specified time interval at which time I wish to log the current "ZonedDateTime.now() value
Problem:
The ZonedDateTime.now() value is always the same value, despite the interval time difference.
Question:
What technique can I used to access the current ZonedDateTime.now() value at each interval?
(NOTE: I ultimately, wish to use this value as a parameter in a REST call)
Sample code:
package aaa.bbb.ccc.dateparmissue;
import java.time.ZonedDateTime;
import java.time.format.DateTimeFormatter;
import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.cdi.ContextName;
#ContextName("rest-dsl")
public class DateParmIssue extends RouteBuilder {
public DateParmIssue() {
}
private final String codeList = "AA,BB,CC";
private final int notifyTime = 10; //<==10 second interval
#Override
public void configure() throws Exception {
org.apache.log4j.MDC.put("app.name", "dateParmIssue");
System.getProperties().list(System.out);
onException(Exception.class)
.log("onException_processing_exception:" + this.exceptionMessage().toString() + "...send to_error_queue:" + body(String.class).toString())
.handled(true);
from("timer://foo?fixedRate=true&period=" + (notifyTime * 1000))
.setBody(constant(this.codeList))
.to("seda:node0");
from("seda:node0")
.split().tokenize(",")
.to("seda:node1");
from("seda:node1")
.log("seda:node1...body-code=${body}...zdt=" + simple(ZonedDateTime.now().format(DateTimeFormatter.ISO_OFFSET_DATE_TIME)));
}
}
Sample output...
2017-12-27 12:17:11,649 | INFO | 1 - seda://node1 | route3 | 232 - org.apache.camel.camel-core - 2.17.0.redhat-630187 | direct:node1...body-code=AA...zdt=Simple: 2017-12-27T12:17:10.306-05:00
2017-12-27 12:17:11,653 | INFO | 1 - seda://node1 | route3 | 232 - org.apache.camel.camel-core - 2.17.0.redhat-630187 | direct:node1...body-code=BB...zdt=Simple: 2017-12-27T12:17:10.306-05:00
2017-12-27 12:17:11,653 | INFO | 1 - seda://node1 | route3 | 232 - org.apache.camel.camel-core - 2.17.0.redhat-630187 | direct:node1...body-code=CC...zdt=Simple: 2017-12-27T12:17:10.306-05:00
2017-12-27 12:17:21,630 | INFO | 1 - seda://node1 | route3 | 232 - org.apache.camel.camel-core - 2.17.0.redhat-630187 | direct:node1...body-code=AA...zdt=Simple: 2017-12-27T12:17:10.306-05:00
2017-12-27 12:17:21,630 | INFO | 1 - seda://node1 | route3 | 232 - org.apache.camel.camel-core - 2.17.0.redhat-630187 | direct:node1...body-code=BB...zdt=Simple: 2017-12-27T12:17:10.306-05:00
2017-12-27 12:17:21,631 | INFO | 1 - seda://node1 | route3 | 232 - org.apache.camel.camel-core - 2.17.0.redhat-630187 | direct:node1...body-code=CC...zdt=Simple: 2017-12-27T12:17:10.306-05:00
2017-12-27 12:17:31,633 | INFO | 1 - seda://node1 | route3 | 232 - org.apache.camel.camel-core - 2.17.0.redhat-630187 | direct:node1...body-code=AA...zdt=Simple: 2017-12-27T12:17:10.306-05:00
2017-12-27 12:17:31,636 | INFO | 1 - seda://node1 | route3 | 232 - org.apache.camel.camel-core - 2.17.0.redhat-630187 | direct:node1...body-code=BB...zdt=Simple: 2017-12-27T12:17:10.306-05:00
2017-12-27 12:17:31,637 | INFO | 1 - seda://node1 | route3 | 232 - org.apache.camel.camel-core - 2.17.0.redhat-630187 | direct:node1...body-code=CC...zdt=Simple: 2017-12-27T12:17:10.306-05:00
It turns out that the "timer" inserts a "firedTime" property in the header that can be used (although it is currently as a java.util.Date object, rather than one of the newer Java 8 date objects - e.g., ZonedDateTime).
So, an alternative might be as follows - i.e., which shows the changing date...
from("seda:node1")
.process(new Processor(){
public void process(Exchange exchange) {
exchange.getIn().setHeader("code", exchange.getIn().getBody().toString());
exchange.getIn().setBody((new Date(exchange.getIn().getHeader("firedTime").toString()).toInstant()).atZone(ZoneId.systemDefault()).format(DateTimeFormatter.ISO_OFFSET_DATE_TIME));
}
})
.log("seda:node1...body-code=${in.header.code}...zdt=${body}");
...and date now changes...
2017-12-28 15:37:18,017 | INFO | 7 - seda://node1 | route17 | 232 - org.apache.camel.camel-core - 2.17.0.redhat-630187 | seda:node1...body-code=AA...zdt=2017-12-28T15:37:18-05:00
2017-12-28 15:37:18,019 | INFO | 7 - seda://node1 | route17 | 232 - org.apache.camel.camel-core - 2.17.0.redhat-630187 | seda:node1...body-code=BB...zdt=2017-12-28T15:37:18-05:00
2017-12-28 15:37:18,020 | INFO | 7 - seda://node1 | route17 | 232 - org.apache.camel.camel-core - 2.17.0.redhat-630187 | seda:node1...body-code=CC...zdt=2017-12-28T15:37:18-05:00
2017-12-28 15:37:28,017 | INFO | 7 - seda://node1 | route17 | 232 - org.apache.camel.camel-core - 2.17.0.redhat-630187 | seda:node1...body-code=AA...zdt=2017-12-28T15:37:28-05:00
2017-12-28 15:37:28,017 | INFO | 7 - seda://node1 | route17 | 232 - org.apache.camel.camel-core - 2.17.0.redhat-630187 | seda:node1...body-code=BB...zdt=2017-12-28T15:37:28-05:00
2017-12-28 15:37:28,018 | INFO | 7 - seda://node1 | route17 | 232 - org.apache.camel.camel-core - 2.17.0.redhat-630187 | seda:node1...body-code=CC...zdt=2017-12-28T15:37:28-05:00

grails webflow notserializableexception

Grails upgrade from 1.3.7 to Grails 2.1.0.
Grails run-app executed. No errors.
Call to webflow action throws GrailsExceptionResolver error. States a grails webflow object isn’t serializable. I have navigated all classes to ensure all have ‘implements serializable’.
Error not identifying object not being serialized, so unable to identify if a class field should be made transient to be ignored by serializable runtime.
Full Stacktrace:
2016-12-20 09:51:25,750 [http-bio-80-exec-1] DEBUG services.RpgService - ************
| Error 2016-12-20 09:51:26,450 [http-bio-80-exec-1] ERROR errors.GrailsExceptionResolver - NotSerializableException occurred when processing request: [GET] /…/…/order
org.codehaus.groovy.grails.commons.spring.GrailsWebApplicationContext. Stacktrace follows:
Message: Could not serialize flow execution; make sure all objects stored in flow or flash scope are serializable
Line | Method
->> 384 | doFilterInternal in org.jsecurity.web.servlet.JsecurityFilter
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
| 183 | doFilter in org.jsecurity.web.servlet.OncePerRequestFilter
| 886 | runTask . . . . in java.util.concurrent.ThreadPoolExecutor$Worker
| 908 | run in ''
^ 662 | run . . . . . . in java.lang.Thread
Caused by NotSerializableException: org.codehaus.groovy.grails.commons.spring.GrailsWebApplicationContext
->> 1164 | writeObject0 in java.io.ObjectOutputStream
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
| 1518 | defaultWriteFields in ''
| 1483 | writeSerialData in ''
| 1400 | writeOrdinaryObject in ''
| 1158 | writeObject0 . . in ''
| 1518 | defaultWriteFields in ''
| 1483 | writeSerialData in ''
| 1400 | writeOrdinaryObject in ''
| 1158 | writeObject0 . . in ''
| 330 | writeObject in ''
| 1001 | writeObject . . in java.util.HashMap
| 940 | invokeWriteObject in java.io.ObjectStreamClass
| 1469 | writeSerialData in java.io.ObjectOutputStream
| 1400 | writeOrdinaryObject in ''
| 1158 | writeObject0 . . in ''
| 1518 | defaultWriteFields in ''
Found the resolution.
My controller service was calling sessionFactory which also needed to be transient.
code change:
def sessionFactory => transient sessionFactory.
Did not need make any domain class variables transient directly.
fixed the issue.

Cassandra 3.9 CQL tracing bottleneck identification

I am using 3 node RF3 Cassandra cluster on AWS.
I set a timeout of a read request for 10ms. I have noticed some of my requests are timing out. This is what I observe in CQL with TRACING ON;
Tracing session: 9fc1d420-9829-11e6-b04a-834837c1747b
activity | timestamp | source | source_elapsed | client
-------------------------------------------------------------------------------------------------------------------+----------------------------+---------------+----------------+-----------
Execute CQL3 query | 2016-10-22 10:32:02.274000 | 10.20.30.40 | 0 | 127.0.0.1
Parsing select * from recipes where id = fcc7d8b5-46d3-4867-903c-4a5c66a1fd2e; [Native-Transport-Requests-8] | 2016-10-22 10:32:02.274000 | 10.20.30.40 | 264 | 127.0.0.1
Preparing statement [Native-Transport-Requests-8] | 2016-10-22 10:32:02.274000 | 10.20.30.40 | 367 | 127.0.0.1
reading data from /10.20.0.1 [Native-Transport-Requests-8] | 2016-10-22 10:32:02.275000 | 10.20.30.40 | 680 | 127.0.0.1
Sending READ message to /10.20.0.1 [MessagingService-Outgoing-/10.20.0.1] | 2016-10-22 10:32:02.286000 | 10.20.30.40 | 12080 | 127.0.0.1
READ message received from /10.20.30.40 [MessagingService-Incoming-/10.20.30.40] | 2016-10-22 10:32:02.296000 | 10.20.0.1 | 51 | 127.0.0.1
Executing single-partition query on recipes [ReadStage-8] | 2016-10-22 10:32:02.298000 | 10.20.0.1 | 2423 | 127.0.0.1
Acquiring sstable references [ReadStage-8] | 2016-10-22 10:32:02.298000 | 10.20.0.1 | 2481 | 127.0.0.1
Skipped 0/4 non-slice-intersecting sstables, included 0 due to tombstones [ReadStage-8] | 2016-10-22 10:32:02.298000 | 10.20.0.1 | 2548 | 127.0.0.1
Bloom filter allows skipping sstable 55 [ReadStage-8] | 2016-10-22 10:32:02.298000 | 10.20.0.1 | 2614 | 127.0.0.1
Bloom filter allows skipping sstable 130 [ReadStage-8] | 2016-10-22 10:32:02.298000 | 10.20.0.1 | 2655 | 127.0.0.1
Bloom filter allows skipping sstable 140 [ReadStage-8] | 2016-10-22 10:32:02.298000 | 10.20.0.1 | 2704 | 127.0.0.1
Bloom filter allows skipping sstable 141 [ReadStage-8] | 2016-10-22 10:32:02.298001 | 10.20.0.1 | 2739 | 127.0.0.1
Merged data from memtables and 4 sstables [ReadStage-8] | 2016-10-22 10:32:02.298001 | 10.20.0.1 | 2796 | 127.0.0.1
Read 0 live and 0 tombstone cells [ReadStage-8] | 2016-10-22 10:32:02.298001 | 10.20.0.1 | 2854 | 127.0.0.1
Enqueuing response to /10.20.30.40 [ReadStage-8] | 2016-10-22 10:32:02.299000 | 10.20.0.1 | 2910 | 127.0.0.1
Sending REQUEST_RESPONSE message to /10.20.30.40 [MessagingService-Outgoing-/10.20.30.40] | 2016-10-22 10:32:02.302000 | 10.20.0.1 | 6045 | 127.0.0.1
REQUEST_RESPONSE message received from /10.20.0.1 [MessagingService-Incoming-/10.20.0.1] | 2016-10-22 10:32:02.322000 | 10.20.30.40 | 47911 | 127.0.0.1
Processing response from /10.20.0.1 [RequestResponseStage-42] | 2016-10-22 10:32:02.322000 | 10.20.30.40 | 48056 | 127.0.0.1
Request complete | 2016-10-22 10:32:02.323239 | 10.20.30.40 | 49239 | 127.0.0.1
While looking at DataStax documentation it seems that source_elapsed column is elapsed time in microseconds before the event occurred on the source node.
There is a big time gap between Sending REQUEST_RESPONSE message to /10.20.30.40 and REQUEST_RESPONSE message received from /10.20.0.1.
Does this indicate a network latency issue?

Ruby on Rails: Rake: rake stats didn't add my field to the correct value?

Before my rake stats modification
+----------------------+-------+-------+---------+---------+-----+-------+
| Name | Lines | LOC | Classes | Methods | M/C | LOC/M |
+----------------------+-------+-------+---------+---------+-----+-------+
| Controllers | 5037 | 3936 | 31 | 292 | 9 | 11 |
| Helpers | 150 | 128 | 0 | 17 | 0 | 5 |
| Models | 1523 | 1166 | 42 | 123 | 2 | 7 |
| Libraries | 633 | 415 | 4 | 65 | 16 | 4 |
| Functional tests | 289 | 228 | 13 | 0 | 0 | 0 |
| Unit tests | 560 | 389 | 30 | 0 | 0 | 0 |
| Model specs | 1085 | 904 | 0 | 3 | 0 | 299 |
| View specs | 88 | 75 | 0 | 0 | 0 | 0 |
| Controller specs | 468 | 388 | 0 | 2 | 0 | 192 |
+----------------------+-------+-------+---------+---------+-----+-------+
| Total | 9833 | 7629 | 120 | 502 | 4 | 13 |
+----------------------+-------+-------+---------+---------+-----+-------+
Code LOC: 5645 Test LOC: 1984 Code to Test Ratio: 1:0.4
now, when I add:
#Factories
::STATS_DIRECTORIES << %w(Factories\ specs test/factories) if File.exist?('test/factories')
::CodeStatistics::TEST_TYPES << "Factory specs" if File.exist?('test/factories')
around line 120, it should increase test LOC, right?
+----------------------+-------+-------+---------+---------+-----+-------+
| Controllers | 5037 | 3936 | 31 | 292 | 9 | 11 |
| Helpers | 150 | 128 | 0 | 17 | 0 | 5 |
| Models | 1523 | 1166 | 42 | 123 | 2 | 7 |
| Libraries | 633 | 415 | 4 | 65 | 16 | 4 |
| Functional tests | 289 | 228 | 13 | 0 | 0 | 0 |
| Unit tests | 560 | 389 | 30 | 0 | 0 | 0 |
| Model specs | 1085 | 904 | 0 | 3 | 0 | 299 |
| View specs | 88 | 75 | 0 | 0 | 0 | 0 |
| Controller specs | 468 | 388 | 0 | 2 | 0 | 192 |
| Factories specs | 144 | 119 | 0 | 0 | 0 | 0 |
+----------------------+-------+-------+---------+---------+-----+-------+
| Total | 9977 | 7748 | 120 | 502 | 4 | 13 |
+----------------------+-------+-------+---------+---------+-----+-------+
Code LOC: 5764 Test LOC: 1984 Code to Test Ratio: 1:0.3
Instead of adding tho 144 lines from factories to test LOC, it adds them to code LOC =\
How do I get the line count to be in Test LOC?
You're adding something called "Factories specs" (plural) to the STATS_DIRECTORIES array, but you call it "Factory specs" (singular) when you add it to TEST_TYPES array -- so when rake:stat hits your test/factories folder, it looks for "Factories specs" in TEST_TYPES, doesn't find it, and assumes it's code, not tests. You need to call it the same thing in both places:
::STATS_DIRECTORIES << %w(Factory\ specs test/factories) if File.exist?('test/factories')
::CodeStatistics::TEST_TYPES << "Factory specs" if File.exist?('test/factories')

Resources