Arduino Ethernet client.available() not working - amazon-ec2

I eventually want to send a GET request to my Amazon EC2 server, but for now I just need to get the GET request to work in some fashion. To do that, I am trying to send a GET request to google as a test.
I am using a Arduino Uno with an Ethernet shield connected to the internet via DHCP.
My issue is that although client.connect() seems to work, client.available doesn't work either with google or with my EC2 server. Although, if my issue lies elsewhere, please tell me.
I am able to ping google and do a Telnet GET request simulation using:
-telnet www.google.com 80
-GET /search?q=arduino HTTP/1.0
Arduino Code:
#include <SPI.h>
#include <Ethernet.h>
// Enter a MAC address and IP address for your controller below.
// The IP address will be dependent on your local network:
byte mac[] = { 0x90, 0xA2, 0xDA, 0x0F, 0xD2, 0xAF };
//byte mac[] = {0xf0, 0x1f, 0xaf, 0x33, 0x62, 0x2f };
IPAddress ip(192,168,1,11);
// initialize the library instance:
EthernetClient client;
//char server[] = "ec2-54-69-168-77.us-west-2.compute.amazonaws.com";
char server[] = "www.google.com";
double dummyValue = 7.5;
void setup()
{
Serial.begin(9600);
// attempt a DHCP connection:
Serial.println("Attempting to get an IP address using DHCP:");
if (!Ethernet.begin(mac)) {
// if DHCP fails, start with a hard-coded address:
Serial.println("failed to get an IP address using DHCP, trying manually");
Ethernet.begin(mac, ip);
}
Serial.print("My address:");
Serial.println(Ethernet.localIP());
}
void loop()
{
// connect to the server
for(int i = 0;i <100 ; i++) {
if (client.connect(server, 80)) {
// print to serial monitor
Serial.println("connected...");
Serial.println("ARDUINO: forming HTTP request message");
// send data the server through GET request
//client.print("GET /~sclaybon3/firstdatatest.php?reading=3 HTTP/1.0\r\n");
client.print("GET /search?q=arduino HTTP/1.0\r\n");
Serial.println("Get request");
//client.print(dummyValue);
//client.print(" HTTP/1.1");
//client.print("Host: ec2-54-69-168-77.us-west-2.compute.amazonaws.com\r\n");
client.print("Host: www.google.com\r\n");
//Serial.println("Host:www.google.com");
//client.print("Connection: close\r\n\r\n");
client.print("\r\n");
Serial.println("ARDUINO: HTTP message sent");
// give server some time to receive and store data
// before asking for response from it
delay(1000);
// get the response from the page and print it to serial port
// to ascertain that data was received properly
if(client.available())
{
Serial.println("ARDUINO: HTTP message received");
Serial.println("ARDUINO: printing received headers and script response...\n");
while(client.available())
{
char c = client.read();
Serial.print(c);
}
}
else
{
Serial.println("ARDUINO: no response received / no response received in time");
}
client.stop();
}
}
// do nothing forever after:
while(true);
}
Arduino output:
Attempting to get an IP address using DHCP:
My address:192.168.1.11
connected...
ARDUINO: forming HTTP request message
Get request
ARDUINO: HTTP message sent
ARDUINO: no response received / no response received in time

Related

esp32 reconnection after power surge

I have an esp32 as STA in my network, collecting sensor data, that publishes through my wifi AP to my home network. Everything was working fine until we had a general power surge in the neighborhood. When power came back the esp32 was out of the network. I had to recycle its power to let it re-establish connection. Is there an explanation for this? Most probably it came up much faster than the wifi or the mqtt broker did. Could this be the issue? If this is the issue, is there a way to postpone booting of the esp32 just after wifi network and mqtt is available?
Edit: I return to my original post to add some code that I used.
void setup() {
Serial.begin(115200);
delay(10);
WiFi.mode(WIFI_STA);
// Configures static IP address
if (!WiFi.config(local_IP, gateway, subnet, primaryDNS, secondaryDNS)) {
Serial.println("STA Failed to configure");
}
// Connect to Wi-Fi network with SSID and password
Serial.print("Connecting to ");
Serial.println(ssid);
WiFi.begin(ssid, password);
while (WiFi.status() != WL_CONNECTED) {
delay(500);
Serial.print(".");
}
// Print local IP address
Serial.println("");
Serial.println("WiFi connected.");
Serial.println("IP address: ");
...
client.setKeepAlive( MQTT_KEEPALIVE );
client.setServer(mqtt_server, 1883);
client.connect(aHostname);
if (!client.connected()) {
Serial.print("mqtt status in setup: ");
Serial.println(client.state());
reconnect();
} else {
client.setCallback(callback);
Serial.print("mqtt status in setup: ");
Serial.println(client.state());
client.setCallback(callback);
client.setKeepAlive( MQTT_KEEPALIVE );
}
Also if mqtt broker is lost there is a reconnection attempt in loop:
void loop(){
if (!client.connected()) {
reconnect();
}
..
}
If it's a question of ESP32 booting faster than the WiFi access point, then this has a very simple solution. Retry the WiFi connection in ESP32 until it succeeds. Then do the same with MQTT. If you think about it, the device is useless without WiFi&MQTT connections so feel free to just keep retrying forever (or until it's successful).
In your case the MQTT broker's retry in the loop() should be an obvious example of to add a similar retry for the WiFi. Since I don't use Arduino and I don't currently have the time to read the WiFi module's documentation, I'll leave the implementation to you.

ZeroMQ server client start sequence

Facing a issue if the client is started before server
Specs : ubuntu 16.04 with c++11,libzmq : 4.2.3
problem : sample codes
server.cpp
int main()
{
zmq::context_t context(1);
zmq::socket_t requester(context,ZMQ_ROUTER);
.
//code to get address
.
requester.bind(address);
while(true)
{
zmq::message_t message;
requester.recv(&message);
.
//remaining code
.
}
return 0;
}
client.cpp
int main()
{
zmq::context_t context(1);
zmq::socket_t requester(context,ZMQ_DEALER);
.
//code to get address
.
requester.connect(address);
zmq::message_t message;
.
//populate the message to send
.
requester.send(message);
return 0;
}
I know that in zmq i can start client even if the server is not running,but my client application has to include a safety check which requires the server to be started.Is there any way i can achieve by making connect fail or someother workaround.Timeout options doesnt work for me
Make the send non blocking, if there there is no server(s) available the send will fail and set an errno
http://api.zeromq.org/4-2:zmq-send
ZMQ_DONTWAIT
For socket types (DEALER, PUSH) that block when there are no available peers
(or all peers have full high-water mark), specifies that the operation should
be performed in non-blocking mode. If the message cannot be queued on the
socket, the zmq_send() function shall fail with errno set to EAGAIN.

boost asio - exclusive binding to the network port

I develop server app using boost asio. App works great. What doesn't work, is the the exclusive binding to the network port.
I launch one instance of the app - it starts listening to incoming connections.
I launch one more instance - it also starts listening to incoming connections on the same port. Handler that passed to async_accept do not invoked with error as expected.
Usually I just try to acquire the port. If operation fails - port is in use. With Asio this approach does not work. How to check availability of the port?
void TcpServerFactory::acceptConnectionsOnPort(int serverPort,
boost::shared_ptr<TcpConfigServerReceiver> tcpConfig,
boost::function<void(boost::shared_ptr<TcpServer>)> onSuccessfullyConnectedHandler)
{
// todo check is port not busy
FORMATTED_LOG(this->_log, info) << "Start to accept connections on port " << serverPort;
auto endpoint = boost::asio::ip::tcp::endpoint(boost::asio::ip::tcp::v4(), serverPort);
boost::shared_ptr<boost::asio::ip::tcp::acceptor> tcpAcceptor(new boost::asio::ip::tcp::acceptor(this->_ioService, endpoint));
this->acceptConnections(tcpAcceptor, tcpConfig, onSuccessfullyConnectedHandler);
}
void TcpServerFactory::acceptConnections(boost::shared_ptr<boost::asio::ip::tcp::acceptor> tcpAcceptor,
boost::shared_ptr<TcpConfigServerReceiver> tcpConfig,
boost::function<void(boost::shared_ptr<TcpServer>)> onSuccessfullyConnectedHandler)
{
boost::shared_ptr<boost::asio::ip::tcp::socket> tcpSocket(new boost::asio::ip::tcp::socket(this->_ioService));
boost::function<void(const boost::system::error_code &)> onAcceptOperationCompletedHandler =
boost::bind(&TcpServerFactory::onAcceptOperationCompleted, this->downcasted_shared_from_this<TcpServerFactory>(),
_1, tcpAcceptor, tcpSocket, tcpConfig, onSuccessfullyConnectedHandler);
tcpAcceptor.get()->async_accept(*tcpSocket, onAcceptOperationCompletedHandler);
}
void TcpServerFactory::onAcceptOperationCompleted(const boost::system::error_code & err,
boost::shared_ptr<boost::asio::ip::tcp::acceptor> tcpAcceptor,
boost::shared_ptr<boost::asio::ip::tcp::socket> tcpSocket,
boost::shared_ptr<TcpConfigServerReceiver> tcpConfig,
boost::function<void(boost::shared_ptr<TcpServer>)> onSuccessfullyConnectedHandler)
{
if (err)
{
FORMATTED_LOG(this->_log, info) << "Failed to accept connections on port " << tcpAcceptor->local_endpoint().port() << "due to error " << BOOST_ERROR_TO_STREAM(err);
return;
}
this->acceptConnections(tcpAcceptor, tcpConfig, onSuccessfullyConnectedHandler);
this->onConnectionEstablished(tcpSocket, tcpConfig, onSuccessfullyConnectedHandler);
}
Update
I tried to replace constructor of acceptor on series of commands. I expected that on tcpAcceptor->bind() exception will be raised, but that didn't happened.
// boost::shared_ptr<boost::asio::ip::tcp::acceptor> tcpAcceptor(new boost::asio::ip::tcp::acceptor(this->_ioService, endpoint));
boost::shared_ptr<boost::asio::ip::tcp::acceptor> tcpAcceptor(new boost::asio::ip::tcp::acceptor(this->_ioService));
tcpAcceptor->open(endpoint.protocol());
tcpAcceptor->set_option(boost::asio::socket_base::reuse_address(true));
tcpAcceptor->bind(endpoint);
boost::system::error_code err;
tcpAcceptor->listen(boost::asio::socket_base::max_connections, err);
reuse_address is not supposed to do that. Its meaning to avoid "wait" interval after port was freed.

What can cause data written to a TCP socket on windows to be altered before transmiission?

Something is altering data written to TCP sockets on my Windows (Windows 7) machine - specifically, when the bytes follow a specific HTTP POST pattern, the pattern is repeated when the bytes are read from the corresponding listener socket side of the connection.
The following bytes are written to the client socket (note: each line ends with a carriage-return and newline and the two nonblank lines are followed by two blank lines):
POST / HTTP/1.1
Transfer-Encoding: chunked
what is read from the listener socket is:
POST / HTTP/1.1
Transfer-Encoding: chunked
POST / HTTP/1.1
Transfer-Encoding: chunked
I've tested this on the loopback (127.0.0.1) address on my machine, but I've also seen the modified bytes when the listener socket was on another machine, so it appears the bytes are modified on the client side. I've reproduced the problem using both netcat and a java program (see below) on my machine, so the issue appears to be in the TCP stack. I've only been able to cause it with a specific set of HTTP headers, so it appears that something is doing deep packet inspection on my TCP communication and altering it. If I alter the input bytes slightly (e.g. so it is not a valid HTTP request by, for instance, changing "POST" to "QOST", it works fine).
Below is a java program I've written that demonstrates this and its output:
import java.io.InputStream;
import java.io.OutputStream;
import java.net.InetAddress;
import java.net.ServerSocket;
import java.net.Socket;
import java.nio.charset.Charset;
public final class Main {
private static final String PAYLOAD
= "POST / HTTP/1.1\r\n"
+ "Transfer-Encoding: chunked\r\n"
+ "\r\n"
+ "\r\n"
;
private static final int PORT = 8080;
public static final void main(final String[] args) throws Exception {
final Thread serverThread = new Thread(new Server());
serverThread.start();
final byte[] payloadBytes = PAYLOAD.getBytes(Charset.forName("UTF-8"));
int i = 0;
try (final Socket socket = new Socket(InetAddress.getLoopbackAddress(), PORT)) {
socket.setTcpNoDelay(true);
final OutputStream os = socket.getOutputStream();
for (final byte byteValue : payloadBytes) {
os.write(byteValue);
os.flush();
i++;
}
}
serverThread.join();
System.out.println("bytes written: " + i);
}
private static final class Server implements Runnable {
#Override
public void run() {
try (final ServerSocket serverSocket = new ServerSocket(PORT)) {
// while (true) {
final Socket socket = serverSocket.accept();
socket.setTcpNoDelay(true);
try (final InputStream is = socket.getInputStream()) {
int i = 0;
int byteValue;
while ((byteValue = is.read()) >= 0) {
System.out.print((char) byteValue);
System.out.flush();
i++;
}
System.out.println("----------------");
System.out.println("bytes read: " + i);
}
// }
} catch (final Exception e) {
throw new RuntimeException(e);
}
}
}
}
output:
POST / HTTP/1.1
Transfer-Encoding: chunked
POST / HTTP/1.1
Transfer-Encoding: chunked
----------------
bytes read: 96
bytes written: 49
Below is the same test using netcat (nc.exe from cygwin) on windows (note: the file test_payload.blob contains the bytes as described above and derived from the PAYLOAD constant in the java program):
start the nc listener:
nc -l 8080 > nc_capture; more nc_capture
run the nc client (in another shell from the listener):
nc -v 127.0.0.1 8080 < test_payload.blob
the output written to nc_capture:
POST / HTTP/1.1
Transfer-Encoding: chunked
POST / HTTP/1.1
Transfer-Encoding: chunked
My first thought was a buggy firewall, so I disabled it, but it still happens. I also tried resetting my winsock and tcp/ip, and it still happens. I tried disabling all of my network adapters (the above tests work on the loopback IP address, so they are not needed), and it still happens. At this point, I am pretty much out of ideas, and I don't even know how I would go about trying to debug this at a lower level. Has anyone ever seen something like this before? Is there some low level diagnostic tool on windows that I can use to see what might have its hooks into my TCP stack?

WinDivert redirect to proxy

I'm trying to redirect all tcp packets to my local proxy to modify html content(adblocker like). I wanted to use WinDivert but it doesn't seem to work.
Im starting the driver like this:
handle = WinDivertOpen("outbound", WINDIVERT_LAYER_NETWORK, 0, 0);
then when capturing and modifying packets:
if (ip_header != NULL && tcp_header != NULL) {
//redirect to proxy
if (ntohs(tcp_header->DstPort) == 80)
{
UINT32 dst_addr = ip_header->DstAddr;
ip_header->DstAddr = ip_header->SrcAddr;
ip_header->SrcAddr = dst_addr;
tcp_header->DstPort = htons(PROXY);
addr.Direction = DIVERT_DIRECTION_INBOUND;
}
else if (ntohs(tcphdr->SrcPort) == PROXY)
{
// proxy to browser
uint32_t dst_addr = iphdr->DstAddr;
iphdr->DstAddr = iphdr->SrcAddr;
iphdr->SrcAddr = dst_addr;
tcphdr->SrcPort = htons(80);
addr.Direction = DIVERT_DIRECTION_INBOUND;
}
WinDivertHelperCalcChecksums(packet, packet_len, 0);
if (!WinDivertSend(handle, packet, packet_len , &addr, &send_len))
{
qWarning() << "warning: failed to reinject packet" << GetLastError() << send_len;
}
But on the proxy side i cant see any incoming traffic and pages are not loading in the web browser.
The code snippet will transform outbound (port HTTP) packets into inbound (port PROXY) packets. This part is OK. But there is currently nothing that handles the reverse path.
For example, consider the TCP handshake. The code snippet will redirect a (DstPort=80) SYN packet to the proxy server, which will reply with a (SrcPort=PROXY) SYN/ACK. However, this SYN/ACK is not handled by the above code and will be lost. You need to add code to redirect outbound (SrcPort=PROXY) packets to inbound (SrcPort=80) packets.
See the TorWall example: https://github.com/basil00/TorWall/blob/082b7ff0fa86abfa2df480ece8cb31e25a29c1bc/tor_wall.c
Edit: Also see the streamdump WinDivert sample: https://github.com/basil00/Divert/blob/master/examples/streamdump/streamdump.c

Resources