I made two programs. One is a Client program that sends UDP Packet to UDP Server and the other is a Server program that receives UDP Packet from UDP Client.
UDP Server is made by Spring boot.
The problem is that there is difference of performance between run in STS(Spring Tools Suite) and run as jar(maven installed).
When I run this program in STS(Run as->Spring boot app), it gets all packets from client. If Client sent 3,000 packets, it gets 3,000 packet. But when I run as jar(maven install), it gets a little of packets about 200-300. Even It's exactly same code.
I don't know why it occurs. Is there any way to build spring boot app or need some option?
Whole code is too large. so I upload connect, send, bind code.
This is Client program
public UdpClient(String[] args) {
connector = new NioDatagramConnector();
connector.setHandler(this);
ConnectFuture connFuture = connector.connect(new InetSocketAddress(
args[0], 6001));
connFuture.awaitUninterruptibly();
if(args[1] != null) {
totalCount = Integer.parseInt(args[1]);
}
if(args[2] != null) {
count = Integer.parseInt(args[2]);
}
connFuture.addListener(new IoFutureListener<ConnectFuture>() {
public void operationComplete(ConnectFuture future) {
if (future.isConnected()) {
session = future.getSession();
try {
sendData(totalCount, count);
} catch (InterruptedException e) {
e.printStackTrace();
}
} else {
LOGGER.error("Not connected...exiting");
}
}
});
}
private void sendData(int totalCount, int count) throws InterruptedException {
int tempCount = 0;
for (int i = 0; i < totalCount; i++) {
try{
File udpData = new File("udp_data.txt");
BufferedReader br = new BufferedReader(new FileReader(udpData));
String line;
int k = 0;
ArrayList<IoBuffer> bufList = new ArrayList<IoBuffer>();
while((line = br.readLine()) != null) {
String[] str = line.split(" ");
IoBuffer buf = IoBuffer.allocate(str.length);
for (int j = 0; j < str.length; j++) {
int hexData = Integer.parseInt(str[j],16);
byte hexByte = (byte)hexData;
buf.put(hexByte);
}
bufList.add(buf);
buf.flip();
}
long startTime = System.currentTimeMillis();
for (int j = 0; j < count; j++) {
session.write(bufList.get(j));
tempCount++;
if(j % 100 == 0) {
Thread.sleep(1);
}
}
long endTime = System.currentTimeMillis();
System.out.println("oper Time : " + (endTime - startTime));
} catch(Exception e) {
e.printStackTrace();
}
}
System.out.println("Total Send Count : " + tempCount);
}
and this is server program.
#Service
public class ModBusUdpMina {
private static Logger logger = LoggerFactory.getLogger(ModBusUdpMina.class);
#Autowired
private IoHandlerAdapter ioUdpHandlerAdapter;
#Autowired
private ProtocolCodecFactory ambProtocolCodecFactory;
#PostConstruct
public void bind() {
NioDatagramAcceptor acceptor = new NioDatagramAcceptor();
acceptor.getFilterChain().addLast(\"codec\", new ProtocolCodecFilter(ambProtocolCodecFactory));
acceptor.setHandler(ioUdpHandlerAdapter);
acceptor.getSessionConfig().setReuseAddress(true);
acceptor.getSessionConfig().setReceiveBufferSize(1024*512);
acceptor.getSessionConfig().setReadBufferSize(2048);
acceptor.getSessionConfig().setIdleTime(IdleStatus.BOTH_IDLE, 10);
try {
acceptor.bind(new InetSocketAddress(6001));
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
Related
I am trying to write a program using chronicle map. I have written a UDP server which will broadcast a message every 1 sec. A UDP client will receive the message and will store the message in a chronicle map. The programs are as under:
The UDP server program:
public class UDPServer {
public static void main(String[] args) {
DatagramSocket socket = null;
try {
socket = new DatagramSocket();
byte[] buf = new byte[256];
String messg = "Hello UDP Server\n";
String transmittedMsg = null;
int count = 0;
while (true) {
transmittedMsg = count + "";
buf = transmittedMsg.getBytes();
InetAddress address = InetAddress.getByName ("127.0.0.1");
DatagramPacket packet = new DatagramPacket (buf, buf.length, address, 4000);
socket.send(packet);
Thread.sleep(1000);
count++;
}
} catch (SocketTimeoutException ex) {
System.out.println("Timeout error: " + ex.getMessage());
ex.printStackTrace();
} catch (IOException ex) {
System.out.println("Client error: " + ex.getMessage());
ex.printStackTrace();
} catch (InterruptedException ex) {
ex.printStackTrace();
} finally {
socket.close();
}
}
}
The UDP client program:
public class UDPClient {
public static void main(String[] args) {
DatagramSocket socket = null;
DatagramPacket packet = null;
byte[] buf = new byte[256];
ChronicleMap<String, String> cr = null;
try {
socket = new DatagramSocket(4000);
InetAddress address = InetAddress.getByName ("127.0.0.1");
while (true) {
packet = new DatagramPacket(buf, buf.length, address, 5000);
socket.receive(packet);
String received = new String(packet.getData());
System.out.println(received);
cr = ChronicleMapBuilder.of(String.class, String.class)
.name("test-map")
.averageKey("Message")
.averageValue("0")
.entries(1)
.actualChunkSize(100)
.actualSegments(1)
.createPersistedTo(new File("D://test.txt"));
cr.put("Message", received);
}
} catch (Exception e) {
e.printStackTrace();
} finally {
if (cr != null) {
cr.close();
}
}
}
}
Below is the exception I am geting:
java.lang.IllegalArgumentException: ChronicleMap{name=test-map, file=D:\test.txt, identityHashCode=11583403}: Entry is too large: requires 68 chunks, 9 is maximum.
at net.openhft.chronicle.map.impl.CompiledMapQueryContext.allocReturnCode(CompiledMapQueryContext.java:1805)
at net.openhft.chronicle.map.impl.CompiledMapQueryContext.allocReturnCodeGuarded(CompiledMapQueryContext.java:123)
at net.openhft.chronicle.map.impl.CompiledMapQueryContext.alloc(CompiledMapQueryContext.java:3468)
at net.openhft.chronicle.map.impl.CompiledMapQueryContext.initEntryAndKey(CompiledMapQueryContext.java:3502)
at net.openhft.chronicle.map.impl.CompiledMapQueryContext.putEntry(CompiledMapQueryContext.java:3995)
at net.openhft.chronicle.map.impl.CompiledMapQueryContext.doInsert(CompiledMapQueryContext.java:4184)
at net.openhft.chronicle.map.MapEntryOperations.insert(MapEntryOperations.java:153)
at net.openhft.chronicle.map.impl.CompiledMapQueryContext.insert(CompiledMapQueryContext.java:4107)
at net.openhft.chronicle.map.MapMethods.put(MapMethods.java:88)
at net.openhft.chronicle.map.VanillaChronicleMap.put(VanillaChronicleMap.java:724)
at udp.client.UDPClient.main(UDPClient.java:38)
Please help.
Apparently, some entry that you receive is much larger than
averageKey("Message")
averageValue("0")
That you specified.
You also mix together high-level configurations: averageKey(), averageValue(), entries(), and low-level ones: actualChunkSize(), actualSegments(), that is not recommended.
I want to load lots of data into Orient DB with multiple threads.
I'm using OrientDB 2.2.20 and Java 1.8.0_131 to run below sample test client.
But when I run this client with 5 threads and 10000 samples then the client's CPU usage goes over 100% and the process becomes almost dead.
Actually I wanted to use graph APIs to create huge number of vertices and edges between them.
But I read in some post that for massive inserts use document API and set the in & out pointers using doc APIs. Hence tried this program.
Could someone point what is wrong in the code?
public OrientDBTestClient(){
db = new ODatabaseDocumentTx(url).open(userName, password);
}
public static void main(String[] args) throws Exception{
int threadCnt = Integer.parseInt(args[0]);
OrientDBTestClient client = new OrientDBTestClient();
try {
db.declareIntent(new OIntentMassiveInsert());
Thread[] threads = new Thread[threadCnt];
for (int i = 0; i < threadCnt; i++) {
Thread loadStatsThread = new Thread(client.new LoadTask(Integer.parseInt(args[1])));
loadStatsThread.setName("LoadTask" + (i + 1));
loadStatsThread.start();
threads[i] = loadStatsThread;
}
}
catch(Exception e){
e.printStackTrace();
}
}
private class LoadTask implements Runnable{
public int count = 0;
public LoadTask(int count){
this.count = count;
}
public void run(){
long start = System.currentTimeMillis();
try{
db.activateOnCurrentThread();
for(int i = 0; i < count; ++ i){
storeStatsInDB(i +"");
}
}
catch(Exception e){
log.println("Error in LoadTask : " + e.getMessage());
e.printStackTrace();
}
finally {
db.commit();
System.out.println(Thread.currentThread().getName() + " loaded: " + count + " services in: " + (System.currentTimeMillis() - start) + "ms");
}
}
}
public void storeStatsInDB(String id) throws Exception{
try{
long start = System.currentTimeMillis();
ODocument doc = db.newInstance();
doc.reset();
doc.setClassName("ServiceStatistics");
doc.field("serviceID", id);
doc.field("name", "Service=" + id);
doc.save();
}
catch(Exception e){
log.println("Exception :" + e.getMessage());
e.printStackTrace();
}
}
db instances aren't sharable between threads.
You have two choices:
create an instance for each thread
use the pool (my first choice): http://orientdb.com/docs/last/Java-Multi-Threading.html#working-with-databases
The following example is extracted from internal tests:
pool = new OPartitionedDatabasePool("remote:localshot/test", "admin", "admin");
Runnable acquirer = () -> {
ODatabaseDocumentTx db = pool.acquire();
try {
List<ODocument> res = db.query(new OSQLSynchQuery<>("SELECT * FROM OUser"));
} finally {
db.close();
}
};
//spawn 20 threads
List<CompletableFuture<Void>> futures = IntStream.range(0, 19).boxed().map(i -> CompletableFuture.runAsync(acquirer))
.collect(Collectors.toList());
futures.forEach(cf -> cf.join());`
I need a specialized queue-like data structure. It can be used by multiple consumers, but each item in queue must be removed from queue after k consumers read it.
Is there any production ready implementation? Or Should I implement a queue with read-counter in each item, and handle item removal myself?
Thanks in advance.
I think this is what you are looking for. Derived from the source code for BlockingQueue. Caveat emptor, not tested.
I tried to find a way to wrap Queue, but Queue doesn't expose its concurrency members, so you can't get the right semantics.
public class CountingQueue<E> {
private class Entry {
Entry(int count, E element) {
this.count = count;
this.element = element;
}
int count;
E element;
}
public CountingQueue(int capacity) {
if (capacity <= 0) {
throw new IllegalArgumentException();
}
this.items = new Object[capacity];
this.lock = new ReentrantLock(false);
this.condition = this.lock.newCondition();
}
private final ReentrantLock lock;
private final Condition condition;
private final Object[] items;
private int takeIndex;
private int putIndex;
private int count;
final int inc(int i) {
return (++i == items.length) ? 0 : i;
}
final int dec(int i) {
return ((i == 0) ? items.length : i) - 1;
}
private static void checkNotNull(Object v) {
if (v == null)
throw new NullPointerException();
}
/**
* Inserts element at current put position, advances, and signals.
* Call only when holding lock.
*/
private void insert(int count, E x) {
items[putIndex] = new Entry(count, x);
putIndex = inc(putIndex);
if (count++ == 0) {
// empty to non-empty
condition.signal();
}
}
private E extract() {
Entry entry = (Entry)items[takeIndex];
if (--entry.count <= 0) {
items[takeIndex] = null;
takeIndex = inc(takeIndex);
if (count-- == items.length) {
// full to not-full
condition.signal();
}
}
return entry.element;
}
private boolean waitNotEmpty(long timeout, TimeUnit unit) throws InterruptedException {
long nanos = unit.toNanos(timeout);
while (count == 0) {
if (nanos <= 0) {
return false;
}
nanos = this.condition.awaitNanos(nanos);
}
return true;
}
private boolean waitNotFull(long timeout, TimeUnit unit) throws InterruptedException {
long nanos = unit.toNanos(timeout);
while (count == items.length) {
if (nanos <= 0)
return false;
nanos = condition.awaitNanos(nanos);
}
return true;
}
public boolean put(int count, E e) {
checkNotNull(e);
final ReentrantLock localLock = this.lock;
localLock.lock();
try {
if (count == items.length)
return false;
else {
insert(count, e);
return true;
}
} finally {
localLock.unlock();
}
}
public boolean put(int count, E e, long timeout, TimeUnit unit)
throws InterruptedException {
checkNotNull(e);
final ReentrantLock localLock = this.lock;
localLock.lockInterruptibly();
try {
if (!waitNotFull(timeout, unit)) {
return false;
}
insert(count, e);
return true;
} finally {
localLock.unlock();
}
}
public E get() {
final ReentrantLock localLock = this.lock;
localLock.lock();
try {
return (count == 0) ? null : extract();
} finally {
localLock.unlock();
}
}
public E get(long timeout, TimeUnit unit) throws InterruptedException {
final ReentrantLock localLock = this.lock;
localLock.lockInterruptibly();
try {
if (waitNotEmpty(timeout, unit)) {
return extract();
} else {
return null;
}
} finally {
localLock.unlock();
}
}
public int size() {
final ReentrantLock localLock = this.lock;
localLock.lock();
try {
return count;
} finally {
localLock.unlock();
}
}
public boolean isEmpty() {
final ReentrantLock localLock = this.lock;
localLock.lock();
try {
return count == 0;
} finally {
localLock.unlock();
}
}
public int remainingCapacity() {
final ReentrantLock lock= this.lock;
lock.lock();
try {
return items.length - count;
} finally {
lock.unlock();
}
}
public boolean isFull() {
final ReentrantLock localLock = this.lock;
localLock.lock();
try {
return items.length - count == 0;
} finally {
localLock.unlock();
}
}
public void clear() {
final ReentrantLock localLock = this.lock;
localLock.lock();
try {
for (int i = takeIndex, k = count; k > 0; i = inc(i), k--)
items[i] = null;
count = 0;
putIndex = 0;
takeIndex = 0;
condition.signalAll();
} finally {
localLock.unlock();
}
}
}
A memory efficient way that retains the info you need:
Each queue entry becomes a
Set<ConsumerID>
so that you ensure the k times are for k distinct consumers: your app logic checks if the
set.size()==k
and removes it from queue in that case.
In terms of storage: you will have tradeoffs of which Set implementation based on
size and type of the ConsumerID
speed of retrieval requirement
E.g if k is very small and your queue retrieval logic has access to a
Map<ID,ConsumerId>
then you could have simply an Int or even a Short or Byte depending on # distinct ConsumerID's and possibly store in an Array . This is slower than accessing a set since it would be traversed linearly - but for small K that may be reasonable.
I want to create a Custom Load function for Pig UDF, I have created a SimpleTextLoader using the link
https://pig.apache.org/docs/r0.11.0/udf.html , I have successfully generate the jar file for this code, register in pig and run a Pig Script.I am getting the empty output. I don't know how to solve this issue, any help would be appreciated.
Below is my Java code
public class SimpleTextLoader extends LoadFunc{
protected RecordReader in = null;
private byte fieldDel = '\t';
private ArrayList<Object> mProtoTuple = null;
private TupleFactory mTupleFactory = TupleFactory.getInstance();
private static final int BUFFER_SIZE = 1024;
public SimpleTextLoader() {
}
public SimpleTextLoader(String delimiter)
{
this();
if (delimiter.length() == 1) {
this.fieldDel = (byte)delimiter.charAt(0);
} else if (delimiter.length() > 1 && delimiter.charAt(0) == '\\') {
switch (delimiter.charAt(1)) {
case 't':
this.fieldDel = (byte)'\t';
break;
case 'x':
fieldDel =
Integer.valueOf(delimiter.substring(2), 16).byteValue();
break;
case 'u':
this.fieldDel =
Integer.valueOf(delimiter.substring(2)).byteValue();
break;
default:
throw new RuntimeException("Unknown delimiter " + delimiter);
}
} else {
throw new RuntimeException("PigStorage delimeter must be a single character");
}
}
private void readField(byte[] buf, int start, int end) {
if (mProtoTuple == null) {
mProtoTuple = new ArrayList<Object>();
}
if (start == end) {
// NULL value
mProtoTuple.add(null);
} else {
mProtoTuple.add(new DataByteArray(buf, start, end));
}
} #Override
public Tuple getNext() throws IOException {
try {
boolean notDone = in.nextKeyValue();
if (notDone) {
return null;
}
Text value = (Text) in.getCurrentValue();
System.out.println("printing value" +value);
byte[] buf = value.getBytes();
int len = value.getLength();
int start = 0;
for (int i = 0; i < len; i++) {
if (buf[i] == fieldDel) {
readField(buf, start, i);
start = i + 1;
}
}
// pick up the last field
readField(buf, start, len);
Tuple t = mTupleFactory.newTupleNoCopy(mProtoTuple);
mProtoTuple = null;
System.out.println(t);
return t;
} catch (InterruptedException e) {
int errCode = 6018;
String errMsg = "Error while reading input";
throw new ExecException(errMsg, errCode,
PigException.REMOTE_ENVIRONMENT, e);
}
}
#Override
public void setLocation(String string, Job job) throws IOException {
FileInputFormat.setInputPaths(job,string);
}
#Override
public InputFormat getInputFormat() throws IOException {
return new TextInputFormat();
}
#Override
public void prepareToRead(RecordReader reader, PigSplit ps) throws IOException {
in=reader;
}
}
Below is my Pig Script
REGISTER /home/hadoop/netbeans/sampleloader/dist/sampleloader.jar
a= load '/input.txt' using sampleloader.SimpleTextLoader();
store a into 'output';
You are using sampleloader.SimpleTextLoader() that doesn't do anything as it is just an empty constructor.
Instead use sampleloader.SimpleTextLoader(String delimiter) which is performing the actual operation of split.
I'm writing a Metro app in Windows 8 Consumer Preview.
However, I'm unable to use the TcpClient in .NET 4.5, there doesn't seem to be a place to add the assembly reference.
http://msdn.microsoft.com/en-us/library/1612451t(v=vs.110).aspx
TcpClient is not supported on the metro side. You can use StreamSocket class instead. Here is a sample on how to use it to create a TCP Socket, make a connection, send and receive data. The samples are in JS and C++ but the same class will work for C#.
Ultimately, we should probably use the new Metro NET stuff. However, if porting a lot of code and depending on how much of the TcpClient members you are using, it might not be so bad to just create a limited implementation around the Metro objects. I wanted to do a quick port of a bunch of code over to Metro in a hurry (just to try out some stuff), so I slapped together something that seemed to work, but most certainly not ideal. (You end up doing some sync-ing of async methods which is commonly frowned upon.)
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using Windows.Networking;
using Windows.Networking.Sockets;
using Windows.Storage.Streams;
namespace MetroNetHelper
{
public class IPAddress // assumes IPv4 currently
{
public string IP_String;
public IPAddress() { }
public IPAddress(string IP) { IP_String = IP; }
public static IPAddress Broadcast { get { return new IPAddress("255.255.255.255"); } }
public static IPAddress Parse(string IP) { return new IPAddress(IP); }
public static bool TryParse(string V, out IPAddress Addr)
{
try
{
Addr = IPAddress.Parse(V);
return true;
}
catch { Addr = null; return false; }
}
public HostName GetHostNameObject() { return new HostName(IP_String); }
public byte[] GetAddressBytes()
{
string[] Fields = IP_String.Split('.');
byte[] temp = new byte[4];
for (int i = 0; i < temp.Length; i++) temp[i] = byte.Parse(Fields[i]);
return temp;
}
}
public class IPEndPoint
{
public IPAddress Address;
public int Port;
public IPEndPoint() { }
public IPEndPoint(IPAddress Addr, int PortNum)
{
Address = Addr; Port = PortNum;
}
}
public class NetworkStream
{
private DataReader Reader;
private DataWriter Writer;
public void Set(StreamSocket HostClient)
{
Reader = new DataReader(HostClient.InputStream);
Reader.InputStreamOptions = InputStreamOptions.Partial;
Writer = new DataWriter(HostClient.OutputStream);
}
public int Write(byte[] Buffer, int Offset, int Len)
{
if (Offset != 0 || Len != Buffer.Length) throw new ArgumentException("Can only write whole byte array");
Writer.WriteBytes(Buffer);
Task Tk = Writer.StoreAsync().AsTask();
Tk.Wait();
return Buffer.Length;
}
public int Read(byte[] Buffer, int Offset, int Len)
{
if (Offset != 0 || Len != Buffer.Length) throw new ArgumentException("Can only read whole byte array");
Task<uint> Tk = Reader.LoadAsync((uint)Len).AsTask<uint>();
Tk.Wait();
uint Count = Tk.Result;
for (int i=0;i<Count;i++)
{
Buffer[i] = Reader.ReadByte();
}
return (int)Count;
}
public bool DataAvailable
{
get
{
return true; // Read() will still work if no data; could we do read ahead 1 byte to determine?
}
}
}
public class TcpClient
{
private StreamSocket sock;
public void Connect(IPEndPoint EndPt)
{
try
{
sock = new StreamSocket();
HostName Hst = EndPt.Address.GetHostNameObject();
Task Tsk = sock.ConnectAsync(Hst, EndPt.Port.ToString()).AsTask();
Tsk.Wait();
}
catch (Exception ex) { MetroHelpers.UnpeelAggregate(ex); }
}
public void Close()
{
sock.Dispose();
sock = null;
}
public NetworkStream GetStream()
{
var N = new NetworkStream();
N.Set(sock);
return N;
}
}
public static class MetroHelpers
{
public static void UnpeelAggregate(Exception Ex)
{
AggregateException Ag_Ex = Ex as AggregateException;
if (Ag_Ex == null) throw Ex;
if (Ag_Ex.InnerExceptions.Count > 0)
{
if (Ag_Ex.InnerExceptions.Count == 1) throw Ag_Ex.InnerExceptions[0];
StringBuilder Str = new StringBuilder();
foreach (Exception X in Ag_Ex.InnerExceptions)
{
Str.AppendLine(X.Message);
}
throw new Exception(Str.ToString(), Ag_Ex);
}
else throw Ag_Ex;
}
}
} // END NAMESPACE
This was just something I did quick and dirty in a morning. If it helps anyone with ideas, great. Again, we are most likely better off developing the way Microsoft wants us for Metro apps. It just gets frustrating when they keep changing the ruddy framework. (If Xamarin can keep consistent frameworks on iOS/Android/etc, why can't MS on their own OS's!)