I have written a small client-server program, where server is running on an Android phone and client program is running on my PC. It works absolutely fine when I send small files from server to client (like small text files, doc files, etc). But when I try to send larger files (like mp3 of size 2 to 3 MB), the client program throws error due to array size overflow.
Here is my client side program running on my PC:
package fileClient;
import java.io.BufferedOutputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.net.InetAddress;
import java.net.Socket;
public class myFileClient {
/**
* #param args
*/
public static void main(String[] args) throws IOException {
// TODO Auto-generated method stub
int filesize=1022386;
int bytesRead;
int currentTot = 0;
String servAdd="10.142.100.161";
Socket socket = null;
InetAddress serverIP=InetAddress.getByName(servAdd);
byte [] bytearray = new byte [filesize];
socket=new Socket(serverIP, 4444);
InputStream is = socket.getInputStream();
FileOutputStream fos = new FileOutputStream("//home//evinish//MANA.mp3");
BufferedOutputStream bos = new BufferedOutputStream(fos);
bytesRead = is.read(bytearray,0,bytearray.length);
currentTot = bytesRead;
do {
bytesRead =
is.read(bytearray, currentTot, (bytearray.length-currentTot));
if(bytesRead >= 0) currentTot += bytesRead;
} while(bytesRead > -1);
bos.write(bytearray, 0 , currentTot);
bos.flush();
bos.close();
socket.close();
}
}
The console output shows following error:
/usr/lib/jvm/java-6-openjdk-amd64/bin/java: line 3: [: too many arguments
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException
at java.lang.System.arraycopy(Native Method)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:128)
at fileClient.myFileClient.main(myFileClient.java:38)
I tried to increase filesize variable to 100MB or so, but that didn't work either. Can anybody tell me a work around for this?
Related
I am getting the error "ERROR 2078: Caught error from UDF: com.Hadoop.pig.SplitRec [Caught exception processing input row [1]]". I am sure that the input string is going out of bound, but I am not sure which record(record number) is causing the problem.
I am trying to create log for displaying the record which is causing the problem, but I am not sure about debugging to print/log the error record.
The input looks like:
**PXW01YIN 12000099PGEN PXW01YINFFFFFFFF PXW01YINIMFGUIPY04301Y301 JFK 00888JFK 008880001 PIMF 0000N/ACTRC5/TXN08/SCR301\/SEQ/TEX021\#
PXW01PIN 12000099PGEN PXW01PINFFFFFFFF PXW01PINIMFGUIAV04301P301 PER 03615PER 036150001 PIMF 0000N/ACTRCK/TXN08/SCR301\/SEQ/TEX021\#**
The above lines are two records and I have tested them(using LIMIT), and they are not causing problem. I have more than 150kb of input data.
The script that I am using:
SPLT_REC1 = load '/user/hduser/output/realdata/pig_out6/part-m-00000' as (tran_array:chararray);
register /home/cloudera/workspace/SplitRec.jar;
define SplitRec com.Hadoop.pig.SplitRec();
SPLT_REC2 = foreach SPLT_REC1 generate SplitRec(tran_array);
store SPLT_REC2 into '/user/hduser/output/realdata/pig_out7';
package com.Hadoop.pig;
import java.io.IOException;
import org.apache.pig.EvalFunc;
import org.apache.pig.data.Tuple;
import org.apache.pig.impl.util.WrappedIOException;
#SuppressWarnings("deprecation")
public class SplitRec extends EvalFunc<String> {
public String exec(Tuple input) throws IOException {
if (input == null || input.size() == 0)
return null;
try {
String Str1 = (String)input.get(0);
String delim1 = "PIMF+";
String[] tokens1 = Str1.split(delim1);
String part3 = tokens1[0];
String part4 = tokens1[1];
int len1 = part4.length();
String part5 = part4.substring(8,len1);
String conCat1 = part3+":"+part5;
return conCat1;
}
catch(Exception e) {
throw WrappedIOException.wrap("Caught exception processing input row ", e);
}
}
I'm using a custom AVIOContext to bridge FFMpeg with java IO. The function avformat_open_input() never returns. I have searched the web for similar problems, all of which were caused by faulty network or wrong server configurations. However, I'm not using network at all, as you can see in the following little program:
package com.example;
import org.bytedeco.javacpp.*;
import java.io.File;
import java.io.IOException;
import java.io.RandomAccessFile;
import static org.bytedeco.javacpp.avcodec.*;
import static org.bytedeco.javacpp.avformat.*;
import static org.bytedeco.javacpp.avutil.*;
import static org.bytedeco.javacpp.avdevice.*;
import static org.bytedeco.javacpp.avformat.AVFormatContext.*;
public class Test {
public static void main(String[] args) throws Exception {
File dir = new File(System.getProperty("user.home"), "Desktop");
File file = new File(dir, "sample.3gp");
final RandomAccessFile raf = new RandomAccessFile(file, "r");
Loader.load(avcodec.class);
Loader.load(avformat.class);
Loader.load(avutil.class);
Loader.load(avdevice.class);
Loader.load(swscale.class);
Loader.load(swresample.class);
avcodec_register_all();
av_register_all();
avformat_network_init();
avdevice_register_all();
Read_packet_Pointer_BytePointer_int reader = new Read_packet_Pointer_BytePointer_int() {
#Override
public int call(Pointer pointer, BytePointer buf, int bufSize) {
try {
byte[] data = new byte[bufSize]; // this is inefficient, just use as a quick example
int read = raf.read(data);
if (read <= 0) {
System.out.println("EOF found.");
return AVERROR_EOF;
}
System.out.println("Successfully read " + read + " bytes of data.");
buf.position(0);
buf.put(data, 0, read);
return read;
} catch (Exception ex) {
ex.printStackTrace();
return -1;
}
}
};
Seek_Pointer_long_int seeker = new Seek_Pointer_long_int() {
#Override
public long call(Pointer pointer, long offset, int whence) {
try {
raf.seek(offset);
System.out.println("Successfully seeked to position " + offset + ".");
return offset;
} catch (IOException ex) {
return -1;
}
}
};
int inputBufferSize = 32768;
BytePointer inputBuffer = new BytePointer(av_malloc(inputBufferSize));
AVIOContext ioContext = avio_alloc_context(inputBuffer, inputBufferSize, 1, null, reader, null, seeker);
AVInputFormat format = av_find_input_format("3gp");
AVFormatContext formatContext = avformat_alloc_context();
formatContext.iformat(format);
formatContext.flags(formatContext.flags() | AVFMT_FLAG_CUSTOM_IO);
formatContext.pb(ioContext);
// This never returns. And I can never get result.
int result = avformat_open_input(formatContext, "", format, null);
// all clean-up code omitted for simplicity
}
}
And below is my sample console output:
Successfully read 32768 bytes of data.
Successfully read 32768 bytes of data.
Successfully read 32768 bytes of data.
Successfully read 32768 bytes of data.
Successfully read 32768 bytes of data.
Successfully read 7240 bytes of data.
EOF found.
I've checked the sum of bytes, which corresponds to the file size; EOF is also hit, meaning the file is completely read. Actually I am a bit skeptical as why avformat_open_input() would even read the entire file and still without returning? There must be something wrong with what I am doing. Can any expert shed some lights or point me to the right direction? I'm new to javacv and ffmpeg and especially to programming with Buffers and stuff. Any help, suggestion or criticism is welcome. Thanks in advance.
Ok, now I have found the problem. I have misinterpreted the docs and overlooked most of the examples I found. My bad.
According to the documentation on ffmpeg:
AVIOContext* avio_alloc_context (unsigned char* buffer,
int buffer_size,
int write_flag,
void* opaque,
int(*)(void *opaque, uint8_t *buf, int buf_size) read_packet,
int(*)(void *opaque, uint8_t *buf, int buf_size) write_packet,
int64_t(*)(void *opaque, int64_t offset, int whence) seek
)
The third parameter, write_flag is used in the following fashion:
write_flag - Set to 1 if the buffer should be writable, 0 otherwise.
Actually, it means if the AVIOContext is for data output (i.e. writing), write_flag should be set to 1. Otherwise, if the context is for data input (i.e. reading), it should be set to 0.
In the question I posted, I passed 1 as the write_flag and it is causing the problem when reading. Passing 0 instead solves the problem.
Later I re-read all the examples I found, all the avio_alloc_context() calls uses 0, not 1 when reading. So that further indicates why I'm having the problem.
To conclude, I will post the revised code with the problems corrected as a future reference.
package com.example;
import org.bytedeco.javacpp.*;
import java.io.File;
import java.io.IOException;
import java.io.RandomAccessFile;
import static org.bytedeco.javacpp.avformat.*;
import static org.bytedeco.javacpp.avutil.*;
import static org.bytedeco.javacpp.avformat.AVFormatContext.*;
public class Test {
public static void main(String[] args) throws Exception {
File dir = new File(System.getProperty("user.home"), "Desktop");
File file = new File(dir, "sample.3gp");
final RandomAccessFile raf = new RandomAccessFile(file, "r");
Loader.load(avformat.class);
Loader.load(avutil.class);
av_register_all();
avformat_network_init();
Read_packet_Pointer_BytePointer_int reader = new Read_packet_Pointer_BytePointer_int() {
#Override
public int call(Pointer pointer, BytePointer buf, int bufSize) {
try {
byte[] data = new byte[bufSize]; // this is inefficient, just use as a quick example
int read = raf.read(data);
if (read <= 0) {
// I am still unsure as to return '0', '-1' or 'AVERROR_EOF'.
// But according to the following link, it should return 'AVERROR_EOF',
// http://www.codeproject.com/Tips/489450/Creating-Custom-FFmpeg-IO-Context
// btw 'AVERROR_EOF' is a nasty negative number, '-541478725'.
return AVERROR_EOF;
}
buf.position(0);
buf.put(data, 0, read);
return read;
} catch (Exception ex) {
ex.printStackTrace();
return -1;
}
}
};
Seek_Pointer_long_int seeker = new Seek_Pointer_long_int() {
#Override
public long call(Pointer pointer, long offset, int whence) {
try {
if (whence == AVSEEK_SIZE) {
// Returns the entire file length. If not supported, simply returns a negative number.
// https://www.ffmpeg.org/doxygen/trunk/avio_8h.html#a427ff2a881637b47ee7d7f9e368be63f
return raf.length();
}
raf.seek(offset);
return offset;
} catch (IOException ex) {
ex.printStackTrace();
return -1;
}
}
};
int inputBufferSize = 32768;
BytePointer inputBuffer = new BytePointer(av_malloc(inputBufferSize));
AVIOContext ioContext = avio_alloc_context(inputBuffer,
inputBufferSize,
0, // CRITICAL, if the context is for reading, it should be ZERO
// if the context is for writing, then it is ONE
null,
reader,
null,
seeker);
AVInputFormat format = av_find_input_format("3gp");
AVFormatContext formatContext = avformat_alloc_context();
formatContext.iformat(format);
formatContext.flags(formatContext.flags() | AVFMT_FLAG_CUSTOM_IO);
formatContext.pb(ioContext);
// Now this is working properly.
int result = avformat_open_input(formatContext, "", format, null);
System.out.println("result == " + result);
// all clean-up code omitted for simplicity
}
}
References:
AVSEEK_SIZE documentation
avio_alloc_context() documentation
Additional References: (I do not have enough reputation points for more links but I found these examples critical in helping me so I pasted them in plain text anyway)
Creating Custom FFmpeg IO-Context (CodeProject Example) at:
http://www.codeproject.com/Tips/489450/Creating-Custom-FFmpeg-IO-Context
Another example showing the use of write_flag in avio_alloc_context() at:
https://www.ffmpeg.org/doxygen/2.5/avio_reading_8c-example.html#a20
Your seek code needs to handle AVSEEK_SIZE as whence, and your read should return 0 on EOF ("Upon reading end-of-file, zero is returned." - literal quote from man 2 read), not AVERROR_EOF.
Goal: Get the data from a .dat file and print it to the console in Eclipse
Resources: fpfret.java and PointF.java and dichromatic.dat
I have resolved all my issues and have just a few console errors, here's my code and my question is: How do I add the getCodeBase() method?
package frp3;
import java.io.BufferedInputStream;
import java.io.DataInputStream;
import java.io.EOFException;
import java.net.URL;
import java.util.Vector;
public class FileRead {
public static void main(String[] args) { //getDocumentBase
System.out.println(readDataFile(getCodeBase() + "dichromatic.dat", 300, 750));
}
private static String getCodeBase() {
// TODO Auto-generated method stub
return null;
}
#SuppressWarnings("unchecked")
private static PointF[] readDataFile(String filename, int min, int max) {
#SuppressWarnings("rawtypes")
Vector v = new Vector();
try {
DataInputStream dis = new DataInputStream(new BufferedInputStream((new URL(filename)).openStream()));
float f0, f1;
while (true) {
try {
f0 = dis.readFloat();
f1 = dis.readFloat();
if (min < 0 || max < 0 || (f0 >= min && f0 <= max)) {
v.addElement(new PointF(f0, f1));
}
}
catch (EOFException eof) {
break;
}
}
}
catch (Exception e) {
e.printStackTrace();
}
PointF[] array = new PointF[v.size()];
for (int i = 0; i < v.size(); i++) {
array[i] = (PointF) v.elementAt(i);
}
return array;
}
}
Here's my console errors:
java.net.MalformedURLException: no protocol: nulldichromatic.dat
at java.net.URL.<init>(Unknown Source)
at java.net.URL.<init>(Unknown Source)
at java.net.URL.<init>(Unknown Source)
at frp3.FileRead.readDataFile(FileRead.java:27)
at frp3.FileRead.main(FileRead.java:12)
[Lfrp3.PointF;#29be513c
Here's my Project View in Eclipse:
Alright. This is actually more complex then I thought at first pass. Basically, readDataFile expects the dichromatic.dat file to be a resource available on the Internet. Look at the following line from readDataFile:
DataInputStream dis = new DataInputStream(new BufferedInputStream((new URL(filename)).openStream()));
Basically, whatever filename gets passed in, is used as a URL. For your use-case, where your file is hosted on your local filesystem, I recommend a few changes.
First, replace the above DataInputStream declaration line with:
DataInputStream dis = new DataInputStream(new BufferedInputStream(new FileInputStream(filename)));
Second, replace getCodeBase with:
private static String getCodeBase() {
return "";
}
I've simply replace null with an empty string. Since "dichromatic.dat" is in the root of your project, it should be sufficient to use an empty string, indicating project root, as the result for getCodeBase(), as the result of that function gets pre-pended to "dichromatic.dat" before being passed to readDataFile as filename.
If you put dichromatic.dat in a different place, just modify that empty string to be the "path" that leads to the file.
Hope this helps.
Forgot to mention -- be sure to update your imports list to include import java.io.FileInputStream -- although Eclipse should handle this gracefully for you.
I've a protocol buffer service running on the remote machine say at https://website/protobufService
In the user guide I see only message structure. like
:
message Request {
required String name = 1;
required int id = 2,
........
}
message Response {
optional int result =1;
enum Code {
val1 = 1;
val2 = 2;
}
optional Code = 2;
}
I am not sure how to write the client code in Java to make use of this service ?
Is there any guidelines on writing the client code for Protobuf services ?
Thanks a ton in advance.
- JE
Protobufs only specify how to turn a message into bytes and back, not how to send it over a network. In particular, there is no standard way to send protobufs over HTTP. The documentation for the particular service you are trying to use needs to specify exactly how it expects the messages to be sent.
Typically, though, you would send an HTTP POST to the URL with an encoded Request protobuf as its body, and it would respond with an encoded Response protobuf. You should be able to use any old HTTP library to do this; just give in the encoded Request as bytes, and decode the response bytes as a Response.
I ran into the same problem.I have written the below client for for my web service.Fortunately it does return an output.Unfortunately I think it is not in proto-buf format.I want to see it in protobuff format.I do not know how protobuff format looks like.You may wan to have a look at the below link:-
https://spring.io/blog/2015/03/22/using-google-protocol-buffers-with-spring-mvc-based-rest-services
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.MalformedURLException;
import java.net.URL;
public class HttpRestClient {
// http://localhost:8080/customers/
public static void main(String[] args) {
try {
URL url = new URL("http://localhost:8080/customers/1");
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setRequestMethod("GET");
conn.setRequestProperty("Content-Type", "application/x-protobuf");
conn.setRequestProperty("Accept", "application/x-protobuf");
String contentType = conn.getContentType();
System.out.println(contentType);
System.out.println("---------------------------------------------------------------------");
if (conn.getResponseCode() != 200) {
throw new RuntimeException("Failed : HTTP error code : " +
conn.getResponseCode());
}
BufferedReader br = new BufferedReader(new
InputStreamReader((conn.getInputStream())));
String output;
System.out.println("Output from Server .... \n");
while ((output = br.readLine()) != null) {
Object response = output;
System.out.println(response);
}
conn.disconnect();
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
}
If anyone has better suggestions please lemme know.This is how my output looks like:-
application/x-protobuf;charset=UTF-8
Output from Server ....
Chris
Richardson*
crichardson#email.com
As you could see it is not in protobuff format.
I used the code from the following link: Signare's Blog. I have 10 image URLs and would like to retrieve and show them on my screen. When I use the code from the above link, it's taking more than 10 minutes to load all of the images. How do I speed up this loading?
URLBitmapField post_img= new URLBitmapField(image_url);
add(post_img);
where the class URLBitmapField is defined as:
import net.rim.device.api.math.Fixed32;
import net.rim.device.api.system.Bitmap;
import net.rim.device.api.system.EncodedImage;
import net.rim.device.api.ui.UiApplication;
import net.rim.device.api.ui.component.BitmapField;
public class URLBitmapField extends BitmapField implements URLDataCallback {
EncodedImage result = null;
public static EncodedImage _encoded_img = null;
int _imgWidth = 52;
int _imgHeight = 62;
int _imgMargin = 10;
public URLBitmapField(String url) {
try {
http_image_data_extrator.getWebData(url, this);
}
catch (Exception e) {}
}
public Bitmap getBitmap() {
if (_encoded_img == null) return null;
return _encoded_img.getBitmap();
}
public void callback(final String data) {
if (data.startsWith("Exception")) return;
try {
byte[] dataArray = data.getBytes();
_encoded_img = EncodedImage.createEncodedImage(dataArray, 0, dataArray.length); // with scale
_encoded_img = sizeImage(_encoded_img, _imgWidth, _imgHeight);
setImage(_encoded_img);
UiApplication.getUiApplication().getActiveScreen().invalidate();
}
catch (final Exception e){}
}
public EncodedImage sizeImage(EncodedImage image, int width, int height) {
int currentWidthFixed32 = Fixed32.toFP(image.getWidth());
int currentHeightFixed32 = Fixed32.toFP(image.getHeight());
int requiredWidthFixed32 = Fixed32.toFP(width);
int requiredHeightFixed32 = Fixed32.toFP(height);
int scaleXFixed32 = Fixed32.div(currentWidthFixed32,requiredWidthFixed32);
int scaleYFixed32 = Fixed32.div(currentHeightFixed32,requiredHeightFixed32);
result = image.scaleImage32(scaleXFixed32, scaleYFixed32);
return result;
}
}
public interface URLDataCallback {
public void callback(String data);
}
and the class http_image_data_extrator is defined as:
import java.io.IOException;
import java.io.InputStream;
import javax.microedition.io.Connector;
import javax.microedition.io.HttpConnection;
import net.rim.device.api.system.RadioInfo;
import net.rim.device.api.system.WLANInfo;
import net.rim.device.api.ui.UiApplication;
public class http_image_data_extrator {
static String url_="";
static StringBuffer rawResponse=null;
public static void getWebData(String url, final URLDataCallback callback) throws IOException {
HttpConnection connection = null;
InputStream inputStream = null;
try {
if ((WLANInfo.getWLANState() == WLANInfo.WLAN_STATE_CONNECTED)&& RadioInfo.areWAFsSupported(RadioInfo.WAF_WLAN)) {
url += ";interface=wifi";
}
connection = (HttpConnection) Connector.open(url, Connector.READ, true);
String location=connection.getHeaderField("location");
if(location!=null){
if ((WLANInfo.getWLANState() == WLANInfo.WLAN_STATE_CONNECTED)&& RadioInfo.areWAFsSupported(RadioInfo.WAF_WLAN)) {
location += ";interface=wifi";
}
connection = (HttpConnection) Connector.open(location, Connector.READ, true);
}else{
connection = (HttpConnection) Connector.open(url, Connector.READ, true);
}
inputStream = connection.openInputStream();
byte[] responseData = new byte[10000];
int length = 0;
rawResponse = new StringBuffer();
while (-1 != (length = inputStream.read(responseData))) {
rawResponse.append(new String(responseData, 0, length));
}
int responseCode = connection.getResponseCode();
if (responseCode != HttpConnection.HTTP_OK){
throw new IOException("HTTP response code: "+ responseCode);
}
final String result = rawResponse.toString();
UiApplication.getUiApplication().invokeLater(new Runnable() {
public void run(){
callback.callback(result);
}
});
}
catch (final Exception ex) {
UiApplication.getUiApplication().invokeLater(new Runnable() {
public void run() {
callback.callback("Exception (" + ex.getClass() + "): " + ex.getMessage());
}
});
}
}
}
Resize on the server
Resizing the images on the server is the best answer. Because downloading big images and scaling them down requires a lot of everything (network, memory, cpu) on the device.
Resize via a proxy
If the image server is not under your control, you could still use your own server as a resizing proxy (send the image url and desired size to your server, it gets the image, resizes, and returns the resized image). Maybe there is a service that does this already.
Cheaper decode option
Some decode options may make decoding (and resizing) cheaper. DECODE_NO_DITHER, DECODE_READONLY, and DECODE_NATIVE all seem worth trying.
http://www.blackberry.com/developers/docs/4.2api/net/rim/device/api/system/EncodedImage.html#DECODE_NO_DITHER
Serial instead of parallel
You mentioned you are loading 10 images. If 10 images takes more than 10x the time 1 image takes, then the system might be "thrashing". Like it might initiate all 10 requests, then wind up working on 10 fullscale images in memory at the same time in callbacks. Could try showing the first image before starting to download the next, which also gives the user something to look at sooner. Similarly, calling invalidate 10 times in parallel (in the callback) might cause a hiccup.