I am accessing an internal site that returns gzipped content. When the content reaches a certain size, the site returns a chunked response. I am using the Apache httpcomponents 5 CloseableHttpAsyncClient and the SimpleHttpRequest and SimpleHttpResponse. The internal site is a vendor product that can't be modified.
String encoding = getEncoding(response.getHeaders());
byte[] bytes;
byte[] bodyBytes = response.getBodyBytes();
if (encoding.equals("gzip")) {
ByteArrayInputStream inputStream = new ByteArrayInputStream(bodyBytes);
GZIPInputStream gzipInputStream = new GZIPInputStream(inputStream);
bytes = IOUtils.toByteArray(gzipInputStream);
} else {
bytes = bodyBytes;
}
String html = new String(bytes, StandardCharsets.UTF_8);
I check for the response type as follows
private String getEncoding(Header[] headers) {
for (Header header : headers) {
if (header.getName().toLowerCase().equals("transfer-encoding")) {
return header.getValue();
}
if (header.getName().toLowerCase().equals("content-encoding")) {
return header.getValue();
}
}
return "";
}
I know that there is a ChunkedInputStream class, but the inputs to the constructor are not obviously available from the response
ChunkedInputStream(SessionInputBuffer buffer, InputStream inputStream)
Wraps session input stream and reads chunk coded input.
ChunkedInputStream(SessionInputBuffer buffer, InputStream inputStream, Http1Config http1Config)
Default constructor.
Do I need to use a different response type? If so, which one? Or is there a different way that is better?
Thanks for your help.
Related
I am invoking a AWS Lambda function from the AWS API Gateway. The returned JSON needs to be zipped since it sometimes became too big (body size too large etc). However, I have some issues getting the response through the API Gateway. This is my Java code:
#Override
public JSONObject handleRequest(Object input, Context context) {
String json_string = "";
try {
Gson gson = new Gson();
json_string = gson.toJson(input, LinkedHashMap.class);
} catch (ClassCastException ex) {
json_string = (String) input;
}
GenerateJson generateJson = new GenerateJson ();
String body = "";
try {
JSONParser parser = new JSONParser();
Object jsonObj = parser.parse(json_string);
JSONObject matchesobj = (JSONObject) jsonObj;
if (matchesobj.containsKey("body")) {
body = (String) matchesobj.get("body");
} else {
JSONObject error = new JSONObject();
error.put("error", "No body with Base64 data in Request.");
System.out.println(error.toJSONString());
return error;
}
} catch (ParseException ex) {
ex.printStackTrace();
}
byte[] decodedBytes = Base64.getDecoder().decode(body);
String decodedString = new String(decodedBytes);
// System.out.println(decodedString);
JSONObject json = generateJson .getJson(decodedString, "", 2);
JSONObject returnObject = new JSONObject();
JSONObject headers = new JSONObject();
returnObject.put("statusCode", 205);
returnObject.put("isBase64Encoded", true);
// returnObject.put("Content-Encoding", "gzip");
returnObject.put("headers", headers);
returnObject.put("body", compressStringAndReturnBase64(json.toString()));
return (returnObject);
}
public static String compressStringAndReturnBase64(String srcTxt) {
ByteArrayOutputStream rstBao = new ByteArrayOutputStream();
GZIPOutputStream zos;
try {
zos = new GZIPOutputStream(rstBao);
zos.write(srcTxt.getBytes());
IOUtils.closeQuietly(zos);
byte[] bytes = rstBao.toByteArray();
String base64comp = Base64.getEncoder().encodeToString(bytes);
System.out.println("Json String is " + srcTxt.toString().getBytes().length + " compressed " + bytes.length + " compressed Base64 " + base64comp.getBytes().length);
return base64comp;
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
return "";
}
I've checked the Base64 output and that seems to work fine (pasted it in https://www.base64decode.org/). In addition, when I check with Postman, I get a binary blob which can be unpacked with 7-zip if I save the response to something that ends with .gz.
Under settings, the API Gateway Binary Media Types has been set to /
But I'd like to have the client "see" that it is GZIPped and decode it on the fly. However, when I add the line
returnObject.put("Content-Encoding", "gzip");
I get {"message": "Internal server error"} and in the AWS API logs: Execution failed due to configuration error: Malformed Lambda proxy response
The Lambda logs are fine, so it did execute successfully, just wasn't able to be returned.
I am thinking I need some more tweaking on the API Gateway side, any ideas?
This sounds like the binary support setting on API Gateway isn't configured correctly; API gateway is trying to parse the response from your lambda rather than passing it on directly to the client.
You can update this setting in the console:
In your HTTP request add "Accept" header with payload content type.
Accept: application/gzip
Also in the HTTP response, there should be "Content-Type" header indicating response content type.
Content-Type: application/gzip
Your lambda returns Base64 encoded binary data to API Gateway. So in order to decode data your HTTP request's Accept header and Response's Content-type headers should there.
I am using third party service to get data using rest API. But sometimes it return more than 6 MB data. They don't have API to provide data in chunks. I am already using future method to increase heap size limit to 12 MB.
With below code, I am facing heap size error at the time of de-serialization of data. I have also emphasised that line where I am getting error.
#future (callout=true)
public static void CreateProjects(set<ID> setOfProjectID)
{
HttpRequest request;
HttpResponse response = new HttpResponse();
Map<Id,Opportunity> mapOfIdAndProjectToCreate = new Map<Id,Opportunity>([SELECT Id, Name, projectId__c FROM Opportunity
WHERE ID IN :setOfProjectID]);
Integer customerId;
List<Opportunity> lstOppToUpdate = new List<Opportunity>();
if(!mapOfIdAndProjectToCreate.isEmpty())
{
for(Id OpportunityID : mapOfIdAndProjectToCreate.keySet())
{
Opportunity oppCreated = new Opportunity();
oppCreated = mapOfIdAndProjectToCreate.get(OpportunityID);
try
{
String projectId;
String requestBody = createRequestBodyForProject(mapOfIdAndProjectToCreate.get(OpportunityID));
HttpRequest req = CreateHttpRequest(ConstantCls.CreateProjectURL,requestBody,'POST');
Http http = new Http();
HTTPResponse resp = new HTTPResponse();
if(!Test.IsRunningTest())
{
resp = http.send(req);
System.debug('heap size after api call '+Limits.getHeapSize());
}
if(resp.getStatusCode() == 200)
{
System.debug('heap size before deserializtion '+Limits.getHeapSize());
// Till now we have 6 MB data in HTTPResponse resp variable, but when we go for deserializing by fetching its body, it gives Heap size limit error.
// It is giving error when we try to fetch its body, because it has already size of response of 6 MB + little bytes are occupied by others variable. and when we fetch its body, it exceeds to more than 12 MB.
// In execution of below line, I am getting Heap size limit error.
*Map<String, Object> mapKeyVal = (Map<String, Object>)System.JSON.deserializeUntyped(resp.getBody());*
System.debug('heap size after deserialization '+Limits.getHeapSize());
Map<String, Object> data = (Map<String, Object>)mapKeyVal.get('data');
projectId = String.valueOf(data.get('projectId'));
oppCreated.projectId__c = projectId;
lstOppToUpdate.add(oppCreated);
}
}
Catch(Exception ex)
{
System.debug('Error In Sync - CreateProjects method Updated - '+ ex.getMessage() + ex.getStackTraceString());
}
}
}
UPDATE lstOppToUpdate;
}
public static HttpRequest CreateHttpRequest(String apiUrl, String requestBody, String method)
{
Map<String,String> apiConfigs = GetAllConfigSettings();
HttpRequest req = new HttpRequest();
req.setEndpoint(apiUrl);
req.setMethod(method);
req.setTimeout(120000);
req.setHeader('Content-Type', 'application/json');
req.setHeader('Authorization', apiConfigs.get('Access Token'));
if(String.isNotEmpty(requestBody))
{
req.setBody(requestBody);
}
return req;
}
I want to send binary data from my stompClient to the spring controller.
this is my JS
var socket = new SockJS('/test/binary');
var stompClient = Stomp.over(socket);
socket.binaryType = "arraybuffer";
var appModel = new ApplicationModel(stompClient);
ko.applyBindings(appModel);
appModel.connect();
appModel.pushNotification("Notifications.");
var ctx = canvas.get()[0].getContext('2d');
ctx.drawImage(img, 0, 0, 320, 240);
var data = canvas.get()[0].toDataURL('image/jpeg', 1.0);
newblob = dataURItoBlob(data);
stompClient.send("/app/vid", {}, newblob);
function dataURItoBlob(dataURI) {
// convert base64/URLEncoded data component to raw binary data held in a string
var byteString;
if (dataURI.split(',')[0].indexOf('base64') >= 0)
byteString = atob(dataURI.split(',')[1]);
else
byteString = unescape(dataURI.split(',')[1]);
// separate out the mime component
var mimeString = dataURI.split(',')[0].split(':')[1].split(';')[0];
// write the bytes of the string to a typed array
var ia = new Uint8Array(byteString.length);
for (var i = 0; i < byteString.length; i++) {
ia[i] = byteString.charCodeAt(i);
}
return new Blob([ia], {type:mimeString});
I have tried sending Blob, Uint8Array the message is sent but on the server side I can not use it.
My method in the controller is:
#MessageMapping("/vid")
public void getVid(Message<byte[]> message, Principal principal) throws IOException {
ByteArrayOutputStream bOut = new ByteArrayOutputStream();
bOut.write(message.getPayload());
bOut.flush();
byte[] imageInByte = bOut.toByteArray();
bOut.close();
InputStream in = new ByteArrayInputStream(imageInByte);
BufferedImage bImageFromConvert = ImageIO.read(in);
ImageIO.write(bImageFromConvert, "jpeg", new File(
"E:/new-darksouls.jpg"));
template.convertAndSend("/topic/binarydemo2", message);
I have used ByteArrayMessageConverter,StringMessageConverter,MappingJackson2MessageConverter and a Base64JavaObjectMessageConverter that I wrote:
public class Base64JavaObjectMessageConverter extends AbstractMessageConverter {
public Base64JavaObjectMessageConverter() {
super(MimeTypeUtils.APPLICATION_OCTET_STREAM);
}
#Override
protected boolean supports(Class<?> clazz) {
return true;
}
#Override
public Object convertFromInternal(Message<?> message, Class<?> targetClass) {
byte[] messageBytes =Base64Utils.decode( (byte[])message.getPayload() );
return SerializationUtils.deserialize( messageBytes );
}
#Override
public Object convertToInternal(Object payload, MessageHeaders headers) {
byte[] messageBytes = SerializationUtils.serialize( payload );
return Base64Utils.encode( messageBytes);
}
}
I only am able to send the byte[] as string removing the 'base64' etc from it:
stompClient.send("/app/vid", {}, data.split(',')[1]);
in the controller :
#MessageMapping("/vid")
public void getVid(String message, Principal principal) throws IOException {
byte[] bytearray = Base64.decode(message);
BufferedImage imag=ImageIO.read(new ByteArrayInputStream(bytearray));
ImageIO.write(imag, "jpg", new File("E:/new-darksouls.jpg"));
But this is not what I wish to achieve. I suppose that this will take a tall on performance.
I am using Spring 4.2.0.RC1 WebSockets,StompJS
I read on different posts that it is possible to send from back end to client and client to back end but I was not able to reproduce it. I would be very thankful if I can get a concrete example how to structure and send the Uint8Array or blob on the client and how to deal with it on the server side.
Thank you for your time and help
SockJS does not support sending binary data. You can send binary messages with use STOMP over a plain WebSocket.
On the server side see StompSubProtocolHandler which checks if incoming messages are binary or text and hence can handle either. From there it doesn't matter much, i.e. from a STOMP perspective the body is a byte array and the content-type/content-length headers determine what the body contains and how it should be intepreted. On the sending side StompSubProtocolHandler can also send either binary or text and at the moment it uses binary if the content-type is application/octet-stream and as long as SockJS is not being used.
So in short over WebSocket it should work but not when using SockJS as the transport.
Is it possible to include multiple response types in a single HTTP response? For example, JSON data as well as an image.
Hey It is not possible to set multiple MIME TYPE to HTTPResponse. But what you can do is, you can set the content type as application/json. And using json you can send the image using BASEEncoder.
public static String encodeToString(BufferedImage image, String type) {
String imageString = null;
ByteArrayOutputStream bos = new ByteArrayOutputStream();
try {
ImageIO.write(image, type, bos);
byte[] imageBytes = bos.toByteArray();
BASE64Encoder encoder = new BASE64Encoder();
imageString = encoder.encode(imageBytes);
bos.close();
} catch (IOException e) {
e.printStackTrace();
}
return imageString;
}
No. I think one generally would send the URL of image in json, and use javascript to update the "src" attribute of "img" element
I am working on GWT RPC. I am facing a problem in retrieving image from my SQL.
Here is my code:
Base64 bas = new Base64();
// sun.misc.BASE64Encoder enc = new sun.misc.BASE64Encoder();
UploadfileJpaController up = new UploadfileJpaController();
// this function returns the value in blob field in the form of byte array
byte[] b = up.findUploadfile(n);
String base64Contents = enc.encode(b).replaceAll("\\s+", "");
//String base64 = Base64Utils.toBase64(b);
base64Contents = "data:image/gif;base64,"+base64Contents;
return base64Contents;
But this is not working.. the image is not displayed. Please help :(
You should let a regular servlet take care of returning the image data, and not use GWT-RPC. The servlet should set the proper image/gif header and write the binary data to the response outputstream.
EDIT
This should look somewhat like this
public class FileDownloadServlet extends HttpServletv {
// This method is called by the servlet container to process a GET request.
public void doGet(HttpServletRequest req, HttpServletResponse resp) throws IOException {
// Set content type
resp.setContentType("image/gif");
//Up to you!
byte[] binaryData = getDataFromDbase();
ByteArrayInputStream bis = new ByteArrayInputStream(binaryData);
OutputStream out = resp.getOutputStream();
// Copy the contents of the file to the output stream
byte[] buf = new byte[1024];
int count = 0;
while ((count = bis.read(buf)) >= 0) {
out.write(buf, 0, count);
}
bis.close();
out.close();
}
}
You url is going to be something like
http://server/application/image_servlet?id=123545 where you use the id parameter in the servlet to look up the image. And of course add the servlet to you web.xml. Good luck.