GZIP encoding in Jersey 2 / Grizzly - jersey

I can't activate gzip-encoding in my Jersey service. This is what I've tried:
Started out with the jersey-quickstart-grizzly2 archetype from the Getting Started Guide.
Added rc.register(org.glassfish.grizzly.http.GZipContentEncoding.class);
(have also tried rc.register(org.glassfish.jersey.message.GZipEncoder.class);)
Started with mvn exec:java
Tested with curl --compressed -v -o - http://localhost:8080/myapp/myresource
The result is the following:
> GET /myapp/myresource HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 zlib/1.2.3.4 ...
> Host: localhost:8080
> Accept: */*
> Accept-Encoding: deflate, gzip
>
< HTTP/1.1 200 OK
< Content-Type: text/plain
< Date: Sun, 03 Nov 2013 08:07:10 GMT
< Content-Length: 7
<
* Connection #0 to host localhost left intact
* Closing connection #0
Got it!
That is, despite Accept-Encoding: deflate, gzip in the request, there is no Content-Encoding: gzip in the response.
What am I missing here??

You have to register the org.glassfish.jersey.server.filter.EncodingFilter as well. This example enables deflate and gzip compression:
import org.glassfish.jersey.message.DeflateEncoder;
import org.glassfish.jersey.message.GZipEncoder;
import org.glassfish.jersey.server.ResourceConfig;
import org.glassfish.jersey.server.filter.EncodingFilter;
...
private void enableCompression(ResourceConfig rc) {
rc.registerClasses(
EncodingFilter.class,
GZipEncoder.class,
DeflateEncoder.class);
}
This solution is jersey specific and works not only with Grizzly, but with the JDK Http server as well.

Try the code like:
HttpServer httpServer = GrizzlyHttpServerFactory.createHttpServer(
BASE_URI, rc, false);
CompressionConfig compressionConfig =
httpServer.getListener("grizzly").getCompressionConfig();
compressionConfig.setCompressionMode(CompressionConfig.CompressionMode.ON); // the mode
compressionConfig.setCompressionMinSize(1); // the min amount of bytes to compress
compressionConfig.setCompressableMimeTypes("text/plain", "text/html"); // the mime types to compress
httpServer.start();

Related

curl syntax in GET based HTTP logins

For practice purposes I decided to create a simple bruteforcing bash script, that I succesuly used to solve DWVA. I then moved to IoT - namely my old IP camera. This is my code as of now:
#!/bin/bash
if [ "${##}" != "2" ]; then
echo "<command><host><path>"
exit
fi
ip=$1
path=$2
for name in $(cat user.txt); do
for pass in $(cat passwords.txt); do
echo ${name}:${pass}
res="$(curl -si ${name}:${pass}#${ip}${path})"
check=$(echo "$res" | grep "HTTP/1.1 401 Unauthorised")
if [ "$check" != '' ]; then
tput setaf 1
echo "[FAILURE]"
tput sgr0
else
tput setaf 2
echo "[SUCCESS]"
tput sgr0
exit
fi
sleep .1
done;
done;
Despite obvious flaws - like reporting succes in case of network failure - it's as good as my 20 minutes coding jobs are. However, I can't seem to get the curl command syntax quite right. Camera in question is a simple Axis, running cramFS and a small scripting os. It's similar to a lot of publicly available cameras' login forms, like ones found here, here or here. A simple GET, yet I feel like I'm bashing my head against a wall. Any bit of ahint will be madly appreciated at this point.
I've taken the liberty to paste contents of first GET package:
AYGET /operator/basic.shtml?id=478 HTTP/1.1
Host: <target_host_ip>
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:58.0) Gecko/20100101 Firefox/58.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US;q=0.7,en;q=0.3
Accept-Encoding: gzip, deflate
Referer: http://<target_host_ip>/view/view.shtml?id=282&imagepath=%2Fmjpg%2Fvideo.mjpg&size=1
Connection: keep-alive
Upgrade-Insecure-Requests: 1
Authorization: Digest username="root", realm="AXIS_ACCC8E4A2177", nonce="w3PH7XVmBQA=32dd7cd6ab72e0142e2266eb2a68f59e92995033", uri="/operator/basic.shtml?id=478", algorithm=MD5, response="025664e1ba362ebbf9c108b1acbcae97", qop=auth, nc=00000001, cnonce="a7e04861c3634d3b"
Package sent in return is a simple, dry 401.
PS.: Any powers that be - feel free to remove the IPs if they violate anything. Also feel free to point out grammar/spelling etc. mistakes since C2 exam is coming.
It looks like those cameras don't simply use "Basic" HTTP auth with a base64 encoded username:password combo, but use digest authentication which involves a bit more.
Luckily, with cURL this just means you need to specify --digest on the command line to handle it properly.
Test the sequence of events yourself using:
curl --digest http://user:password#example.com/digest-url/
You should see something similar to:
* Trying example.com...
* Connected to example.com (x.x.x.x) port 80 (#0)
* Server auth using Digest with user 'admin'
> GET /view/viewer_index.shtml?id=1323 HTTP/1.1
> Host: example.com
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 401 Unauthorized
< Date: Wed, 08 Nov 1972 17:30:37 GMT
< Accept-Ranges: bytes
< Connection: close
< WWW-Authenticate: Digest realm="AXIS_MACADDR", nonce="00b035e7Y417961b2083fae7e4b2c4053e39ef8ba0b65b", stale=FALSE, qop="auth"
< WWW-Authenticate: Basic realm="AXIS_MACADDR"
< Content-Length: 189
< Content-Type: text/html; charset=ISO-8859-1
<
* Closing connection 0
* Issue another request to this URL: 'http://admin:admin2#example.com/view/viewer_index.shtml?id=1323'
* Server auth using Digest with user 'admin'
> GET /view/viewer_index.shtml?id=1323 HTTP/1.1
> Host: example.com
> Authorization: Digest username="admin", realm="AXIS_MACADDR", nonce="00b035e7Y417961b2083fae7e4b2c4053e39ef8ba0b65b", uri="/view/viewer_index.shtml?id=1323", cnonce="NWIxZmY1YzA3NmY3ODczMDA0MDg4MTUwZDdjZmE0NGI=", nc=00000001, qop=auth, response="3b03254ef43bc4590cb00ba32defeaff"
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 401 Unauthorized
< Date: Wed, 08 Nov 1972 17:30:37 GMT
< Accept-Ranges: bytes
< Connection: close
* Authentication problem. Ignoring this.
< WWW-Authenticate: Digest realm="AXIS_MACADDR", nonce="00b035e8Y8232884a74ee247fc1cc42cab0cdf59839b6f", stale=FALSE, qop="auth"
< WWW-Authenticate: Basic realm="AXIS_MACADDR"
< Content-Length: 189
< Content-Type: text/html; charset=ISO-8859-1
<

Elastic Search Bulk API - Not indexing

I have output a JSON file in bulk format which I can load in to Kibana with the developer tools. and by inserting a few lines using the -d command
example lines of file:
{"index":{"_index":"els","_type":"logs","_id":1481018400003}}
{"timestamp":1481018400003,"zoneId":29863567,............[]}
{"index":{"_index":"els","_type":"logs","_id":"30cee368073c0c9b"}}
{"timestamp":1481018400005,"zoneId":29863567,............[]}
...
However when I run the bulk api to pot a file it does not do anything. I added verbose to the command and get the following:
* Connected to localhost (::1) port 9200 (#0)
> POST /_bulk HTTP/1.1
> Host: localhost:9200
> User-Agent: curl/7.49.0
> Accept: */*
> Content-Length: 0
> Content-Type: application/x-www-form-urlencoded
>
< HTTP/1.1 400 Bad Request
< content-type: application/json; charset=UTF-8
< content-length: 165
* HTTP error before end of send, stop sending
Any help would be great.
Thanks!

Using PDF Reactor as Web Service

I am discovering PDF reactor and I'd like to use it as a web service. To test a file, I use cURL
curl -v -X POST --header "Content-Type:application/xml" http://localhost:9423/service/rest/convert/async -d #test.html
Is that correct ?
test.html :
<html>
<body>
Coucou, je suis terrien.
</body>
</html>
Thank you for your help,
Cédrik
edit #1:
response from the comman above :
* About to connect() to localhost port 9423 (#0)
* Trying 127.0.0.1... connected
* Connected to localhost (127.0.0.1) port 9423 (#0)
> POST /service/rest/convert/async HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.14.3.0 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: localhost:9423
> Accept: */*
> Content-Type:application/xml
> Content-Length: 50
>
< HTTP/1.1 400 Bad Request
< Content-Type: text/plain
< Date: Tue, 15 Dec 2015 11:47:29 GMT
< Content-Length: 307
< Server: Jetty(9.3.2.v20150730)
<
* Connection #0 to host localhost left intact
* Closing connection #0
JAXBException occurred : élément inattendu (URI : "", local : "html"). Les éléments attendus sont <{http://webservice.pdfreactor.realobjects.com/}configuration>. élément inattendu (URI : "", local : "html"). Les éléments attendus sont <{http://webservice.pdfreactor.realobjects.com/}configuration>.
When using the REST API of PDFreactor via cURL you have to send a configuration XML or JSON to the server which includes configuration for PDFreactor and your document, as described here: http://www.pdfreactor.com/product/doc_html/index.html#d0e688
A sample configuration for XML could look like this:
config.xml:
<tns:configuration xmlns:tns="http://webservice.pdfreactor.realobjects.com/">
<document><html> <body> Coucou, je suis terrien. </body> </html></document>
</tns:configuration>
You can then call the following:
curl -v -X POST --header "Content-Type:application/xml" http://localhost:9423/service/rest/convert/async.xml -d #config.xml
The output will look like the following:
* About to connect() to localhost port 9423
* Trying 127.0.0.1... connected
* Connected to localhost (127.0.0.1) port 9423
> POST /service/rest/convert/async.xml HTTP/1.1
> User-Agent: curl/7.15.5 (x86_64-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5
> Host: localhost:9423
> Accept: */*
> Content-Type:application/xml
> Content-Length: 195
>
> <tns:configuration xmlns:tns="http://webservice.pdfreactor.realobjects.com/"> <document><html><body>Coucou, je suis terrien.</body></html></document></tns:configuration>HTTP/1.1 202 Accepted
< Access-Control-Allow-Credentials: true
< Access-Control-Allow-Headers: Accept, Content-Length, content-type, Host, User-Agent
< Access-Control-Allow-Methods: GET, PUT, POST, DELETE
< Access-Control-Expose-Headers: Location
< Cache-Control: no-cache
< Date: Wed, 16 Dec 2015 16:34:19 GMT
< Location: http://localhost:9423/service/rest/progress/c2a58dbd-ef9d-4b79-87d9-079c139fe9ed
< Content-Length: 0
< Server: Jetty(9.3.2.v20150730)
* Connection #0 to host localhost left intact
* Closing connection #0
The "Location" response header contains the URL which can be used to retrieve the progress of the conversion, so you can retrieve the progress with (the ID will of course vary):
curl -v http://localhost:9423/service/rest/progress/c2a58dbd-ef9d-4b79-87d9-079c139fe9ed
This will return the conversion progress and if the conversion has finished the "Location" repsonse header will contain a new URL to retrieve the document. You can use ".pdf" to retrieve the PDF binary data or ".xml" to retrieve XML data containing the PDF as base64 encoded String, the number of pages of the document, etc.
curl -v http://localhost:9423/service/rest/document/c2a58dbd-ef9d-4b79-87d9-079c139fe9ed.pdf

Ruby http, net/http, httpclient: can't parse www.victoriassecret.com

I am using httpclient gem, it works fine on Windows, just moved to AWS EC2, tried it on https://victoriassecret.com and it gets this response:
= Response
HTTP/1.1 920 Unknown
Content-Type: text/html
Date: Wed, 21 Oct 2015 21:42:51 GMT
Connection: Keep-Alive
Content-Length: 23
<h1>File not found</h1>#<HTTP::Message:0x000000023f5168
#http_body=
#<HTTP::Message::Body:0x000000023f50a0
#body="<h1>File not found</h1>",
#chunk_size=nil,
#positions=nil,
#size=0>,
#http_header=
#<HTTP::Message::Headers:0x000000023f5140
#body_charset=nil,
#body_date=nil,
#body_encoding=#<Encoding:ASCII-8BIT>,
#body_size=0,
#body_type=nil,
#chunked=false,
#dumped=false,
#header_item=
[["Content-Type", "text/html"],
["Date", "Wed, 21 Oct 2015 21:42:51 GMT"],
["Connection", "Keep-Alive"],
["Content-Length", "23"]],
#http_version="1.1",
#is_request=false,
#reason_phrase="Unknown",
#request_absolute_uri=nil,
#request_method="GET",
#request_query=nil,
#request_uri=
#<URI::HTTPS:0x000000023f58c0 URL:https://www.victoriassecret.com/pink/new-and-now>,
#status_code=920>,
#peer_cert=
#<OpenSSL::X509::Certificate: subject=#<OpenSSL::X509::Name:0x000000024ebe00>, issuer=#<OpenSSL::X509::Name:0x000000024ebec8>, serial=#<OpenSSL::BN:0x000000024de110>, not_before=2015-05-27 00:00:00 UTC, not_after=2017-05-26 23:59:59 UTC>,
#previous=nil>
It does not work only with this website, httpclient get https://google.com for example works fine. But on Windows I get normal response from httpclient get https://www.victoriassecret.com. Butt when using standard NET/HTTP library I get the same 920 response on Windows.
This isn't ec2 related. It's most likely related to the User Agent header sent by the various http library implementations.
For example, they clearly don't like 'wget':
curl -A "Wget/1.13.4 (linux-gnu)" -v https://www.victoriassecret.com
* Rebuilt URL to: https://www.victoriassecret.com/
* Trying 98.158.54.100...
* Connected to www.victoriassecret.com (98.158.54.100) port 443 (#0)
* TLS 1.2 # truncated
> GET / HTTP/1.1
> Host: www.victoriassecret.com
> User-Agent: Wget/1.13.4 (linux-gnu)
> Accept: */*
>
< HTTP/1.1 910 Unknown
< Content-Type: text/html
< Date: Thu, 22 Oct 2015 01:16:31 GMT
< Connection: Keep-Alive
< Content-Length: 23
<
* Connection #0 to host www.victoriassecret.com left intact
<h1>File not found</h1>%

Changing HTTP status message using Sinatra

I'm writing a simple Sinatra app, and given a user posts a request with an specific data, I want to return an error '453' (custom error code) with a message CLIENT_ERROR, or something similar.
The problem is: looking into the Sinatra documentation and doing some testing I couldn't find a way to setup the response error message, only the response status.
So, if a set the Sinatra response
get '/' do
response.status = 453
end
I get the error code right:
curl -v localhost:4567
* About to connect() to localhost port 4567 (#0)
* Trying 127.0.0.1... connected
> GET / HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: localhost:4567
> Accept: */*
>
< HTTP/1.1 453
< X-Frame-Options: sameorigin
< X-XSS-Protection: 1; mode=block
< Content-Type: text/html;charset=utf-8
< Content-Length: 0
< Connection: keep-alive
< Server: thin 1.3.1 codename Triple Espresso
<
* Connection #0 to host localhost left intact
* Closing connection #0
But what I want to have is:
< HTTP/1.1 453 CLIENT_ERROR
The same way I have
< HTTP/1.1 200 OK
When everything goes according to the plan.
Is there anyway to do this using Sinatra/Rack?
The status message is generated by the server you are using, e.g. in Thin the messages are in Thin::HTTP_STATUS_CODES and the reponse line is generated in Thin::Response, and in WEBrick they are in WEBrick::HHTPStatus::StatusMessage and the response is generated in WEBrick::HTTPResponse.
If you know what server you are using, you could add your error to the appropriate hash.
With Thin:
require 'thin'
Thin::HTTP_STATUS_CODES[453] = "Client Error"
and the output:
$ curl -v localhost:4567
* About to connect() to localhost port 4567 (#0)
* Trying 127.0.0.1... connected
* Connected to localhost (127.0.0.1) port 4567 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.19.7 (universal-apple-darwin10.0) libcurl/7.19.7 OpenSSL/0.9.8r zlib/1.2.3
> Host: localhost:4567
> Accept: */*
>
< HTTP/1.1 453 Client Error
< X-Frame-Options: sameorigin
< X-XSS-Protection: 1; mode=block
< Content-Type: text/html;charset=utf-8
< Content-Length: 0
< Connection: keep-alive
< Server: thin 1.4.1 codename Chromeo
<
* Connection #0 to host localhost left intact
* Closing connection #0
and with WEBrick:
require 'webrick'
WEBrick::HTTPStatus::StatusMessage[453] = "Client Error"
which gives the output:
$ curl -v localhost:4567
* About to connect() to localhost port 4567 (#0)
* Trying 127.0.0.1... connected
* Connected to localhost (127.0.0.1) port 4567 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.19.7 (universal-apple-darwin10.0) libcurl/7.19.7 OpenSSL/0.9.8r zlib/1.2.3
> Host: localhost:4567
> Accept: */*
>
localhost - - [13/Aug/2012:01:41:48 BST] "GET / HTTP/1.1" 453 0
- -> /
< HTTP/1.1 453 Client Error
< X-Frame-Options: sameorigin
< X-Xss-Protection: 1; mode=block
< Content-Type: text/html;charset=utf-8
< Content-Length: 0
< Server: WEBrick/1.3.1 (Ruby/1.9.3/2012-04-20)
< Date: Mon, 13 Aug 2012 00:41:48 GMT
< Connection: Keep-Alive
<
* Connection #0 to host localhost left intact
* Closing connection #0
I would recommend not to use custom HTTP status codes. If you think you have something of general use, consider writing an Internet Draft and going through the IETF specification process.

Resources