Volley's NetworkImageView with images on FTP - ftp

The scenario is: "Have adapters that work for HTTP, been asked to add FTP support" - sounds reasonable.
Is there a way to do it in an equally reasonable timeframe?
Or a similar mechanism/customized Volley library to swap Volley for?
The FTP is password-protected, if this makes any difference.
Tried so far:
HttpClientStack: Scheme 'ftp' not registered
HurlStack" libcore.net.url.FtpURLConnection cannot be cast to java.net.HttpURLConnection
attempted tunneling through a custom URLonnection (multiple difficulties, like libcore package not being public)
custom ImageCache that takes FTP handling off Volley's hands (what's the point, really?)

Only thing I've come up with thus far: override performRequest to do something completely different for ftp links.
Mind, the code here is bad, it just barely serves my purpose:
public class HurlStackFtp extends HurlStack {
#Override
public HttpResponse performRequest(Request<?> request, Map<String, String> additionalHeaders)
throws IOException, AuthFailureError {
String urlString = request.getUrl();
if (urlString != null && urlString.startsWith("ftp://")) {
return performFTPRequest(request, additionalHeaders);
} else {
return super.performRequest(request, additionalHeaders);
}
}
public HttpResponse performFTPRequest(Request<?> request, Map<String, String> additionalHeaders)
throws IOException, AuthFailureError {
String url = request.getUrl();
HashMap<String, String> map = new HashMap<String, String>();
map.putAll(request.getHeaders());
map.putAll(additionalHeaders);
// UrlRewriter not supported
InputStream input = getStreamFromFTP(url);
StatusLine responseStatus = new BasicStatusLine(new ProtocolVersion("HTTP", 1, 1),
/*connection.getResponseCode()*/200, /*connection.getResponseMessage()*/"OK");
BasicHttpResponse response = new BasicHttpResponse(responseStatus);
response.setEntity(makeEntity(input));
return response;
}
public static InputStream getStreamFromFTP(String url) throws IOException {
URL parsedUrl = new URL(url);
URLConnection cn = parsedUrl.openConnection();
cn.connect();
return cn.getInputStream();
}
private static HttpEntity makeEntity(InputStream inputStream) throws IOException {
BasicHttpEntity entity = new BasicHttpEntity();
entity.setContent(inputStream);
entity.setContentLength(inputStream.available());
entity.setContentType("binary/octet-stream"); // sigh...
return entity;
}
}

Related

How to use MockMVC test the controller which use org.apache.commons.fileupload?

My Controller use " org.apache.commons.fileupload " realized the file UPload.
see it:
#PostMapping("/upload")
public String upload2(HttpServletRequest request) throws Exception {
ServletFileUpload upload = new ServletFileUpload();
FileItemIterator iter = upload.getItemIterator(request);
boolean uploaded = false;
while (iter.hasNext() && !uploaded) {
FileItemStream item = iter.next();
if (item.isFormField()) {
item.openStream().close();
} else {
String fieldName = item.getFieldName();
if (!"file".equals(fieldName)) {
item.openStream().close();
} else {
InputStream stream = item.openStream();
// dosomething here.
uploaded = true;
}
}
}
if (uploaded) {
return "ok";
} else {
throw new BaseResponseException(HttpStatus.BAD_REQUEST, "400", "no file field or data file is empty.");
}
}
and my MockMvc code is
public void upload() throws Exception {
File file = new File("/Users/jianxiaowen/Documents/a.txt");
MockMultipartFile multipartFile = new MockMultipartFile("file", new FileInputStream(file));
HashMap<String, String> contentTypeParams = new HashMap<String, String>();
contentTypeParams.put("boundary", "----WebKitFormBoundaryaDEFKSFMY18ehkjt");
MediaType mediaType = new MediaType("multipart", "form-data", contentTypeParams);
MvcResult mvcResult = mockMvc.perform(MockMvcRequestBuilders.post(baseUrl+"/upload")
.content(multipartFile.getBytes())
.contentType(mediaType)
.header(Origin,OriginValue)
.cookie(cookie))
.andReturn();
logResult(mvcResult);
}
my controller is right , it has successed in my web project,
but I want to test it use MvcMock, it has some mistake, see :
can someOne can help me?
"status":"400","msg":"no file field or data file is empty.","data":null
I don't know why it says my file is empty.
my English is poor, thank you very much if someone can help me.
The MockMvc can be used for integration testing for controllers using Apache Commons Fileupload too!
Import the org.apache.httpcomponents:httpmime into your pom.xml or gradle.properties
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpmime</artifactId>
<version>4.5.13</version>
</dependency>
Update the code to use MultipartEntityBuilder to build the multipart request on the client, and then serialize the entity into bytes, which is then set in the request content
public void upload() throws Exception {
File file = new File("/Users/jianxiaowen/Documents/a.txt");
String boundary = "----WebKitFormBoundaryaDEFKSFMY18ehkjt";
// create 'Content-Type' header for multipart along with boundary
HashMap<String, String> contentTypeParams = new HashMap<String, String>();
contentTypeParams.put("boundary", boundary); // set boundary in the header
MediaType mediaType = new MediaType("multipart", "form-data", contentTypeParams);
// create a multipart entity builder, and add parts (file/form data)
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
HttpEntity multipartEntity = MultipartEntityBuilder.create()
.addPart("file", new FileBody(file, ContentType.create("text/plain"), file.getName())) // add file
// .addTextBody("param1", "value1") // optionally add form data
.setBoundary(boundary) // set boundary to be used
.build();
multipartEntity.writeTo(outputStream); // or getContent() to get content stream
byte[] content = outputStream.toByteArray(); // serialize the content to bytes
MvcResult mvcResult = mockMvc.perform(
MockMvcRequestBuilders.post(baseUrl + "/upload")
.contentType(mediaType)
.content(content) // finally set the content
.header(Origin,OriginValue)
.cookie(cookie)
).andReturn();
logResult(mvcResult);
}
Can you try the below?
mockMvc.perform(
MockMvcRequestBuilders.multipart(baseUrl+"/upload")
.file(multiPartFile)
).andReturn();
Update:
You need to update the controller to handle the MultipartFile:
#PostMapping("/upload")
public String upload2(#RequestParam(name="nameOfRequestParamWhichContainsFileData")
MultipartFile uploadedFile, HttpServletRequest request) throws Exception {
//the uploaded file gets copied to uploadedFile object.
}
You need not use another library for managing file uploads. You can use the file upload capabilities provided by Spring MVC.

Streaming upload via #Bean-provided RestTemplateBuilder buffers full file

I'm building a reverse-proxy for uploading large files (multiple gigabytes), and therefore want to use a streaming model that does not buffer entire files. Large buffers would introduce latency and, more importantly, they could result in out-of-memory errors.
My client class contains
#Autowired private RestTemplate restTemplate;
#Bean
public RestTemplate restTemplate(RestTemplateBuilder restTemplateBuilder) {
int REST_TEMPLATE_MODE = 1; // 1=streams, 2=streams, 3=buffers
return
REST_TEMPLATE_MODE == 1 ? new RestTemplate() :
REST_TEMPLATE_MODE == 2 ? (new RestTemplateBuilder()).build() :
REST_TEMPLATE_MODE == 3 ? restTemplateBuilder.build() : null;
}
and
public void upload_via_streaming(InputStream inputStream, String originalname) {
SimpleClientHttpRequestFactory requestFactory = new SimpleClientHttpRequestFactory();
requestFactory.setBufferRequestBody(false);
restTemplate.setRequestFactory(requestFactory);
InputStreamResource inputStreamResource = new InputStreamResource(inputStream) {
#Override public String getFilename() { return originalname; }
#Override public long contentLength() { return -1; }
};
MultiValueMap<String, Object> body = new LinkedMultiValueMap<String, Object>();
body.add("myfile", inputStreamResource);
HttpHeaders headers = new HttpHeaders();
headers.setContentType(MediaType.MULTIPART_FORM_DATA);
HttpEntity<MultiValueMap<String, Object>> requestEntity = new HttpEntity<>(body,headers);
String response = restTemplate.postForObject(UPLOAD_URL, requestEntity, String.class);
System.out.println("response: "+response);
}
This is working, but notice my REST_TEMPLATE_MODE value controls whether or not it meets my streaming requirement.
Question: Why does REST_TEMPLATE_MODE == 3 result in full-file buffering?
References:
How to forward large files with RestTemplate?
How to send Multipart form data with restTemplate Spring-mvc
Spring - How to stream large multipart file uploads to database without storing on local file system -- establishing the InputStream
How to autowire RestTemplate using annotations
Design notes and usage caveats, also: restTemplate does not support streaming downloads
In short, the instance of RestTemplateBuilder provided as an #Bean by Spring Boot includes an interceptor (filter) associated with actuator/metrics -- and the interceptor interface requires buffering of the request body into a simple byte[].
If you instantiate your own RestTemplateBuilder or RestTemplate from scratch, it won't include this by default.
I seem to be the only person visiting this post, but just in case it helps someone before I get around to posting a complete solution, I've found a big clue:
restTemplate.getInterceptors().forEach(item->System.out.println(item));
displays...
org.SF.boot.actuate.metrics.web.client.MetricsClientHttpRequestInterceptor
If I clear the interceptor list via setInterceptors, it solves the problem. Furthermore, I found that any interceptor, even if it only performs a NOP, will introduce full-file buffering.
public class SimpleClientHttpRequestFactory { ...
I have explicitly set bufferRequestBody = false, but apparently this code is bypassed if interceptors are used. This would have been nice to know earlier...
#Override
public ClientHttpRequest createRequest(URI uri, HttpMethod httpMethod) throws IOException {
HttpURLConnection connection = openConnection(uri.toURL(), this.proxy);
prepareConnection(connection, httpMethod.name());
if (this.bufferRequestBody) {
return new SimpleBufferingClientHttpRequest(connection, this.outputStreaming);
}
else {
return new SimpleStreamingClientHttpRequest(connection, this.chunkSize, this.outputStreaming);
}
}
public abstract class InterceptingHttpAccessor extends HttpAccessor { ...
This shows that the InterceptingClientHttpRequestFactory is used if the list of interceptors is not empty.
/**
* Overridden to expose an {#link InterceptingClientHttpRequestFactory}
* if necessary.
* #see #getInterceptors()
*/
#Override
public ClientHttpRequestFactory getRequestFactory() {
List<ClientHttpRequestInterceptor> interceptors = getInterceptors();
if (!CollectionUtils.isEmpty(interceptors)) {
ClientHttpRequestFactory factory = this.interceptingRequestFactory;
if (factory == null) {
factory = new InterceptingClientHttpRequestFactory(super.getRequestFactory(), interceptors);
this.interceptingRequestFactory = factory;
}
return factory;
}
else {
return super.getRequestFactory();
}
}
class InterceptingClientHttpRequest extends AbstractBufferingClientHttpRequest { ...
The interfaces make it clear that using InterceptingClientHttpRequest requires buffering body to a byte[]. There is not an option to use a streaming interface.
#Override
public ClientHttpResponse execute(HttpRequest request, byte[] body) throws IOException {

HttpURLConnnection request failures on Android 6.0 (MarshMallow)

I am using Google's Volley library for my application project , that targets minimum api level 14.So, Volley library uses HttpURLConnection as the NetworkClient.
Therefore , there should not be any issue related to Removal of Apache HTTPClient. However, I have done the configuration required for 6.0 Sdk i.e compileSdkVersion 23, build-tool-version 23.0.1 and build:gradle:1.3.1' and even tried adding useLibrary 'org.apache.http.legacy'. Have updated the same for Volley library project in my application.
Recently ,I tried to run my app on Android 6.0 (MarshMallow), my project compiles and runs. But those requests that require authentication headers are failing on MarshMallow with:
BasicNetwork.performRequest: Unexpected response code 401 com.android.volley.AuthFailureError
However the same is running on all Api level below 23.
I have checked the headers many times.Strangely, those requests that do not require authentication are giving response with 200 OK.
Right now I am totally clueless what is breaking the requests, does anybody have any idea what has changed in new Version that HttpURLConnection request fails for only Api level 23? Is anybody else using Volley and facing similar issue?
Here is my CustomRequest Class
public class CustomRequest extends Request<Void> {
int id, cmd;
Map<String, String> params;
BaseModel model;
public CustomRequest(int method, int cmd, String url, Map<String, String> params, int id, BaseModel model) {
super(method, url, null);
this.id = id;
this.cmd = cmd;
this.params = params;
this.model = model;
if (method == Method.GET) {
setUrl(buildUrlForGetRequest(url));
}
Log.v("Volley", "Making request to: " + getUrl());
}
private String buildUrlForGetRequest(String url) {
if (params == null || params.size() == 0) return url;
StringBuilder newUrl = new StringBuilder(url);
Set<Entry<String, String>> paramPairs = params.entrySet();
Iterator<Entry<String, String>> iter = paramPairs.iterator();
newUrl.append("?");
while (iter.hasNext()) {
Entry<String, String> param = iter.next();
newUrl
.append(param.getKey())
.append("=")
.append(param.getValue());
if (iter.hasNext()) newUrl.append("&");
}
return newUrl.toString();
}
#Override
public Map<String, String> getHeaders() throws AuthFailureError {
Map<String, String> headers = new HashMap<String, String>();
headers.put("X-Api-Version", Contract.API_VERSION);
headers.put("X-Client", "android");
String accessToken = APP.getInstance().getToken();
if (!accessToken.equals("")) {
headers.put("Authorization", "Bearer " + accessToken);
}
return headers;
}
#Override
protected Response<Void> parseNetworkResponse(NetworkResponse response) {
Exception ex;
try {
String jsonString = new String(
response.data, HttpHeaderParser.parseCharset(response.headers));
JsonNode json = new ObjectMapper().readTree(jsonString);
if (model != null) model.parse(id, json);
EventBus.getDefault().post(new EventResponse(cmd, true, null));
return Response.success(null, HttpHeaderParser.parseCacheHeaders(response)); //Doesn't return anything. BaseModel.parse() does all the storage work.
} catch (NoMoreDataException e) {
ex = e;
EventBus.getDefault().post(new NoMoreDataModel(cmd, e));
EventBus.getDefault().post(new EventResponse(cmd, false, null));
return Response.success(null, HttpHeaderParser.parseCacheHeaders(response));
} catch (Exception e) {
ex = e;
Log.e("CustomRequest", Log.getStackTraceString(e));
String message = APP.getInstance().getString(R.string.failedRequest);
if (!TextUtils.isEmpty(e.getMessage()))
message = e.getMessage();
EventBus.getDefault().post(new ErrorEventModel(cmd, message, e));
return Response.error(new ParseError(ex));
}
}
#Override
protected Map<String, String> getParams() throws AuthFailureError {
return params;
}
#Override
protected void deliverResponse(Void response) {
Log.v("Volley", "Delivering result: " + getUrl());
}
#Override
public void deliverError(VolleyError error) {
Log.e("CustomRequest", "Delivering error: Request=" + getUrl() + " | Error=" + error.toString());
String message = APP.getInstance().getString(R.string.failedRequest);
EventBus.getDefault().post(new ErrorEventModel(cmd, message, error));
}
}
Only difference I found between Api 23 and others is the HostNameVerifier.
For Api level 23 : com.android.okhttp.internal.tls.OkHostnameVerifier
For Api level <23 : javax.net.ssl.DefaultHostnameVerifier.
After checking the Server side of my application, found the reason behind the issue.
For Android 6.0(MarshMallow) the headers were becoming empty and this was not the case with other versions.
So the fix that worked for me:
Created and Added a new custom header X-Proxy-No-Redirect => 1 and passed along with other headers.
Theory behind it:
There is a API server to which we send request, then the API server redirects the request to corresponding site based on the oAuth token
While redirecting
For the redirection to happen, there are two ways to do that
1 - We just send a response back to the caller stating to redirect to a certain page. Then the caller(Networking library) takes care of redirection
2 - API server will itself makes the redirect request and get the response and then pass to caller
Earlier we were using the Method 1.
On Android 6.0 - The networking lib(Volley) doesn't seem to set all the headers while making the redirection request.
Once this new header is set, Method 2 will come into effective.
P.S This fix was applicable for my application project , maybe not for others.Just providing what was wrong and what helped me.

How to get mesos log dir

I want to get mesos log dir like this
mesos-master --zk=zk://localhost:2181/mesos --log_dir=/Users/wangyao/WorkAndLog/mesos/log
/Users/wangyao/WorkAndLog/meoso/log
How could i get this in my code?
I try it with do GetRequest in my code like this
GetRequestHandler requestHandler = new GetRequestHandler() {
#Override
public String doGetRequestCallBack(HttpResponse response) throws IOException {
return EntityUtils.toString(response.getEntity());
}
};
Map<String, String> httpParams = new HashMap<String, String>();
String obj = HttpClientUtils.doGetRequest("http", "localhost", 5050, "/master/state.json", httpParams,
requestHandler);
in this way I can get the log_dir in the obj.
but I knew that the application server and mesos server is not using the same machine ,so I can't do the request with 'localhost', Generally it's not a viable way.
is there any way to get the log_dir?

Android Volley Caching with different POST requests

I am using Android Volley to cache requests this works fine when I was using GET but I switched to use POST for some reasons. Now I want to cache the same URL with different POST data.
Request 1 -> URL1, POST Data = "Cat=1"
Request 2 -> URL1, POST Data = "Cat=2"
Request 3 -> URL1, POST Data = "Cat=3"
is this can be done with Android Volley
as the Volley.Request.getCacheKey() returns the URL which in my case is the same; this did not work for me.
Instead I had to override getCacheKey() in my child class to return URL+POST(key=Value)
That way I was able to cache all the POST requests made to the same URL with different POST data.
when you try to retrieve the cached request you need to construct the cache key with the same way.
so here is a snapshot of my code:
public class CustomPostRequest extends Request<String> {
.
.
private Map<String, String> mParams;
.
.
public void SetPostParam(String strParam, String strValue)
{
mParams.put(strParam, strValue);
}
#Override
public Map<String,String> getParams() {
return mParams;
}
#Override
public String getCacheKey() {
String temp = super.getCacheKey();
for (Map.Entry<String, String> entry : mParams.entrySet())
temp += entry.getKey() + "=" + entry.getValue();// not do another request
return temp;
}
}
When ever you construct a new request you can use getCacheKey() to search for the cached request first before putting it in the requests queue.
I hope this helps.
Also if you don't want to use one of the existing Request classes you can follow this code (I'm using JsonArrayRequest here, you can use whatever you want)
Map<String, String> params = yourData;
JsonArrayRequest request = new JsonArrayRequest(Request.Method.POST, url,
new Response.Listener<JSONArray>() {
... Needed codes
},
new Response.ErrorListener() {
...
}
){
#Override
protected Map<String, String> getParams() throws AuthFailureError {
return params;
}
#Override
public String getCacheKey() {
return generateCacheKeyWithParam(super.getCacheKey(), params);
}
};
and based on Mahmoud Fayez's answer, here the generateCacheKeyWithParam() method:
public static String generateCacheKeyWithParam(String url, Map<String, String> params) {
StringBuilder urlBuilder = new StringBuilder(url);
for (Map.Entry<String, String> entry : params.entrySet()) {
urlBuilder.append(entry.getKey()).append("=").append(entry.getValue());
}
url = urlBuilder.toString();
return url;
}

Resources