I am trying to use spring xd to ingest text/xml responses from a web service using HTTP/1.1 protocol. The goal is to convert the xml response to json and insert into mongodb. But right now I am unable to get any responses from the stream. I want to do this programatically rather than in the shell, below is my code
public static void main(String[] args) {
SpringXDTemplate template = null;
try {
template = new SpringXDTemplate(new URI("http://localhost:9393"));
} catch (URISyntaxException e) {
e.printStackTrace();
}
String name = "test";
String definition = "time --fixedDelay=5 | http-client --url='''http://www.ctabustracker.com/bustime/api/v2/getvehicles?key=key&vid=1''' | file";
template.streamOperations().destroy(name);
template.streamOperations().createStream(name, definition, true);
}
I'm expecting to find the responses written to C:\tmp\xd\output\test.out but no file was created. I see that the stream was created in the admin ui, I'm not seeing any exceptions. How do I obtain the responses from sending requests to this url?
Turn on DEBUG logging; you should see the time source emit a message every 5 seconds; follow the messages through the flow to figure out what's going wrong.
Related
Hi,
I have a list with size of 500k and I have to make a request to a server with hash parameters.
The server accepts JSON Array of 200 objects. so I could send 200 items each time.
But still I need to split the list each time and send that part to server.
I have a method for this which makes the http post request. and I want to use spring boot options (if available) to call the method with different threads and get the response back and merge them into one.
I did it using java CompletableFuture class without any springboot tags. but you Could use #async for your method too. sample code :
var futures = new ArrayList<CompletableFuture<List<Composite>>>();
var executor = Executors.newFixedThreadPool(Runtime.getRuntime().availableProcessors());
for (List<CompositeRecord> records : Lists.partition(recordsList, 200)) {
var future = CompletableFuture.supplyAsync(() -> /* call your method here */, executor);
futures.add(future);
Thread.sleep(2000);
}
CompletableFuture.allOf(futures.toArray(new CompletableFuture[0])).exceptionally(ex -> null).join(); // avoid throwing an exception in the join() call
var futureMap = futures.stream().collect(Collectors.partitioningBy(CompletableFuture::isCompletedExceptionally));
var compositeWithErrorList = new ArrayList<Composite>();
futureMap.get(false).forEach(l -> {
try {
compositeWithErrorList.addAll(l.get());
} catch (InterruptedException | ExecutionException e) {
e.printStackTrace();
}
});
after the code is executed you will have a map of done and undone futures.
I have a java spring integration project that is receving emails through the below code:
ClassPathXmlApplicationContext ac =
new ClassPathXmlApplicationContext(
"/integration/gmail-imap-idle-config.xml");
DirectChannel inputChannel = ac.getBean("receiveChannel", DirectChannel.class);
inputChannel.subscribe(message -> {
org.springframework.messaging.Message<MimeMailMessage> received =
(org.springframework.messaging.Message<MimeMailMessage>) message;
log.info("content" + message);
List<String> sentences = null;
try {
} catch (Exception e) {
}
I get the email, and I can get the subject, but I can never actually extract the message body. How do I do this?
Thank you!
You have to use this option on the channel adapter:
simple-content="true"
See its description:
When 'true', messages produced by the source will be rendered by 'MimeMessage.getContent()'
which is usually just the body for a simple text email. When false (default) the content
is rendered by the 'getContent()' method on the actual message returned by the underlying
javamail implementation.
For example, an IMAP message is rendered with some message headers.
This attribute is provided so that users can enable the previous behavior, which just
rendered the body.
But still it is doubtful, since I see in case of GMail message it is never simple. The content is a MimeMultipart and we need to read its parts to get access to the real body.
So, this is how you should change your code as well:
log.info("content" + ((MimeMultipart) ((MimeMessage) message.getPayload()).getContent()).getBodyPart(0).getContent());
I need to send a gzipped byte array over HTTPS. I searched the web and only thing ı can found is SharpGIS.GZipWebClient.
However, the problem is - this third party solution only works with WebClient which allow you to send only String data.
(I'm on Windows Phone 8. Most of the WebClient methods do not exist.)
Any ideas to solve this problem?
Edit:
This is how I tried the POST JSON data over HTTPS using SharpGIS;
WebClient webClient = new SharpGIS.GZipWebClient();
webClient.Headers["Accept-Encoding"] = "gzip";
var uri = new Uri(pUrl, UriKind.Absolute);
webClient.UploadStringCompleted += new UploadStringCompletedEventHandler(wc_UploadStringCompleted);
webClient.UploadStringTaskAsync(uri, jsonAsString);
But it doesn't compresses the string as well(as using OpenWriteSync method).
You write the post data in the OpenWriteCompleted handler, like this:
void webClient_OpenWriteCompleted(object sender, OpenWriteCompletedEventArgs e)
{
Stream s = e.Result;
s.Write(jsonAsByteArray, 0, jsonAsByteArray.Length);
s.Flush();
s.Close();
}
You should also add the appropriate error handling.
My builds have been failing due to some of the integration tests I've been running. I'm stuck on why it won't work. Here is an example of the output:
I'm using Maven to first build, then it calls the JUnit tests. I'm seeing this 401 Unauthorized message in every single test, and I believe that's what is causing the builds to fail. In my mind, this means there are some permissions / authentication parameters that need to be set. Where would I go about doing this in JUnit?
Edit
#Test
public void testXmlHorsesNonRunners() throws Exception {
String servletUrl = SERVER + "sd/date/2013-01-13/horses/nonrunners";
Document results = issueRequest(servletUrl, APPLICATION_XML, false);
assertNotNull(results);
// debugDocument(results, "NonRunners");
String count = getXPathStringValue(
"string(count(hrdg:data/hrdg:meeting/hrdg:event/hrdg:nonrunner/hrdg:selection))",
results);
assertEquals("non runners", "45", count);
}
If you can, try to ignore the detail. Effectively, this is making a request. This is a sample of a test that uses the issueRequest method. This method is what makes HTTP requests. (This is a big method, which is why I didn't post it originally. I'll try to make it as readable as possible.
logger.info("Sending request: " + servletUrl);
HttpGet httpGet = null;
// InputStream is = null;
DefaultHttpClient httpclient = null;
try {
httpclient = new DefaultHttpClient();
doFormLogin(httpclient, servletUrl, acceptMime, isIrishUser);
httpGet = new HttpGet(servletUrl);
httpGet.addHeader("accept", acceptMime);
// but more importantly now add the user agent header
setUserAgent(httpGet, acceptMime);
logger.info("executing request" + httpGet.getRequestLine());
// Execute the request
HttpResponse response = httpclient.execute(httpGet);
// Examine the response status
StatusLine statusLine = response.getStatusLine();
logger.info(statusLine);
switch (statusLine.getStatusCode()) {
case 401:
throw new HttpResponseException(statusLine.getStatusCode(),
"Unauthorized");
case 403:
throw new HttpResponseException(statusLine.getStatusCode(),
"Forbidden");
case 404:
throw new HttpResponseException(statusLine.getStatusCode(),
"Not Found");
default:
if (300 < statusLine.getStatusCode()) {
throw new HttpResponseException(statusLine.getStatusCode(),
"Unexpected Error");
}
}
// Get hold of the response entity
HttpEntity entity = response.getEntity();
Document doc = null;
if (entity != null) {
InputStream instream = entity.getContent();
try {
// debugContent(instream);
doc = documentBuilder.parse(instream);
} catch (IOException ex) {
// In case of an IOException the connection will be released
// back to the connection manager automatically
throw ex;
} catch (RuntimeException ex) {
// In case of an unexpected exception you may want to abort
// the HTTP request in order to shut down the underlying
// connection and release it back to the connection manager.
httpGet.abort();
throw ex;
} finally {
// Closing the input stream will trigger connection release
instream.close();
}
}
return doc;
} finally {
// Release the connection.
closeConnection(httpclient);
}
I notice that your test output shows HTTP/1.1 500 Internal Server Error a couple of lines before the 401 error. I wonder if the root cause could be hiding in there. If I were you I'd try looking for more details about what error happened on the server at that point in the test, to see if it could be responsible for the authentication problem (maybe the failure is in a login controller of some sort, or is causing a session to be cancelled?)
Alternately: it looks like you're using the Apache HttpClient library to do the request, inside the issueRequest method. If you need to include authentication credentials in the request, that would be the code you'd need to change. Here's an example of doing HTTP Basic authentication in HttpClient, if that helps. (And more examples, if that one doesn't.)
(I'd second the observation that this problem probably isn't specific to JUnit. If you need to do more research, I'd suggest learning more about HttpClient, and about what this app expects the browser to send. One possibility: use something like Chrome Dev Tools to peek at your communications with the server when you do this manually, and see if there's anything important that the test isn't doing, or is doing differently.
Once you've figured out how to login, it might make sense to do it in a #Before method in your JUnit test.)
HTTP permission denied has nothing to do with JUnit. You probably need to set your credentials while making the request in the code itself. Show us some code.
Also, unit testing is not really meant to access the internet. Its purpose is for testing small, concise parts of your code which shouldn't rely on any external factors. Integration tests should cover that.
If you can, try to mock your network requests using EasyMock or PowerMock and make them return a resource you would load from your local resources folder (e.g. test/resources).
Actually my question is short.
How can I get a HttpPostedFile from a ASP.NET Web API POST or PUT?
I did see that I can get various information from the Request like Request.Header, Request.Content, Request.Properties. Where in there can I find the file I passed and how can I create a HttpPostedFile from it?
Thanks in advance!
Check out the great article from Henrik Nielsen to post multi-part content (i.e posting a form with file)
UPDATE: Add simple code for a controller to receive a file without multipart content
If you only need your controller to receive a file (i.e. no multipart content), you could do something like the above. The request only contains the file binary and the filename is passed within the URL.
public Task<HttpResponseMessage> Post([FromUri]string filename)
{
Guid uploadedFile = Guid.NewGuid();
Task<HttpResponseMessage> task = Request.Content.ReadAsStreamAsync().ContinueWith<HttpResponseMessage>(t =>
{
if (t.IsFaulted || t.IsCanceled)
throw new HttpResponseException(HttpStatusCode.InternalServerError);
try
{
using (Stream stream = t.Result)
{
//TODO: Write the stream to file system / db as you need
}
}
catch (Exception e)
{
Object o = e;
return Request.CreateResponse(HttpStatusCode.InternalServerError, e.GetBaseException().Message);
}
return Request.CreateResponse(HttpStatusCode.Created, uploadedFile.ToString());
});
return task;
}
Your short question does not have a short answer I am afraid.
ASP.NET Web API exposes you to the wonders of HTTP while ASP.NET MVC abstracted some of it - in this case for HttpPostedFile.
So a bit of background:
HTTP posts where a file is involved usually has multipart form data content. This means that you are mixing different kind of content-type: your normal form data will be sent using formurlencoded while the files will be sent application/octent-stream.
So in Web API, all you have to do is to say
var contents = message.Content.ReadAsMultipartAsync(); // message is HttpRequestMessage
One of the contents will contain your file.