I'm developting a web crawler that will download a PDF file from a website.
I checked the source code of the website before and I discovery that the button which download the PDF is actually a submit input to a form. That form retrives the file in Content-Disposition header in response.
Here's a picture of it:
My question is, how to get this file using web request (or HTML Agility pack). I tried in this way, but the hearder returns null.
HttpWebResponse response = (HttpWebResponse)req.GetResponse();
string file = response.Headers["Content-Disposition"];
Thanks in advance
I already have my answers, here is what I've done to get the file
response = (HttpWebResponse)request.GetResponse();
stream = response.GetResponseStream();
byte[] retorno = ReadToEnd(stream);
response.Close();
stream.Close();
public static byte[] ReadToEnd(System.IO.Stream stream)
{
long originalPosition = 0;
if (stream.CanSeek)
{
originalPosition = stream.Position;
stream.Position = 0;
}
try
{
byte[] readBuffer = new byte[4096];
int totalBytesRead = 0;
int bytesRead;
while ((bytesRead = stream.Read(readBuffer, totalBytesRead, readBuffer.Length - totalBytesRead)) > 0)
{
totalBytesRead += bytesRead;
if (totalBytesRead == readBuffer.Length)
{
int nextByte = stream.ReadByte();
if (nextByte != -1)
{
byte[] temp = new byte[readBuffer.Length * 2];
Buffer.BlockCopy(readBuffer, 0, temp, 0, readBuffer.Length);
Buffer.SetByte(temp, totalBytesRead, (byte)nextByte);
readBuffer = temp;
totalBytesRead++;
}
}
}
byte[] buffer = readBuffer;
if (readBuffer.Length != totalBytesRead)
{
buffer = new byte[totalBytesRead];
Buffer.BlockCopy(readBuffer, 0, buffer, 0, totalBytesRead);
}
return buffer;
}
finally
{
if (stream.CanSeek)
{
stream.Position = originalPosition;
}
}
}
thanks
Related
I am trying to convert a PDF page to an image, to create thumbnails. This is the code that I am using:
PdfRenderer pdfRenderer = new PdfRenderer(GetSeekableFileDescriptor(filePath));
var appDirectory = System.Environment.GetFolderPath(System.Environment.SpecialFolder.MyDocuments);
string fileName = System.IO.Path.GetFileNameWithoutExtension(filePath);
string directoryPath = System.IO.Path.Combine(appDirectory, "thumbnailsTemp", System.IO.Path.GetFileNameWithoutExtension(fileName));
if (!Directory.Exists(directoryPath))
{
Directory.CreateDirectory(directoryPath);
int pageCount = pdfRenderer.PageCount;
for (int i = 0; i < pageCount; i++)
{
Page page = pdfRenderer.OpenPage(i);
Android.Graphics.Bitmap bmp = Android.Graphics.Bitmap.CreateBitmap(page.Width, page.Height, Android.Graphics.Bitmap.Config.Rgb565 or Argb8888);
page.Render(bmp, null, null, PdfRenderMode.ForDisplay);
try
{
using (FileStream output = new FileStream(System.IO.Path.Combine(directoryPath, fileName + "Thumbnails" + i + ".png"), FileMode.Create))
{
bmp.Compress(Android.Graphics.Bitmap.CompressFormat.Png, 100, output);
}
page.Close();
}
catch (Exception ex)
{
//TODO -- GERER CETTE EXPEXPTION
throw new Exception();
}
}
return directoryPath;
}
I tried with ARGB 8888 and that was a success. But the rendering time was too slow for big PDF files. This is why I tried to improve it by changing the format to RGB 565. But my app is crashing with this Execption:
Unsuported pixel format
Any idea to fix this, or how to render a PDF to a bitmap faster? I was looking on google but didn't find a solution related to my code.
UPDATE
I did this but know, my app is crashing at this line of code :
await Task.Run(() =>
{
bytes = page.AsPNG(72);
});
My class :
public async Task<string> GetBitmaps(string filePath)
{
//TODO -- WORK ON THIS
PdfRenderer pdfRenderer = new PdfRenderer(GetSeekableFileDescriptor(filePath));
var appDirectory = System.Environment.GetFolderPath(System.Environment.SpecialFolder.MyDocuments);
string fileName = System.IO.Path.GetFileNameWithoutExtension(filePath);
string directoryPath = System.IO.Path.Combine(appDirectory, "thumbnailsTemp", System.IO.Path.GetFileNameWithoutExtension(fileName));
var stream = new MemoryStream();
using (Stream resourceStream = new FileStream(filePath, FileMode.Open))
{
resourceStream.CopyTo(stream);
}
for (int i = 0; i < pdfRenderer.PageCount; i++)
{
TallComponents.PDF.Rasterizer.Page page = new TallComponents.PDF.Rasterizer.Page(stream, i);
byte[] bytes = null;
await Task.Run(() =>
{
bytes = page.AsPNG(72);
});
using (FileStream output = new FileStream(System.IO.Path.Combine(directoryPath, fileName + "Thumbnails" + i + ".png"), FileMode.Create, FileAccess.Write))
{
output.Write(bytes, 0, bytes.Length);
}
}
return directoryPath;
}
you could draw a PDF page in app by converting a PDF page to a bitmap,here the PDF document itself is embedded as a resource.
var assembly = Assembly.GetExecutingAssembly();
var stream = new MemoryStream();
using (Stream resourceStream = assembly.GetManifestResourceStream("DrawPdf.Android.tiger.pdf"))
{
resourceStream.CopyTo(stream);
}
Page page = new Page(stream, 0);
// render PDF Page object to a Bitmap
byte[] bytes = null;
await Task.Run(() =>
{
bytes = page.AsPNG(72);
});
Bitmap bmp = global::Android.Graphics.BitmapFactory.DecodeByteArray(bytes, 0, bytes.Length);
Good afternoon everyone.
I'm trying to download a few files from a FTP server and I don't know why this isn't working, because the same code worked before. Now when i run the service I get the following error:
Error 1053: The service did not respond to the start or control request in a timely fashion
I've tried doing other Methods like GetFileSize or even the DeleteFile that it's suppose to execute after the DownloadFile and they work, but this one isn't and i can't figure out why. Any ideas? My code is below, many thanks!
FtpWebRequest reqFTP;
reqFTP = (FtpWebRequest)FtpWebRequest.Create(new Uri(server + file));
reqFTP.Credentials = new NetworkCredential(username, password);
reqFTP.KeepAlive = true;
reqFTP.Method = WebRequestMethods.Ftp.DownloadFile;
reqFTP.UseBinary = true;
reqFTP.Proxy = null;
reqFTP.UsePassive = false;
reqFTP.Timeout = -1;
FtpWebResponse response2 = (FtpWebResponse)reqFTP.GetResponse();
Stream responseStream2 = response2.GetResponseStream();
FileStream writeStream = new FileStream(folder + file, FileMode.Create);
int Length = 2048;
Byte[] buffer = new Byte[Length];
int bytesRead = responseStream2.Read(buffer, 0, Length);
while (bytesRead > 0)
{
writeStream.Write(buffer, 0, bytesRead);
bytesRead = responseStream2.Read(buffer, 0, Length);
}
writeStream.Close();
response2.Close();
try
{
string uri2 = server + file;
Uri serverUri2 = new Uri(uri);
if (serverUri2.Scheme != Uri.UriSchemeFtp)
{
return;
}
FtpWebRequest reqFTP2;
reqFTP2 = (FtpWebRequest)FtpWebRequest.Create(new Uri(server + file));
reqFTP2.Credentials = new NetworkCredential(username, password);
reqFTP2.KeepAlive = false;
reqFTP2.Method = WebRequestMethods.Ftp.DeleteFile;
reqFTP2.UseBinary = true;
reqFTP2.Proxy = null;
reqFTP2.UsePassive = false;
FtpWebResponse response3 = (FtpWebResponse)reqFTP2.GetResponse();
Stream responseStream3 = response3.GetResponseStream();
response3.Close();
}
My J2ME midlet could retrieves message in Chinese character from a PHP server but it's garbled. The server basically returns the response string and by detecting the first 2 characters. AA = good, anything else indicates error of which the message is to be passed to the calling function for display
InputStream is = null;
StringBuffer sb = null;
String str = "";
HttpConnection http = null;
DataOutputStream dos = null;
try
{
URL = login.getURL();
URL += ctlFunction + "/" + uriStr;
URL = EncodeURL(URL);
//System.out.println(URL);
if(!ctlFunction.equals("login"))
{
msg += "&user=" + login.getUsername();
msg += "&token=" + login.getToken();
}
msg += "&lang=" + System.getProperty("microedition.locale");
// establish the connection
http = (HttpConnection) Connector.open(URL);
http.setRequestMethod(HttpConnection.POST);
http.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
http.setRequestProperty("Content-length", ""+EncodeURL(msg).getBytes().length);
dos = http.openDataOutputStream();
byte[] request_body = EncodeURL(msg).getBytes();
for (int i = 0; i < request_body.length; i++)
{
dos.writeByte(request_body[i]);
}
// server response
if (http.getResponseCode() == HttpConnection.HTTP_OK)
{
is = http.openDataInputStream();
int length = (int) http.getLength();
if (length != -1)
{
// Read data in one chunk
byte serverData[] = new byte[length];
is.read(serverData);
str = new String(serverData);
}
else // Length not available...
{
ByteArrayOutputStream bStrm = new ByteArrayOutputStream();
int ch;
while ((ch = is.read()) != -1)
bStrm.write(ch);
str = new String(bStrm.toByteArray());
bStrm.close();
}
}
else
{
networkError();
}
}
catch (Exception e)
{
System.err.println("Error3: " + e.toString());
networkError(e.toString());
}
finally
{
if (is != null)
is.close();
if (!str.equals(""))
post = str;
else
networkError();
if (http != null)
http.close();
}
if (post != null)
{
String fate = post.substring(0, 2);
if(fate.equals("AA"))
{
if(ctlFunction.equals("login"))
{
String rawPost = post.substring(2);
Vector v = new Vector();
int index = 0;
//find the first occurrence of the SPLITTER
int endIndex = rawPost.indexOf(SPLITTER, index);
String item = "";
//extract the items until the end of the last SPLITTER found in the rawPost string
while(endIndex != -1)
{
item = rawPost.substring(index, endIndex);
index = endIndex + 1;
endIndex = rawPost.indexOf(SPLITTER, index);
v.addElement(item);
}
//extract the rest of the rawPost (the text item)
item = rawPost.substring(index);
v.addElement(item);
String[] ret = new String[v.size()];
v.copyInto(ret);
login.setToken(ret[0]);
login.setToday(ret[1]);
login.setNextDrawDay(ret[2]);
}
midlet.returnResults(post.substring(2), getCurrentDisplay(), ctlFunction);
}
else
{
String errmessage = post.substring(2);
System.out.println(post);
midlet.showInfo(post, getCurrentDisplay());
}
}
else
{
networkError();
}
On the PHP server, I have set the header to UTF-8 encoding
<?php header("Content-Type:text/plain; charset=utf-8"); ?>
What could possibly be wrong?
I found that this user has the same problem and it's been answered
Reading UTF8 strings from a server through http using MIDP. Kudos to the answer.
I basically edited my MIDP code from
// is = http.openDataInputStream();
// int length = (int) http.getLength();
// if (length != -1)
// {
// // Read data in one chunk
// byte serverData[] = new byte[length];
// is.read(serverData);
// str = new String(serverData);
// }
// else // Length not available...
// {
// ByteArrayOutputStream bStrm = new ByteArrayOutputStream();
// int ch;
// while ((ch = is.read()) != -1)
// bStrm.write(ch);
//
// str = new String(bStrm.toByteArray());
// bStrm.close();
// }
to
Reader r = new InputStreamReader(http.openInputStream(), "UTF-8");
int ch;
while((ch = r.read()) != -1)
str = str + (char)ch;
just wondering though why does reading bytes messes up the UTF-8 characters?
I need to POST a 5MB file from within a ResourceIntensiveTask, where the OS sets a max memory usage of 5MB.
So trying to stream the file directly from storage, but the Stream associated to the HttpWebRequest keeps growing in size. This is the code:
public void writeStream(Stream writer, string filesource, string filename)
{
var store = System.IO.IsolatedStorage.IsolatedStorageFile.GetUserStoreForApplication();
var f = store.OpenFile(filesource, FileMode.Open, FileAccess.Read);
store.Dispose();
byte[] buffer = Encoding.UTF8.GetBytes(String.Format(#"Content-Disposition: form-data; name=""file""; filename=""{0}""\n", filename));
writer.Write(buffer, 0, buffer.Length);
buffer = Encoding.UTF8.GetBytes("Content-Type: application/octet-stream\n");
writer.Write(buffer, 0, buffer.Length);
long initialMemory = Microsoft.Phone.Info.DeviceStatus.ApplicationCurrentMemoryUsage;
buffer = new byte[2048];
int DataRead = 0;
do
{
DataRead = f.Read(buffer, 0, 2048);
if (DataRead > 0)
{
writer.Write(buffer, 0, DataRead);
Array.Clear(buffer, 0, 2048);
}
} while (DataRead > 0);
double increasedMemory = ((double)Microsoft.Phone.Info.DeviceStatus.ApplicationCurrentMemoryUsage - initialMemory) / 1000000;
buffer = Encoding.UTF8.GetBytes("\n--" + boundary + "\n--");
writer.Write(buffer, 0, buffer.Length);
writer.Flush();
}
increasedMemory debug variable is used to get the differential memory before and after the file is read and streamed to the HttpWebRequest, and it gives almost the exact size of the file (5MB) which means the process memory is increasing 5MB.
I am also setting AllowReadStreamBuffering=false to the HttpWebRequest.
How to keep memory low? How to upload large files when memory usage limit is 5MB?
The problem is that without being able to turn off write buffering, the connection to the server is not even made until BeginGetResponse() is called after closing the request stream (verified with WireShark).
The only way I can think of to get around this would be to use sockets directly (although that will be way more complicated if using an SSL connection).
This code works for me and doesn't increase memory usage while sending data to the server. I haven't tested it in a background task but don't see any reason it wouldn't work.
Socket _socket;
const int BUFFERSIZE = 4096;
byte[] writebuffer = new byte[BUFFERSIZE];
string hostName = "www.testdomain.com";
string hostPath = "/test/testupload.aspx";
IsolatedStorageFileStream isoFile;
public void SocketPOST(string hostName, string filesource)
{
using (IsolatedStorageFile store = IsolatedStorageFile.GetUserStoreForApplication())
{
if (store.FileExists(filesource))
{
isoFile = store.OpenFile(filesource, FileMode.Open, FileAccess.Read);
}
}
_socket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
_socket.SetNetworkRequirement(NetworkSelectionCharacteristics.NonCellular);
SocketAsyncEventArgs socketEventArg = new SocketAsyncEventArgs();
socketEventArg.RemoteEndPoint = new DnsEndPoint(hostName, 80);
socketEventArg.Completed += new EventHandler<SocketAsyncEventArgs>(Socket_Completed);
_socket.ConnectAsync(socketEventArg);
}
private void Socket_Completed(object sender, SocketAsyncEventArgs e)
{
if (e.SocketError == SocketError.Success)
{
switch (e.LastOperation)
{
case SocketAsyncOperation.Connect: // Connected so started sending data, headers first
if (e.ConnectSocket.Connected)
{
StringBuilder sbHeaders = new StringBuilder("POST " + hostPath + " HTTP/1.1\r\n");
sbHeaders.Append("HOST: " + hostName + "\r\n");
sbHeaders.Append("USER-AGENT: MyWP7App/1.0\r\n");
sbHeaders.Append("Content-Type: text/plain; charset=\"utf-8\"\r\n");
sbHeaders.Append("Content-Length: " + isoFile.Length.ToString() + "\r\n\r\n");
byte[] headerBuffer = Encoding.UTF8.GetBytes(sbHeaders.ToString());
e.SetBuffer(headerBuffer, 0, headerBuffer.Length);
if (!e.ConnectSocket.SendAsync(e)) Socket_Completed(e.ConnectSocket, e);
}
break;
case SocketAsyncOperation.Send:
case SocketAsyncOperation.SendTo: // Previous buffer sent so send next one if stream not finished
Array.Clear(writebuffer, 0, BUFFERSIZE);
int DataRead = 0;
DataRead = isoFile.Read(writebuffer, 0, BUFFERSIZE);
if (DataRead > 0)
{
e.SetBuffer(writebuffer, 0, DataRead);
if (!_socket.SendAsync(e)) Socket_Completed(e.ConnectSocket, e);
}
else
{
isoFile.Dispose();
if (!_socket.ReceiveAsync(e)) Socket_Completed(e.ConnectSocket, e);
}
break;
case SocketAsyncOperation.Receive:
case SocketAsyncOperation.ReceiveFrom:
if (e.BytesTransferred > 0)
{
string response = Encoding.UTF8.GetString(e.Buffer, e.Offset, e.BytesTransferred).Trim('\0');
// Check response if necessary
e.ConnectSocket.Shutdown(SocketShutdown.Both);
e.ConnectSocket.Dispose();
}
break;
default:
break;
}
}
}
Note: I've left a lot of the error handling out to keep the example short.
SSL Note: Because SSL works at the TCP level and WP7 doesn't currently support SSL sockets (SslStream) you would need to handle the certificate handshake, cipher exchange, etc yourself to set up the SSL connection on the socket and then encrypt everything being sent (and decrypt everything received) with the agreed algorithms. There has been some success using the Bouncy Castle API so that could be possible (see this blog post).
One thing I noticed: you forgot to dispose f!
I personally would use the code like this:
public void writeStream(Stream writer, string filesource, string filename)
{
using (var store = System.IO.IsolatedStorage.IsolatedStorageFile.GetUserStoreForApplication())
{
long initialMemory = Microsoft.Phone.Info.DeviceStatus.ApplicationCurrentMemoryUsage;
using (var f = store.OpenFile(filesource, FileMode.Open, FileAccess.Read))
{
byte[] buffer = Encoding.UTF8.GetBytes(string.Format(#"Content-Disposition: form-data; name=""file""; filename=""{0}""\n", filename));
writer.Write(buffer, 0, buffer.Length);
buffer = Encoding.UTF8.GetBytes("Content-Type: application/octet-stream\n");
writer.Write(buffer, 0, buffer.Length);
buffer = new byte[2048];
int DataRead = 0;
do
{
DataRead = f.Read(buffer, 0, 2048);
if (DataRead > 0)
{
writer.Write(buffer, 0, DataRead);
}
} while (DataRead > 0);
buffer = Encoding.UTF8.GetBytes("\n--" + boundary + "\n--");
writer.Write(buffer, 0, buffer.Length);
writer.Flush();
}
double increasedMemory = ((double)Microsoft.Phone.Info.DeviceStatus.ApplicationCurrentMemoryUsage - initialMemory) / 1000000;
}
}
The boundary var seems to be missing, so a coding error still remains here!
i get the stream from webSite ,then put it in isolatedStorage into IsolatedstorageStream ,
but it don't work ,no error no sound , what's wrong ????
HttpWebResponse reponse = request.EndGetResponse(result) as HttpWebResponse;
if (reponse.StatusCode == HttpStatusCode.OK)
{
Stream stream=reponse.GetResponseStream();
SaveMusic(stream, "music");
ReadMusic("music");
Deployment.Current.Dispatcher.BeginInvoke(
() =>
{
me.AutoPlay = true;
me.Volume = 100;
me.SetSource(songStream);
me.Play();
});
}
ok thanks keyboardP for your help ;here is my code
protected void SaveMusic(Stream stream,string name)
{
IsolatedStorageFile fileStorage = IsolatedStorageFile.GetUserStoreForApplication();
if (!fileStorage.DirectoryExists("Source/Music"))
{
fileStorage.CreateDirectory("Source/Music");
}
using (IsolatedStorageFileStream fileStream = IsolatedStorageFile.GetUserStoreForApplication().OpenFile("Source\\Music\\" + name + ".mp3", FileMode.Create))
{
byte[] bytes = new byte[stream.Length];
stream.Read(bytes, 0, bytes.Length);
fileStream.Write(bytes, 0, bytes.Length);
fileStream.Flush();
}
}
protected void ReadMusic(string name)
{
using (IsolatedStorageFile fileStorage = IsolatedStorageFile.GetUserStoreForApplication())
{
songStream = null;
songStream = new IsolatedStorageFileStream("Source\\Music\\" + name + ".mp3", FileMode.Open, fileStorage);
}
}
Assuming your saving and reading code is correct, your stream's position might be at the end. Try adding
songStream.Position = 0;
before SetSource(songStream);
Try using this to save the file:
using (var fileStorage = IsolatedStorageFile.GetUserStoreForApplication())
{
var buffer = new byte[1024];
using (var myIsStream = fileStorage.OpenFile("Source\\Music\\" + name + ".mp3", FileMode.CreateNew))
{
int bytesRead = 0;
while ((bytesRead = stream.Read(buffer, 0, 1024)) > 0)
myIsStream.Write(buffer, 0, bytesRead);
}
}