How can I read a file from 100th byte, using ReadFile function and C++
I used this code but it reads the first 100 bytes of the file
I want to read the second 100 bytes
hndl = CreateFileW(L"1.txt", GENERIC_READ, 0, NULL, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, NULL);
SetFilePointer(hndl, 100, NULL, FILE_BEGIN);
ReadFile(hndl, pbytReadBuffer, 100, NULL, &ol);
The ReadFile API offers two distinct ways to set the starting offset when doing synchronous I/O:
By using the implicitly stored file pointer (which can be manipulated through the SetFilePointer API call).
Or by passing an explicit offset through the OVERLAPPED structure.
Your code fails because you are setting the implicitly stored file pointer, but then (presumably) pass a zero-initialized OVERLAPPED structure, which ignores the file pointer (see Synchronization and File Position for more information).
Either of the following solutions will work. First, by using the implicitly stored file pointer. This is useful, when you want to read chunks of a file in consecutive calls:
hndl = CreateFileW(L"1.txt", GENERIC_READ, 0, nullptr,
OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, nullptr);
// Move the file pointer to offset 100
SetFilePointer(hndl, 100, NULL, FILE_BEGIN);
// Read contents from the current offset
DWORD dwBytesRead{0};
ReadFile(hndl, pbytReadBuffer, 100, &dwBytesRead, nullptr);
Alternatively, you can pass an OVERLAPPED structure to pass the offset. This ignores the implicitly stored file pointer. It is slightly more efficient, because it doesn't need two calls to the file I/O API.
hndl = CreateFileW(L"1.txt", GENERIC_READ, 0, nullptr,
OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, nullptr);
OVERLAPPED ol{};
// Set the offset from the start of the file
ol.Offset = 100;
ReadFile(hndl, pbytReadBuffer, 100, nullptr, &ol);
Note that error handling is elided in these samples for brevity. In real code, you always must check for errors.
you need code like this
HANDLE hFile = CreateFile(L"c:\\windows\\notepad.exe", FILE_GENERIC_READ, FILE_SHARE_READ, 0, OPEN_EXISTING, 0, 0);
if (hFile != INVALID_HANDLE_VALUE)
{
OVERLAPPED ov = {};
ov.Offset = 100;
UCHAR buf[100];
ULONG cb;
if (ReadFile(hFile, buf, sizeof(buf), &cb, &ov))
{
}
else
{
GetLastError();
}
CloseHandle(hFile);
}
you not need call SetFilePointer - much better set offset direct in OVERLAPPED structure
EDIT
for every read/write operation we need to specify the starting byte offset. we can do this directed by setting values in Offset and OffsetHigh from OVERLAPPED.
or indirect only if file opened if synchronous mode - I/O Manager can use the current file position in FILE_OBJECT - so we can not direct set offset - it will be get from FILE_OBJECT.CurrentByteOffset. FILE_OBJECT.CurrentByteOffset we can set with SetFilePointer also every read/write operation update this offset - move forward to count of bytes which readed/ written. of course this is correct only if file used in synchronous mode when all operations with file is sequential
if we use direct offset in OVERLAPPED he and used - so FILE_OBJECT.CurrentByteOffset is ignored - this mean that previous call to SetFilePointer - also lost all effect - will be used offset from OVERLAPPED and after read operation FILE_OBJECT.CurrentByteOffset will be updated based on offset + bytes readed
Related
I know lots of similar questions on this topic have been asked before but so far I have been unable to find a solution that actually works. I want to start a console program from my program and capture its output. My implementation should be in a way that is compatible with WaitForMultipleObjects(), i.e. I want to get notified whenever there is new data to read in the pipe.
My implementation is based on this example from MSDN. However, I had to modify it a little because I need overlapped I/O in order to be able to wait for ReadFile() to finish. So I'm using named pipes created using Dave Hart's MyCreatePipeEx() function from here.
This is my actual code. I have removed error checks for readability reasons.
HANDLE hReadEvent;
HANDLE hStdIn_Rd, hStdIn_Wr;
HANDLE hStdOut_Rd, hStdOut_Wr;
SECURITY_ATTRIBUTES saAttr;
PROCESS_INFORMATION piProcInfo;
STARTUPINFO siStartInfo;
OVERLAPPED ovl;
HANDLE hEvt[2];
DWORD mask, gotbytes;
BYTE buf[4097];
saAttr.nLength = sizeof(SECURITY_ATTRIBUTES);
saAttr.bInheritHandle = TRUE;
saAttr.lpSecurityDescriptor = NULL;
MyCreatePipeEx(&hStdOut_Rd, &hStdOut_Wr, &saAttr, 0, FILE_FLAG_OVERLAPPED, FILE_FLAG_OVERLAPPED);
MyCreatePipeEx(&hStdIn_Rd, &hStdIn_Wr, &saAttr, 0, FILE_FLAG_OVERLAPPED, FILE_FLAG_OVERLAPPED);
SetHandleInformation(hStdOut_Rd, HANDLE_FLAG_INHERIT, 0);
SetHandleInformation(hStdIn_Wr, HANDLE_FLAG_INHERIT, 0);
memset(&piProcInfo, 0, sizeof(PROCESS_INFORMATION));
memset(&siStartInfo, 0, sizeof(STARTUPINFO));
siStartInfo.cb = sizeof(STARTUPINFO);
siStartInfo.hStdError = hStdOut_Wr;
siStartInfo.hStdOutput = hStdOut_Wr;
siStartInfo.hStdInput = hStdIn_Rd;
siStartInfo.dwFlags |= STARTF_USESTDHANDLES;
CreateProcess(NULL, "test.exe", NULL, NULL, TRUE, 0, NULL, NULL, &siStartInfo, &piProcInfo);
hReadEvent = CreateEvent(NULL, TRUE, FALSE, NULL);
for(;;) {
int i = 0;
hEvt[i++] = piProcInfo.hProcess;
memset(&ovl, 0, sizeof(OVERLAPPED));
ovl.hEvent = hReadEvent;
if(!ReadFile(hStdOut_Rd, buf, 4096, &gotbytes, &ovl)) {
if(GetLastError() == ERROR_IO_PENDING) hEvt[i++] = hReadEvent;
} else {
buf[gotbytes] = 0;
printf("%s", buf);
}
mask = WaitForMultipleObjects(i, hEvt, FALSE, INFINITE);
if(mask == WAIT_OBJECT_0 + 1) {
if(GetOverlappedResult(hStdOut_Rd, &ovl, &gotbytes, FALSE)) {
buf[gotbytes] = 0;
printf("%s", buf);
}
} else if(mask == WAIT_OBJECT_0) {
break;
}
}
The problem with this code is the following: As you can see, I'm reading in chunks of 4kb using ReadFile() because I obviously don't know how much data the external program test.exe will output. Doing it this way was suggested here:
To read a variable amount of data from the client process just issue
read requests of whatever size you find convenient and be prepared to
handle read events that are shorter than you requested. Don't
interpret a short but non-zero length as EOF. Keep issuing read
requests until you get a zero length read or an error.
However, this doesn't work. The event object passed to ReadFile() as part of the OVERLAPPED structure will only trigger once there are 4kb in the buffer. If the external program just prints "Hello", the event won't trigger at all. There need to be 4kb in the buffer for hReadEvent to actually trigger.
So I thought I should read byte by byte instead and modified my program to use ReadFile() like this:
if(!ReadFile(hStdOut_Rd, buf, 1, &gotbytes, &ovl)) {
However, this doesn't work either. If I do it like this, the read event is not triggered at all which is really confusing me. When using 4096 bytes, the event does indeed trigger as soon as there are 4096 bytes in the pipe, but when using 1 byte it doesn't work at all.
So how am I supposed to solve this? I'm pretty much out of ideas here. Is there no way to have the ReadFile() event trigger whenever there is some new data in the pipe? Can't be that difficult, can it?
Just for the record, while there are some problems with my code (see discussion in comments below the OP), the general problem is that it's not really possible to capture the output of arbitrary external programs because they will typically use block buffering when their output is redirected to a pipe, which means that output will only arrive at the capturing program once that buffer is flushed so real time capturing is not really possible.
Some workarounds have been suggested though:
1) (Windows) Here is a workaround that uses GetConsoleScreenBuffer() to capture the output from arbitrary console programs but it currently only supports one console page of output.
2) (Linux) On Linux, it's apparently possible to use pseudo-terminals to force the external program to use unbuffered output. Here is an example.
I'm reading a volume (logical drive) with ReadFile. I'm using DeviceIoControl with FSCTL_ALLOW_EXTENDED_DASD_IO code, because I want to have access to all (including the last) bytes and had an issue trying to read last 512 bytes (ReadFile successed, but reported 0 bytes read) and saw advice to use it. Unfortunately, ReadFile fails being called after that DeviceIoControl called.
In code it looks like this (all success checks are omitted for the brevity):
HANDLE fd;
DWORD junk;
int lenToBeRead = 0x1000;
DWORD nread;
char* alignedBuf = new char[lenToBeRead];
fd = CreateFile("path to volume", FILE_READ_DATA,
FILE_SHARE_READ | FILE_SHARE_WRITE,
NULL, OPEN_EXISTING, 0, NULL)) //success
DeviceIoControl(fd, FSCTL_ALLOW_EXTENDED_DASD_IO,
NULL, 0, NULL, 0, &junk, (LPOVERLAPPED) NULL) //success
ReadFile(fd, alignedBuf, (DWORD) lenToBeRead, &nread, NULL)
// fails with 0x57 code, ERROR_INVALID_PARAMETER
All work with fd handle is synchronous.
EDIT. I solved the problem. I was trying to read last bytes. So my volume was of length L = 0x...200 and I had my handle on position pos = L - 0x200. What I had done before I did the FSCTL_ALLOW_EXTENDED_DASD_IO thing - I cut lenToBeRead to fit in remaining space (so, if it was 0x1000, it would change to 0x200), because I had found that ReadFile did not guarantee read all the bytes to the EOF in case of lenToBeRead is greater than amount of bytes remained from current handle position. This did not help, ReadFilewas still returning with success and 0 bytes read. I deleted that fix and then used FSCTL_ALLOW_EXTENDED_DASD_IO, which deliver me then ReadFile failing with ERROR_INVALID_PARAMETER on lenToBeRead = 0x1000. I totally forgot about the first fix and remembered now and now it works.
I found the solution and add it to the question body.
What one has to keep in mind when working with ReadFile is to control arguments (length) to not cross the boundary of the file.
I had tried it as a fix before doing the FSCTL_ALLOW_EXTENDED_DASD_IO thing and it did not help. But combination of the FSCTL_ALLOW_EXTENDED_DASD_IO thing and the boundary check gave me wanted result - I could read that last bytes.
BOOL SetDeviceID(HANDLE device,char *id){//
char data[2];
data[0]=0x02;
data[1]=0x27;
DWORD dwPtr=SetFilePointer(device,0x33,//distance
NULL,//
FILE_BEGIN);
if(dwPtr==INVALID_SET_FILE_POINTER) cout<<GetLastError()<<endl;
BOOL result=WriteFile(device,data,2,NULL,NULL);
//cout<<GetLastError()<<endl;
if(result==false)cout<<"Fail WRITE "<<endl;
return TRUE;
}
HANDLE GetDeviceHandle(char *path){
HANDLE handle= CreateFile(LPCSTR(path),
GENERIC_ALL,//
0,
NULL,
OPEN_EXISTING,
NULL,
NULL);
if(handle==INVALID_HANDLE_VALUE){
cout<<"fail to createfile()"<<endl;
exit(1);
}
else return handle;
}
this is some code of my works.
I am going to read/write directly device(usb)
on ReadFile() case, It was successful.
But, I have tried to call SetFilePointer
But GetLastError returned 87. it means invalid input
What is the problem? on my code
shortly, CreateFile,ReadFile is ok but SetFilePointer and WriteFile failed
When you are directly accessing a disk device you cannot seek to positions in the middle of a sector. The position must always be a multiple of the sector length. And 0x33 is not a mutiple of your sector length.
What you will need to do is read an entire sector. Modify the bytes that need to be modified. And finally write back the entire sector.
I am trying to get simple file IO working in Win32. So far the writing is working fine, but the reading is not: although it successfully reads the contents, additional "garbage" is appended to the string. The code I have so far is below. The program has UNICODE defined.
For writing:
DWORD dwTextSize = GetWindowTextLength(hWndTextBox);
WCHAR *lpszText = new WCHAR[dwTextSize];
GetWindowText(hWndTextBox, lpszText, dwTextSize + 1);
hTextFile = CreateFile(lpszTextFileName, GENERIC_WRITE, 0, NULL, CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL);
DWORD dwBytesWritten;
WriteFile(hTextFile, lpszText, 2 * dwTextSize, &dwBytesWritten, NULL); // x2 for 2 bytes per Unicode character
CloseHandle(hTextFile);
DeleteObject(hTextFile);
For this example, Hello, World! is saved successfully as Hello, World!.
For reading:
lpszTextFileName = L"text.txt"; // LPCTSTR Variable
hTextFile = CreateFile(lpszTextFileName, GENERIC_READ, 0, NULL, OPEN_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL);
DWORD dwFileSize = GetFileSize(hTextFile, &dwFileSize);
DWORD dwBytesRead;
WCHAR *lpszText = new WCHAR[dwFileSize / 2];
ReadFile(hTextFile, lpszText, dwFileSize, &dwBytesRead, NULL);
CloseHandle(hTextFile);
The string is then used to set the text of an EDIT control:
SendMessage(hWndTextBox, WM_SETTEXT, NULL, (LPARAM)lpszText); // SetWindowText() also possible
When Hello, World! is read back in, it reads back in as Hello, World!﷽﷽ꮫꮫꮫꮫﻮ or a visual variation upon this, but basically "garbage"!
I have probably missed something rather obvious, but I cannot see where! Is there a solution to this problem and if so, what is it?
Ok I started this with a comment, but its getting out of control.
For Writing
This:
WCHAR *lpszText = new WCHAR[dwTextSize];
should be:
WCHAR *lpszText = new WCHAR[dwTextSize+1];
This:
DeleteObject(hTextFile);
should not be there at all.. Get rid of it.
I'm assuming you delete [] lpszText; somewhere when you're done with it. if not, do so.
For Reading
The second parameter to GetFileSize() should not be the same variable as your return value. It is the HIGH 32bit of a 64-bit value for large file sizes. If you know you're file size is smaller than 4gB, you can leave it NULL, so change this:
DWORD dwFileSize = GetFileSize(hTextFile, &dwFileSize);
to this:
DWORD dwFileSize = GetFileSize(hTextFile, NULL);
You must account for the null terminator of your file buffer, so this:
WCHAR *lpszText = new WCHAR[dwFileSize / 2];
should be changed to this:
WCHAR *lpszText = new WCHAR[dwFileSize / 2 + 1];
lpszText[dwFileSize / 2] = 0;
and the rest should work as you're hoping it would. No error checking, which is not good, but I've seen worse. And as before, I'm assuming you delete [] lpszText; somewhere when you're done with it. if not, do so.
I am trying to use combination of functions CreateFileMapping , MapViewOfFile, FlushViewOfFile.
the total buffer size is more than the mapped view.
example buffer is 50KB. and mapped view is 2KB. in such scenario,
i want to write the total buffer to a physical file, using the above function.
First part i am able to write to file. but the remaining part how to write to file. I mean, how to move to next page and write the next part of data.
#define MEM_UNIT_SIZE 100
-first module...Memory map creator
GetTempPath (256, szTmpFile);
GetTempFileName (szTmpFile, pName, 0, szMMFile);
hFile = CreateFile (szMMFile, GENERIC_WRITE | GENERIC_READ, FILE_SHARE_WRITE,
NULL, CREATE_ALWAYS, FILE_ATTRIBUTE_TEMPORARY, NULL);
HANDLE hFileMMF = CreateFileMapping( hFile ,NULL,PAGE_READWRITE,0,
(MEM_UNIT_SIZE),pName)
-second module... Memory writer
long lBinarySize = 1000;
long lPageSize = MEM_UNIT_SIZE;
HANDLE hFileMMF = OpenFileMapping(FILE_MAP_WRITE,FALSE,pMemName);
LPVOID pViewMMFFile = MapViewOfFile(hFileMMF,FILE_MAP_WRITE,0,0, lPageSize );
CMutex mutex (FALSE, _T("Writer"));
mutex.Lock();
try
{
ASSERT(FALSE);
CopyMemory(pViewMMFFile,pBinary,lPageSize); // write
FlushViewOfFile(pViewMMFFile,lPageSize);
// first 100 bytes flushed to file.
//how to move to next location and write next 900 bytes..<---??
}
catch(CException e)
{
...
}
please share if you have any suggestion.
thanks in advance,
haranadh
Repeat your call to MapViewOfFile with a different range.
as described in the following link,
http://msdn.microsoft.com/en-us/library/windows/desktop/aa366761(v=VS.85).aspx
can you please check "allocation granularity", I think you should use this parameter to set the values for "dwFileOffsetLow" or "dwFileOffsetHigh".