I am building a filter driver in NDIS 6. I want to add some data at the back of my NET_BUFFER.
To add data I have allocated an MDL through the NdisAllocateMdl API. Now I want to know, is there any wrapper to add this MDL to the existing MDL chain?
If not, is it correct to make the last MDL's Next pointer to point to my new allocated MDL, as I was able to do so? Also what are the fields in NET_BUFFER I have to change to make it recognize the added MDL?
The actual packet payload of an NET_BUFFER is a subset of the payload of the entire MDL chain. The packet might not start at the beginning of the MDL chain, and the packet might not end at the end of the MDL chain.
Therefore, in the most general case, you'll actually need to remove some MDLs from the end of the NET_BUFFER, before you append your new MDL. Let me give a specific example:
NET_BUFFER
* DataOffset=300 bytes
* DataLength=200 bytes
* MdlChain=[200 bytes]->[200 bytes]->[300 bytes]->[200 bytes]
So in this example, the NET_BUFFER points to an MDL chain with 4 MDLs in it. Let's look at the buffers in ASCII-art:
0 100 200 300 400 500 600 700 800 900
| | | | | | | | | |
[ First MDL ][ Second MDL ][ Third MDL ][ Fourth MDL ]
↑ ↑ [ Packet ] ↑
| | |
| | ↑ ↑ |
| | | | |
MdlChain | DataOffset DataLength End-of-MDL-chain
|
CurrentMdl
As hopefully the diagram illustrates, the actual packet payload is only spread across the second & third MDLs. Both the first & fourth MDLs are completely ignored. So if you want to append data to the packet, you need to:
Remove all MDLs from the end of the MDL chain, where the last byte of the MDL's buffer isn't part of the logical packet buffer. (In the example above, remove both the third & fourth MDLs).
Allocate a new buffer that's big enough to hold your data, plus any actual packet payload you deleted from step #1 above. (In the example, you'd have to allocate an extra 100 bytes, to hold the first 100 bytes from the third MDL.)
Chain your new MDL to the end of the chain.
Increment NET_BUFFER::DataLength.
Give the packet to NDIS.
Here's a diagram of what the packet will look like after finishing step 3:
0 100 200 300 400 500 600 700 800 900
| | | | | | | | | |
[ First MDL ][ Second MDL ][ Your MDL ]
↑ ↑ [ Packet ] ↑
| | |
| | ↑ ↑ |
| | | | |
MdlChain | DataOffset DataLength |
| |
CurrentMdl End-of-MDL-chain
Then after step #4:
0 100 200 300 400 500 600 700 800 900
| | | | | | | | | |
[ First MDL ][ Second MDL ][ Your MDL ]
↑ ↑ [ Packet ]
| |
| | ↑ ↑
| | | |
MdlChain | DataOffset DataLength
| |
CurrentMdl End-of-MDL-chain
When the packet is completed back to your driver, you need to undo the modifications you made earlier:
Decrement DataLength.
Remove the MDL you added.
Free the MDL you added.
Restore the original MDLs you removed.
Related
I'm working in R with the arrow package. I have multiple tsv files:
sample1.tsv
sample2.tsv
sample3.tsv
sample4.tsv
sample5.tsv
...
sample50.tsv
each of the form
| id | start| end | value|
| --- | -----|-----|------|
| id1 | 1 | 3 | 0.2 |
| id2 | 4 | 6 | 0.5 |
| id. | ... | ... | ... |
| id2 | 98 | 100 | 0.5 |
and an index file:
| id | start| end |
| --- | -----|-----|
| id1 | 1 | 3 |
| id2 | 4 | 6 |
| id. | ... | ... |
| id2 | 98 | 100 |
I use the index file to left join on id, start and end with each sample to get a datatable like this:
| id | start| end | sample 1| sample 2| sample ...|
| --- | -----|-----|---------|---------|-----------|
| id1 | 1 | 3 | 0.2 | 0.1 | ... |
| id2 | 4 | 6 | 0.5 | 0.8 | ... |
| id. | ... | ... | ... | ... | ... |
| id2 | 98 | 100 | 0.5 | 0.6 | ... |
With multiple samples. I'd like to read them in chunks (eg: chunk_size=5), and when I have a table of chunk_size samples read, write that joined datatable as a parquet file to disk.
Currently, I'm able to write each chunked datatable to disk and I read them with open_dataset(datadir). In a loop with i as the sample_number:
# read and join
...
if (i %% chunk_size == 0) {
write_parquet(joined_table, paste0("datadir", "chunk", i / chunk_size, ".parquet"))
}
...
# clear the data table of samples
However, even though the arrow package says it read as many files as were written, when I check the columns available, only the columns from the first chunk are found.
data <- arrow::open_dataset("datadir")
data
# FileSystemDataset with 10 Parquet files
# id: string
# start: int32
# end: int32
# sample1: double
# sample2: double
# sample3: double
# sample4: double
# sample5: double
Samples 6-50 are missing. Reading the parquet files individually shows that each contains the samples from their chunk.
data2 <- arrow::open_dataset("datadir/chunk2.parquet")
data2
# FileSystemDataset with 1 Parquet file
# id: string
# start: int32
# end: int32
# sample6: double
# sample7: double
# sample8: double
# sample9: double
# sample10: double
Are parquet files the right format for this task? I'm not sure what I'm missing to make a splintered set of files that are all the same dataset when read in.
I would like to count the same log messages in Kibana. With the Size set to 200, it turns out that there are two results that happened twice
But, if I lower the Size to 5, I don't see those two:
It should show me top 5 rows, ordered by count. I expected something like this:
| LogMessage | Count |
|------------|-------|
| xx | 2 |
| yy | 2 |
| zz | 1 |
| qq | 1 |
| ww | 1 |
What am I missing?
The issue is the little warning about Analyzed Field. You should use a keyword field.
With analyzed fields, the analyzer breaks down the original string during indexing into sub-strings to facilitate search use cases (handling things like word boundaries, punctuation, case insensitivity, declination, etc)
A keyword field is just a simple string.
What's probably happening is that you have data like
| LogMessage | Count |
|------------|-------|
| a | 1 |
| b | 1 |
| c x | 1 |
| d x | 1 |
With an analyzed field, if you have a terms agg of size 2 you might (depending on the sort order) get a and b
With a larger terms agg, the top sub-string will be x
This is a simplified example, but I hope it gets the issue across.
The Terms Aggregation docs have a good section about how to avoid/solve this issue.
I'm implementing my own HTTP/2 server. Read through the RFC 7540, a portion of content about HTTP header frame got me confused:
https://www.rfc-editor.org/rfc/rfc7540#section-4.3
Header lists are collections of zero or more header fields. When
transmitted over a connection, a header list is serialized into a
header block using HTTP header compression [COMPRESSION]. The
serialized header block is then divided into one or more octet
sequences, called header block fragments
So in my imagination, the process would look like:
+-------------------+ +-------------------+ +------------------+ +---------+
| Header List | | Header Block | | Block Fragment 1 | -> | Frame 1 |
+-------------------+ +-------------------+ +------------------+ +---------+
| :method = POST | | Header 1 Bin Data | | Block Fragment 2 | -> | Frame 2 |
| :path = /resource | -> | Header 2 Bin Data | -> +------------------+ +---------+
| :scheme = https | | Header 3 Bin Data | | Block Fragment 3 | -> | Frame 3 |
| ..... | | Header 4 Bin ... | +------------------+ +---------+
+-------------------+ +-------------------+ ... ...
But how the serialized header block got divided does not mentioned in the section.
So my question is: Is that possible to separate/divide one HTTP header into multiple Header Block Fragments? For example, part of Header 1 Bin Data is carried by Block Fragment 1, and the rest is carried by Block Fragment 2.
Thank you!
Yes, it is possible to split the bytes corresponding to 1 HTTP header into 2 (or more) frames.
You encode the HTTP headers into bytes via HPACK, so you have now just a byte[].
Let's assume the byte[] has length 23.
You can create a HEADERS frame with flags end_headers=false and put for example 13 of the 23 bytes into it.
Then, you create a CONTINUATION frame with flags end_headers=true and put the remaining 10 bytes into it.
It is completely opaque where you split the byte[].
The important thing is that the HEADERS and CONTINUATION frames must be sent one after the other, with no other frame between them.
The receiving side will see the HEADERS frame with end_headers=false, will extract the 13 bytes and will put them aside, waiting for a CONTINUATION frame.
When the CONTINUATION frame arrives, the receiving side will extract the 10 bytes, concatenate it with the previous 13 to obtain a copy of the original 23 bytes that can now be decoded via HPACK, obtaining the original HTTP headers.
I'm writing scenarios for QA engineers, and now I face a problem such as step encapsulation.
This is my scenario:
When I open connection
And device are ready to receive files
I send to device file with params:
| name | ololo |
| type | txt |
| size | 123 |
All of each steps are important for people, who will use my steps.
And I need to automate this steps and repeat it 100 times.
So I decide create new step, which run it 100 times.
First variant was to create step with other steps inside like:
Then I open connection, check device are ready and send file with params 100 times:
| name | ololo |
| type | txt |
| size | 123 |
But this version is not appropriate, because:
people who will use it, will don't understand which steps executes inside
and some times name of steps like this are to long
Second variant was to create step with other steps in parameters table:
I execute following steps 100 times:
| When I open connection |
| And device are ready to receive files |
| I send to device file |
It will be easy to understand for people, who will use me steps and scenarios.
But also I have some steps with parameters,
and I need to create something like two tier table:
I execute following steps 100 times:
| When I open connection |
| And device are ready to receive files |
| I send to device file with params: |
| | name | ololo | |
| | type | txt | |
| | size | 123 | |
This is the best variant in my situation.
But of cause cucumber can't parse it without errors ( it's not correct as cucumber code ).
How can I fix last example of step? (mark with bold font)
Does cucumber have some instruments, which helps me?
Can you suggest some suggest your type of solution?
Does someone have similar problems?
I decide to change symbols "|" to "/" in parameters table, which are inside.
It's not perfect, but it works:
This is scenarios steps:
I execute following steps 100 times:
| I open connection |
| device are ready to receive files |
| I send to device file with params: |
| / name / ololo / |
| / type / txt / |
| / size / 123 / |
This is step definition:
And /^I execute following steps (.*) times:$/ do |number, table|
data = table.raw.map{ |raw| raw.last }
number.to_i.times do
params = []
step_name = ''
data.each_with_index do |line,index|
next_is_not_param = data[index+1].nil? || ( data[index+1] && !data[index+1].include?('/') )
if !line.include?('/')
step_name = line
#p step_name if next_is_not_param
step step_name if next_is_not_param
else
params += [line.gsub('/','|')]
if next_is_not_param
step_table = Cucumber::Ast::Table.parse( params.join("\n"), nil, nil )
#p step_name
#p step_table
step step_name, step_table
params = []
end
end
end
#p '---------------------------------------------------------'
end
end
A number of students want to get into sections for a class, some are already signed up for one section but want to change section, so they all get on the wait lists. A student can get into a new section only if someone drops from that section. No students are willing to drop a section they are already in unless that can be sure to get into a section they are waiting for. The wait list for each section is first come first serve.
Get as many students into their desired sections as you can.
The stated problem can quickly devolve to a gridlock scenario. My question is; are there known solutions to this problem?
One trivial solution would be to take each section in turn and force the first student from the waiting list into the section and then check if someone end up dropping out when things are resolved (O(n) or more on the number of section). This would work for some cases but I think that there might be better options involving forcing more than one student into a section (O(n) or more on the student count) and/or operating on more than one section at a time (O(bad) :-)
Well, this just comes down to finding cycles in the directed graph of classes right? each link is a student that wants to go from one node to another, and any time you find a cycle, you delete it, because those students can resolve their needs with each other. You're finished when you're out of cycles.
Ok, lets try. We have 8 students (1..8) and 4 sections. Each student is in a section and each section has room for 2 students. Most students want to switch but not all.
In the table below, we see the students their current section, their required section and the position on the queue (if any).
+------+-----+-----+-----+
| stud | now | req | que |
+------+-----+-----+-----+
| 1 | A | D | 2 |
| 2 | A | D | 1 |
| 3 | B | B | - |
| 4 | B | A | 2 |
| 5 | C | A | 1 |
| 6 | C | C | - |
| 7 | D | C | 1 |
| 8 | D | B | 1 |
+------+-----+-----+-----+
We can present this information in a graph:
+-----+ +-----+ +-----+
| C |---[5]--->1| A |2<---[4]---| B |
+-----+ +-----+ +-----+
1 | | 1
^ | | ^
| [1] [2] |
| | | |
[7] | | [8]
| V V |
| 2 1 |
| +-----+ |
\--------------| D |--------------/
+-----+
We try to find a section with a vacancy, but we find none. So because all sections are full, we need a dirty trick. So lets take a random section with a non empty queue. In this case section A and assume, it has an extra position. This means student 5 can enter section A, leaving a vacancy at section C which is taken by student 7. This leaves a vacancy in section D which is taken by student 2. We now have a vacancy at section A. But we assumed that section A has an extra position, so we can remove this assumption and have gained a simpler graph.
If the path never returned to section A, undo the moves and mark A as an invalid startingpoint. Retry with another section.
If there are no valid sections left we are finished.
Right now we have the following situation:
+-----+ +-----+ +-----+
| C | | A |1<---[4]---| B |
+-----+ +-----+ +-----+
| 1
| ^
[1] |
| |
| [8]
V |
1 |
+-----+ |
| D |--------------/
+-----+
We repeat the trick with another random section, and this solves the graph.
If you start with several students currently not assigned, you add an extra dummy section as their startingpoint. Of course, this means that there must be vacancies in any sections or the problem is not solvable.
Note that due to the order in the queue, it can be possible that there is no solution.
This is actually a Graph problem. You can think of each of these waiting list dependencies as edges on a directed graph. If this graph has a cycle, then you have one of the situations you described. Once you have identified a cycle, you can chose any point to "break" the cycle by "over filling" one of the classes, and you will know that things will settle correctly because there was a cycle in the graph.