Elm byte decoding within Http.get - byte

I'm pretty new to elm and stuck with a problem regarding populating my model with data from my backend. I'm curently able to make a get request to the server which returns a byte[] (the data is any kind of image,audio or video), which works fine when just displaying this data by for example Html.img. When I try to use Http.get (src: https://package.elm-lang.org/packages/elm/http/latest/Http) to populate my model it requires a decoder. The problem is, that Bytes.Decode.bytes requires an Int to know how many bytes have to be decoded.
So my question is: Is there any way to access the byte width while still matching the type pattern of Http.get?
Here's a simple example of my problem:
import Bytes exposing (Bytes)
import Bytes.Decode exposing (Decoder, bytes, decode)
import GeneralTypes exposing (Msg(..))
import Http
getMediaFromUrl : Cmd Msg
getMediaFromUrl = Http.get
{ url = "http://localhost:8090/image/2006/aa#a.de/session"
, expect = Http.expectBytes GetThumbnail decodeBytes
}
decodeBytes: Bytes.Bytes -> Decoder Bytes
decodeBytes bytesToDecode= let
fileSize =
bytesToDecode |> Bytes.width
in
Bytes.Decode.bytes fileSize
module GeneralTypes exposing (..)
import Bytes exposing (Bytes)
import Http
type Msg = GetThumbnail (Result Http.Error Bytes)

The expectBytes function requires you to specify a Bytes decoder, which is useful if you immediately want to translate the bytes into something more meaningful in your code.
If, however, you want to retain the raw Bytes in your application without having to clone or otherwise read the bytes at this time, you may find expectBytesResponse more useful. It has the signature:
expectBytesResponse : (Result x a -> msg) -> (Response Bytes -> Result x a) -> Expect msg
This does not take a decoder as input. It takes two functions which let you translate Response Bytes to a Result and another function (the first argument) which lets you translate that Result into a Msg. Through each of those steps you can retain the original Bytes reference, to do with as you please at a later time.
You will, however, have to manually handle more HTTP response scenarios, but at least you gain full control over what to do with your Bytes.

Related

Create script protocol buffers

Using the image as reference where would the .proto be used?
old
.
I need to create a script in protocol buffers.
What I want is to create a delay of 20 minutes (1200 seconds), for every bus, of the specific route, and during a specific time, from 08:30:00 until 09:30:00 (am time).
The bus route is identified as 9403 (from x to y), and 9404 (from y to x).
Is this correct?
syntax = "proto3";
package teste1;
message TripUpdate {
string agency_id = 1;
string route_id = 9403, 9404;
message DelayConstruction{
uint64 start 08:30:00;
uint64 end 09:30:00;
int32 delay 1200; //20min
}
}
.proto is not a scripting language - it is a schema DSL: it just describes the metadata of a potential future payload; it is the equivalent of XSD in terms of XML, for example.
Each field has only one field number, as indicated by = {the number} , so string route_id = 9403, 9404; doesn't make sense. Likewise, the start, end and delay fields are field definitions here - they just have field numbers, not values like 08:30:00. Serialized data may have values, but that is not encoded in .proto - it is going to be either the binary protobuf format, or (sometimes, rarely) the JSON variant of protobuf.
It is very hard to understand what you are trying to do, but no: this isn't how you do it, regardless of what "it" is.
It may be worth trying to explain what you're trying to do, so we can offer guidance.

Flatbuffer mutate doesn't change bytes

I am running into a problem mutating a flatbuffer from bytes. According to the flatbuffer documentation (https://github.com/google/flatbuffers/blob/master/docs/source/Tutorial.md), you can
mutate fixed size fields, such as an int32. And as you can see below, the generated golang TestMutate has a
MutateServerId() function. My problem is that after I mutate it, the bytes don't seem to have changed.
Here is my flatbuffer table definition:
namespace foo;
table TestMutate {
serverId:int32;
}
Here is a unit test I wrote:
func TestMutateFlatbuffer2(test *testing.T) {
builder := flatbuffers.NewBuilder(1024)
packageWalletStorageServicesRPC.TestMutateStart(builder)
packageWalletStorageServicesRPC.TestMutateAddServerId(builder, 1)
endMessage := packageWalletStorageServicesRPC.TestMutateEnd(builder)
builder.Finish(endMessage)
bytes := builder.FinishedBytes()
testMutate := packageWalletStorageServicesRPC.GetRootAsTestMutate(bytes, 0)
success := testMutate.MutateServerId(2)
if !success {
panic("Server id not mutated.")
} else {
logger.Logf(logger.INFO, "serverId mutated to:%d", testMutate.ServerId())
}
mutatedBytes := testMutate.Table().Bytes
if string(mutatedBytes) == string(bytes) {
panic("Bytes were not mutated.")
}
}
Here is the output from the test.
=== RUN TestMutateFlatbuffer2
2019/08/01 19:33:56.801926 foo_test.go:389 : [ I ]: serverId mutated to:2
--- FAIL: TestMutateFlatbuffer2 (0.00s)
panic: Bytes were not mutated. [recovered]
panic: Bytes were not mutated.
Notice that it appears I've mutated the underlying structure, but when I get the bytes of the flatbuffer,
they're not changed. Question 1: Am I getting the bytes the correct way? Question 2: If I'm getting them in the right way, why
are they not changed since the call to mutate seemed to succeed?
Your test string(mutatedBytes) == string(bytes) fails because.. you are comparing the mutated buffer against itself. bytes refers to a buffer, that before your mutation contains a 1, and after it contains a 2. mutatedBytes points to the same buffer, and thus also contains a 2. The fact that testMutate.ServerId() returns 2 should tell you that the buffer got mutated succesfully, because there is no other way it could return 2 :) You'd have to make a (deep) copy of bytes before the mutation if you wanted this comparison to show that the buffers are different.
There's at least two solutions to this problem. The second solution is better in my case because it results in less copying of bytes.
By creating an intermediate string (and therefore, a copy of the bytes). Generally with flatbuffers you want to avoid copies, but for my use case, I'm ok with it.
Wrap table definitions like this:
table LotsOfData {
notMutated:[ubyte];
}
table TestMutate {
notMutated:LotsOfData;
serverId:int32;
}

How to merge a serialized protobuf with a another protobuf without deserializing the first

I'm trying to understand if it's possible to take a serialized protobuf that makes up part of another protobuf and merge them together without having to deserialize the first protobuf.
For example, given a protobuf wrapper:
syntax = "proto2";
import "content.proto";
message WrapperContent {
required string metatData = 1;
required Content content = 2;
}
And then imagine we get a serialized version of Content below (i.e. Content is coming that is coming from a remote client):
syntax = "proto2";
message Content {
required string name = 1;
required bytes payload = 2;
}
Do you know if any way I can inject the serialized Content into the WrapperContent without first having to deserialize Content.
The reason I'm trying to inject Content without deserializing it, is that I'm try and save on the overhead of deserializing the message.
If that answer is, no, it's not possible. That is still helpful.
Thanks, Mike.
In protobuf, submessages are stored like bytes fields.
So you can make a modified copy of your wrapper:
message WrapperContentBytes {
required string metatData = 1;
required bytes content = 2;
}
and write the already serialized content data into the content field.
Decoders can use the unmodified WrapperContent message to decode also the submessage. The binary data on the wire will be the same so decoders do not know the difference.

create device mapper target

I am trying to implement device mapper target by referring to the already existing ones dm-linear, dm-snapshot, dm-cache etc. In my implementation, I need to perform a read/modify/write operation on a certain sector range. Since the device mapper directly talks to the block layer, I am not sure what data structures/functions to use to read the sectors in the memory, modify the buffer and write it back to another sector range.
At the application level, we have syscalls and below we have vfs_read/vfs_write. Is there anything similar for device mapper layer?
I have been stuck here for very long. Any help will be appreciated.
NOTE: My answer is related to kernel version < 3.14, because since 3.14 API is slightly changed.
In kernel you read/write certain sectors with struct bio. This struct is used for all block level I/O. Comprehensive documentation can be found in kernel and on lwn. These are the several most significant members of this structure:
bio->bi_sector - first sector of block I/O request
bio->bi_size - size of I/O request
bio->bi_bdev - device to read/write
bio->bi_end_io - callback that kernel will call on the end of request
What you do in device mapper target is map incoming bio. When you creating your device mapper target you supply at least 2 callbacks: ctr, and map. For example, the simplest device-mapper target dm-zero declares it's callbacks as following:
static struct target_type zero_target = {
.name = "zero",
.version = {1, 1, 0},
.module = THIS_MODULE,
.ctr = zero_ctr,
.map = zero_map,
};
map is a key callback - it's a heart of every device-mapper target. map receive incoming bio and it can do anything with it. For example, dm-linear just shift sector of every incoming bio by predefined offset. See the code:
static sector_t linear_map_sector(struct dm_target *ti, sector_t bi_sector)
{
struct linear_c *lc = ti->private;
return lc->start + dm_target_offset(ti, bi_sector);
}
static void linear_map_bio(struct dm_target *ti, struct bio *bio)
{
struct linear_c *lc = ti->private;
bio->bi_bdev = lc->dev->bdev;
if (bio_sectors(bio))
bio->bi_sector = linear_map_sector(ti, bio->bi_sector);
}
static int linear_map(struct dm_target *ti, struct bio *bio)
{
linear_map_bio(ti, bio);
return DM_MAPIO_REMAPPED;
}
Because map receives pointer to bio it can change value under that pointer and that's it.
That's how you map I/O requests. If you want to create your own requests then you must allocate bio, fill it's sector, device, size, end callback and add buffers to read into/write from. Basically, it's just a few steps:
Call to bio_alloc to allocate bio.
Set bio->bi_bdev, bio->bi_sector, bio->bi_size, bio->bi_end_io
Add pages via bio_add_page.
Call submit_bio.
Handle results and errors in bio->bi_end_io callback
Example can be found in dm-crypt target in crypt_alloc_buffer function.

some content of function 'send' is missing

I used the tcp protocol to deal the request the client, I found a phenomenon which is some of the content is missing while using the function of 'send'. the code is as fellow:
_stprintf(cData,"[%s]",send_back);
memset(send_back,0,sizeof(cData));
int send_count;
if((send_count=send(service_sock,cData,_tcslen(cData),0))!=SOCKET_ERROR){
fwrite(cData,sizeof(char),_tcslen(cData),hFile);
fflush(hFile);
g_log->print_log("%c%c%c%c",cData[0],cData[1],cData[2],cData[send_count-1]);
g_log->print_log("buffer len is :%d , send %d bytes",_tcslen(cData),send_count);
fclose(hFile);
memset(cData,0,sizeof(cData));
return true;
}
the send function is always successful, and the value of _tcslen(cData) is equal to send_count and the cData[send_count-1] is ']'.
But when I use the wireshark(a capture tool) to capture the packet which is send out by the socket, I found some content is always missing including the Character of ']'. the content is encapsulated by JSON protocol, so the ']' is important. the total size of every time send out is 8900 bytes. But when I change the request item one time (before is 100) to 50, there is nothing missed, the size of send back is about 4000 bytes.
I do not know why this happened.
from my log file, I am sure the array named 'cData' contain the total content, But why the the content from the packets captured by the wireshark is not complete?
Seeing that you're using TCP, it already looks wrong.
First off, TCP is stream protocol which is not suited for one time packets ( especially small ) but the benefits are far more greater than just use UDP instead.
Keep in mind that in case of TPC you are not in control you can only make sure that your requests are handled correctly, the actual communication is done by the Winsock library.
Always remember that the send functions len parameter is NOT a requirement it's a hint on how big is your buffer and how much you can send in one go, it may return less than you want to send, and this may depend on lot of factors how often it happens, lets say you use the loopback device it would probably never ever do this, meaning that send will actually send what you requested. In a real network it may send it on one go in about 90% or with even less probability.
You have to make sure you send as much as you want, i.e. check for the return value and call send again if it didn't send as much as you wanted and do the same on the other side with recv, call recv until you get as much data as you wanted. This method only works if you know exactly how much data you want to send over the network.
As for the loss off data, TCP, I would say almost always sends data, assuming that you checked the return value of send. If there is a network problem, like loss of data you would see the TCP retransmit packet.
For your way of sending data this is more suitable, this is to make sure you really send the amount of data you want :
xint xsend(SOCKET s,const char* buf,xint len)
{
xint lastSize;
xint result;
if (len==0 || s==(SOCKET)NULL || buf==(const char*)NULL)
return SOCKET_ERROR;
lastSize=0;
result=0;
do
{
result=send(s,buf+lastSize,len-lastSize,0);
if (result==0)
return 0;
if (result==SOCKET_ERROR)
return SOCKET_ERROR;
if (result==len)
return len;
if (result>len)
{
xlog(1,"xsend : socket sent too much data [ %i of %i ]",result,len);
return SOCKET_ERROR;
}
lastSize+=result;
if (lastSize>len)
{
xlog(1,"xsend : socket sent too much data ( overall ) [ %i of %i ]",result,len);
return SOCKET_ERROR;
}
if (lastSize==len)
return len;
}
while (1);
xlog(2,"failed to do xsend");
return SOCKET_ERROR;
}
This code is just a copy paste from one of my projects, xlog is simple logging function, you can figure it out.

Resources