I have a client and a server code. I create ffmpeg process on
both ends and use pipes to communicate with ffmpeg. I passed an RGB data and read HVec out , send that the client and do the opposite. basically I'm streaming HVec, The problem I have with the pipes is The buffer size are too small so I have to do a bunch of file read operations from the pipe . and it's really slows me down. for example if I have to read 100 bytes , function reads 20 bytes. so I have to loop five times to read hundred bytes . of course these numbers are made up . actual data size I try to Read is 1080*1920*4=829400
1 is writing to a socket and communicating via sockets a better choice.
2 would name pipes work better?
I know third option is to use avcodec lib but I'm trying not to do that
Related
I'm trying to send an array of data as an rpc reply using golang's built-in net/rpc server and client and the net/rpc/jsonrpc codec. But I'm running into some trouble.
The data I'm sending is around 48 bytes, and the client will just hang in client.Call.
I've made a playground that replicates the problem:
https://go.dev/play/p/_IQ9SF7TSdc
If you change the constant "N" in the above program to 5,
things work as expected!
Another playground shows how the issue seems to crop up only when the slice/array in question exceeds 49 bytes:
https://go.dev/play/p/R8CQa0mv7vB
Does anyone know what might be the issue? Golang's tests for the array and slice data types are not exactly designed for "large" arrays in mind. Thanks in advance.
On the line where the listener is set up:
listener, err := net.ListenUnix("unixpacket", &net.UnixAddr{RPCPath, "unixpacket"})
Don't use unixpacket. It corresponds to the underlying SOCK_SEQPACKET which is not a stream protocol. Likely large files were separated into packets in a way the receiver was not able to process. Use unix instead, which corresponds to SOCK_STREAM.
See this SO post
for more.
I've recently started to use tsduck, which I know can give me information about each and every packet within a ts file. But when I pipe the output to a text file, its way too much data and crashes often when I am trying to view it. Similarly, if i were to read that file into a script for e.g a Python script, it just seems so slow and sluggish, especially for long duration HD content where the number of ts packets will go into the millions.
Yes utilities like tsduck has its uses, but I have some very specific things I want to find out about the stream for e.g how consistent is PAT and PMT, or if continuity indictors are present... the list goes on.
Yes there are tools out there like Manzanita that gives this information, but what if you wanted to create your own script to do these things?
So the question is, what's a good way to grab data from an MPEG transport stream file such that you can use a script to analyse the data?
My use case requires transcoding a remote MOV file that can’t be stored locally. I was hoping to use http protocol to stream the file into ffmpeg. This works, but I’m observing this to be a very expensive operation with (seemingly) redundant network traffic, so am looking for suggestions.
What I see is that ffmpeg starts out with a Range request “0-“ (which brings in the entire file), followed by a number of open-ended requests (no ending offset) at different positions, each of which makes the http server return large chunks of the file again and again, from the starting position to the very end.
For example, http range requests for a short 10MB file look like this:
bytes=0-
bytes=10947419-
bytes=36-
bytes=3153008-
bytes=5876422-
Is there another input method that would be more network-efficient for my use case? I control the server where the video file resides, so I’m flexible in what code runs there.
Any help is greatly appreciated
Is it possible to figure out on the writer (server) end of a Windows named pipe how much data the client is reading from the other end in each request?
Background: Simple scenario. We have one process that writes to a named pipe that it creates via CreateNamedPipe. The data only flows outward (PIPE_ACCESS_OUTBOUND) and is PIPE_TYPE_BYTE. Another process reads from the pipe and displays some information about it. This repeats about once per second.
What I need to change: I have to add a little bit more data to each write and subsequent read. This is no problem to update both the client and the server, but the person who created this 14 years ago apparently didn't think the structure of the data in the pipe would ever change. No metadata is included whatsoever and the client pays no attention to the amount of data available. For example, let's say the structure size has been 8 bytes for all these years. The server writes 8 bytes, the client reads 8 bytes. Now I want to write 12 bytes. If it is an old client doing the reading, it will get weird results since it just tries to blindly read 8 bytes each time.
What I currently have working: I have an ugly solution working now, but am not overly pleased. I used GetNamedPipeClientProcessId to get the process ID of the reader and then the appropriate calls to get its file name and then version information (OpenProcess, GetModuleFileNameEx, GetFileVersionInfo, ...) to determine the version number of the client and then write the appropriate amount of data. It appears to work, but it feels a bit cumbersome and fragile.
What I think I want: What I would like to do is have the server somehow detect that the client only read 8 bytes from the pipe and then adjust the behavior accordingly. Is it possible to figure this out?
You could have some form of handshake on connection from new clients to say "I support XYZ". If you don't get that, keep at the 8 bytes
I've checked out a few of the forum posts here and can't find quite what I'm looking for. Suppose you are reading in a text document via Ruby. I understand the stream is essentially the characters coming in byte by byte. What is the purpose/best practice of buffering in this case? My book shows plenty examples of the buffer being utilized, but no real description of what the buffer is or why it even exists. What should I be considering when setting the buffer? For example, the book illustrates the following method as:
read(n, buffer=nil) reads in n bytes, until the bytes are ready
I don't understand what the statement "until the bytes are ready" means. Does the buffer play a role in this? Please feel free to point me to another place where this is explained, I couldn't for the life of me find it on my own.
IO can be not only file, but a network socket. and in networks you regularly have a situation where you are ready to process more data, but the remote side have a pause in data sending.
(You usually see a progress bar or a spinner element in your browser in these cases)
So, if you are using regular files, the bytes are always 'ready'.
The Picaxe book for IO#read says:
Reads at most int bytes from the I/O stream or to the end of file if int is omitted. Returns nil if called at end of file. If buffer (a String) is provided, it is resized accordingly, and input is read directly into it.