I'm trying to track down a memory leak that I think has to do with how I am using MS Bond. Specifically, the issue is likely on the subscriber side due to 'new' ArraySegment and InputBuffer objects being generated on every iteration inside a while loop.
On the publisher side, the code roughly looks as follows and I don't think there is a problem here:
open ZeroMQ
open Bond
open Bond.Protocols
open Bond.IO.Unsafe
let bond = new BondStructs.SomeStruct()
let output = new OutputBuffer()
let writer = new CompactBinaryWriter<OutputBuffer>(output)
let ctx = new ZContext()
let sock = new ZSocket(ctx, ZSocketType.PUB)
sock.Bind "tcp://localhost:12345"
while true do
// do work, populate bond structure
Marshal.To(writer, bond)
use frame = new ZFrame(output.Data.Array, output.Data.Offset, output.Data.Count)
sock.Send frame
output.position <- 0L
The issue I think is on the subscriber side due to the fact the new ArraySegment and InputBuffer objects are being generated on every iteration and somehow the GC is unable to properly clean up.
open ZeroMQ
open Bond
open Bond.Protocols
open Bond.IO.Unsafe
let sock = new ZSocket(ctx, ZSocketType.SUB)
sock.SubscribeAll()
sock.SetOption(ZSocketOption.CONFLATE, 1) |> ignore
sock.Connect("tcp://localhost:12345")
while true do
let zf = sock.ReceiveFrame()
let segment = new ArraySegment<byte>(zf.Read())
let input = new InputBuffer(segment)
let msg = Unmarshal<Record>.From(input)
zf.Close()
// do work
Is there a way for me to push ArraySegment and InputBuffer lines above the while loop and reuse those objects within the loop?
If the resulting msg instances are stored anywhere after processing the one request, that can keep the buffer alive.
The msg instances can have references to the underlying InputBuffer or ArraySegment when bonded- or blob-type fields are used.
Failing that, I've had luck with the WinDBG extension command !gcroot to figure out what's keeping something alive longer than I expected. If !gcroot provides more insights, but not a solution, please edit those details into the question.
Related
I am trying to loop through an array of Buffers each containing a sound sample read from disk, but I am having problems getting the SynthDef to reset its pointer to the buffers.
I did the following:
Assume I have a folder of sound files and I have read them all into an array of Buffers called "~buffers"
I just want to go through the array in order, playing the samples back to back and stopping after the last one.
I define a simple SynthDef, and then put the Synth that calls it into a Routine:
(
SynthDef(\playBuffer,{arg out = 0, buf;
var sig;
sig = PlayBuf.ar(2, buf, doneAction: Done.freeSelf);
Out.ar(out, sig);
}).add
~routine = Routine({
~buffers.do({
arg item;
var synth;
synth = Synth(\playBuffer, [\buf, item]);
item.duration.wait;
synth.free;
});
});
~routine.play;
)
It does not work as expected---the synths are always playing the same sound,the first one, although for the durations corresponding to the different samples.
I think the problem could be that the function inside my \playbuffer SynthDef (at least according to the Help files) is not re-evaluated with a different bufnum argument inside the loop.
In fact I can loop through the buffers if I use Buffer.play which creates synthDef's and Synth's on the fly. Replacing my Routine with this code works:
(
~routine2 = Routine({
~buffers.do({
arg item;
item.play;
item.duration.wait;
});
});
~routine2.play;
)
BUT: it is very crude, as now I cannot manipulate the buffer output except for changing the amplitude through the mul parameter of Buffer.play.
What I would like to do is to replicate Buffer.play's behavior---creating SynthDef's and Synth's on the fly---in my own code. But I'm having no luck with it. In fact I am not sure where to start, possibly because I don't fully grasp SuperCollider's server's handling of functions. Should I make a Synth-making function and use that inside the routine's loop? Or should I move the definition of the SynthDef inside the loop (which seems equivalent)? I tried the latter, but still got the same sound playing.
Perhaps I am going at this the wrong way---I'm very new to SuperCollider.
The code in your first example is correct. If I fill by buffers like this:
(
s.makeBundle(nil, {
~buffers = [1, 2, 3, 4, 5].collect {
|i|
var b;
b = Buffer.alloc(s, 44100, 1);
b.sine3([100, 150, 175] * i, 0.25);
};
})
)
and then play them with your code example:
(
SynthDef(\playBuffer,{arg out = 0, buf;
var sig;
sig = PlayBuf.ar(1, buf, doneAction: Done.freeSelf);
Out.ar(out, sig);
}).add;
~routine = Routine({
~buffers.do({
arg item;
Synth(\playBuffer, [\buf, item]);
item.duration.wait;
});
});
~routine.play;
)
This works fine, I hear ascending tones. (I changed your example to be single channel buffers, and removed the .free as you were already doing Done.freeSelf). If you're hearing the same sound playing each time, the problem is likely in the code where you're loading your buffers and not in playback.
One gotcha: the duration property of a buffer is not available immediately after you load them - reading an audio file is asynchronous, and SC doesn't know the duration until it's loaded. If you're doing Buffer.read immediately before playing, there's a chance your duration might be e.g. 0 or nil, which would cause unexpected results.
i tryed it in a Task, but i think it is complitly like the think which you will do in routine.
you have to put the buffers in array.
like buffer=[1,2,3,4,5]
but it is better to code in this way.
\buffer=[a.bufnum,b.bufnum,c.bufnum,d.bufnum]
and set the buffer variable in second arrgument of PlayBuf in SynthDef.
because you might load others buffer in your server and if you put the number of the buffer in the array, it will usually play wrong buffer which you do not want to play,.
Using the Google CAF Receiver SDK, how do we prevent the receiver from timing out and automatically killing the cast session when we're not using the receiver player?
The standard Google Cast use case is to send media from a device to the cast receiver and have the receiver render the media using a player. The CAF receiver SDK provides this functionality in a beautiful, simple way using the element cast-media-player.
But for those instances when we want to cast from a device and render content where it's not relevant to use the cast-media-player (e.g. an HTML dashboard), how do we keep the receiver alive?
The following custom receiver for example (HAML for brevity), results in the cast session automatically terminating after 5 minutes...
!!! 5
%html
%head
:css
cast-media-player {
display: none;
}
= javascript_include_tag 'https://www.gstatic.com/cast/sdk/libs/caf_receiver/v3/cast_receiver_framework.js'
%body
%cast-media-player
:javascript
const context = cast.framework.CastReceiverContext.getInstance();
const player = context.getPlayerManager();
player.setMessageInterceptor(cast.framework.messages.MessageType.LOAD, loadRequestData => {
...[load custom view]...
return false;
});
context.start();
The receiver log shows the line cast.framework.common.IdleTimeoutManager] timer expired and then shuts down. Example receiver log shown here.
I've tried:
Increasing cast.framework.CastReceiverOptions#maxInactivity to a very large number
Periodically loading new data from the sender
Periodically sending custom messages from the receiver to the sender
Periodically sending custom messages from the sender to the receiver
Any help is very much appreciated!
I ran into the same problem while developing a custom receiver app that does not play media. Here is the solution I implemented:
var idleTime = 0;
const context = cast.framework.CastReceiverContext.getInstance();
const CUSTOM_CHANNEL = '[MY CHANNEL HERE]';
context.addCustomMessageListener(CUSTOM_CHANNEL, function(customEvent) {
var eventData = customEvent.data;
parseCommand(eventData);
idleTime = 0;
});
const options = new cast.framework.CastReceiverOptions();
options.disableIdleTimeout = true;
context.start(options);
var idleInterval = setInterval(timerIncrement, 60000); // 1 minute
function timerIncrement() {
idleTime = idleTime + 1;
if (idleTime > 4) { // 5 minutes
context.stop();
}
}
With CastReveiverOptions I disable idle timeout, which according to the documentation: "If true, the receiver will not set an idle timeout to close receiver if there is no activity. Should only be used for non media apps."
https://developers.google.com/cast/docs/reference/caf_receiver/cast.framework.CastReceiverOptions#constructor_1
Since mine is a "non media app," I believe this is correct usage.
I then set my own time out based on 5 minutes of inactivity in my custom channel.
I figured out an alternative way to stop this which is more efficient than periodically sending a silent clip, but it feels dirty. Basically we have to stop Chromecast's setTimeout from firing and closing the connection due to no media. The quickest solution is to simply re-declare setTimeout as a dummy no-op function before loading the Chromecast receiver script. It does not seem to break anything Chromecast-related in this scenario because it looks like Chromecast's timeouts are all related to video which aren't relevant to this use case.
window._setTimeout = window.setTimeout;
window.setTimeout = function(a, b) {
// disable setTimeout so chromecast won't kill us after 5 minutes...
};
Then in our own app if we need to use a timeout we call _setTimeout instead.
I would be interested if anyone has discovered a better way to achieve this, aside from manually hosting cast_receiver_framework.js with the offending line commented out (which is inside the Wn(a, b) function) or sending a silent clip every few minutes. But self-hosting isn't recommended by Google.
A better solution may be to dig deep in the minified code to work out how Xn(a) is called as that disables the timeout whenever media is playing, and then find a way to call that from within the Chromecast app.
Loading a short inaudible audio clip from the sender to the receiver every 4 minutes seems to do the trick. This should not impact performance much if the file is small. Here is some android code.
MediaMetadata metadata = new MediaMetadata(MediaMetadata.MEDIA_TYPE_MUSIC_TRACK);
MediaInfo mediaInfo = new MediaInfo.Builder("https://some-inaudible-clip.mp3")
.setStreamType(MediaInfo.STREAM_TYPE_BUFFERED)
.setContentType("audio/mpeg")
.setMetadata(metadata)
.build();
RemoteMediaClient remoteMediaClient = castSession.getRemoteMediaClient();
remoteMediaClient.load(mediaInfo, true);
It is possible to send a custom namespace message from the receiver to the sender. That should keep the heartbeat live. However, your use case is not directly supported by the Cast SDK, so you would have to experiment on a solution.
I'm trying to test a little proof-of-concept command line app prior to integrating it into a larger app. What I'm trying to do is download some data using NSURLSession using this example. However it appears that if I use the examples given in a simple OS X command line app then the app exits prior to the data being retrieved.
How can I download data from a stand-alone command line app using NSURLSession? What I've read about is using NSRunLoop however I've not yet found a clear example in Swift so if NSRunLoop is actually the way to go then any examples would be appreciated.
Any other strategies for downloading data from a URL for a Swift command line app is also welcome (infinite while loop?).
You can use a semaphore to block the current thread and wait for your URL session to finish.
Create the semaphore, kick off your URL session, then wait on the semaphore. From your URL session completion callback, signal the semaphore.
You could use a global flag (declare a volatile boolean variable) and poll that from a while loop, but that is less optimal. For one thing, you're burning CPU cycles unnecessarily.
Here's a quick example I did using a playground:
import Foundation
var sema = DispatchSemaphore( value: 0 )
class Delegate : NSObject, URLSessionDataDelegate
{
func urlSession(_ session: URLSession, dataTask: URLSessionDataTask, didReceive data: Data)
{
print("got data \(String(data: data, encoding: .utf8 ) ?? "<empty>")");
sema.signal()
}
}
let config = URLSessionConfiguration.default
let session = URLSession(configuration: config, delegate: Delegate(), delegateQueue: nil )
guard let url = URL( string:"http://apple.com" ) else { fatalError("Could not create URL object") }
session.dataTask( with: url ).resume()
sema.wait()
Try this
let sema = DispatchSemaphore( value: 0)
let url = URL(string: "https://upload.wikimedia.org/wikipedia/commons/4/4d/Cat_November_2010-1a.jpg")!;
let task = URLSession.shared.dataTask(with: url) { (data, response, error) in
print("after image is downloaded");
sema.signal(); // signals the process to continue
};
task.resume();
sema.wait(); // sets the process to wait
For proof of concept(s) or tryouts/testing purposes, you can simplify asynchronous complexity by hard coding some timeout period until your stuff finishes. (see notes below)
SWIFT 5
//...your magic here
// add a little 🤓iness to make it fun at least...
RunLoop.main.run(until: Date() + 0x10) //oh boi, default init && hex craze 🤗
// yeah, 16 seconds timeout
// or even worse (!)
RunLoop.main.run(until: .distantFuture)
SWIFT 3 or earlier
//...your stuff here
RunLoop.main.run(until: Date(timeIntervalSinceNow: 15)) //will execute things on main loop for 15 seconds
NOTES :
DO NOT USE THIS IN PRODUCTION
respect the first rule
This is very quick and dirty way to overcome serious concerns of parallelism. Explore better and more complex solutions described in other answers of this question.
How would I limit upload speed from the server in node.js?
Is this even an option?
Scenario: I'm writing some methods to allow users to automated-ly upload files to my server. I want to limit the upload speed to (for instance) 50kB/s (configurable of course).
I do not think you can force a client to stream at a predefined speed, however you can control the "average speed" of the entire process.
var startTime = Date.now(),
totalBytes = ..., //NOTE: you need the client to give you the total amount of incoming bytes
curBytes = 0;
stream.on('data', function(chunk) { //NOTE: chunk is expected to be a buffer, if string look for different ways to get bytes written
curBytes += chunk.length;
var offsetTime = calcReqDelay(targetUploadSpeed);
if (offsetTime > 0) {
stream.pause();
setTimeout(offsetTime, stream.resume);
}
});
function calcReqDelay(targetUploadSpeed) { //speed in bytes per second
var timePassed = Date.now() - startTime;
var targetBytes = targetUploadSpeed * timePassed / 1000;
//calculate how long to wait (return minus in case we actually should be faster)
return waitTime;
}
This is of course pseudo code, but you probably get the point. There may be another, and better, way which I do not know about. In such case, I hope someone else will point it out.
Note that it is also not very precise, and you may want to have a different metric than the average speed.
Use throttle module to control the pipe stream speed
npm install throttle
var Throttle = require('throttle');
// create a "Throttle" instance that reads at 1 b/s
var throttle = new Throttle(1);
req.pipe(throttle).pipe(gzip).pipe(res);
Instead of rolling your own, the normal way to do this in production, is to let your load balancer or entry server throttle the incoming requests. See http://en.wikipedia.org/wiki/Bandwidth_throttling. It's not typically something an application needs to handle itself.
I am new to Cocoa, but managed to get a connection (to a FTP) up and running, and I've set up an eventhandler for the NSInputStream iStream to alert every response (which also works).
What I manage to get is simply the hello message and a connection timeout 60 sec, closing control connection.
EDIT: I guess my question is "without closing->opening what would be an non-terminating way of flushing the outputStream?
After searching stackoverflow and finding a lot of NSOutputStream write problems (e.g. How to use NSOutputStream's write message?) and a lot of confusion in my google hits, I figured I'd try to ask my own question:
I've tried reading the developer.apple.com doc on OutputStream, but it seems almost impossible for me to send some data (in this case just a string) to the "connection" via the NSOutputStream oStream.
- (IBAction) send_something: sender
{
const char *send_command_char = [#"USER foo" UTF8String];
send_command_buffer = [NSMutableData dataWithBytes:send_command_char length:strlen(send_command_char) + 1];
uint8_t *readBytes = (uint8_t *)[send_command_buffer mutableBytes];
NSInteger byteIndex = 0;
readBytes += byteIndex;
int data_len = [send_command_buffer length];
unsigned int len = ((data_len - byteIndex >= 1024) ?
1024 : (data_len-byteIndex));
uint8_t buf[len];
(void)memcpy(buf, readBytes, len);
len = [oStream write:(const uint8_t *)buf maxLength:len];
byteIndex += len;
}
the above seems not to result in any useable events. typing it under NSStreamEventHasSpaceAvailable sometimes give a response if I spam the ftp by keep creating new connection instances and keep sending some command whenever oStream has free space. In other words, nothing "right" and so I'm still unclear how to properly send a command to the connection. Should I open -> write -> close every time i want to write to oStream (and thus to the ftp) and can I then expect a reply (hasBytesAvailable event on iStream)?
EDIT: Doesn't look like it no.
For some reason I find it very difficult to find any clear tutorials on this matter. Seems like there are more than a few in the same position as me: unclear how to use oStream write?
Please! Any little bit that can help clear this up is greatly appreciated!
If needed I can write the rest of the code.
Chuck
Okay, so 10 hours 28 views and no answers/comments, but that's OK, because I just solved it with some good help from a very very friendly irssi coder (no butt licking intended ;)).
He proposed that I tried to std::endl'e it (newline + flush), and so I tried simply adding the newline char (\n, 0x0A) and it worked perfectly!