Related
Background:
I have writing a crypto trading bot for fun and profit.
So far, it connects to an exchange and gets streaming price data.
I am using this price to create a technical indicator (MACD).
Generally for MACD, it is recommended to use closing prices for 26, 12 and 9 days.
However, for my trading strategy, I plan to use data for 26, 12 and 9 minutes.
Question:
I am getting multiple (say 10) price ticks in a minute.
Do I simply average them and round the time to the next minute (so they all fall in the same minute bucket)? Or is there is better way to handle this.
Many Thanks!
This is how I handled it. Streaming data comes in < 1s period. Code checks for new low and high during streaming period and builds the candle. Probably ugly since I'm not a trained developer, but it works.
Adjust "...round('20s')" and "if dur > 15:" for whatever candle period you want.
def on_message(self, msg):
df = pd.json_normalize(msg, record_prefix=msg['type'])
df['date'] = df['time']
df['price'] = df['price'].astype(float)
df['low'] = df['low'].astype(float)
for i in range(0, len(self.df)):
if i == (len(self.df) - 1):
self.rounded_time = self.df['date'][i]
self.rounded_time = pd.to_datetime(self.rounded_time).round('20s')
self.lhigh = self.df['price'][i]
self.lhighcandle = self.candle['high'][i]
self.llow = self.df['price'][i]
self.lowcandle = self.candle['low'][i]
self.close = self.df['price'][i]
if self.lhigh > self.lhighcandle:
nhigh = self.lhigh
else:
nhigh = self.lhighcandle
if self.llow < self.lowcandle:
nlow = self.llow
else:
nlow = self.lowcandle
newdata = pd.DataFrame.from_dict({
'date': self.df['date'],
'tkr': tkr,
'open': self.df.price.iloc[0],
'high': nhigh,
'low': nlow,
'close': self.close,
'vol': self.df['last_size']})
self.candle = self.candle.append(newdata, ignore_index=True).fillna(0)
if ctime > self.rounded_time:
closeit = True
self.en = time.time()
if closeit:
dur = (self.en - self.st)
if dur > 15:
self.st = time.time()
out = self.candle[-1:]
out.to_sql(tkr, cnx, if_exists='append')
dat = ['tkr', 0, 0, 100000, 0, 0]
self.candle = pd.DataFrame([dat], columns=['tkr', 'open', 'high', 'low', 'close', 'vol'])
As far as I know, most or all technical indicator formulas rely on same-sized bars to produce accurate and meaningful results. You'll have to do some data transformation. Here's an example of an aggregation technique that uses quantization to get all your bars into uniform sizes. It will convert small bar sizes to larger bar sizes; e.g. second to minute bars.
// C#, see link above for more info
quoteHistory
.OrderBy(x => x.Date)
.GroupBy(x => x.Date.RoundDown(newPeriod))
.Select(x => new Quote
{
Date = x.Key,
Open = x.First().Open,
High = x.Max(t => t.High),
Low = x.Min(t => t.Low),
Close = x.Last().Close,
Volume = x.Sum(t => t.Volume)
});
See Stock.Indicators for .NET for indicators and related tools.
I have a costly server ajax request which has one input (full: boolean). If full is false, the server can return either a partial or a full response (response.isFull == true); but if full is true, the server will return a full response. Normally the partial response is good enough, but there are certain conditions that will require a full response. I need to avoid requesting a full response explicitly as much as possible, so I thought I'd start with a BehaviorSubject which I can eventually feed with true and combine it with distinctUntilChanged if I ever need to get the full response. This will give me an observable with false initially and that can give me true if I feed that into it:
const fullSubject = new BehaviorSubject<boolean>(false);
Then I've got a function that takes a boolean parameter and returns an observable with the server request (retried, transformed, etc.). As said, the answer can be partial or full, but it can be full even if the input parameter was false at the server's discretion. For example:
interface IdentityData {
...
isFull: boolean;
}
private getSimpleIdentity(full: boolean): Observable<IdentityData> {
return Axios.get(`/api/identity${full?"?full=true":""}`)
.pipe( ... retry logic ...,
... transformation logic ...,
shareReplay(1) );
}
I need to know how can I combine these so that the following is true:
The server needs to be queried at most twice.
If the first answer is a full answer, no further queries must be performed to the server.
If the first answer is a partial answer, and true is fed into fullSubject, a full answer must be requested.
The expected output from all this is an observable that emits either one full response, or a partial response and, when asked, a full response.
Environment: Vue 2.6.11, RxJS 6.5.5, Axios 0.19.2, TypeScript 3.7.5.
Thanks in advance
Here would be my approach:
const fullSubject = new BehaviorSubject(false);
const src$ = fullSubject.pipe(
switchMap(isFull => Axios.get('...')),
take(2), // Server required at most twice
takeWhile(response => !response.isFull, true), // When `isFull`, it will complete & unsubscribe -> no more requests to the server
shareReplay(1),
);
src$.subscribe(() => { /* ... */ });
function getFullAnswer () {
fullSubject.next(true);
}
takeWhile takes a second argument, inclusive. When set to true, when the predicate function evaluates to false(e.g isFull is true) it will send that value as well. –
if I've got it correctly
private getSimpleIdentity(): Observable<IdentityData> {
return fullSubject.pipe(
switchMap(full => Axios.get(`/api/identity${full ? "?full=true" : ""}`)),
shareReplay(1),
);
}
Uses the retryWhen() operator
const source = of("").pipe(map(() => Math.floor(Math.random() * 10 + 1)));
const example = source
.pipe(
tap((val) => console.log("tap", val)),
map((val) => {
//error will be picked up by retryWhen
if (val !== 5) throw val;
return val;
}),
retryWhen((errors) =>
errors.pipe(
tap(() => console.log("--Wait 1 seconds then repeat")),
delay(1000)
)
)
)
.subscribe((val) => console.log("subscription", val));
/*
output:
tap 3
--Wait 1 seconds then repeat
tap 8
--Wait 1 seconds then repeat
tap 1
--Wait 1 seconds then repeat
tap 4
--Wait 1 seconds then repeat
tap 7
--Wait 1 seconds then repeat
tap 5
subscription 5
*/
In Vue.js i use this part of code in computed to calculate cumule of amounts. It works good in LocalHost. But when i upload the project to a Web Server, this part of code isn't working.
Code:
personsWithAmount(){
const reducer = (accumulator, currentValue) => accumulator + currentValue.amount;
return this.persons.map((pers)=>{
let p=pers;
if(pers.usercashfloat.length===0){
p.totalAmount=0;
}else if(pers.usercashfloat.length===1){
p.totalAmount=pers.usercashfloat[0].amount;
}else{
window.cashfloat=pers.usercashfloat
p.totalAmount=pers.usercashfloat.reduce(reducer,0);
};
Result in LocalHost :
Array 1 =200
Array2 = 200
Result = 400
Resultat In Server
Array 1 =200
Array2 = 200
Result = 200200
Thanks
It seems like some string concatination is happening. Perhaps this works:
const reducer = (accumulator, currentValue) => Number(accumulator) + Number(currentValue.amount);
I'm trying to fit a model using fitDataset(). I can train using the "normal" approach, with a for loop and getting random batches of data (20000 data points).
I'd like to use the fitDataset() and be able to use the entire dataset and not rely on "randomness" of my getBatch function.
I'm getting closer, using the API docs and the example on tfjs-data but, i'm stuck on a probably dumb data manipulation...
So here's how i'm doing it:
const [trainX, trainY] = await bigData
const model = await cnnLSTM // gru performing well
const BATCH_SIZE = 32
const dataSet = flattenDataset(trainX.slice(200), trainY.slice(200))
model.compile({
loss: 'categoricalCrossentropy',
optimizer: tf.train.adam(0.001),
metrics: ['accuracy']
})
await model.fitDataset(dataSet.train.batch(32), {
epochs: C.trainSteps,
validationData: dataSet.validation,
callbacks: {
onBatchEnd: async (batch, logs) => (await tf.nextFrame()),
onEpochEnd: (epoch, logs) => {
let i = epoch + 1
lossValues.push({'epoch': i, 'loss': logs.loss, 'val_loss': logs.val_loss, 'set': 'train'})
accuracyValues.push({'epoch': i, 'accuracy': logs.acc, 'val_accuracy': logs.val_acc, 'set': 'train'})
// await md `${await plotLosses(train.lossValues)} ${await plotAccuracy(train.accuracyValues)}`
}
}
})
here's my interpretation of the dataset creation:
flattenDataset = (features, labels, split = 0.35) => {
return tf.tidy(() => {
let slice =features.length - Math.floor(features.length * split)
const featuresTrain = features.slice(0, slice)
const featuresVal = features.slice(slice)
const labelsTrain = labels.slice(0, slice)
const labelsVal = labels.slice(slice)
const data = {
train: tf.data.array(featuresTrain, labelsTrain),
validation: tf.data.array(featuresVal, labelsVal)
}
return data
})
}
I'm getting an error:
Error: Dataset iterator for fitDataset() is expected to generate an Array of length 2: `[xs, ys]`, but instead generates Tensor
[[0.4106583, 0.5408, 0.4885066, 0.9021732, 0.1278526],
[0.3711334, 0.5141, 0.4848816, 0.9021571, 0.2688071],
[0.4336613, 0.5747, 0.4822159, 0.9021728, 0.3694479],
...,
[0.4123166, 0.4553, 0.478438 , 0.9020132, 0.8797594],
[0.3963479, 0.3714, 0.4871198, 0.901996 , 0.7170534],
[0.4832076, 0.3557, 0.4892016, 0.9019232, 0.9999322]],Tensor
[[0.3711334, 0.5141, 0.4848816, 0.9021571, 0.2688071],
[0.4336613, 0.5747, 0.4822159, 0.9021728, 0.3694479],
[0.4140858, 0.5985, 0.4789927, 0.9022084, 0.1912155],
...,
The input data is 6 timesteps with 5 dimensions and the labels are just one-hot encoded classes [0,0,1], [0,1,0] and [1, 0, 0]. I guess the flattenDataset() is not sending the data in the correct way.
Does data.train needs to output for each data point [6 timesteps with 5 dims, label] ? I get this error when i tried that:
Error: The feature data generated by the dataset lacks the required input key 'conv1d_Conv1D5_input'.
Could really use some pro insight...
--------------------
Edit #1:
I feel i'm close to an answer.
const X = tf.data.array(trainX.slice(0, 100))//.map(x => x)
const Y = tf.data.array(trainY.slice(0, 100))//.map(x => x)
const zip = tf.data.zip([X, Y])
const dataSet = {
train: zip
}
dataSet.train.forEach(x => console.log(x))
With this i get on the console:
[Array(6), Array(3)]
[Array(6), Array(3)]
[Array(6), Array(3)]
...
[Array(6), Array(3)]
[Array(6), Array(3)]
but the fitDataset is giving me: Error: The feature data generated by the dataset lacks the required input key 'conv1d_Conv1D5_input'.
my model look like this:
const model = tf.sequential()
model.add(tf.layers.conv1d({
inputShape: [6, 5],
kernelSize: (3),
filters: 64,
strides: 1,
padding: 'same',
activation: 'elu',
kernelInitializer: 'varianceScaling',
}))
model.add(tf.layers.maxPooling1d({poolSize: (2)}))
model.add(tf.layers.conv1d({
kernelSize: (1),
filters: 64,
strides: 1,
padding: 'same',
activation: 'elu'
}))
model.add(tf.layers.maxPooling1d({poolSize: (2)}))
model.add(tf.layers.lstm({
units: 18,
activation: 'elu'
}))
model.add(tf.layers.dense({units: 3, activation: 'softmax'}))
model.compile({
loss: 'categoricalCrossentropy',
optimizer: tf.train.adam(0.001),
metrics: ['accuracy']
})
return model
What is wrong here?
What model.fitDataset expects are a Dataset, each element inside this dataset is a tuple of two items, [feature, label].
So in your case, you need to create featureDataset and labelDataset, then merge then with tf.data.zip to create trainDataset. Same for validation dataset.
Solved it
so after a lot of trial an error i found a way to make it work.
So, i had an input shape of [6, 5], meaning an array with 6 arrays of 5 floats each.
[[[0.3467378, 0.3737, 0.4781905, 0.90665, 0.68142351],
[0.44003019602788285, 0.3106, 0.4864576, 0.90193448, 0.5841830879700972],
[0.30672944860847245, 0.3404, 0.490295674, 0.90720676, 0.8331748581920732],
[0.37475716007758336, 0.265, 0.4847249, 0.902056932, 0.6611207914113887],
[0.5639427928616854, 0.2423002, 0.483168235, 0.9020202294447865, 0.82823],
[0.41581425627336555, 0.4086, 0.4721923, 0.902094287, 0.914699]], ... 20k more]
What i did was to flatten the array becoming an array of 5 dimensions arrays. Then applied the .batch(6) to it.
const BATCH_SIZE = 20 //batch size fed to the NN
const X = tf.data.array([].concat(...trainX)).batch(6).batch(BATCH_SIZE)
const Y = tf.data.array(trainY).batch(BATCH_SIZE)
const zip = tf.data.zip([X, Y])
const dataSet = {
train: zip
}
Hope it can help others on complex data!!
I am learning Rxjs and wanted to try out a few examples on my own
but I can't seem to get my head around to think reactively.
I am trying to calculate the time a user's mouse pointer spends inside and outside a div.
see fiddle - https://jsfiddle.net/ishansoni22/44af3n3k/
<div class = "space">
<div>
let $space = $(".space")
let in$ = Rx.Observable.fromEvent($space, "mouseenter")
.map((event) => "in")
let out$ = Rx.Observable.fromEvent($space, "mouseleave")
.map((event) => "out")
let inOut$ = Rx.Observable.merge(in$, out$)
let time$ = Rx.Observable.interval(1000)
.buffer(inOut$)
.map((list) => list.length)
time$.subscribe((value) => console.log(value));
I am able to calculate the time but how do I relate it to the respective in/ out streams? I want the output to look something like :
inside, in - 20, out - 30
outside, in - 20, out - 35
inside, in - 100, out - 35
Also, can someone point me to some examples I could do so that I can start thinking in the reactive paradigm?
There are some examples in the official documentation (http://reactivex.io/rxjs) but they are a little bit scarce indeed.
I think I would some your sample something like this:
let $space = $(".space")
let in$ = Rx.Observable.fromEvent($space, "mouseenter")
let out$ = Rx.Observable.fromEvent($space, "mouseleave")
let durations$ = in$
.map(_ => Date.now())
.switchMap(inTime => out$
.take(1)
.map(_ => Date.now())
.map(outTime => outTime - inTime)
)
durations$
.scan((sum, next) => sum + next, 0)
.subscribe(total => console.log(total))
This would start listening to in$, then upon a mouseenter-event it starts to listen to mouseleaves, takes 1 of those events and calculate the duration.
I have written multiple maps below each other for clarity, but of course you can compose that into a single function.
One of the things I found most challenging when starting out with Rx was using streams of streams, and becoming comfortable with flatMap and switchMap. The problem you describe is most easily solved using exactly this approach. With your streams defined as follows (I prefer const over let to make it clear no mutation is occuring):
const in$ = Rx.Observable.fromEvent($space, 'mouseenter');
const out$ = Rx.Observable.fromEvent($space, 'mouseleave');
you can describe entering and then leaving as follows:
const inThenOut$ = in$.switchMap(() => out$);
To understand exactly what this is doing I urge you to learn about flatMap, become comfortable with streams of streams, and then learn how switchMap works by only maintaining a subscription to the most recent inner stream. For this I found the official rxjs documentation the best source. The included marble diagrams often tell complex stories with just a few dots and lines.
From here it's a relatively small step to get the time spent inside. First, we map our original streams into timestamp values:
const timestamp = () => + new Date();
const in$ = Rx.Observable.fromEvent($space, 'mouseenter').map(() => timestamp());
const out$ = Rx.Observable.fromEvent($space, 'mouseleave').map(() => timestamp());
(note: there is a timestamp method in rxjs you could use instead of doing this manually, but I feel this better illustrates how you can map your stream elements into anything you please).
From there, we can adjust our switchMap usage to access both the in and out values, and return the difference between them:
const inThenOut$ = in$.switchMap(() => out$, (x, y) => y - x);
Here's the whole thing working:
https://jsbin.com/qoruyoluho/edit?js,console,output
You could use RXJS - Timestamp operator to attach timestamp to each item emitted by an Observable indicating when it was emitted.
const { fromEvent } = Rx;
const { map, switchMap, timestamp, take, tap } = RxOperators;
const in$ = fromEvent($space, 'mouseenter').pipe(
timestamp(),
tap(x => console.log(`In: ${x.timestamp}`))
)
const out$ = fromEvent($space, 'mouseleave').pipe(
timestamp(),
tap(x => console.log(`Out: ${x.timestamp}`))
)
const duration$ = in$.pipe(
switchMap(start => out$.pipe(
take(1),
map(finish => finish.timestamp - start.timestamp),
tap(value => console.log(`Duration ms: ${value}`))
)
)
)
/* output example
In: 1552295324302
Out: 1552295325158
Duration ms: 856
*/
Try it here: https://rxviz.com/v/rOW5g9x8