In consul's documentation, it states:
This endpoint returns the most recent events known by the agent.
What does it mean exactly by most recent? Most 100 recent events? 1000? Events fired in the last 7 days?
Is there a way for me to configure this?
My concern is this event list could grow infinitely large if older events are not removed within a reasonable amount of time (which can vary across different applications).
So after some digging into the source code of consul. I found out it is max 256
https://github.com/hashicorp/consul/blob/94835a2715892f48ffa9f81a9a32808d544b1ca5/agent/agent.go#L221
eventBuf: make([]*UserEvent, 256),
On below you can see the rotation
https://github.com/hashicorp/consul/blob/94835a2715892f48ffa9f81a9a32808d544b1ca5/agent/user_event.go#L229
a.eventBuf[idx] = msg
a.eventIndex = (idx + 1) % len(a.eventBuf)
Below code shows that the data is pulled from the same buffer only
https://github.com/hashicorp/consul/blob/94835a2715892f48ffa9f81a9a32808d544b1ca5/agent/user_event.go#L235
func (a *Agent) UserEvents() []*UserEvent {
So you can safely assume, this will be max 256
Related
I deployed an apache beam pipeline to GCP dataflow in a DEV environment and everything worked well. Then I deployed it to production in Europe environment (to be specific - job region:europe-west1, worker location:europe-west1-d) where we get high data velocity and things started to get complicated.
I am using a session window to group events into sessions. The session key is the tenantId/visitorId and its gap is 30 minutes. I am also using a trigger to emit events every 30 seconds to release events sooner than the end of session (writing them to BigQuery).
The problem appears to happen in the EventToSession/GroupPairsByKey. In this step there are thousands of events under the droppedDueToLateness counter and the dataFreshness keeps increasing (increasing since when I deployed it). All steps before this one operates good and all steps after are affected by it, but doesn't seem to have any other problems.
I looked into some metrics and see that the EventToSession/GroupPairsByKey step is processing between 100K keys to 200K keys per second (depends on time of day), which seems quite a lot to me. The cpu utilization doesn't go over the 70% and I am using streaming engine. Number of workers most of the time is 2. Max worker memory capacity is 32GB while the max worker memory usage currently stands on 23GB. I am using e2-standard-8 machine type.
I don't have any hot keys since each session contains at most a few dozen events.
My biggest suspicious is the huge amount of keys being processed in the EventToSession/GroupPairsByKey step. But on the other, session is usually related to a single customer so google should expect handle this amount of keys to handle per second, no?
Would like to get suggestions how to solve the dataFreshness and events droppedDueToLateness issues.
Adding the piece of code that generates the sessions:
input = input.apply("SetEventTimestamp", WithTimestamps.of(event -> Instant.parse(getEventTimestamp(event))
.withAllowedTimestampSkew(new Duration(Long.MAX_VALUE)))
.apply("SetKeyForRow", WithKeys.of(event -> getSessionKey(event))).setCoder(KvCoder.of(StringUtf8Coder.of(), input.getCoder()))
.apply("CreatingWindow", Window.<KV<String, TableRow>>into(Sessions.withGapDuration(Duration.standardMinutes(30)))
.triggering(Repeatedly.forever(AfterProcessingTime.pastFirstElementInPane().plusDelayOf(Duration.standardSeconds(30))))
.discardingFiredPanes()
.withAllowedLateness(Duration.standardDays(30)))
.apply("GroupPairsByKey", GroupByKey.create())
.apply("CreateCollectionOfValuesOnly", Values.create())
.apply("FlattenTheValues", Flatten.iterables());
After doing some research I found the following:
regarding constantly increasing data freshness: as long as allowing late data to arrive a session window, that specific window will persist in memory. This means that allowing 30 days late data will keep every session for at least 30 days in memory, which obviously can over load the system. Moreover, I found we had some ever-lasting sessions by bots visiting and taking actions in websites we are monitoring. These bots can hold sessions forever which also can over load the system. The solution was decreasing allowed lateness to 2 days and use bounded sessions (look for "bounded sessions").
regarding events dropped due to lateness: these are events that on time of arrival they belong to an expired window, such window that the watermark has passed it's end (See documentation for the droppedDueToLateness here). These events are being dropped in the first GroupByKey after the session window function and can't be processed later. We didn't want to drop any late data so the solution was to check each event's timestamp before it is going to the sessions part and stream to the session part only events that won't be dropped - events that meet this condition: event_timestamp >= event_arrival_time - (gap_duration + allowed_lateness). The rest will be written to BigQuery without the session data (Apparently apache beam drops an event if the event's timestamp is before event_arrival_time - (gap_duration + allowed_lateness) even if there is a live session this event belongs to...)
p.s - in the bounded sessions part where he demonstrates how to implement a time bounded session I believe he has a bug allowing a session to grow beyond the provided max size. Once a session exceeded the max size, one can send late data that intersects this session and is prior to the session, to make the start time of the session earlier and by that expanding the session. Furthermore, once a session exceeded max size it can't be added events that belong to it but don't extend it.
In order to fix that I switched the order of the current window span and if-statement and edited the if-statement (the one checking for session max size) in the mergeWindows function in the window spanning part, so a session can't pass the max size and can only be added data that doesn't extend it beyond the max size. This is my implementation:
public void mergeWindows(MergeContext c) throws Exception {
List<IntervalWindow> sortedWindows = new ArrayList<>();
for (IntervalWindow window : c.windows()) {
sortedWindows.add(window);
}
Collections.sort(sortedWindows);
List<MergeCandidate> merges = new ArrayList<>();
MergeCandidate current = new MergeCandidate();
for (IntervalWindow window : sortedWindows) {
MergeCandidate next = new MergeCandidate(window);
if (current.intersects(window)) {
if ((current.union == null || new Duration(current.union.start(), window.end()).getMillis() <= maxSize.plus(gapDuration).getMillis())) {
current.add(window);
continue;
}
}
merges.add(current);
current = next;
}
merges.add(current);
for (MergeCandidate merge : merges) {
merge.apply(c);
}
}
I'm integrating a lambda function with a standard queue in SQS.
I came across these two parameters batchSize and maxBatchingWindow. My original thinking was either the number of messages in the queue has reached the batchSize or the time since the first message came in has last for maxBatchingWindow seconds will trigger the lambda. In other words, whichever condition is satisfied first will invoke the lambda. And I couldn't find enough clarification about these two parameters in this documentation.
As a result, I did some experiment, setting batchSize = 3 and maxBatchingWindow = 300 seconds while setting the reservedConcurrency = 1 for lambda. Then I manually create 3 messages in the queue quickly (<< 5 min). However, I didn't observe the lambda being invoked after 5 minutes (300 s). Particularly, the metric Number Of Messages Sent of sqs shows a new data point at xx:54:15 while the logGroup for lambda updates around xx:59:53 (The lambda does nothing intensive but just to print out the value of event so I'm sure that would be the right execution).
Does that mean, once maxBatchingWindow is set greater than 0, it will become the only requirement to invoke lambda even if the batchSize has met?
I have an EA set in place that loops history trades and builds one large string with trade information. I then send this string every second from MT4 to the python backend using a plain PUSH/PULL pattern.
For whatever reason, the data isn't received on the pull side when the string transferred becomes too long. The backend PULL-socket slices each string and further processes it.
Any chance that the PULL-side is too slow to grab and process all the data which then causes an overflow (so that a delay arises due to the processing part)?
Talking about file sizes we are well below 5kb per second.
This is the PULL-socket, which manipulates the data after receiving it:
while True:
# check 24/7 for available data in the pull socket
try:
msg = zmq_socket.recv_string()
data = msg.split("|")
print(data)
# if data is available and msg is account info, handle as follows
if data[0] == "account_info":
[...]
except zmq.error.Again:
print("\nResource timeout.. please try again.")
sleep(0.000001)
I am a bit curious now since the pull socket seems to not even be able to process a string containing 40 trades with their according information on a single MT4 client - Python connection. I actually planned to set it up to handle more than 5.000 MT4 clients - python backend connections at once.
Q : Any chance that the pull side is too slow to grab and process all the data which then causes an overflow (so that a delay arises due to the processing part)?
Zero chance.
Sending 640 B each second is definitely no showstopper ( 5kb per second - is nowhere near a performance ceiling... )
The posted problem formulation is otherwise undecidable.
Step 1) POSACK/NACK prove whether a PUSH side accepts the payload for sending error-free.
Step 2) prove the PULL side is not to be blamed - [PUSH.send(640*chr(64+i)) for i in range( 10 )] via a python-2-python tcp://-transport-class solo-channel crossing host-to-host hop, over at least your local physical network ( no VMCI/emulated vLAN, no other localhost colocation )
Step 3) if either steps above got POSACK-ed, your next chances are the ZeroMQ configuration space and/or the MT4-based PUSH-side incompatibility, most probably "hidden" inside a (not mentioned) third party ZeroMQ wrapper used / first-party issues with string handling / processing ( which you must have already read about, as it has been so many times observed and mentioned in the past posts about this trouble with well "hidden" MQL4 internal eco-system changes ).
Anyway, stay tuned. ZeroMQ is a sure bet and a truly horsepower for professional and low-latency designs in distributed-system's domain.
I'm using ZeroMQ / ZMQ from Python and Java and have a question. When sending a shorter string, ZMQ uses one byte as described here (http://zguide.zeromq.org/page:all#A-Minor-Note-on-Strings)
Then what goes onto the wire is a length (one byte for shorter
strings) and the string contents as individual characters.
Does anyone know how many bytes are used when sending a longer string?
How many bytes are used for longer string when sending via ZMQ?
That depends on hell more things, than just on the string itself :
Your post refers to indeed historical text - the zguide pages.
While this was sure a very helpful first-read source in the early days of ZeroMQ v.2.x, today, we live with distributed-sysems spanning many versions, from v.2.1+, 3.x, 4.x, 4.2 being so far the last stable API version in 2018-Q2.
No one can a priori guess what API-version was used on the message-sender's side, until a receiver actually sets/accepts the link-setup and .recv()-s the respective message. Relying on a C-lang based s_recv()-helper tricks in post v4.0 API is not a sure direction to follow.
Being in python, many protocol-hardwired details remain beyond your sight, yet there are byte-maps, that get filled under the hood exactly as the benevolent dictatorship, indoctrinated in the published ZeroMQ RFC/ZMTP-specifications, dictates.
If we cannot guess or know beforehand, can we ... ?
Yes, we can experiment. Best setup a controlled distributed-system experiment.
Node A : The Sender
can be pythonic, being a sender:
- setup a REQ-archetype AccessNode ( details in "ZeroMQ Hierarchy in less than a five seconds" ),
- setup .setsockopt( zmq.IDENTITY, ... ) with a randomness-generated static identity,
- setup a .setsockopt( zmq.REQ_RELAXED, 1 ),
- .bind() it to a known endpoint of a transport-class of one's choice
- start an xrange()-generator controlled for L in xrange( 1, int( 1E+9 ) )-loop of .send()-s
- there load a payload .send( r"{0:}|{1:}".format( str( L ), L * r"*" ) )
- handle the respective .recv() for a REP-side "answer",
- check the error-states and adapt the time.sleep()-s / re-loops, according to the sender-side socket capacity and capability to send further payloads
Node B : The Receiver | The MitM-sniffer
ought be as low level as possible, so as to dis-assemble the RFC/ZMTP wire-line protocol, so python gets out of the candidate list. Other option may include a wire-level sniffer(s), if the selected transport-class permits ( an ipc:// or a vmci:// one will not )
setup a ROUTER-archetype AccessNode,
.connect() it to the know Node A's transport-class endpoint used,
start .recv()-ing messages,
If your experiment has correctly acquired / sniffed the wire-level details about the ZMTP-compliant transport sizes of the know payload compositions, your question gets repeatable, verifiable, quantitatively correct records on string-size to message-size mapping-function.
A BONUS POINT: for those indeed interested . . .Next, re-run the controlled white-box distributed-system experiment above, now with having the Node A: The Sender-side extended it's behaviour to also{ randomly | deterministically } { change | alter } its own configuration ( or map both such options onto a pair of the same payload re-.send()-s )with a.setsockopt( zmq.REQ_CORRELATE, { on | off } ) inside its for-loop and record the observed changes in the expected outputs.
This adds a final touch to the definitive answer, as far as the API v.4.2.x permits in the 2018-Q2.
Context
I have multiple servers listening to a specific collection (/items). Each of them use NTS for time calibration and the ".info/serverTimeOffset" to measure the expected time difference with Firebase. It is consistently around 20ms.
I have many clients pushing items to the collection with the specific field:
{
...
created: Firebase.database.ServerValue.TIMESTAMP
}
What is expected:
When the server receives the item from Firebase and subtracts the item.created with the Firebase expected time (Date.now() + offset), this value should be positive and probably around 10ms (time for the item to be sent from Firebase to the server).
What is happening:
When the server receives the items, the item.created field is superior to the Firebase expected time. Like it was created in the future. Usually the difference is around -5ms
Question:
What is the Firebase.database.ServerValue.TIMESTAMP set to ? and how is it related to the ".info/serverTimeOffset" ?
The 27th September 2016 at 1am UTC, that difference jumped from -5ms to around -5000ms like a kind of re-calibration happened (it lasted until I reset the .info/serverTimeOffset) Did someone experienced something similar?