When you create a window using kafka streams I assume it uses the timestamp the record was published? Is there anyway to window by something else.
My use case is our record value object contains a timestamp and that is what we want to window on.
If I do something like this it will window on published time stamp. I want to window by myObject.getCallTimestamp()
KTable<Windowed<String>, MyObject> windowedPageViewCounts = pageViews
.groupByKey(Serialized.with(Serdes.String(), myObjectSerde))
.windowedBy(TimeWindows.of(TimeUnit.MINUTES.toMillis(5)))
.count();
EDIT:
Based on recommendation below I believe this is what I need to be doing?
public class RecordTimeStampExtractor implements TimestampExtractor {
//default timestamp extractor
private FailOnInvalidTimestamp failOnInvalidTimestamp = new FailOnInvalidTimestamp();
#Override
public long extract(ConsumerRecord<Object, Object> consumerRecord, long l) {
//could also use consumerRecord.topic().equals("mytopic")
if(consumerRecord.value() instanceof MyClass) {
MyClass myClass = (MyClass) consumerRecord.value();
return myClass.getRecordTimestamp().toEpochMilli();
}
return failOnInvalidTimestamp.extract(consumerRecord,l);
}
}
You can implement and configure (via default.timestamp.extractor) a custom TimestampExtractor that returns myObject.getCallTimestamp().
See the docs for details: https://docs.confluent.io/current/streams/developer-guide/config-streams.html#streams-developer-guide-timestamp-extractor
Related
I'm going to use #InsertOnlyProperty with Spring Boot 2.7 as it will take time for us to migrate to Spring Boot 3.0!
So I'm going to create my DataAccessStrategy based on the DefaultAccessStrategy and also override the SqlParametersFactory so that I can pass the RelationalPersistentProperty::isInsertOnly condition to the getParameterSource method, also overriding RelationalPersistentProperty by adding isInsertOnly. And is there a way to override RelationalPersistentProperty to add isInsertOnly property. Am I correct or is there a better solution than switching to Spring Boot 3.0 now. Thank you!
Since #InsertOnlyProperty is only supported for the aggregate root (in Spring Boot 3.0), one approach could be to copy the data to a surrogate object and use a custom method to save it. It would look something like this:
public record MyAggRoot(#Id Long id,
/* #InsertOnlyProperty */ Instant createdAt, int otherField) {}
public interface MyAggRootRepository
extends Repository<MyAggRoot, Long>, MyAggRootRepositoryCustom { /* ... */ }
public interface MyAggRootRepositoryCustom {
MyAggRoot save(MyAggRoot aggRoot);
}
#Component
public class MyAggRootRepositoryCustomImpl implements TaskRepositoryCustom {
#Autowired
private final JdbcAggregateOperations jao;
// Override table name which would otherwise be derived from the class name
#Table("my_agg_root")
private record MyAggRootForUpdate(#Id Long id, int otherField) {}
#Override
public MyAggRoot save(MyAggRoot aggRoot) {
// If this is a new instance, insert as-is
if (aggRoot.id() == null) return jao.save(aggRoot);
// Create a copy without the insert-only field
var copy = new MyAggRootForUpdate(aggRoot.id(), aggRoot.otherField());
jao.update(copy);
return aggRoot;
}
}
It is however a bit verbose so it would only be a reasonable solution if you only need it in a few places.
My Audit logs are getting out of hand so I decided I want to only audit all requests which basically are not a Get request. Is there a very simply way to do this from configuration?
The documentation here:
https://aspnetboilerplate.com/Pages/Documents/Audit-Logging
Says:
Note: In addition to the standard audit configuration, MVC and ASP.NET
Core modules define configurations to enable/disable audit logging for
actions.
But I could not find more information about what exactly this means.
As a last resort, I know it would work if I went to every class and added [DisableAuditing] and then [Audited] on the non-Get endpoints, but that seems a bit messy.
Best soltuion: I just want to have a simply way to select only non-GET requests and audit them.
Second best solution:
I just want to have only [Audited] methods audited. I don't want to have to go and write [DisabledAuditing] on every class.
You can create an AuditStore to do that, and then replace the original AuditStore in service YourAplicationNameCoreModule
Here is the example
public class YourAuditStore : AuditingStore
{
public ILogger<AuditingStore> Logger { get; set; }
private readonly IRepository<AuditLog, long> _auditLogRepository;
private readonly ISettingManager _settingManager;
public YourAuditStore(IRepository<AuditLog, long> auditLogRepository, ISettingManager settingManager) : base(auditLogRepository)
{
_auditLogRepository = auditLogRepository;
_settingManager = settingManager;
}
public override async Task SaveAsync(AuditInfo auditInfo)
{
AuditLog auditLog = new AuditLog();
bool logErrorsOnly = await _settingManager.GetSettingValueAsync<bool>(AppSettings.Logging.LogOnErrorsOnly);
var exceptionMessage = auditInfo.Exception != null ? auditInfo.Exception.ToString() : null;
if ((logErrorsOnly && exceptionMessage != null) || !logErrorsOnly)
{
auditLog = await _auditLogRepository.InsertAsync(AuditLog.CreateFromAuditInfo(auditInfo));
}
}
}
As you can see, you can filter whatever you want in SaveAsync method as it recieve the AuditInfo, you can check if method is different to Get then save
Add the next code to YourApplicationNameCoreModule on PreInitialize method
public override void PreInitialize()
{
Configuration.ReplaceService<IAuditingStore, YourAuditStore>();
}
I want to make the save operation efficient, so I'd like to write a bulk of objects to Mongo once in a while (i.e. when exceeding some capacity)
Would saveAll() do that for me? Should I use BulkOperations instead?
Short answer, yes, but only if all documents are new. If not, it will insert or update one by one.
Take a look at SimpleMongoRepository (MongoRepository's default implementation):
public <S extends T> List<S> saveAll(Iterable<S> entities) {
Assert.notNull(entities, "The given Iterable of entities not be null!");
Streamable<S> source = Streamable.of(entities);
boolean allNew = source.stream().allMatch((it) -> {
return this.entityInformation.isNew(it);
});
if (allNew) {
List<S> result = (List)source.stream().collect(Collectors.toList());
return new ArrayList(this.mongoOperations.insert(result, this.entityInformation.getCollectionName()));
} else {
return (List)source.stream().map(this::save).collect(Collectors.toList());
}
}
Notice that when all documents are new, the repository will use MongoOperations.insert method (MongoTemplate is the implementation), Then, if you look at that method's code you'll realize it does a batch insert:
public <T> Collection<T> insert(Collection<? extends T> batchToSave, String collectionName) {
Assert.notNull(batchToSave, "BatchToSave must not be null!");
Assert.notNull(collectionName, "CollectionName must not be null!");
return this.doInsertBatch(collectionName, batchToSave, this.mongoConverter);
}
UPDATE 2021:
As of spring-data-mongodb 1.9.0.RELEASE (current 3.2.2), BulkOperations comes with a lot of extra features.
If more advanced tasks are needed other than just saving a bunch of documents, then BulkOperations class is the way to go.
It covers bulk inserts, updates, and deletes:
insert(List<? extends Object> documents)
remove(List removes)
updateMulti(List<Pair<Query,Update>> updates)
The client periodically calls an async method (long polling), passing it a value of a stock symbol, which the server uses to query the database and return the object back to the client.
I am using Spring's DeferredResult class, however I'm not familiar with how it works. Notice how I am using the symbol property (sent from client) to query the database for new data (see below).
Perhaps there is a better approach for long polling with Spring?
How do I pass the symbol property from the method deferredResult() to processQueues()?
private final Queue<DeferredResult<String>> responseBodyQueue = new ConcurrentLinkedQueue<>();
#RequestMapping("/poll/{symbol}")
public #ResponseBody DeferredResult<String> deferredResult(#PathVariable("symbol") String symbol) {
DeferredResult<String> result = new DeferredResult<String>();
this.responseBodyQueue.add(result);
return result;
}
#Scheduled(fixedRate=2000)
public void processQueues() {
for (DeferredResult<String> result : this.responseBodyQueue) {
Quote quote = jpaStockQuoteRepository.findStock(symbol);
result.setResult(quote);
this.responseBodyQueue.remove(result);
}
}
DeferredResult in Spring 4.1.7:
Subclasses can extend this class to easily associate additional data or behavior with the DeferredResult. For example, one might want to associate the user used to create the DeferredResult by extending the class and adding an additional property for the user. In this way, the user could easily be accessed later without the need to use a data structure to do the mapping.
You can extend DeferredResult and save the symbol parameter as a class field.
static class DeferredQuote extends DeferredResult<Quote> {
private final String symbol;
public DeferredQuote(String symbol) {
this.symbol = symbol;
}
}
#RequestMapping("/poll/{symbol}")
public #ResponseBody DeferredQuote deferredResult(#PathVariable("symbol") String symbol) {
DeferredQuote result = new DeferredQuote(symbol);
responseBodyQueue.add(result);
return result;
}
#Scheduled(fixedRate = 2000)
public void processQueues() {
for (DeferredQuote result : responseBodyQueue) {
Quote quote = jpaStockQuoteRepository.findStock(result.symbol);
result.setResult(quote);
responseBodyQueue.remove(result);
}
}
I really like #SubscribeMapping approach to implement RPC-like semantic with STOMP-over-Websocket.
Unfortunately its "magic" requires that annotated method returns a value. But what if return value is not readily available? I want to avoid blocking inside the method waiting for it. Instead I'd like to pass a callback that will publish a value when it's ready. I thought I could use messaging template's convertAndSendToUser() inside a callback to do that. Turns out #SubscribeMapping handling is quite special and is not possible with instance of SimpMessageSendingOperations.
I was able to achieve my goal by calling handleReturnValue() on a SubscriptionMethodReturnValueHandler, but the overall mechanics of this is very tedious if not hackish (like providing dummy instance of MethodParameter to handleReturnValue()):
public class MessageController {
private final SubscriptionMethodReturnValueHandler subscriptionMethodReturnValueHandler;
#Autowired
public MessageController(SimpAnnotationMethodMessageHandler annotationMethodMessageHandler) {
SubscriptionMethodReturnValueHandler subscriptionMethodReturnValueHandler = null;
for (HandlerMethodReturnValueHandler returnValueHandler : annotationMethodMessageHandler.getReturnValueHandlers()) {
if (returnValueHandler instanceof SubscriptionMethodReturnValueHandler) {
subscriptionMethodReturnValueHandler = (SubscriptionMethodReturnValueHandler) returnValueHandler;
break;
}
}
this.subscriptionMethodReturnValueHandler = subscriptionMethodReturnValueHandler;
}
#SubscribeMapping("/greeting/{name}")
public void greet(#DestinationVariable String name, Message<?> message) throws Exception {
subscriptionMethodReturnValueHandler.handleReturnValue("Hello " + name, new MethodParameter(Object.class.getMethods()[0], -1), message);
}
}
So my question is simple: Is there a better way?