kafka-strams leftJoin produce NPE - apache-kafka-streams

I have the next values in topics
// photos
1, {"id": 1, user_id: 1, "url": "http://example.com"}
2, {"id": 2, user_id: 2, "url": "http://example1.com"}
3, {"id": 3, user_id: 1, "url": "http://example2.com"}
// users
1, {"id": 1, "name": "user1"}
2, {"id": 2, "name": "user2"}
I want to get info: [photo_id, photo_url, user_id, user_name]
I implement Result class for it
public class Result {
public int photo_id;
public String photo_url;
public int user_id;
public String user_name;
public static Result from(Photo photo, User user) {
Result r = new Result();
r.photo_id = photo.id;
r.photo_url = r.url;
r.user_id = user.id;
r.user_name = user.name;
return r;
}
}
And my stream implementation:
final KStream<Integer, Photo) photo_by_user = ...;
final KStream<Integer, User) users = ...;
users.leftJoin(photo_by_user, new ValueJoiner<User, Photo, Result> {
public Result apply(User user, Photo photo) {
return Result.from(user, photo);
}
}, JoinWindows.of(1L))
But when I run this code, I get:
Exception in thread "example-StreamThread-1" java.lang.NullPointerException
at myapps.util.Result.create(Result.java:15)
at myapps.Example$1.apply(Example.java:56)
at myapps.Example$1.apply(Example.java:53)
at org.apache.kafka.streams.kstream.internals.KStreamKStreamJoin$KStreamKStreamJoinProcessor.process(KStreamKStreamJoin.java:87)
at org.apache.kafka.streams.processor.internals.ProcessorNode$1.run(ProcessorNode.java:46)
Because User have a value, but Photo is null
But I do not understand why? And how to avoid it.

with leftJoin you can have null Photo values inside ValueJoiner apply method.
for each input record of users stream that does not satisfy the join predicate the provided ValueJoiner will be called with a null value for the photo_by_user stream.
inside Result.from() method you need to check that Photo instance is non null, and only after that get photo's fields id and url.
Also you have left join with JoinWindows.of(1L), where 1 is millisecond.
pay attention that two records are only joined if their timestamps are close to each other as defined by specified JoinWindows. potentially in your case two records were created not exactly at the same time, so try to increase value e.g. 10000L to test your joining logic.

Related

Spring Boot - MongoDB aggregation nested documents returns empty result

I have a collection named results where each document has a token (unique) and a list of embedded documents (services).
Sample:
{
_id: ObjectId("61e7eed15b9df6f80f0164c0"),
token: '7683af2f-8f93-4387-9b6a-c89840d9525f',
providers: [
{
_id: 2,
companyName: 'Autopro'
}
],
services: [
{
_id: 1,
serviceTypeId: 103
},
{
_id: 5,
serviceTypeId: 103
},
{
_id: 6,
serviceTypeId: 103,
}
]
}
I want to extract using mongo template (spring boot) one service based on token and service id.
here is a code snippet:
UnwindOperation unwindServicesOp = Aggregation.unwind("services");
ProjectionOperation projectionOp = Aggregation.project(Fields.fields("services")).andExclude("_id");
HashSet<Criteria> set = new HashSet<>();
set.add(Criteria.where("token").is(token));
set.add(Criteria.where("services._id").is(serviceId));
Criteria c = new Criteria();
c.andOperator(set);
MatchOperation matchOp = Aggregation.match(c);
return mongoTemplate.aggregate(Aggregation.newAggregation(Arrays.asList(unwindServicesOp, matchOp, projectionOp)),"results", Service.class).getUniqueMappedResult();
Service class:
public class Service implements Serializable {
private static final long serialVersionUID = -7786799739639015883L;
#Id
private Integer id;
private Integer serviceTypeId;
//setter and getters omitted for simplicity
}
Above code execute this query:
[{ "$unwind" : "$services"}, { "$match" : { "$and" : [{ "token" : "7683af2f-8f93-4387-9b6a-c89840d9525d"}, { "services._id" : 1}]}}, { "$project" : { "services" : 1, "_id" : 0}}]
getUniqueMappedResult() method returns empty Service object!
Is there any changes to do in my aggregation in order to get a Service object?
There is one more stage (last one) to add in order to solve the issue:
ReplaceRootOperation replaceRootOp = Aggregation.replaceRoot("services");
Add it at last in the aggregation object.
I works.

Django rest_framework not serializing relationships

I am trying to serialize Foreign keys inline with Django rest_framework. Foreign keys are used to link lookup tables as per a normal DB normalisation setup.
An example of my model with the lookup:
class OrderStatus(models.Model):
StatusId = models.PositiveSmallIntegerField(primary_key=True)
StatusDescription = models.CharField(max_length=50)
class Order(models.Model):
OrderId = models.AutoField(primary_key=True)
OrderDate = models.DateTimeField(auto_now_add=True)
Status = models.ForeignKey(OrderStatus, on_delete=models.CASCADE)
My serializers:
class OrderStatusSerializer(serializers.ModelSerializer):
class Meta:
model = OrderStatus
fields = ['StatusId', 'StatusDescription']
class OrderSerializer(serializers.ModelSerializer):
class Meta:
model = Order
fields = ('OrderId', 'OrderDate', 'Status')
What I obtain when I call the REST API is the following:
{
"type": "Order",
"id": "1",
"attributes": {
"OrderId": 1,
"OrderDate": "2020-05-19T08:23:54"
},
"relationships": {
"Status": {
"data": {
"type": "OrderStatus",
"id": "1"
}
}
}
}
I would like to have the Status inline in the "attributes", either as a simple id or even as a JSON object with the two values inline. Both are good options, as long as it's not in that "relationships" field.
I tried to add the following to OrderSerializer:
Status = serializers.PrimaryKeyRelatedField(queryset=OrderStatus.objects.all())
No difference.
I tried the following:
Status = OrderStatusSerializer(many=False)
No difference.
I tried all the other options in
https://github.com/encode/django-rest-framework/blob/master/docs/api-guide/relations.md
including the SlugField to include the description instead that the ID, with no result.
It seems that what I change has no effect on the serialization.
As per my own comment, the issue was caused by rest_framework_json_api.renderers.JSONRenderer.
By switching it back to rest_framework.renderers.JSONRenderer the expected documented behaviour has been re-established.

How to hide json field when using Springdata pageable?

Currently I'm using SpringData to build my restful project.
I'm using Page findAll(Pageable pageable, X condition, String... columns); ,this method .The result looks like this:
{
"content": [
{
"id": 2,
"ouId": 1,
"pClassId": 3,
"isPublic": 0,
"accessMethod": 3,
"modifierName": null
}
],
"last": true,
"totalPages": 1,
"totalElements": 3,
"number": 0,
"size": 10,
"sort": [
{
"direction": "DESC",
"property": "id",
"ignoreCase": false,
"nullHandling": "NATIVE",
"ascending": false,
"descending": true
}
],
"first": true,
"numberOfElements": 3
}
The question is how to hide some specific json field in the content ?
And #JsonIgnore annotation is not flexible , the fields I need in different APIs are different.
I tried to write an annotation ,but when processing the Page ,I found that the content is unmodifiable .
So , hope that someone could help me with it.
If you don't want to put annotations on your Pojos you can also use Genson.
Here is how you can exclude a field with it without any annotations (you can also use annotations if you want, but you have the choice).
Genson genson = new Genson.Builder().exclude("securityCode", User.class).create();
// and then
String json = genson.serialize(user);
OR using flexjson
import flexjson.JSONDeserializer;
import flexjson.JSONSerializer;
import flexjson.transformer.DateTransformer;
public String toJson(User entity) {
return new JSONSerializer().transform(new DateTransformer("MM/dd/yyyy HH:mm:ss"), java.util.Date.class)
.include("wantedField1","wantedField2")
.exclude("unwantedField1").serialize(entity);
}
You have to use a custom serialize like the following:
#JsonComponent
public class MovieSerializer extends JsonSerializer<Movie> {
#Override
public void serialize(Movie movie, JsonGenerator jsonGenerator, SerializerProvider serializerProvider) throws IOException {
jsonGenerator.writeStartObject();
// The basic information of a movie
jsonGenerator.writeNumberField("id", movie.getId());
jsonGenerator.writeStringField("name", movie.getName());
jsonGenerator.writeStringField("poster", movie.getPoster());
jsonGenerator.writeObjectField("releaseDate", movie.getReleaseDate());
jsonGenerator.writeObjectField("runtime", movie.getRuntime());
jsonGenerator.writeStringField("storyline", movie.getStoryline());
jsonGenerator.writeStringField("rated", movie.getRated());
jsonGenerator.writeNumberField("rating", movie.getRating());
jsonGenerator.writeEndObject();
}
}
And then annotate your model class with: #JsonSerialize(using = MovieSerializer.class)

Passing multiple parameters to web API GET method

I have created a WEB API using MySQL Database. The API works as expected for now. I sent a meter serial number and a date time parameter and then GET the expected result. Below is my controller
public MDCEntities medEntitites = new MDCEntities();
public HttpResponseMessage GetByMsn(string msn, DateTime dt)
{
try
{
var before = dt.AddMinutes(-5);
var after = dt.AddMinutes(5);
var result = medEntitites.tj_xhqd
.Where(m =>
m.zdjh == msn &&
m.sjsj >= before &&
m.sjsj <= after).Select(m => new { MSN = m.zdjh, DateTime = m.sjsj, Signal_Strength = m.xhqd }).Distinct();
return Request.CreateResponse(HttpStatusCode.Found, result);
}
catch (Exception ex)
{
return Request.CreateErrorResponse(HttpStatusCode.NotFound, ex);
}
}
Below is my WebApiConfig file
config.Routes.MapHttpRoute(
name: "GetByMsn",
routeTemplate: "api/{controller}/{action}/{msn}/{dt}",
defaults: null,
constraints: new { msn = #"^[0-9]+$" , dt = #"^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}$" }
);
The URL is http://localhost:14909/api/meters/GetByMsn/002999000171/2017-10-10T10:08:20
The response I GET is
[{
"MSN": "002999000171",
"DateTime": "2017-10-10T10:04:39",
"Signal_Strength": "20"
},
{
"MSN": "002999000171",
"DateTime": "2017-10-10T10:06:35",
"Signal_Strength": "19"
},
{
"MSN": "002999000171",
"DateTime": "2017-10-10T10:08:31",
"Signal_Strength": "20"
},
{
"MSN": "002999000171",
"DateTime": "2017-10-10T10:10:27",
"Signal_Strength": "20"
},
{
"MSN": "002999000171",
"DateTime": "2017-10-10T10:12:23",
"Signal_Strength": "20"
}]
This all scenario works when a single serial number is passed. But at client side there would be more than one different serial numbers. For this I had to make my method to work both for one and more than one serial numbers provided the date time will be the same for all.
One solution is to create a new method a pass the multiple serial number strings, but this will not help because the number of serial numbers are dynamic i.e. they may be one, two to 100's. So setting a hard coded method won't be a solution.
I have searched for it but most of the times I have found the static method again and again. But this solution looks some what helpful but again I don't know whether it will work or not.
Any help would be highly appreciated.
You can pass a custom model to an action method, but I suggest not using the GET for you task because GET does not have a body.
Instead, use the SEARCH verb and put the list of serials number and the date inside a custom model in the body.
public class MeterSearchModel
{
public List<string> Serials {get;set;}
public DateTime Date {get;set;}
}
In .NET Core 2 your controller would have something like -
[AcceptVerbs("SEARCH")]
public async Task<IActionResult> Search([FromBody] MeterSearchModel model)
{
//..perform search
}

Consuming REST Service in Spring

I'm frightfully new to Spring and Java but I'm trying to consume some code for some rule validations in Easy Rules but I can't quite figure it out.
#RequestMapping(method = {RequestMethod.GET}, value = "author/field", produces= MediaType.APPLICATION_JSON_VALUE)
public #ResponseBody ResponseEntity<Enum> getField(#RequestParam(value="field", required=true) String field){
Enum enum = mongoService.findByField(field);
if(enum == null){
return new ResponseEntity<Enum>(HttpStatus.NO_CONTENT);
}else{
return new ResponseEntity<Enum>(enum,HttpStatus.OK);
}
}
So I'm trying something like:
import com.mongoservice.Enum
import com.mongoservice.Enums
RestTemplate restTemplate = new RestTemplate();
String uri = "http://localhost:9000";
//This is my confusion
List<Enums> response = restTemplate.getForObject(uri +
"/author/field?={field}", Enum.class,"a").getEnums();
String value = response.getValue().toString().trim();
//this is the record i'm checking against that is pulling a specific string value and what i'm expecting
String record = "a";
return (value == record);
The JSON data I'm trying to pull back is modeled like this but I need to validate to make sure that record equals one of the values from enums[] json array
{
"field": "a",
"descriptor": "blah",
"enums": [
{
"value": "h",
"description": "blah"
},
{
"value": "e",
"description": "blah"
},
{
"value": "l",
"description": "blah"
},
{
"value": "p",
"description": "blah"
}
]
}
What is the problem that you are seeing is it just not matching? If so it could be because you are using == instead of String.equals. Try modifying your code to:
return record.equals(value);
See Java String.equals versus == for more.
Can you change String uri = "http://localhost:9000"
and missed the path variable name field it should be like author/field?field={field} as per your controller description.

Resources