How to create multiple EmbeddedKafka, right now i have defined two topics on EmbeddedKafka.
#EmbeddedKafka(
partitions = 1,
topics = [Constants.TEST_TOPIC_one, Constants.TEST_TOPIC_two],
controlledShutdown = false,
brokerProperties = [
"offsets.topic.replication.factor=1",
"transaction.state.log.replication.factor=1",
"transaction.state.log.min.isr=1"
]
)
but i need to have two instance like below,
#EmbeddedKafka(
partitions = 1,
topics = [Constants.TEST_TOPIC_one],
controlledShutdown = false,
brokerProperties = [
**"offsets.topic.replication.factor=1",
"transaction.state.log.replication.factor=1",
"transaction.state.log.min.isr=1"**
]
)
#EmbeddedKafka(
partitions = 1,
topics = [Constants.TEST_TOPIC_two],
controlledShutdown = false,
brokerProperties = [
**"offsets.topic.replication.factor=2",**
"transaction.state.log.replication.factor=1",
"transaction.state.log.min.isr=1"
]
)
in the above block i will be changing replication.factor
If that the case then, you need to create an EmbeddedKafkaBroker manually, not relying on the annotation processing (and probably not as a bean): https://docs.spring.io/spring-kafka/docs/current/reference/html/#testing
Related
I want to use following for expression to create CloudWatch dashboard using TF 0.13.7
Variables:
variable "redis_replicas" {
type = list(map(string))
default = [
{
name = "test-eucentral1-redis-001"
region = "eu-central-1"
},
locals {
replicas = [for i in var.redis_replicas: i.name]
regions = [for i in var.redis_replicas: i.region]
resources
resource "aws_cloudwatch_dashboard" "cloudwatch_db_redis" {
dashboard_name = "Redis-Performance"
dashboard_body = jsonencode({
widgets : concat(local.body1)
})
}
body1 = [
{
height = 6
properties = {
metrics = [ for replica in local.replicas :
[
{
expression = "m1"
id = "e1"
},
], # getting error here
[
"AWS/ElastiCache",
"ClusterBasedCmds",
"CacheClusterId",
replica,
],
]
view = "timeSeries"
}
type = "metric"
}
]
Using
metrics = [for instance in var.redis_replicas : [
[],
[],
]
Plan works but apply is failing.
Error: Putting dashboard failed: InvalidParameterInput: The dashboard body is invalid, there are 50 validation errors:
[
{
"dataPath": "/widgets/9/properties/metrics/0/0",
"message": "Invalid metric field type, only \"String\" type is allowed"
},
Any idea how to combine those two?
It works if I do for in body but then I have different graphs for different replica.
I have two models
Apartment Model
class Apartment(models.Model):
owner = models.ForeignKey(LandOwner, on_delete=models.CASCADE )
property_size = models.CharField(max_length = 255, null =False, blank = False)
location = models.CharField(max_length=255, blank=False, null=False )
name = models.CharField(max_length=255, blank=False, null=False )
town = models.CharField(max_length=255, blank=False, null=False )
category = models.ManyToManyField(HouseCategory)
no_of_house = models.IntegerField(blank=False, null=False )
year = models.IntegerField(blank=False, null=False )
House Model
class House(models.Model):
category = models.ForeignKey(HouseCategory, on_delete = models.CASCADE)
apartment = models.ForeignKey(Apartment, on_delete=models.CASCADE )
cost = models.CharField(max_length = 30, null=False, blank = False)
description = models.TextField(null = True, blank = True)
features = models.ManyToManyField(HouseFeatures)
occupied = models.IntegerField()
total = models.IntegerField()
Their serializers
ApartmentSerializer
class PropertySerializer(ModelSerializer):
class Meta:
model = models.Apartment
fields = '__all__'
HouseSerializer
class HouseSerializer(ModelSerializer):
class Meta:
model = models.House
fields = [
'description',
'apartment',
'category',
'occupied',
'cost',
'total',
'features',
]
depth = 1
A Get request to the following view
class Listing(ListCreateAPIView):
permission_classes = [ permissions.AllowAny ]
# permission_classes = [ permissions.IsAuthenticatedOrReadOnly ]
serializer_class = serializers.HouseSerializer
queryset = House.objects.all()
I get this JSON which my designed output
{
"description": "",
"apartment": {
"id": 1,
"property_size": "30Ha",
"location": "Nairobi",
"name": "Real Estate Apartment",
"town": "Nairobi",
"no_of_house": 4,
"year": 2019,
"owner": 1,
"category": [
1
]
},
"category": {
"id": 1,
"name": "Single Room"
},
"occupied": 2,
"cost": "40000",
"total": 5,
"features": [
{
"id": 1,
"name": "Source of Water"
},
{
"id": 2,
"name": "wifi availability"
},
{
"id": 3,
"name": "Swimming pool available"
}
]
}
The problem is when I send a POST request to the same view I get "NOT NULL constraint failed: House.apartment_id". The JSON sent with POST request is
{
"apartment": 10,
"cost": "40000",
"description": "",
"occupied": 2,
"total": 5,
"category": 1,
"features": [
1,
2,
3
]
}
What I intended the view to do with the post request was to save a new house with the information given and set the related field apartment to be the id I sent with the JSON payload
Change your Serializer like this:
class HouseSerializer(ModelSerializer):
apartment = PropertySerializer(read_only=True, many=True)
class Meta:
model = models.House
fields = [
'description',
'apartment',
'category',
'occupied',
'cost',
'total',
'features',
]
I found a solution for this question, on the property serializer, I overrode the method def to_reperesentation(self, instance) to this
def to_representation(self, instance):
self.fields['owner'] = LandOwnerSerializer(read_only = True)
self.fields['category'] = HouseCategorySerializer(read_only = True, many = True)
return super(PropertySerializer, self).to_representation(instance)
And in the HouseSerializer I also did the same ( overriding the to_representation) method and added a bit of modification to look like this
def to_representation(self, instance):
self.fields['apartment'] = PropertySerializer(read_only = True, )
return super(HouseSerializer, self).to_representation(instance)
Now it's working just fine, adding a house only takes the Id of the target apartment while fetching fetches the apartment details. Thanks all for your participation.
I am trying to figure out how to read from additional values in Terraform using for / for_each using Terraform 0.12.26
dns.tfvars
mx = {
"mywebsite.org." = {
ttl = "3600"
records = [
"home.mywebsite.org.",
"faq.mywebsite.org."
]
}
"myotherwebsite.org." = {
ttl = "3600"
records = [
"home.myotherwebsite.org."
]
}
}
variables.tf
variable "mx" {
type = map(object({
ttl = string
records = set(string)
}))
}
mx.tf
locals {
mx_records = flatten([
for mx_key, mx in var.mx : [
for record in mx.records : {
mx_key = mx_key
record = record
ttl = mx.ttl
}]
])
}
resource "aws_route53_record" "mx_records" {
for_each = { for mx in local.mx_records : mx.mx_key => mx... }
zone_id = aws_route53_zone.zone.zone_id
name = each.key
type = "MX"
ttl = each.value.ttl
records = [
each.value.record
]
}
In mx.tf, I can comment out the second value, faq.mywebsite.org, and the code works perfectly. I cannot figure out how to set up my for loop and for each statements to get it to "loop" through the second value. The first error I had received stated below:
Error: Duplicate object key
on mx.tf line 13, in resource "aws_route53_record" "mx_records":
13: for_each = { for mx in local.mx_records : mx.mx_key => mx }
|----------------
| mx.mx_key is "mywebsite.org."
Two different items produced the key "mywebsite.org." in this 'for'
expression. If duplicates are expected, use the ellipsis (...) after the value
expression to enable grouping by key.
To my understanding, I do not have two duplicate values helping to form the key so I should not have to use the ellipsis, but I tried using the ellipsis anyway to see if it would apply properly. After adding on the ellipsis after the value expression, I got this error:
Error: Unsupported attribute
on mx.tf line 20, in resource "aws_route53_record" "mx_records":
20: each.value.record
|----------------
| each.value is tuple with 2 elements
This value does not have any attributes.
Any advice on this issue would be appreciated.
UPDATE
Error: [ERR]: Error building changeset: InvalidChangeBatch: [Tried to create resource record set [name='mywebsiteorg.', type='MX'] but it already exists]
status code: 400, request id: dadd6490-efac-47ac-be5d-ab8dad0f4a6c
It's trying to create the record, but it already created because of the first record in the list.
I think you could just construct a map of your objects with key being the index of mx_records list (note the idx being the index):
resource "aws_route53_record" "mx_records" {
for_each = { for idx, mx in local.mx_records : idx => mx }
zone_id = aws_route53_zone.zone.zone_id
name = each.value.mx_key
type = "MX"
ttl = each.value.ttl
records = [
each.value.record
]
}
The above for_each expressions changes your local.mx_records from list(objects) to map(objects), where the map key is idx, and the value is the original object.
Update:
I verified in Route53 and you can't duplicate codes. Thus may try using orginal mx variable:
resource "aws_route53_record" "mx_records" {
for_each = { for idx, mx in var.mx : idx => mx }
zone_id = aws_route53_zone.zone.zone_id
name = each.key
type = "MX"
ttl = each.value.ttl
records = each.value.records
}
Moreover, if you want to avoid flatten function and for loop local variable, you can access the object in the map as:
resource "aws_route53_record" "mx_records" {
for_each = var.mx
zone_id = aws_route53_zone.zone.zone_id
name = each.key
type = "MX"
ttl = each.value["ttl"]
records = each.value["records"]
}
I have created a custom edge class as defined here. The only change I made was a No Arg constructor in order to get the import code below to run. I have successfully generated a DirectedMultigraph via the JSONExporter class and now want to take that exported JSON and re-import it via the JSONImporter class.
I'm having trouble doing this and retaining my edge labels due to my limited understanding of how to build the EdgeProvider required for my JSONImporter constructor.
This is the JSON I'm trying to import:
{
"edges": [
{
"id": "{shipmentNumber:12345}",
"source": "DEHAM",
"target": "USNYC"
}
],
"nodes": [
{
"id": "DEHAM"
},
{
"id": "USNYC"
}
],
"creator": "JGraphT JSON Exporter",
"version": "1"
}
This is the code that I have so far:
Graph<String, RelationshipEdge> graph = new DirectedMultigraph<>(SupplierUtil.createStringSupplier(), SupplierUtil.createSupplier(RelationshipEdge.class), false);
VertexProvider<String> vertexProvider = (label, attributes) -> label;
EdgeProvider<String, RelationshipEdge> edgeProvider =
(from, to, label, attributes) -> graph.getEdgeSupplier().get();
JSONImporter<String, RelationshipEdge> importer = new JSONImporter<>(vertexProvider, edgeProvider);
importer.importGraph(graph, new StringReader([inputJSON]);
I know the problem is the EdgeProvider assignment because I don't know how to pass the argument constructor for the RelationshipEdge class which is where the actual label is set. If I could figure out how to call the argument constructor of the RelationshipEdge class then I think that would solve my problem.
FYI, this is my JSONExporter Code:
ComponentNameProvider<String> vertexIdProvider = name -> name;
ComponentNameProvider<RelationshipEdge> edgeLabelProvider = component -> component.getLabel();
ComponentAttributeProvider<String> vertexAttributeProvider = component -> new HashMap<>();
ComponentAttributeProvider<RelationshipEdge> edgeAttributeProvider = component -> new HashMap<>();
GraphExporter<String, RelationshipEdge> jsonExporter = new JSONExporter<>(vertexIdProvider, vertexAttributeProvider, edgeLabelProvider, edgeAttributeProvider);
StringWriter writer = new StringWriter();
jsonExporter.exportGraph(graph, writer);
System.out.println(writer.toString());
The following JSON is exported (with the label/id missing):
{
"creator": "JGraphT JSON Exporter",
"version": "1",
"nodes": [
{
"id": "DEHAM"
},
{
"id": "USNYC"
}
],
"edges": [
{
"source": "DEHAM",
"target": "USNYC"
}
]
}
Since your graph is using the SupplierUtil factory methods, the graph importer will use no-arg constructors as it visits/parses the json file, iterates over all the nodes/edges in the file and creates instances for them. In order to set your node/edge attributes during import you need to provide appropriate Vertex/Edge Attribute Consumers to the importer. Additionally, your node/edge classes need methods to access/modify their attributes:
JSONImporter<String, RelationshipEdge> importer = new JSONImporter<>(vertexProvider, edgeProvider);
importer.addEdgeAttributeConsumer((edgeField, value) -> {
// edgeField: Pair<Edge, String> / Edge instance, field name
// value: instance of org.jgrapht.nio.Attribute / getValue(); getType();
RelationshipEdge = edgeField.getFirst();
e.setLabel(value.getValue()); // Note that since you only have one field I am not finding the
// correct setter. I don't like reflection so I would probably
// add a 'setField(name, value)' method to my edge/node implementation.
});
To persist your graph, the idea is the same. But in this case you need Vertex/Edge AttributeProviders (You added them but they always returned empty maps):
GraphExporter<String, RelationshipEdge> jsonExporter = new JSONExporter<>();
exporter.setEdgeAttributeProvider(e -> {
Map<String, Attribute> attribs = new HashMap<>();
attribs.put("label", new DefaultAttribute<String>(e.getLabel(), AttributeType.STRING));
return attribs;
});
I have a dictionary containing a lot of fields, and if possible I would like to index it as it is inside ElasticSearch. I am using Python for this. Could it work or is it a limitation of ElasticSearch that it can support only simple data types? Thanks!
Later edit: My dictionary looks like below:
{
'lingpipe': Tagging(tagger_id = 'lingpipe', raw_tagging = '', tagger_config = None, tagger_version = None, generation_time = StreamTime(epoch_ticks = 1364699587.0, zulu_timestamp = '2013-03-31T04:13:07.663542Z')),
'serif': Tagging(tagger_id = 'serif', raw_tagging = '', tagger_config = 'English', tagger_version = '6.1', generation_time = None),
'stanford': Tagging(tagger_id = 'stanford', raw_tagging = '1\tFree\tFree\tNNP\tO\t23\t27\n2\tBiscayne\tBiscayne\tNNP\tO\t28\t36\n3\tNational\tNational\tNNP\tO\t37\t45\n4\tPark\tPark\tNNP\tO\t46\t50\n5\tGuide\tGuide\tNNP\tO\t51\t56\n6\tGet\tget\tVB\tO\t57\t60\n7\tyour\tyou\tPRP$\tO\t61\t65\n8\tquick\tquick\tJJ\tO\t66\t71\n9\tguide\tguide\tNN\tO\t72\t77\n10\tto\tto\tTO\tO\t78\t80\n11\tthe\tthe\tDT\tO\t81\t84\n12\ttop\ttop\tJJ\tO\t85\t88\n13\thotels\thotel\tNNS\tO\t89\t95\n14\t,\t,\t,\tO\t95\t96\n15\trestaurants\trestaurant\tNNS\tO\t97\t108\n16\tand\tand\tCC\tO\t109\t112\n17\tthings\tthing\tNNS\tO\t113\t119\n18\tto\tto\tTO\tO\t120\t122\n19\tdo\tdo\tVB\tO\t123\t125\n20\t.\t.\t.\tO\t125\t126\n\n1\tGrab\tgrab\tVB\tO\t127\t131\n2\tit\tit\tPRP\tO\t132\t134\n3\tand\tand\tCC\tO\t135\t138\n4\tGo\tgo\tVB\tO\t139\t141\n5\t!\t!\t.\tO\t141\t142', tagger_config = 'annotators: {tokenize, cleanxml, ssplit, pos, lemma, ner}, properties: pos.maxlen=100', tagger_version = 'Stanford CoreNLP ver 1.2.0', generation_time = StreamTime(epoch_ticks = 1338505200.0, zulu_timestamp = '2012-06-01T00:00:00.0Z'))
}
so it's basically a mix of data types. I am not yet sure how I can serialize it, ES is throwing a Serialization error. How can I convert this object to JSON?