Here's what one of my document might look like
{
"CC":{"colors":["Blue","Green","Yellow"]},
"CN":{"colors":["White","Green","Blue"]},
"WA":{"colors":["Orange","Green","Blue"]},
...
}
I want a terms aggregation, on the intersection of two fields CC.colors and CN.colors. That is, for this document, that field will have ["Green", "Blue"] in the intersection, and I want a term aggregation on this intersection.
As far as I understand, there are two ways to do it.
1) A painless script in terms aggregation, which returns the intersection of these two arrays for each document.
2) A new field created during index time, maybe called CC_CN.colors, which holds intersection for all docs.
I can't go ahead with 2 because my combinations will be too many. I can have any need during search time, like CC_CN, or CC_WA, or WA_CN_CC etc.
For 1), it works, but gets painfully slow. One reason is that 1) cannot use global ordinals.
Is there any trick, that I can ask elastic to build a custom global ordinal for my painless terms aggregation? I know there are just 25 colors in my system, so can give all colors to elastic somewhere, and "assure" them that I'll not return anything else but these colors from my aggregation?
Or, if I encode and store numbers instead of strings in index, would this be faster for elastic? e.g. 0 instead of "Black", 1 instead of "Green" etc.?
Other than intersection, my other use cases involve union etc. as well. Thanks for reading!
To answer it myself, we ended up asking for these arrays in _source and performing union/intersection in Ruby.
It is also possible to do this in painless, and that offers a bit better performance. Elastic uses map to do aggregation, and I couldn't figure out any way to use global ordinals. I don't think its possible.
We wrote code that generates painless code to perform intersection and union between arrays.
For any future wanderer, here's what the generated code looks like:
This is for union:
Stream stream = [].stream();
String[] stream_keys = new String[] {'CC.colors', 'CN.colors'};
for (int i = 0; i < stream_keys.length; ++i) {
if (doc.containsKey(stream_keys[i])) {
stream = Stream.concat(stream, doc[stream_keys[i]].stream());
}
}
stream = stream.distinct();
And this is for intersection (stream, list_0_stream and list_1_stream intersection):
List list_0 = list_0_stream.collect(Collectors.toList());
List list_1 = list_1_stream.collect(Collectors.toList());
return stream.filter(list_0::contains).filter(list_1::contains).toArray();
The performance seems to be acceptable.
There are 2 method offer to you
Cost alot of my time
^ ^
ColorMap={"Blue":0,"Green":1,"Yellow":2,"White":3,"Orange":4};
ReverseColorMap=["Blue","Green","Yellow","White","Orange"];
var All={
"CC":{"colors":["Blue","Green","Yellow"]},
"CN":{"colors":["White","Green","Blue"]},
"WA":{"colors":["Orange","Green","Blue"]}
};
//Cover Encode
function EncodeColor(T1){
var T2 = JSON.parse(JSON.stringify(T1));//Clone Original
for(var i in T2){
for(var j in T2[i]["colors"]){
T2[i]["colors"][j]=ColorMap[T2[i]["colors"][j]];
}
}
return T2;
}
var NewAll=EncodeColor(All);
console.log(All);
console.log(NewAll);
function SortColor(T1){
for(var i in T1){
T1[i]["colors"].sort((a, b) => {
return a-b;
});
}
}
function BuildSameColor(T1){
var CombineNew={};
var Name_Temp=[];
for(var i in T1){
Name_Temp.push(i);
}
for(var i =0;i<Name_Temp.length;i++){
for(var j =i+1;j<Name_Temp.length;j++){//j=i+1 because CC_CC not valid CC_CN is valid etc...
CombineNew[Name_Temp[i]+"_"+Name_Temp[j]]={"colors":T1[Name_Temp[i]]["colors"].concat(T1[Name_Temp[j]]["colors"])};//combine color array
}
}
SortColor(CombineNew);//Sort Result
//Sort will reduce compare time(later) when color is a lot
for(var i in CombineNew){
var NewAr=[];
for(var j=0;j<CombineNew[i]["colors"].length-1;j++){
if(CombineNew[i]["colors"][j]==CombineNew[i]["colors"][j+1]){
NewAr.push(CombineNew[i]["colors"][j]);
}
}
CombineNew[i]["colors"]=NewAr;
}
return CombineNew;
}
var TTT=BuildSameColor(NewAll);
console.log(TTT);
//Then Decode Color
function DecodeColor(T1){
var T2 = JSON.parse(JSON.stringify(T1));//Clone Original
for(var i in T2){
for(var j in T2[i]["colors"]){
T2[i]["colors"][j]=ReverseColorMap[T2[i]["colors"][j]];
}
}
return T2;
}
var TTTQQ=DecodeColor(TTT);
console.log(TTTQQ);
//This Also work any length of color
var Examp={
"CC":{"colors":["Blue","Green","Yellow","Orange"]},
"CN":{"colors":["White","Green","Blue"]},
"WA":{"colors":["Orange","Green","Blue"]}
};
var E_Examp=EncodeColor(Examp);
var Com_E_E_Examp=BuildSameColor(E_Examp);
var D_Examp=DecodeColor(Com_E_E_Examp);
console.log(D_Examp);
Enjoin it!!!!!!!!
ColorMap={"Blue":0,"Green":1,"Yellow":2,"White":3,"Orange":4};
ReverseColorMap=["Blue","Green","Yellow","White","Orange"];
var All={
"CC":{"colors":["Blue","Green","Yellow"]},
"CN":{"colors":["White","Green","Blue"]},
"WA":{"colors":["Orange","Green","Blue"]}
};
//Cover Encode
function EncodeColor(T1){
var T2 = JSON.parse(JSON.stringify(T1));//Clone Original
for(var i in T2){
for(var j in T2[i]["colors"]){
T2[i]["colors"][j]=ColorMap[T2[i]["colors"][j]];
}
}
return T2;
}
var NewAll=EncodeColor(All);
console.log(All);
console.log(NewAll);
function SortColor(T1){
for(var i in T1){
T1[i]["colors"].sort((a, b) => {
return a-b;
});
}
}
function StaticSearch(T1,Name1,Name2){
if(Name1 in T1 && Name2 in T1 ){
var Temp= T1[Name1]["colors"].concat(T1[Name2]["colors"]);
var T2=[];
Temp.sort((a, b) => {
return a-b;
});
for(var i=0;i<Temp.length-1;i++){
if(Temp[i]==Temp[i+1]){
T2.push(Temp[i]);
}
}
var ReturnObj={};
ReturnObj[Name1+"_"+ Name2]={};
ReturnObj[Name1+"_"+ Name2]["colors"]=T2;
return ReturnObj;
}
return null;
}
var SearchResult=StaticSearch(NewAll,"CC","CN");
console.log(SearchResult);
//Then Decode Color/*
function DecodeColor(T1){
var T2 = JSON.parse(JSON.stringify(T1));//Clone Original
for(var i in T2){
for(var j in T2[i]["colors"]){
T2[i]["colors"][j]=ReverseColorMap[T2[i]["colors"][j]];
}
}
return T2;
}
var TTTQQ=DecodeColor(SearchResult);
console.log(TTTQQ);
//This Also work any length of color
var Examp={
"CC":{"colors":["Blue","Green","Yellow","Orange"]},
"CN":{"colors":["White","Green","Blue"]},
"WA":{"colors":["Orange","Green","Blue"]}
};
var E_Examp=EncodeColor(Examp);
var Com_E_E_Examp=StaticSearch(E_Examp,"WA","CC");
var D_Examp=DecodeColor(Com_E_E_Examp);
console.log(D_Examp);
Another Method Static Search
I want to convert the data of a Google Sheet into JSON format so that I can use it on my website. However, I get a 500 error whenever the website tries to fetch the JSON file.
I have already tried different methods to convert my sheet into JSON that are available on the internet
$url = 'https://spreadsheets.google.com/feeds/list/1dpmzzKNR8hRILX6qMD0KTruxxXYT3UAXR0EcX0zS0dE/1/public/full?alt=json';
$file= file_get_contents($url);
$json = json_decode($file);
$rows = $json->{'feed'}->{'entry'};
return $rows;
I had the same problem; I was able to work around the problem by parsing the html from the pubhtml page directly to output a similar JSON format:
Instead of using the Google Sheet ID, use the link for "Publish to Webpage".
There are some rows that I skip since some are for frozen rows, but you should be able to modify the code to your needs.
function importGoogleSheets(publishedUrl, sheetId, onSuccess, onError) {
var headers = [];
var rows;
$.ajax({
url: publishedUrl+'?gid='+sheetId+'&single=true',
success: function(data) {
data = $.parseHTML(data)[5];
htmldata = data;
rows = data.getElementsByTagName('tr');
for (var i = 0; i < rows[1].cells.length; i++) {
headers[i] = 'gsx$' + rows[1].cells[i].textContent.toLowerCase().replace(/[^0-9a-z]*/g, '');
}
for (var i = 3; i < rows.length; i++) {
temp = {};
for (var h = 1; h < headers.length; h++) {
temp[headers[h]] = {'$t': rows[i].cells[h].textContent};
}
all_data[i - 3] = temp;
}
onSuccess(all_data);
},
error: function(data) {
onError(data);
}
});
}
One note though is that it includes any empty rows unlike the feed, so you may want to filter the ouptut based on some column.
I'm working on some cloud code right now where I have an id I query against which returns all the data for that row.
I then need to iterate over all the fields (columns) of data and make some changes to the values then update that row.
Im able to get the data from parse but im not sure how to pull out the PFObject keys to iterate over the data in a for loop , make changes then save.
Here is some sample code where I hardcoded a field value in but I'm not sure how to get the fields, then iterate over them in a for loop..
Also excuse the JS code, I its been years since I wrote any JS.
<script type="text/javascript">
Parse.initialize("xxxx", "xxxx");
var LocationTag = Parse.Object.extend("LocationTags");
var query = new Parse.Query(LocationTag);
query.equalTo("SomeId", "302d87f2-0188-4cbe-bc2c-e6dcbf822539");
query.find({
success: function(results) {
for (var i = 0; i < results.length; i++) {
var object = results[i];
var data = object.get('T0fYiV9PeeU'); <--- hardcoded field key.. i need to iterate over all fields returned from the row..
count = data.length;
for (var c = 0; c < count; c++) {
var res = Number(data[c].split(":")[0]);
text += "Value: " + res + "<br>";
sum += parseInt(res);
}
document.getElementById("main").innerHTML = text + ' sum: ' + sum + ' average: ' + sum/100 + results
}
},
error: function(error) {
alert("Error: " + error.code + " " + error.message);
}
});
Any ideas.. sorry again if its just a simple JS issue.. but I need to iterate over all fields, returned in the PFObject
If I understand correctly, you want a list of properties from a Parse.Object. The easiest way to to this is to call .toJSON() on the Parse.Object and then extract the keys from the json.
Example:
for (var i = 0; i < results.length; i++) {
var pfObject = results[i];
var jsonObject = pfObject.toJSON();
var pfKeys = [];
for(var key in pfKeys){
if(jsonObject.hasOwnProperty(key)){
pfKeys.push(key);
}
}
//Now we have a list of the pfObject keys in pfKeys
}
I have an MVC application with a dynamic table on one of the pages, which the users defines how many columns the table has, the columns order and where to get the data from for each field.
I have written some very bad code in order to keep it dynamic and now I would like it to be more efficient.
My problem is that I don't know how to define the columns I should get back into my IEnumerable on runtime. My main issue is that I don't know how many columns I might have.
I have a reference to a class which gets the field's text. I also have a dictionary of each field's order with the exact property It should get the data from.
My code should look something like that:
var docsRes3 = from d in docs
select new[]
{
for (int i=0; i<numOfCols; i++)
{
gen.getFieldText(d, res.FieldSourceDic[i]);
}
};
where:
docs = List from which I would like to get only specific fields
res.FieldSourceDic = Dictionary in which the key is the order of the column and the value is the property
gen.getFieldText = The function which gets the entity and the property and returns the value
Obviously, it doesn't work.
I also tried
StringBuilder fieldsSB = new StringBuilder();
for (int i = 0; i < numOfCols; i++)
{
string field = "d." + res.FieldSourceDic[i] + ".ToString()";
if (!string.IsNullOrEmpty(fieldsSB.ToString()))
{
fieldsSB.Append(",");
}
fieldsSB.Append(field);
}
var docsRes2 = from d in docs
select new[] { fieldsSB.ToString() };
It also didn't work.
The only thing that worked for me so far was:
List<string[]> docsRes = new List<string[]>();
foreach (NewOriginDocumentManagment d in docs)
{
string[] row = new string[numOfCols];
for (int i = 0; i < numOfCols; i++)
{
row[i] = gen.getFieldText(d, res.FieldSourceDic[i]);
}
docsRes.Add(row);
}
Any idea how can I pass the linq the list of fields and it'll cut the needed data out of it efficiently?
Thanks, Hoe I was clear about what I need....
Try following:
var docsRes3 = from d in docs
select (
from k in res.FieldSourceDic.Keys.Take(numOfCols)
select gen.getFieldText(d, res.FieldSourceDic[k]));
I got my answer with some help from the following link:
http://www.codeproject.com/Questions/141367/Dynamic-Columns-from-List-using-LINQ
First I created a string array of all properties:
//Creats a string of all properties as defined in the XML
//Columns order must be started at 0. No skips are allowed
StringBuilder fieldsSB = new StringBuilder();
for (int i = 0; i < numOfCols; i++)
{
string field = res.FieldSourceDic[i];
if (!string.IsNullOrEmpty(fieldsSB.ToString()))
{
fieldsSB.Append(",");
}
fieldsSB.Append(field);
}
var cols = fieldsSB.ToString().Split(',');
//Gets the data for each row dynamically
var docsRes = docs.Select(d => GetProps(d, cols));
than I created the GetProps function, which is using my own function as described in the question:
private static dynamic GetProps(object d, IEnumerable<string> props)
{
if (d == null)
{
return null;
}
DynamicGridGenerator gen = new DynamicGridGenerator();
List<string> res = new List<string>();
foreach (var p in props)
{
res.Add(gen.getFieldText(d, p));
}
return res;
}
I have a problem with sorting collection. User saves few values and he sorts it.
At next launch program must order a new collection.
Values which have been saved by user can be in new collection, but It can be situation , when those values aren't in new collection.
I wrote some code, and I want your feedback If it has sense
var oldValues = new List<string>(new[] { "id5", "id3", "id1" });
var valuesToOrder = new List<string>(new[] { "id1", "id2", "id3", "id4" });
int numberOfReorderedValues = 0;
for (int i = 0; i < oldValues.Count; i++)
{
if (valuesToOrder.Contains(oldValues[i]))
{
int indexOfValueWhichShouldBeBefore = valuesToOrder.IndexOf(oldValues[i]);
string valueWhichShouldBeBefore = valuesToOrder[indexOfValueWhichShouldBeBefore];
string valueWhichShouldBeAfter = valuesToOrder[numberOfReorderedValues];
valuesToOrder[numberOfReorderedValues] = valueWhichShouldBeBefore;
valuesToOrder[indexOfValueWhichShouldBeBefore] = valueWhichShouldBeAfter;
numberOfReorderedValues++;
This code works, but I must tomorrow show it to my boss, and I don't go to fool
It's not clear what you're after.
It sounds like you've got 2 lists. One of them contains a list of 'required', and the other contains a list of 'pick from'.
How about use LINQ to sort?
var oldValues = new List<string>(new[] { "id5", "id3", "id1" });
var valuesToOrder = new List<string>(new[] { "id1", "id2", "id3", "id4" });
var sorted = valuesToOrder.Intersect(oldValues).OrderBy(x=>x);
// sorted now has id1 and id3, sorted alphabetically.