Apologies if this has come up before, but I couldn't find anything.
I am using GraphCMS (highly recommend it) and I have several fields that are floats. I am using them for prices. Each entry in the schema can either be a buy or sell in an enumeration field. I would like to total all the price fields where the entry is a buy, and total all the price fields where the entry is a sell.
I thought there would be something clear in the docs because totalling those fields would be very useful for something like calculating an average score etc. but I can only see docs about counting entries. Have I missed it somewhere?
Surely this is possible?
After speaking with the helpful guys at GraphCMS I have figured out how to do it. This is not a question about GraphCMS as #puelo suggested, but can be achieved by running a graphql query and then programatically running calculations with the data. It's a simple solution but wasn't immediately apparent to me at first. Hopefully that info will be useful to other newbies.
Related
EVE Online Manufacturing Spreadsheet
In Batch!F3:G, I'm attempting to break down the data input from columns B3:C to their components (and eventually materials/minerals in I3:J) by using filter to compare results in Engine!P:R. Multiplied of course by the total number of each finished product I need.
I've been trying to figure out ways to arrayformula this together, and even tried quite a few query functions without success. The best I've been able to come up with is to string the actual formula together, appending them with {}, but this gets bloated quickly. I need this to be open ended because I have a tendency to build a lot of things at once. Any help would be appreciated, even just point me in the right direction!
Well, based on my limited knowledge about google sheet, I can only think of one way to do this automatically.
Here's a sheet I constructed based on your sheet.
https://docs.google.com/spreadsheets/d/1AfX8o05gUGPiN5S90w4o0yxuIYjsJRaXsaYUFTJuEPo/edit?usp=sharing
First, on Engine sheet, add one more column which will give you the number of materials required for that part, which is looked up in the PART LIST of BATCH sheet. For this I use VLOOKUP, as you see in D2.
Then on BATCH sheet, query the materials that VLOOKUP return positive, multiply it by the amount of item and then sum them.
This is done by the QUERY used in F3
This method only if you don't have duplicate item in your PART LIST, due to the way VLOOKUP work.
Of course if you want to break the material list further, you can do the same approach..
I have two real time streams. One contains news articles and the other comments about the same articles. I'd like to create a parent-child relationship between each article and that articles comments except for headline. There is no common id. I'd like to use the headline which exists in both streams and match the two streams based on that every 15 minutes. I am assuming that 15 min would be sufficient to handle delay between the two streams. How would you go about doing this? Any ideas would be appreciated.
A typical message containing, entity_name, source_name, headline, which comes through Logstash looks like this:
"Thomson Reuters Corp.","Japan Today","Trump claims victory after
forcing NATO crisis talks"
Some typical comments, comment, headline, which comes through Logstash but a separate pipeline looks like this:
"We applaud Trumps claim ...", "Trump claims victory after forcing NATO crisis talks"
"Nato crisis is important...", "Trump claims victory after forcing NATO crisis talks"
Specifically:
1. Keep indexes separate or create a third index with from the first two?
2. How to run 15 min refresh cycles?
3. If there is a better way/tool/data store, please advise.
You can create a common id between comments and article by hashing the headline (supposing you never observe typos).
Yes, keep articles and comments in separate indices.
reference: https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html
Need more specifics on what you mean by matching the streams. Not sure if there's a way to schedule jobs using Elasticsearch Task API... Maybe make a cronjob to do this? You can go through the articles index, hash the headline, and then query for that hash in the comments index.
Seems like you have a solid storage method right now.
For a certain program I have some type-keywords values like this:
Program Type Keyword
PIM Kind Additional
PIM Period Education
PIM Phase Specialized
PIM Skills Professional
The type is a fixed value, but the keyword depends of the Program and type. I want to transpose this result in analytics by making 4 columns with the type. The result has to look like this:
Program Kind period phase skills
PIM Additional Education Specialized Professional
I have tried by editing the column formula and putting this formula:
CASE WHEN "Type"='Partial period' THEN "Keyword" END
and so on for each different type. But it doesn't give me the result I want. all the new columns are empty.
I also tried with a pivot table, but the keyword isn't a measure, so I don't think this will work.
can someone help?
This simply doesn't make sense in an analytical way. You have no fact, nothing you measure. So no chance of using FILTER...USING... for example.
Don't forget you're not in Excel or a drawing tool. You're in an analytics tool which tries to make sense out of data and not "show data in a weird way".
You have to model things nicely either in the data source itself or be clever in the construction of your RPD.
It's doable in the RPD but it will be quite static and if the list of values changes you will have to adapt it.
tl;dr - garbage data, garbage result
Let's imagine I am logging displays and clicks, say by cities.
I can aggregate those by countries and I can also compute grand totals.
Now I would like to compute click rates (clicks/displays) by cities, countries and I would also like to get a global click rate.
How can I do this?
It seems that I could use a scripted metric (I have not tried yet) but I would also like to expose these rates in Kibana.
It is possible?
I also want this feature. Kibana don't have this feature at this moment but maybe in the future version. please take a look at:
https://github.com/elasticsearch/kibana/issues/2646
update: I think you can use scripted fields please see for more info http://www.elasticsearch.org/blog/kibana-4-beta-3-now-more-filtery/
I have a database of about 200k books. I wish to give my users a way to quickly search a book by the title. Now, some titles might have prefix like A, THE, etc. and also can have numbers in the title, so search for 12 should match books with "12", "twelve" and "dozen" in the title. This will work via AJAX, so I need to make sure database query is really fast.
I assume that most of the users will try to search using some words of the title, so I'm thinking to split all the titles into words and create a separate database table which would map words to titles. However, I fear this might not give the best results. For example, the book title could be some 2 or 3 commonly used words, and I might get a list of books with longer titles that contain all 2-3 words and the one I'm looking for lost like a needle in a haystack. Also, searching for a book with many words in the title might slow down the query because of a lot of OR clauses.
Basically, I'm looking for a way to:
find the results quickly
sort them by relevance.
I assume this is not the first time someone needs something like this, and I'd hate to reinvent the wheel.
P.S. I'm currently using MySQL, but I could switch to anything else if needed.
Using a SOUNDEX is the best way i think.
SELECT
id,
title
FROM products AS p
WHERE p.title SOUNDS LIKE 'Shaw'
// This will match 'Saw' etc.
For best database performances you can best calculate the SOUNDEX value of your titles and put this in a new column. You can calculate the soundex with SOUNDEX('Hello').
Example usage:
UPDATE `books` SET `soundex_title` = SOUNDEX(title);
You might want to have a look at Apache Lucene. this is a high performance java based Information Retrieval System.
you would want to create an IndexWriter, and index all your titles, and you can add parameters (have a look at the class) linking to the actual book.
when searching, you would need an IndexReader and an IndexSearcher, and use the search() oporation on them.
have a look at the sample at: src/demo and in: http://lucene.apache.org/java/2_4_0/demo2.html
using Information Retrieval techniques makes the indexing take longer, but every search will not require going through most of the titles, and overall you can expect better performance for searching.
also, choosing good Analyzer enables you to ignore words such "the","a"...
One solution that would easily accomodate your volume of data and speed requirment is to use the Redis key-value pair store.
The way I see it, you can go ahead with your solution of mapping titles to keywords and storing them under the form:
keyword : set of book titles
Redis already has a built-in set data-type that you can use.
Next, to get the titles of the books that contains the search keywords you can use the sinter command which will peform set intersection for you.
Everything is done in memory; therefore the response time is very fast.
Also, if you want to save your index, redis has a number of different persistance/caching mechanisms.
Apache Lucene with Solr is definitely a very good option for your problem
You can directly link Solr/Lucene to directly index your MySQL database. Here is a simple tutorial on how to link your MySQL database with Lucene/Solr: http://www.cabotsolutions.com/2009/05/using-solr-lucene-for-full-text-search-with-mysql-db/
Here are the advantages and pains of using Lucene-Solr instead of MySQL full text search: http://jayant7k.blogspot.com/2006/05/mysql-fulltext-search-versus-lucene.html
Keep it simple. Create an index on the title field and use wildcard pattern matching. You can not possibly make it any faster as your bottleneck is not the string matching but the number of strings you want to match against the title.
And just came up with a different idea. You say that some words can be interpreted differently. Like 12, Twelve, dozen. Instead of creating a query with different interpretations, why not store different interpretations of the titles in a separate table with a one to many to the books. You can then GROUP BY book_id to get unique book titles.
Say the book "A dime in a dozen". In books table it will be:
book_id=356
book_title='A dime in a dozen'
In titles table will be stored:
titles_id=123
titles_book_id=356
titles_title='A dime in a dozen'
--
titles_id=124
titles_book_id=356
titles_title='A dime in a 12'
--
titles_id=125
titles_book_id=356
titles_title='A dime in a twelve'
The query for this:
SELECT b.book_id, b.book_title
FROM books b JOIN titles t on b.book_id=t.titles_book_id
WHERE t.titles_title='%twelve%'
GROUP BY b.book_id
Now, insertions becomes a much bigger task, but creating the variants can be done outside the database and inserted in one swoop.