Many things can be done with the groupBy function in SparkR.
Here's an example from the documentation:
# Compute the average for all numeric columns grouped by department.
avg(groupBy(df, "department"))
But I'm very curious about the "GroupedData" object generated by the groupBy function, which is mentioned in the groupBy documentation and also has it's own page.
According to that documentation, the following code will generate a GroupedData object:
groupBy(df, "department")
Unfortunately the page for the "GroupedData" object appears incomplete or I don't know how to find the object documentation for it. The doc says it's "A Java object reference to the backing Scala GroupedData"- I tried searching the Scala documentation on spark.apache.org and found nothing there.
I'm looking for a list of the "GroupedData" class members and methods similar to documentation I have found for other programming languages. Depending on what I find I may have some novel ways to use this object for analysis I am doing in SparkR. Also, the answer to this question will help me with many similar questions I have about finding documentation of other SparkR objects.
You can alwys refer to the PySpark documentation if you think SparkR documentation is not adequate, at least I do that :), there are many common APIs
Related
I would like to use reStructuredText, Sphinx, and Sphinx autodoc for Python code in the same way one uses Javadoc for Java code, that is, for specifying parameters and return values, but also for linking to other classes, methods, etc.
I am finding it very hard to find a reference for learning how to do this. The Sphinx documentation mentions :param and :return: in passing, as mere examples of field lists. I cannot find a proper reference for these roles (and similar ones I imagine exist, such as one for "see also") anywhere.
More importantly, I find it unclear how to create links for other classes and methods, especially if those are in other packages.
Is this the main way Python developers extract documentation from source code? If so, I would expect some clear documentation and reference for this purpose somewhere.
Update: please note that I don't mean the autodoc reference. This page explains how to import docstrings into arbitrary Sphinx documents. It does not explain which roles can be used in docstrings to document parameters, return values, types, or to link to other methods and classes. For instance, a search on this page for :param does not find anything.
ReST fields such as param and return that can be used in docstrings are described under "Info field lists" in the "Python domain" section: https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#info-field-lists.
The roles that are available for creating cross-references to documented Python objects are described here: https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#cross-referencing-python-objects.
You should also take a look at the Napoleon extension, which supports docstring styles that many people find more legible: https://sphinx-doc.org/en/master/usage/extensions/napoleon.html.
My baseline for this questions comes from MIT's Structure and Interpretation of Computer Programs. In the book, a filter function is defined. I know that map is part of the spec, but I see nothing resembling filter.
EDIT:
Specifically I'm referring to the spec here: http://www.schemers.org/Documents/Standards/R5RS/HTML/
No. This SFRI specifically requests that a filter function be added because it is not present in R5RS. There is however a filter function defined in MIT-Scheme, and in R6RS, behaving exactly as one would expect.
I'm looking for a definition of the syntax for the Cypher query language. I tried the docs but they're very vague.
Ideally, I'd like a BNF (or any variant) definition, or one of those "graph" definitions like this or this. Really, anything resembling a formal definition.
What you are looking for will be available in openCypher. Several items will be released as part of the project, one of the first of which is the BNF grammar.
Update 2016-01-30: A first draft of the grammar is now avialable at \https://github.com/opencypher/openCypher/blob/master/grammar.ebnf.
Update: 2016-10-17: EBNF and Antlr grammars, TCK, railroad diagrams, and a list of community projects are available at http://www.opencypher.org/#resources
Take a look at the recently announced (Oct 2015) openCypher project. It involves releasing the language specification, among other things.
From the announcement:
1. Cypher reference documentation:
A comprehensive user documentation describing use of the Cypher query language with examples and tutorials.
2. Technology compatibility kit (TCK):
The TCK consists of a number of tests that a software supplier would run in order to self-certify support for a given version of Cypher.
3. Reference implementation:
Distributed under the Apache 2.0 license, the reference implementation is a fully functional implementation of key parts of the stack needed to support Cypher inside a data platform or tool. The first planned deliverable is a parser that will take a Cypher statement and parse it into an AST (abstract syntax tree) representation. The reference implementation complements the documentation and tests by providing working implementations of Cypher – which are permissively licensed – and can be used as examples or as a foundation for one’s own implementation.
4. Cypher language specification:
Licensed under a Creative Commons license, the Cypher language specification is a technical expression of the language syntax to enable parsers to auto-generate the query syntax. A full semantic specification is also planned as a part of the openCypher project.
The same announcement also says that the process is open and that it is possible to submit, review and comment on language proposals.
Update!
Neo4j has changed a lot since this answer was written. In 2017 the simple answer is yes, you can download the grammar files from https://www.opencypher.org/
Below is the old answer, which was accurate in 2014
As far as I can tell, the only formal definition is in the code. That's the bad news.
The good news is that the code uses a scala library to do the parsing which makes the code rules look kinda/sorta like BNF. And there's some documentation on how to read it.
Here's a link into a scala object that defines what a query is.
This general package on github looks to me like it contains all of the cypher command implementations, and should have everything you're asking for.
Code in this package is written in scala, and looks like this:
object Query {
def start(startItems: StartItem*) = new QueryBuilder().startItems(startItems:_*)
def matches(patterns:Pattern*) = new QueryBuilder().matches(patterns:_*)
def optionalMatches(patterns:Pattern*) = new QueryBuilder().matches(patterns:_*).makeOptional()
def updates(cmds:UpdateAction*) = new QueryBuilder().updates(cmds:_*)
def unique(cmds:UniqueLink*) = new QueryBuilder().startItems(Seq(CreateUniqueStartItem(CreateUniqueAction(cmds:_*))):_*)
(...)
This matches roughly with the upper right hand quadrant of the Cypher refcard. You can sorta see that there can be a start clause, a match clause, and so on. This includes links to other implementation classes (like UpdateAction which further define clauses considered update actions.
Make sure to also read How Neo4J Uses Scala's Parser Combinator: Cypher's Internals Part 1 for more information on what's going on here, and the mapping between the scala classes and what we'd normally consider EBNF. This blog post is old (2011) and the specific code examples it gives shouldn't be trusted, but I think it has good general information on how the implementation works, and what to look for if you want to understand the EBNF behind cypher.
Disclaimer: I'm not a scala hardcore, YMMV, IANAL, devs please overrule me if I'm wrong.
(Michael Hunger answered in a comment, so I can't accept his answer. Here's his answer:)
Cypher uses parboiled as parser, the parboiled rule DSL are pretty easy to read and understand. https://github.com/neo4j/neo4j/blob/d18583d260a957ab1a14bd27d34eb5625df42bc5/community/cypher/cypher-compiler-2.2/src/main/scala/org/neo4j/cypher/internal/compiler/v2_2/parser/Clauses.scala
None of these seem to work any more.
I don't see anything on the opencypher.org site that looks like a grammar to download.
None of the github links from Michael Hunger work.
I'd really like access to SOME resource where I can learn how to construct queries for functions like avg that allegedly take a list expression as an argument, yet barf at every variant I can figure out.
(Mathematica version: 8.0.4)
lst = Names["Internal`*"];
Length[lst]
Pick[lst, StringMatchQ[lst, "*Bag*"]]
gives
293
{"Internal`Bag", "Internal`BagLength", "Internal`BagPart", "Internal`StuffBag"}
The Mathematica guidebook for programming By Michael Trott, page 494 says on the Internal context
"But similar to Experimental` context, no guarantee exists that the behavior and syntax of the functions will still be available in later versions of Mathematica"
Also, here is a mention of Bag functions:
Implementing a Quadtree in Mathematica
But since I've seen number of Mathematica experts here suggest Internal`Bag functions and use them themselves, I am assuming it would be sort of safe to use them in actual code? and if so, I have the following question:
Where can I find a more official description of these functions (the API, etc..) like one finds in documenation center? There is nothing now about them now
??Internal`Bag
Internal`Bag
Attributes[Internal`Bag]={Protected}
If I am to start using them, I find it hard to learn about new functions by just looking at some examples and trial and error to see what they do. I wonder if someone here might have a more complete and self contained document on the use of these, describe the API and such more than what is out there already or a link to such place.
The Internal context is exactly what its name says: Meant for internal use by Wolfram developers.
This means, among other things, the following things hold about anything you might find in there:
You most likely won't be able to find any official documentation on it, as it's not meant to be used by the public.
It's not necessarily as robust about invalid arguments. (Crashing the kernel can easily happen on some of them.)
The API may change without notice.
The function may disappear completely without notice.
Now, in practice some of them may be reasonably stable, but I would strongly advise you to steer away from them. Using undocumented APIs can easily leave you in for a lot of pain and a nasty surprise in the future.
I was thinking about making something like Linq for Lua, and I have a general idea how Linq works, but was wondering if there was a good article or if someone could explain how C# makes Linq possible
Note: I mean behind the scenes, like how it generates code bindings and all that, not end user syntax.
It's hard to answer the question because LINQ is so many different things. For instance, sticking to C#, the following things are involved:
Query expressions are "pre-processed" into "C# without query expressions" which is then compiled normally. The query expression part of the spec is really short - it's basically a mechanical translation which doesn't assume anything about the real meaning of the query, beyond "order by is translated into OrderBy/ThenBy/etc".
Delegates are used to represent arbitrary actions with a particular signature, as executable code.
Expression trees are used to represent the same thing, but as data (which can be examined and translated into a different form, e.g. SQL)
Lambda expressions are used to convert source code into either delegates or expression trees.
Extension methods are used by most LINQ providers to chain together static method calls. This allows a simple interface (e.g. IEnumerable<T>) to effectively gain a lot more power.
Anonymous types are used for projections - where you have some disparate collection of data, and you want bits of each of the aspects of that data, an anonymous type allows you to gather them together.
Implicitly typed local variables (var) are used primarily when working with anonymous types, to maintain a statically typed language where you may not be able to "speak" the name of the type explicitly.
Iterator blocks are usually used to implement in-process querying, e.g. for LINQ to Objects.
Type inference is used to make the whole thing a lot smoother - there are a lot of generic methods in LINQ, and without type inference it would be really painful.
Code generation is used to turn a model (e.g. DBML) into code
Partial types are used to provide extensibility to generated code
Attributes are used to provide metadata to LINQ providers
Obviously a lot of these aren't only used by LINQ, but different LINQ technologies will depend on them.
If you can give more indication of what aspects you're interested in, we may be able to provide more detail.
If you're interested in effectively implementing LINQ to Objects, you might be interested in a talk I gave at DDD in Reading a couple of weeks ago - basically implementing as much of LINQ to Objects as possible in an hour. We were far from complete by the end of it, but it should give a pretty good idea of the kind of thing you need to do (and buffering/streaming, iterator blocks, query expression translation etc). The videos aren't up yet (and I haven't put the code up for download yet) but if you're interested, drop me a mail at skeet#pobox.com and I'll let you know when they're up. (I'll probably blog about it too.)
Mono (partially?) implements LINQ, and is opensource. Maybe you could look into their implementation?
Read this article:
Learn how to create custom LINQ providers
Perhaps my LINQ for R6RS Scheme will provide some insights.
It is 100% semantically, and almost 100% syntactically the same as LINQ, with the noted exception of additional sort parameters using 'then' instead of ','.
Some rules/assumptions:
Only dealing with lists, no query providers.
Not lazy, but eager comprehension.
No static types, as Scheme does not use them.
My implementation depends on a few core procedures:
map - used for 'Select'
filter - used for 'Where'
flatten - used for 'SelectMany'
sort - a multi-key sorting procedure
groupby - for grouping constructs
The rest of the structure is all built up using a macro.
Bindings are stored in a list that is tagged with bound identifiers to ensure hygiene. The binding are extracted and rebound locally where ever an expression occurs.
I did track the progress on my blog, that may provide some insight to possible issues.
For design ideas, take a look at c omega, the research project that birthed Linq. Linq is a more pragmatic or watered down version of c omega, depending on your perspective.
Matt Warren's blog has all the answers (and a sample IQueryable provider implementation to give you a headstart):
http://blogs.msdn.com/mattwar/