(println (get-in #app-state ["my-seq"]))
Returns the following sequence with type cljs.core/IndexedSeq
([-Jg95JpA3_N3ejztiBBM {create_date 1421803605375,
website "www.example.com", first_name "one"}]
[-Jg95LjI7YWiWE233eK1 {create_date 1421803613191,
website "www.example.com", first_name "two"}]
[-Jg95MwOBXxOuHyMJDlI {create_date 1421803618124,
website "www.example.com", first_name "three"}])
How can I access the maps in the sequence by uid? For example, the map belonging to
-Jg95LjI7YWiWE233eK1
If you need the order of the data, you have the following possibilities:
Store the data once in order and once as a map. So, when adding a new entry, do something like:
(defn add-entry
[uid data]
(swap! app-state update-in ["my-seq"]
#(-> %
(update-in [:sorted] conj data)
(update-in [:by-uid] assoc uid data))))
With lookup functions being:
(defn sorted-entries
[]
(get-in #app-state ["my-seq" :sorted]))
(defn entry-by-uid
[uid]
(get-in #app-state ["my-seq" :by-uid uid]))
This has the best lookup performance but will use more memory and make the code a little bit more complex.
Search the entry in the seq:
(defn entry-by-uid
[uid]
(->> (get #app-state "my-seq")
(filter (comp #{uid} first))
(first)))
In the worst case, this has to traverse the whole seq to find your entry.
If order does not matter, I recommend storing the data as a map in the first place:
(defn add-entry
[uid data]
(swap! app-state assoc-in ["my-seq" uid] data))
(defn entry-by-uid
[uid]
(get-in #app-state ["my-seq" uid]))
Related
I'm trying to read data (cca 760k rows) from a single column into one (flattened) vector. Result of clojure.java.jdbc/query is seq of maps, e.g. ({:key "a"} {:key "b"} ...). With option :as-arrays? true provided, [[:key] ["a"] ["b"] ...] is returned. To flatten the result, I've also used option :row-fn first and got [:key "a" "b" ...]. Finally, I've applied rest to get rid of the :key.
Wrapping and unwrapping of rows with vectors seems like a lot of unnecessary work. I'm also not happy with performance. Is there a faster / more idiomatic way? I've tried...
(jdbc/with-db-connection [con -db-spec-]
(with-open [^Statement stmt (.createStatement (:connection con))
^ResultSet res (.executeQuery stmt query)]
(let [ret (ArrayList.)]
(while (.next res)
(.add ret (.getString res 1)))
(into [] ret))))
... but it's not much faster, and it's ugly.
EDIT
Nicer way to do it is via transducers (see here):
(into []
(map :key)
(jdbc/reducible-query
connection
["SELECT key FROM tbl"]
{:raw? true}))
You can just use :row-fn :key. Not sure what performance you are expecting but on my i5 PC, retrieving 760K records took ~3 seconds (H2 file based database)
(time
(count
(jdbc/query db ["select top 760000 key from table1"] {:row-fn :key})))
;; => 760000
"Elapsed time: 3003.456295 msecs"
I am using Clojure.java.jdbc for database access in clojure.
I wanted to use prepared statements with select.
From my previous question I got the answer like this,
(jdbc/query (:conn dbinfo)
["select * from users where username = ? and password = ?"
"harikk09"
"amma123"])
It is working also.
Now,
this parameter list I want to make dynamic. so I write a function like,
(defn values-builder (fn[param] (:value #(:value (param 1)))))
which actually works correctly and return a collection of values using a println.
(println (map values-builder params))
gives
(harikk09 amma123)
But when I tried to execute it like this, where sql-query is the previously mentioned query
(jdbc/query (:conn dbinfo) sql-query (map values-builder params))
, it throws an exception:
Caused by: java.lang.IllegalArgumentException: No value supplied for key:
Clojure.lang.LazySeq#ab5111fa
Can anyone help me to rectify this error?
I think clojure expects a list of parameters without () or [].
The JDBC query and prepared values together need to be a collection. So you need to make a collection out of a string and a collection of parametrized values. To prepend a single item onto the front of a collection, use cons
(jdbc/query (:conn dbinfo) (cons sql-query (map values-builder params)))
Use apply to splice in the arguments
(apply jdbc/query (:conn dbinfo) sql-query (map values-builder params))
Update: as noted below apply won't work since sql needs to be in a vector with the params
in that case you need to cons the sql query onto the generated params list
(jdbc/query (:conn dbinfo) (cons sql-query (map values-builder params)))
I'm doing what I thought was a fairly straightforward task: run a sql query (over about 65K rows of data) using sqlkorma library (http://sqlkorma.com), and for each row transforming it in some way, and then writing to CSV file. I don't really think that 65K rows is all that large given that I have a 8GB laptop, but I also assumed that a sql result set would be lazily fetched and so the whole thing would never get held in memory at the same time. So I was really really surprised when I ended up with this stack trace:
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at clojure.lang.PersistentHashMap$BitmapIndexedNode.assoc(PersistentHashMap.java:777)
at clojure.lang.PersistentHashMap.createNode(PersistentHashMap.java:1101)
at clojure.lang.PersistentHashMap.access$600(PersistentHashMap.java:28)
at clojure.lang.PersistentHashMap$BitmapIndexedNode.assoc(PersistentHashMap.java:749)
at clojure.lang.PersistentHashMap$TransientHashMap.doAssoc(PersistentHashMap.java:269)
at clojure.lang.ATransientMap.assoc(ATransientMap.java:64)
at clojure.lang.PersistentHashMap.create(PersistentHashMap.java:56)
at clojure.lang.PersistentHashMap.create(PersistentHashMap.java:100)
at clojure.lang.PersistentArrayMap.createHT(PersistentArrayMap.java:61)
at clojure.lang.PersistentArrayMap.assoc(PersistentArrayMap.java:201)
at clojure.lang.PersistentArrayMap.assoc(PersistentArrayMap.java:29)
at clojure.lang.RT.assoc(RT.java:702)
at clojure.core$assoc.invoke(core.clj:187)
at clojure.core$zipmap.invoke(core.clj:2715)
at clojure.java.jdbc$resultset_seq$thisfn__204.invoke(jdbc.clj:243)
at clojure.java.jdbc$resultset_seq$thisfn__204$fn__205.invoke(jdbc.clj:243)
at clojure.lang.LazySeq.sval(LazySeq.java:42)
at clojure.lang.LazySeq.seq(LazySeq.java:60)
at clojure.lang.Cons.next(Cons.java:39)
at clojure.lang.PersistentVector.create(PersistentVector.java:51)
at clojure.lang.LazilyPersistentVector.create(LazilyPersistentVector.java:31)
at clojure.core$vec.invoke(core.clj:354)
at korma.db$exec_sql$fn__343.invoke(db.clj:203)
at clojure.java.jdbc$with_query_results_STAR_.invoke(jdbc.clj:669)
at korma.db$exec_sql.invoke(db.clj:202)
at korma.db$do_query$fn__351.invoke(db.clj:225)
at clojure.java.jdbc$with_connection_STAR_.invoke(jdbc.clj:309)
at korma.db$do_query.invoke(db.clj:224)
at korma.core$exec.invoke(core.clj:474)
at db$query_db.invoke(db.clj:23)
at main$_main.doInvoke(main.clj:32)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
As far as I can tell from the stack, it has not made it outside the query code (meaning it hasn't reached my transformation/write to CSV code at all). If it matters, my sql is fairly straightforward, basically SELECT * FROM my_table WHERE SOME_ID IS NOT NULL AND ROWNUM < 65000 ORDER BY some_id ASC. This is oracle (to explain rownum above), but I don' think that matters.
EDIT:
Code sample:
(defmacro query-and-print [q] `(do (dry-run ~q) ~q))
(defn query-db []
(query-and-print
(select my_table
(where (and (not= :MY_ID "BAD DATA")
(not= :MY_ID nil)
(raw (str "rownum < " rows))))
(order :MY_ID :asc))))
; args contains rows 65000, and configure-app sets up the jdbc
; connection string, and sets a var with rows value
(defn -main [& args]
(when (configure-app args)
(let [results (query-db)
dedup (dedup-with-merge results)]
(println "Result size: " (count results))
(println "Dedup size: " (count dedup))
(to-csv "target/out.csv" (transform-data dedup)))))
clojure.java.sql creates lazy sequences:
(defn resultset-seq
"Creates and returns a lazy sequence of maps corresponding to
the rows in the java.sql.ResultSet rs. Based on clojure.core/resultset-seq
but it respects the current naming strategy. Duplicate column names are
made unique by appending _N before applying the naming strategy (where
N is a unique integer)."
[^ResultSet rs]
(let [rsmeta (.getMetaData rs)
idxs (range 1 (inc (.getColumnCount rsmeta)))
keys (->> idxs
(map (fn [^Integer i] (.getColumnLabel rsmeta i)))
make-cols-unique
(map (comp keyword *as-key*)))
row-values (fn [] (map (fn [^Integer i] (.getObject rs i)) idxs))
rows (fn thisfn []
(when (.next rs)
(cons (zipmap keys (row-values)) (lazy-seq (thisfn)))))]
(rows)))
Korma fully realizes the sequence by dropping each row to a vector:
(defn- exec-sql [{:keys [results sql-str params]}]
(try
(case results
:results (jdbc/with-query-results rs (apply vector sql-str params)
(vec rs))
:keys (jdbc/do-prepared-return-keys sql-str params)
(jdbc/do-prepared sql-str params))
(catch Exception e
(handle-exception e sql-str params))))
Besides the with-lazy-results route in https://github.com/korma/Korma/pull/66, as a completely different way to resovle the problem, you can simply increase the heap size available to your JVM by setting the appropriate flag. JVMs are not allowed to use all the free memory on your machine; they are strictly limited to the amount you tell them they're allowed to use.0
One way to do this is to set :jvm-opts ["-Xmx4g"] in your project.clj file. (Adjust the exact heap size as necessary.) Another way is to do something like:
export JAVA_OPTS=-Xmx:4g
lein repl # or whatever lanuches your Clojure process
The with-lazy-results route is better in the sense that you can operate on any sized result set, but it's not merged into mainline Korma and requires some updating to work with recent versions. It's good to know how to adjust the JVM's allowed heap size anyway.
Can you have hash tables or dicts in Lisp? I mean the data structure that is a collection of pairs (key, value) where values can be acceded using keys.
Common Lisp has at least four different ways to do that (key value storage):
property lists (:foo 1 :bar 2)
assoc lists ((:foo . 1) (:bar . 2))
hash tables
CLOS objects (slot-value foo 'bar) to get and (setf (slot-value foo 'bar) 42) to set. The slot name can be stored in a variable: (let ((name 'bar)) (slot-value foo name)) .
For simple usage assoc lists or property lists are fine. With a larger number of elements they tend to get 'slow'. Hash tables are 'faster' but have their own tradeoffs. CLOS objects are used like in many other object systems. The keys are the slot-names defined in a CLOS class. Though it is possible to program variants that can add and remove slots on access.
Of course - Common Lisp has hash tables.
(setq a (make-hash-table))
(setf (gethash 'color a) 'brown)
(setf (gethash 'name a) 'fred)
(gethash 'color a) => brown
(gethash 'name a) => fred
(gethash 'pointy a) => nil
Property lists are good for very small examples of demonstrative purpose, but for any real need their performance is abysmal, so use hash tables.
If you're referring to Common Lisp, hash tables are provided by a type called hash-table.
Using these tables involves creating one with function make-hash-table, reading values with gethash, setting them by using gethash as a place in concert with setf, and removing entries with remhash.
The mapping from key value to hash code is available outside of hash tables with the function sxhash.
Clojure has a built-in map type:
user=> (def m {:foo "bar" :baz "bla"})
#'user/m
user=> (m :foo)
"bar"
See http://clojure.org/data_structures
Sure. Here's the SRFI defining the standard hash table libraries in Scheme:
http://srfi.schemers.org/srfi-69/srfi-69.html
There's built-in hash tables, that use a system hash function (typically SXHASH) and where you can have a couple of different equality checkers (EQ, EQL, EQUAL or EQUALP depending on what you consider to be "the same" key).
If the built-in hash tables are not good enough, there's also a generic hash table library. It will accept any pair of "hash generator"/"key comparator" and build you a hash table. However, it relies on having a good hash function to work well and that is not necessarily trivial to write.
In Lisp it's usually called a property list.
I would like to use Yahoo to get stock prices from within an Emacs Lisp program. I have two questions.
How do I make the http GET?
What is the best what to store the data in Elisp so I can make comparisons of the data? In other words, should I use one hash table, several hash tables, or lists to represent that data returned from Yahoo?
Here's the basic outline of what I'd like to do.
;; Call Yahoo to get equity prices
;;
;; Yahoo Input:
;; http://download.finance.yahoo.com/d/quotes.csv?s=AAPL+GOOG&f=sb2b3jkm6
;; Yahoo Output:
;; "AAPL",211.98,211.82,78.20,215.59,+17.90%
;; "GOOG",602.94,601.69,282.75,629.51,+18.27%
;;
;; Symbol, ask, bid, 52 week low, 52 week high, % change from 200 day mavg
;;
;; Yahoo format described here: http://www.gummy-stuff.org/Yahoo-data.htm
(defun get-price-url (tickers)
"
s = symbol
b2 = ask real-time
b3 = bid real-time
j = 52 week low
k = 52 week high
"
(concat "http://download.finance.yahoo.com/d/quotes.csv?s="
(mapconcat 'identity tickers "+") "&f=sb2b3jk"))
(setq lst '("AAPL" "GOOG" "MSFT" "ORCL"))
(setq url (get-price-url lst))
;; Call Yahoo with Url, process results and place in a data structure
;;
;; Return results sorted by largest change in 200 day mavg, in descending order
;;
Here's some code to get you started; I show how to grab the url to a buffer, parse each line and then display the ticker and price of each item. You can modify it from there to do what you need.
This parses each line of stock data into a list, and it's straight forward to grab the values using the first, second, third functions, or using nth. You can write functions to grab each element you want, such as get-ticker(quote) which would just return (first ticker)
I wouldn't over think what kind of data structure to use; whatever is easiest is fine. If you need high performance then you shouldn't be using emacs lisp for this anyway.
(defun test()
(interactive)
(let ((quotes (get-quotes '("AAPL" "GOOG" "MSFT" "ORCL" "ERTS" "THQI") "sb")))
(show-quotes quotes)))
(defun show-quotes(quotes)
(dolist (quote quotes)
(message (format "%s $%.2f" (first quote) (string-to-number (second quote))))))
(defun get-quotes(tickers field-string)
"Given a list of ticker names and a string of fields to return as above, this grabs them
from Yahoo, and parses them"
(let ((results-buffer (get-yahoo-quotes-to-buffer (get-price-url tickers field-string))))
(switch-to-buffer results-buffer)
(parse-quote-buffer results-buffer)))
(defun get-price-url (tickers field-string)
"Set up the get url"
(concat "http://download.finance.yahoo.com/d/quotes.csv?s="
(mapconcat 'identity tickers "+")
"&f=" field-string))
(defun get-yahoo-quotes-to-buffer(url)
"Retrieve the quotes to a buffer and return it"
(url-retrieve-synchronously url))
(defun parse-quote-buffer(b)
"Parse the buffer for quotes"
(goto-line 1)
(re-search-forward "^\n")
(beginning-of-line)
(let ((res nil))
(while (> (point-max) (point))
(setf res (cons (split-string (thing-at-point 'line) ",") res))
(forward-line 1))
(reverse res)))
Check out http://edward.oconnor.cx/elisp/. Edward has some examples of interacting with various services using HTTP, and if you can't find a Yahoo client library you could write one using these techniques.