I have using the google-cloud-bigquery gem (version 0.20.2, can't upgrade at the moment).
I have an audit dataset which contains many tables of the following format:
audit.some_table20170101
audit.some_table20170102
audit.some_table20170103
etc..
I am trying to run a query which will scan all of these tables, and give me the last value of field some_field.
What I was going for is using the tables wildcard:
FROM audit.some_table*
and to hopefully
SELECT LAST(some_field) AS last_some_field
While using Bigquery web console, I was able to do so by
using backticks (FROM `audit.some_table*`), but doing the same programmatically with the gem causes a Google::Cloud::InvalidArgumentError: invalid: Invalid table name: `audit.some_table*`
Even in the web console, when I try to use the LAST command it requires using legacy SQL, which then gives an error due to the backticks of the previous section. If I disable legacy sql, LAST is no available anymore (unfamiliar command) and then I have to order by a timestamp column descending and limit 1.
Any ideas how to solve these problems and to be able to query using the above mention gem and version?
LAST is only meaningful when there is an order. Tables in BigQuery do not have inherit ordering, and if you run SELECT * FROM table you may get results in different order every time. Therefore the right thing to do it is to use ORDER BY some_value DESC LIMIT 1 construct.
The wildcard tables are indeed only available in Standard SQL, to get similar functionality with Legacy SQL you can use TABLE_DATE_RANGE function in FROM clause.
Related
May I have your opinion on below queries please:
Option 1:
I have select script handy with me which fetch data by joining many source tables and performs some transformations like aggregations (group by), data conversion, sub-string etc.
Can I invoke this script through ODI mapping and return results (transformed data output) can be inserted into target of ODI mapping ?
Option2:
Convert the select script into equivalent ODI mapping by using equivalent ODI transformations , functions , look ups etc and use various tables (tables in join clause) as source of mappings.
Basically develop ODI mapping which is equivalent to provided select script plus a target table to insert records into it.
I need to know pros and cons of both options in above (if option 1 is possible).
Is it still possible to track transformation errors, join source tables and where clause condition related errors etc through ODI with option 1?
Log file for mapping failure will have as granular level details as offered by option 2?
Can I still enable Flow Control at Knowledge Module and redirect select script errors into E$_ error tables provided by ODI?
Thanks,
Rajneesh
Option 1: ODI 12c includes that concept out of the box. On the physical tab of a mapping, click on the source node (datastore). Then in the properties pane, there is the CUSTOM_TEMPLATE option under "Extract Options" menu. This allows to enter a custom SQL statement that will be used instead of the code generated by ODI.
However it is probably less maintainable over time than option 2. SQL is less visual than mapping components. Also if you need to bulk change it, it will be trickier. Changing a component in several mappings can be done with the SDK. Changing SQL code would require to parse it. You might indeed have less information in your operator logs as the SQL would be seen as just one block of code. It also wouldn't provide any lineage.
I believe using Flow Control would work but I haven't tested it.
Option 2 would take more time to complete but with that you would benefit from all the functionalities of ODI.
My own preference would be to occasionally use option 1 for really complex SQL queries but to use option 2 for most of the normal use cases.
I get a problem when doing queries with different platforms using Oracle. The results of the query indicate the difference in output.
First I used the "Toad for Oracle Database" application and the results like this:
The results showed perfect results, as I wanted.
But when I do queries on different platforms, namely PHP Codeigniter and Navicat (it may also apply to other platforms). The results are different as in this picture:
Following are the queries that I am trying to run but do not work on different platforms.
select STANDARD_HASH(sysdate) from dual;
Try adding RAWTOHEX. This will convert the returned value from a RAW to a VARCHAR2, which should be more easily understood by all SQL clients. (As Justin pointed out this is probably a client issue, not a problem with STANDARD_HASH.)
select rawtohex(standard_hash(sysdate)) the_hash from dual;
THE_HASH
----------------------------------------
FBC14021D134F922420086D291906B0B0D783421
I'm using Official Aerospike Package for golang.
Is there any way to get list of all existing indexes this like?
aql> show indexes
Also I haven't found any way to execute pure aql query.
How can I do this?
Update:
I'm looking for something similar to this but for aerospike (example from Rails)
custom_query = "select * from users"
result = ActiveRecord::Base.connection.execute(custom_query)
aql>show indexes
is a valid aql command and should show you all the secondary indexes you currently have on the server.
aql runs the C api underneath. You can do pretty much everything with aql at a rudimnetary level.
Type: aql>help it will throw all the aql commands at you, cut and paste!
aql also stores command history in a text file - so persists over sessions.
aql>run 'filepath/filename' is a handy way to store all aql commands in a text file and run them.
Re: aql query -- look at: select * from ns. where ... you can do equality and range queries provided you have pre-built the secondary indexes.
Aerospike ver 3.12+ introduced predicate filtering - ie ~complex queries - I don't think aql has been updated to run those yet.
HTH.
AQL is an admin and data browsing tool. It's not really Aerospike's SQL, as Aerospike doesn't natively implement a query language. Instead, all the Aerospike clients give you an API to make direct get, put, scan, query calls, and those are procedural, not declarative like SQL (where you state how you want the result and the server figures out a query plan). Piyush mentioned the predicate filtering API which is fantastic and lets you create complex queries over scans and secondary-index queries.
Specifically to your question about getting all the indexes, that's the type of thing you should use the info command for. Aerospike allows you to get and set config parameters through it, and get a wide range of metrics, run microbenchmark, etc. Everything you need for admin and monitoring.
You can run sindex through the standalone asinfo tool, or you can call it using the info command that any client provides.
asinfo -v "sindex"
What are the differences between Connection#exec_params and Connection#exec_prepared? It seems like they both require the same kinds of input and they both perform the same action. Why would someone choose one versus the other?
You can use exec_prepared to optimize SQL query, see documentation.
exec_params lets you execute specified SQL query with binded parameters. exec_prepared lets you execute prepared (parsed, optimized etc) SQL query specified by its string identificator/name which you have got earlier.
If you make a lot of similar SQL select queries and difference is parameter values, you can make it effectively by preparing SQL statement once (you receive it identifier) and execute it with different parameters multiple times.
I need to do multiple updates on a PostgreSQL database at once. To make it faster, I would like to avoid continually changing context between my ruby application and the database.
I've learned about the #update_sql method, however my strategy to queue those queries is not working. I'm making a list sqls which holds many update strings, and then db[sqls.join(';') + ';']. If instead of executing this line I decide to print the resulting sql and copy/paste to the database it works.
Why doesn't this work? How can I send multiple updates in a single sql?
Sequel doesn't offer an API that supports multiple queries in a single statement. If the driver you are using supports, it, you can use Sequel::Database#synchronize to get access to the driver connection object, and call methods directly on that.