How to measure moped insert runtime in a ruby mongoid app? - ruby

I'm currently writing a moped log parser in order to monitor moped queries runtime.
It's work great for QUERY command using the runtime parameter, but INSERT and UPDATE have no runtime parameter. All INSERT and UPDATE are followed by a getLastError COMMAND which contains a runtime.
Here are some samples of moped logs:
QUERY with runtime
MOPED: 127.0.0.1:27017 QUERY database=X collection=X selector=X
flags=[] limit=-1 skip=0 batch_size=nil fields=nil runtime: 0.6950ms
INSERT without runtime but with COMMAND
MOPED: 127.0.0.1:27017 INSERT database=X collection=X documents=X flags=[]
COMMAND database=X command={:getlasterror=>1, :w=>1}
runtime: 0.4750ms
I'm pretty sure that COMMAND runtime is for the getlasterror call and not for my INSERT one.
So is there a way to get this runtime info for an INSERT query?

Instead of using a log parser, I use something like this and it works great:
ActiveSupport::Notifications.subscribe('query.moped') do |name, start, finish, id, payload|
runtime = (finish - start)*1000
moped_ops = payload[:ops]
moped_ops.each do |op|
unless op.collection == '$cmd'
query = op.class.name.split('::').last.downcase
query = op.selector.first[0].to_s.gsub(/\$/, '') if query == 'command'
DO SOMETHING WITH #{op.database}.#{op.collection}.#{query}
end
end
end

Related

Write custom H2 DB function Java

I am trying to run the below code using H2DB (via junit test), while doing so i get error message as below. I understand that, there are no function available as "days" in H2. So i am trying to write a custom function, but it does not work out, can any one help on writing this function.
SQLBuilder class code:
public String dummy() {
return new StringBuilder(new SQL() {
{
SELECT("date(CREATE_TMS)");
SELECT("CASE WHEN date(CREATE_TMS) >= (CURRENT DATE - cast('1' AS integer) days) THEN 'Y' ELSE 'N' END NEW_B");
FROM("Q.DUMMY");
}
}.toString().concat(" FOR READ ONLY WITH UR")).toString();
}
Error message:
org.springframework.jdbc.BadSqlGrammarException:
### Error querying database. Cause: org.h2.jdbc.JdbcSQLSyntaxErrorException: Syntax error in SQL statement "SELECT DATE(CREATE_TMS), CASE WHEN DATE(CREATE_TMS) >= (CURRENT DATE - CAST('1' AS INTEGER) DAYS[*]) THEN 'Y' ELSE 'N' END NEW_BILLING
FROM Q.DUMMY FOR READ ONLY WITH UR "; expected "[, ::, *, /, %, +, -, ||, ~, !~, NOT, LIKE, ILIKE, REGEXP, IS, IN, BETWEEN, AND, OR, ,, )"; SQL statement:
SELECT date(CREATE_TMS), CASE WHEN date(CREATE_TMS) >= (CURRENT DATE - cast('1' AS integer) days) THEN 'Y' ELSE 'N' END NEW_BILLING
FROM Q.DUMMY FOR READ ONLY WITH UR [42001-199]
For some reason days are converted to DAYS[*], we can see that in error message.
Customer method i tried in schema-db2.sql:
drop ALIAS if exists days;
CREATE ALIAS days as '
import java.lang.String;
#CODE
java.lang.String days() throws Exception {
return "days";
}
';
applicaiton.properties:
spring.datasource.url=jdbc:h2:mem:testdb;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=false;Mode=DB2
DAYS is not a function and is not a something that other databases support. Db2 also uses non-standard interval literals.
If you can build H2 from its current sources, you can use cast('1' AS integer) day in it (not the days) and such construction is also supported by Db2. You can also simply use 1 DAY, it is supported by current H2 and Db2 too.
(CURRENT_DAY - 1 DAY)
Sources of H2 are available on GitHub:
https://github.com/h2database/h2database
Building instructions are here:
https://h2database.com/html/build.html#building
You need a jar target.
To compile H2 from the current sources you need JDK 8, 9, 10, 11, or 12. Compiled jar will be compatible with more recent versions.

How to insert records into a table using Sequel

This is my code to insert a few lines in my simple "series" table:
db = sequel.postgres(config['dbname'],:user=>config['user'],:password=>config['password'],:host=>config['host'],:port=>config['port'],:max_connections=>10)
#db.create_table? 'series' do
primary_key "series_id" , :autoincrement=>true
String "series_name"
end
seriesDS = db['series']
seriesDS.insert('series_name' => 'test_value')
At seriesDS.insert I get a
Sequel::DatabaseError - PG::SyntaxError: ERREUR: erreur de syntaxe sur ou près de « series »
I didn't manage to get the full SQL query for analysys in STDOUT. It's strange because I added this:
logger = Logger.new STDOUT
logger.level = Logger::DEBUG
db.loggers << logger
It appears to be generating the wrong SQL, but I have no clue to the error's source.
I'm using:
Ruby 2.2.5
Sequel 4.4.1
Postgresql 9.6
The program is launched using ruby -E utf8.
Sequel uses Ruby symbols to represent SQL identifiers. At the very least, you must use seriesDS = db[:series].
Other cases where you want an SQL identifier you should probably switch from using strings to symbols.

cucumber ActiveRecord::ConnectionNotEstablished

The exception I am getting is "ActiveRecord::ConnectionNotEstablished: No connection pool for ActiveRecord::Base". I am really on the deep end of the pool (no pun intended) with this one. I really don't understand the connection and connection pool handling, even as much as I have studied this problem. I'm assuming this might be scope related inside of Cucumber, but I do not know. Any and all assistance is appreciated.
Here are the details:
The exception occurs when I perform a count from a Then clause:
WorkTable.where('? is not null',col['COLUMN_NAME']).count
It does not occur if I send the sql directly through the connection:
WorkTable.connection.select_all(st.encode('utf-8')).first['nulls']
My scenario reads as follows:
Scenario: CompanyMaster test for null value
Given table dbo.base_table in stage
Then these columns are expected to be not null
| COLUMN_NAME | nulls |
| id | 0 |
| company_name | 0 |
I establish my class in my env.rb:
class WorkTable < ActiveRecord::Base
end
ActiveRecord::Base.configurations = YAML.load_file(yaml) # yaml is database.yml file name
I establish my connection in a Given clause:
Given(/^table (\w+)\.?([\w_]+) in (\w+)(?: as (\w+))?$/) do |schema,name,env,id|
#sc_name = schema_file_name(schema,name)
WorkTable.logger.info title_line("* Active table(#{#sc_name}) *")
case id
# ActiveRecord::Base.configurations[env]
...
else
WorkTable.table_name = #sc_name
WorkTable.establish_connection(env.to_sym)
# ary = get_tables(WorkTable,schema:schema)
# expect( ary.any?{|s| s.casecmp(name)==0 } ).to eq(true)
end
end
I execute my test in a Then clause:
Then(/^these columns are expected to be not null$/) do |columns|
# expected is an instance of Cucumber::Ast::Table
WorkTable.logger.info title_line('Columns cannot be null')
results = []
columns.hashes.each {|col|
results << {
'COLUMN_NAME' => col['COLUMN_NAME'],
'nulls' => WorkTable.where('? is not null',col['COLUMN_NAME']).count.to_s
}
}
columns.diff!(results,surplus_row: false)
end
It is the WorkTable.where that throws the "ActiveRecord::ConnectionNotEstablished: No connection pool for ActiveRecord::Base". Again, if I use the WorkTable.connection method, I do not get it. Also, it executes fine if I copy all the function code to single ruby script.
I see the following when I "pp WorkTable.connection":
#<ActiveRecord::ConnectionAdapters::SQLServerAdapter version: 4.2.2, mode: dblib, azure: false>
And I see the following when I "pp WorkTable.connection_pool":
#<ActiveRecord::ConnectionAdapters::ConnectionPool:0x42f5238
#automatic_reconnect=true,
#available=
#<ActiveRecord::ConnectionAdapters::ConnectionPool::Queue:0x42f4f20
#cond=
#<MonitorMixin::ConditionVariable:0x42f4ed8
#cond=
#<ConditionVariable:0x42f4de8
#waiters=[],
#waiters_mutex=#<Mutex:0x42f4d58>>,
#monitor=
#<ActiveRecord::ConnectionAdapters::ConnectionPool:0x42f5238 ...>>,
#lock=#<ActiveRecord::ConnectionAdapters::ConnectionPool:0x42f5238 ...>,
#num_waiting=0,
#queue=[]>,
#checkout_timeout=5,
#connections=
[#<ActiveRecord::ConnectionAdapters::SQLServerAdapter version: 4.2.2, mode: dblib, azure: false>],
#mon_count=0,
#mon_mutex=#<Mutex:0x42f51c0>,
#mon_owner=nil,
#reaper=
#<ActiveRecord::ConnectionAdapters::ConnectionPool::Reaper:0x42f51a8
#frequency=nil,
#pool=#<ActiveRecord::ConnectionAdapters::ConnectionPool:0x42f5238 ...>>,
#reserved_connections=
#<ThreadSafe::Cache:0x42f4fc8
#backend=
{16931712=>
#<ActiveRecord::ConnectionAdapters::SQLServerAdapter version: 4.2.2, mode: dblib, azure: false>},
#default_proc=nil>,
#size=5,
#spec=
#<ActiveRecord::ConnectionAdapters::ConnectionSpecification:0x42f55c8
#adapter_method="sqlserver_connection",
#config=
{:host=>"server_name",
:database=>"mssb_stg",
:encoding=>"utf-8",
:adapter=>"sqlserver",
:timeout=>5000}>>
Ruby 1.9.3, activerecord (4.2.0), activerecord-sqlserver-adapter (4.2.2), and cucumber (1.3.18). And sql server 2014 [this has been a bugger for me].
Thank you for you time and consideration.
dvn
== Additional detail ==
Ignore the sql-server reference. I get the same exception when I reconfigure to work with SqLite. So it is not related to db platform.
Check your env.rb, conf in supports, it seems you are making connections in steps, ideally you should do it in before_scenario or before feature file rather than per steps.
It could be possible after steps your connection is not working properly.

Test the existence of a Teradata table and create the table if non-existent

Our Continuous Inegration server (Hudosn) is having a strange issue when attempting to run a simple create table statement in Teradata.
This statement tests the existence of the max_call table:
unless $teradata_connection.table_exists? :arm_custom_db__max_call_attempt_parameters
$teradata_connection.run('CREATE TABLE all_wkscratchpad_db.max_call_attempt_parameters AS (SELECT * FROM arm_custom_db.max_call_attempt_parameters ) WITH NO DATA')
end
The table_exists? method does the following:
def table_exists?(name)
v ||= false # only retry once
sch, table_name = schema_and_table(name)
name = SQL::QualifiedIdentifier.new(sch, table_name) if sch
from(name).first
true
rescue DatabaseError => e
if e.to_s =~ /Operation not allowed for reason code "7" on table/ && v == false
# table probably needs reorg
reorg(name)
v = true
retry
end
false
end
So as per the from(name).first line, the test which this method is performing is just a simple select statement, which, in SQL, looks like: SELECT TOP 1 MAX(CAST(MAX_CALL_ATTEMPT_CNT AS BIGINT)) FROM ALL_WKSCRATCHPAD_DB.MAX_CALL_ATTEMPT_PARAMETERS
The above SQL statement executes perfectly fine within Teradata SQL Assistant, so it's not a SQL syntax issue. The generic ID which our testing suite (Rubymine) uses is also not the issue; that ID has select access to the arm_custom_db.
The exeption which I can see is being thrown (within the builds console output on Hudson) is
Sequel::DatabaseError: Java::ComTeradataJdbcJdbc_4Util::JDBCException. Since this execption is a subclass of DatabaseError, the exception shouldn't be the problem either.
Also: We use unless statements like this every day for hundreds of different tables, and all except this one work correctly. This statement just seems to be a problem.
The complete error message which appears in the builds console output of Hudson is as follows:
[2015-01-07T13:56:37.947000 #16702] ERROR -- : Java::ComTeradataJdbcJdbc_4Util::JDBCException: [Teradata Database] [TeraJDBC 13.10.00.17] [Error 3807] [SQLState 42S02] Object 'ALL_WKSCRATCHPAD_DB.MAX_CALL_ATTEMPT_PARAMETERS' does not exist.: SELECT TOP 1 MAX(CAST(MAX_CALL_ATTEMPT_CNT AS BIGINT)) FROM ALL_WKSCRATCHPAD_DB.MAX_CALL_ATTEMPT_PARAMETERS
Sequel::DatabaseError: Java::ComTeradataJdbcJdbc_4Util::JDBCException: [Teradata Database] [TeraJDBC 13.10.00.17] [Error 3807] [SQLState 42S02] Object 'ALL_WKSCRATCHPAD_DB.MAX_CALL_ATTEMPT_PARAMETERS' does not exist.
I don't understand why this specific bit of code is giving me issues...there does not appear to be anything special about this table or database, and all SQL code executes perfectly fine in Teradata when I am signed in with the same exact user ID that is being used to execute the code from Hudson.

ruby driver for mongodb : update method returns Fixnum, not Hash

I'm debugging a problem where occasional writes to a mongo collection seem to be failing. As I was going over my error checking code, I found that the update method in the Collection class, seems to be returning a Fixnum instead of a hash.
Here is a code fragment (with debug statements)
begin
puts "collection type: #{#db_collection.class}"
status = #db_collection.update(selector, document)
puts "[Warn] Database update returned NULL status" unless status
puts "[Error] Mongo update returned an error: #{status.class}" unless (status == true)
puts "[Error] Mongo update returned an error: #{status}" unless (status == true)
rescue => e
puts "[Warn] Unable to update mongoDB (#{e})"
end
When I run this code, I get the following output:
collection type: Mongo::Collection
[Error] Mongo update returned an error: Fixnum
[Error] Mongo update returned an error: 236
I was expecting the update function to return a true for successful operations and a Hash for fails as per the documentation:
Returns:
(Hash, true) — Returns a Hash containing the last error object if acknowledging writes. > Otherwise, returns true.
I'm using version 1.8.0 of the ruby driver.
I'm not sure how to properly check that the writes have occurred correctly. I know that the driver should throw an exception if the write fails but I'm not seeing that happen. I don't know how to properly check the status variable since it's not returning a type I expect.
Thanks in advance for any help here.

Resources