Am attempting to store values provided as int, time, hash into Cassandra using Datastax driver.
Hash appears as { "Q17.1_4"=>"val", "Q17.2"=>"other", ...}
Have defined table as:
int
timestamp
map
PK( int, timestamp )
I can get the PK inserted ok but am having trouble coercing the hash values into something that Cassandra can cope with.
Have created a prepared statement and using it while (trying to) looping through values:
stmt = session.prepare( "insert into forms( id, stamp, questions ) values ( ?,?,? )" )
...
json_val.each{ |key,val|
result = session.execute( stmt, val[ 'ID' ].to_i, Time.parse( val[ 'tDate' ]).to_i, val )
}
If I insert 'val' as a hash I get this error:
/lib/cassandra/statements/prepared.rb:53:in `bind': expecting exactly 3 bind parameters, 2 given (ArgumentError)
This says (to me) that there isn't a method to convert the hash to what Cassandra wants.
If I insert the val as a string (using hash.to_s) I get this error:
/lib/cassandra/protocol/type_converter.rb:331:in varchar_to_bytes': undefined methodencode' for 1:Fixnum (NoMethodError)
... same goes for converting it to a json string, changing double quotes to single and so on.
I can insert the values using the cqlsh (command line) ( EG insert into forms (id, stamp, questions) values ( 123, 12345678, { 'one':'two','three':'four'});)
So the question is - how do I get this Ruby hash into a format that the Cassandra driver will accept?
Using:
Ruby 2.1.3
Cassandra 2.0 (with latest Datastax driver)
EDIT:
JSON value of 'questions' hash is { "Q17.1_4":"val", "Q17.2":"other", ...}
Also - I can insert the first two columns just fine. Omitting the third value from the insert statement works so I know those two values aren't the culprit here.
Found two problems:
There is a note in the docs:
Last argument will be treated as options if it is a Hash. Therefore, make sure to pass empty options when executing a statement with the last parameter required to be a map datatype.
When I either put an empty hash as the last parameter or swapped my hash value for a different param the error message changed to:
type_converter.rb:331:in varchar_to_bytes': undefined methodencode' for 1:Fixnum (NoMethodError)
This lead me to believe that there was an invalid value in there somewhere. To fix this I created a temp hash and checked every value to make sure it was a string before adding it to this temp hash. Then using this temp value I was able to add to the DB using the stored procedure.
Related
I have a DB column which is generic type for some stats(qualitative and quantitative info).
Some values are string - type A and some values are numbers stored as string - type B.
What i want to do is cast the B types to number then add one to them and cast back to string and store.
Metadata.objects.filter(key='EVENT', type='COUNT').update(value=CAST(F(CAST('value', IntegerField()) + 1), CharField())
What i want to do is avoid race conditions using F expression and
update in DB.
https://docs.djangoproject.com/en/4.0/ref/models/expressions/#avoiding-race-conditions-using-f
It says in below post that casting and updating in db is possible for mysql
Mysql Type Casting in Update Query
I also know we can do arithmetic very easily on F expressions as it supports it and we can override functionality of add as well. How to do arthmetic on Django 'F' types?
How can i achieve Cast -> update -> cast -> store in Django queryset?
Try using annotation as follows:
Metadata.objects
.filter(key='EVENT', type='COUNT')
.annotate(int_value=CAST('value', IntegerField()))
.update(value=CAST(F('int_value') + 1, CharField())
Or maybe switching F and CAST works?
Metadata.objects
.filter(key='EVENT', type='COUNT')
.update(value=CAST( # cast the whole expression below
CAST( # cast a value
F('value'), # of field "value"
IntegerField() # to integer
) + 1, # then add 1
CharField() # to char.
)
I've added indentation, it helps sometimes to find the errors.
Also, doc says, CAST accepts field name, not an F-object. Maybe it works without F-object at all?
UPD: switched back to first example, it actually works :)
I believe the answer from #som-1 was informative but not substantiated with info or debugged data. I believe assuming is not always right.
I debugged the mysql queries formed in these two cases -
1 - Metadata.objects.update(value=Cast(Cast(F('value'), output_field=IntegerField()) + 1, output_field=CharField()))
2 - Metadata.objects.update(value=Cast(Cast('value', IntegerField()) + 1, CharField())) and
both give the same output as expected.
UPDATE Metadata SET value = CAST((CAST(value AS signed integer) + 1) AS char) WHERE ( key = 'EVENT' AND type = 'COUNT' )
Please find the link to add mysqld options to my.cnf and debug your queries. Location of my.cnf file on macOS
enabling queries - https://tableplus.com/blog/2018/10/how-to-show-queries-log-in-mysql.html
I have a Hash
person_params = {"firstname"=>"",
"lastname"=>"tom123",
"addresses_attributes"=>
{"0"=>
{"address_type"=>"main",
"catalog_delivery"=>"0",
"street"=>"tomstr",
"city"=>"tomcity"
}
}
}
With person_params[:addresses_attributes], I get:
# => {"0"=>{"address_type"=>"main", "catalog_delivery"=>"0", "street"=>"tomstr", "zip"=>"", "lockbox"=>"", "city"=>"tomcity", "country"=>""}}
1) How can I get a new hash without the leading 0?
desired_hash = {"address_type"=>"main", "catalog_delivery"=>"0", "street"=>"tomstr", "zip"=>"", "lockbox"=>"", "city"=>"tomcity", "country"=>""}
2) How can I check whether the attributes in the new hash are empty?
Answer 1:
person_params[:addresses_attributes]['0']
Answer 2:
hash = person_params[:addresses_attributes]['0']
hash.empty?
This looks just like a params hash from Rails =D. Anyway, it seems that your addresses_attributes contains some nested attributes. This means that what you have in practice is more of an array of hashes than a single hash, and that's what you see right? Instead of it being an actually Ruby Array, it is a hash with the index as a string.
So how do you get the address attributes? Well if you only want to get the first address, here are some ways to do that:
person_params[:addresses_attributes].values.first
# OR
person_params[:addresses_attributes]["0"]
In the first case, we will just take the values from the addreses_attributes hash, which gives us an Array from which we can take the first item. If there are no values in addresses_attributes, then we will get nil.
In the second case, we will just ask for the hash value with the key "0". If there are no values in addresses_attributes, we will get nil with this method also. (You might want to avoid using the second case, if you are not confident that the addresses_attributes hash will always be indexed from "0" and incremented by "1")
I need someone could help me out on how to trace the error of "mismatched data type" in visual foxpro 6.0 When I issues a command like this "insert into tmpcur from memvar".
tmpcur is a cursor having bulk numbers of columns and it is ready hard to trace which one is having mismatch in data type for insertion problem.
It is pretty difficult to trace the insertion loop of each record into VFP tables one by one unliked MSSQL profiler.
Appreciate to someone could help. Thanks.
This should help you. I have a temp cursor created with some bogus field / column names testing for types of character, integer, double, currency, date and time. Trying to follow what is the result of your scenario, I am taking the memory variable of "bbbb" which should be double (or numeric at the least), and changed it to a string.
I am then HOLDING the error trapping routine that MAY be in effect, then setting my own (as I don't think try/catch existed in VFP6.. it may, but I just don't remember. So, I did an ON ERROR, set a variable to true. Then, I default it to false, try the insert, then check the flag. If the flag IS set, then I go into a loop and try for each column in the given table/alias (in my example it is "C_Tmp", so replace with your table/alias). It goes through each variable, and if the data type is different from the table structure, it will dump the column name and table / memory value for you to review.
You could put this to a log file or something.
Now, another consideration. Some types are completely valid and common for implied conversion, such as character and memo fields can both get strings. Integer, double, float, currency can all work with generic "numeric" values.
So, if you encounter these differences, then we can go one level further and look for comparable types, but let me know and we can adjust as needed.
At least this should give you a huge jump to your insert issue.
CREATE CURSOR C_tmp ( cccc c(10), iiii i, bbbb b(2), ccyyyy y, ddd d, tttt t )
SCATTER MEMVAR memo
m.bbbb = "wrong data type, was double with 2 decimal"
lcHoldError = ON("ERROR")
ON ERROR lFailInsert = .t.
lFailInsert = .f.
INSERT INTO C_Tmp FROM memvar
IF lFailInsert
FOR lnI = 1 TO FCOUNT( "C_Tmp" )
lcTmp = FIELD( lnI, "C_Tmp" )
IF NOT TYPE( "C_Tmp." + lcTmp ) == TYPE( "m.&lcTmp" )
? "Invalid " + lcTmp + ", C_Tmp.&lcTmp, m.&lcTmp
ENDIF
ENDFOR
ENDIF
ON ERROR &lcHoldError
For some reason, Hive is not recognizing columns emitted as integers, but does recognize columns emitted as strings.
Is there something about Hive or RCFile or GZ that is preventing proper rendering of int?
My Hive DDL looks like:
create external table if not exists db.table (intField int, strField string) stored as rcfile location '/path/to/my/data';
And the relevant portion of my Java looks like:
BytesRefArrayWritable dataWrite = new BytesRefArrayWritable(2);
byte[] byteArray;
BytesRefWritable bytesRefWritable = new BytesRefWritable(); intWritable.set(myObj.getIntField());
byteArray = WritableUtils.toByteArray(intWritable.get());
bytesRefWritable.set(byteArray, 0, byteArray.length);
dataWrite.set(0, bytesRefWritable); // sets int field as column 0
bytesRefWritable = new BytesRefWritable();
textWritable.set(myObj.getStrField());
bytesRefWritable.set(textWritable.getBytes(), 0, textWritable.getLength());
dataWrite.set(1, bytesRefWritable); // sets str field as column 1
The code runs fine, and through logging I can see the various Writables have bytes within them.
Hive can read the external table as well, but the int field shows up as NULL, indicating some error.
SELECT * from db.table;
OK
NULL my string field
Time taken: 0.647 seconds
Any idea what might be going on here?
So, I'm not sure exactly why this is the case, but I got it working using the following method:
In the code that writes the byte array representing the integer value, instead of using WritableUtils.toByteArray(), I instead Text.set(Integer.toString(intVal)).getBytes().
In other words, I convert the integer to its String representation, and use the Text writable object to get the byte array as if it were a string.
Then, in my Hive DDL, I can call the column an int and it interprets it correctly.
I'm not sure what was initially causing the problem, be it a bug in WritableUtils, some incompatibility with compressed integer byte arrays, or a faulty understanding of how this stuff works on my part. In any event, the solution described above successfully meets the task's needs.
I am certain this is very simple, but I'm a noob to Ruby and can't seem to find the answer to a very basic question.
I have a table with a list of words in PostgreSQL. I am getting a return value from a query to get a count of rows. When I try to assign this returned value to a variable "wordcount," I can't seem to get just the integer value. For example, if I try to use wordcount (as obtained below), Ruby throws an error "can't convert Array into Integer (TypeError)." In short, how do I convert the value obtained from the query to an integer? Thanks in advance.
q = 'SELECT COUNT(*) FROM words'
res = conn.exec(q)
wordcount = res.values[0]
puts wordcount
Using PostgreSQL (and probably most other connectors), it would be:
wordcount = res[0]["count"]
are you sure you don't want to just do:
Word.count
though?
The exec method returns a PGResult instance and PGResult#values:
Returns all tuples as an array of arrays.
So your res is an array of arrays: one entry for each row in the result set and each entry will itself be an array with one entry per column. They're strings as well, try this:
wordcount = res.getvalue(0, 0).to_i
puts wordcount
The database library is giving you a return value that supports the return of multiple rows. You are probably getting an array with a single row that contains an array with a single column that takes your value.