For some reason when I insert a value to my Aurora serverless db cluster mysql 5.6, big int column it gets rounded after 16 digits.
Example:
734783792502575105 saves as 734783792502575100
9223372036854775807 saves as 9223372036854776000
My column is defined as -> bigint(20) unsigned NOT NULL.
Isn't unsigned BIGINT supposed to be able to have a max value 18446744073709551615?
EDIT:
Even stranger is that I am able to insert 18446744073709551615 into the column but when I try to run a regular select on the table after the insert I start getting the following error:
"Value '18,446,744,073,709,551,615' is outside of valid range for type java.lang.Long"
I tested the queries in the Query Editor in the AWS dashboard and also through aws RDSDataService lib for nodejs.
Thx
After running into this weird problem myself I found out that there is a setting in serverless-mysql to force bigInt into strings so the rounding error won't occur. It's on this page mysql options.
Here a snippet for setting the mysql config in nodeJS:
import mysql from 'serverless-mysql';
export const db = mysql({
config: {
// left out the normal stuff,
supportBigNumbers: true,
bigNumberStrings: true,
},
});
Related
So, the thing is, I'm trying to query a GSI table on DynamoDB and get a really weird behaviour.
The main table schema is as follows
- Partition key: test_id (string)
- Sort key: version (string)
- Other attributes (createdAt, tags, etc)
I want to obtain every entry that has a sort_key equal to v0_test WITHOUT filtering by partition key. To do this and avoid a full scan I created a GSI (Global Secondary Index) as follows:
- Partition key: version (string)
- Sort key: createdAt (number)
- Other attributes (test_id, tags, etc)
When querying this from AWS console I can query for every partition key equal to v0_test and I get the expected results, but when I query from inside a lambda function (runtime: nodejs16.x) I get an error.
The code for the query is as follows:
const dynamoDb = new AWS.DynamoDB.DocumentClient();
let params = {
TableName: dynamoTable,
IndexName: dynamoTableIndex ,
KeyConditionExpression: 'version = :v0 AND createdAt BETWEEN :tLower AND :tUpper',
ExpressionAttributeValues: {
':v0': 'v0_test',
':tUpper': Math.floor(Date.now() / 1000).toString(),
':tLower': '0'
}
};
let result = await dynamoDb.query(params).promise();
console.log("Success", result);
And the error I get is
ERROR ValidationException: Query condition missed key schema element: test_id
As you can see it's asking for the partition key for the main table.
Things I've tried:
Using AWS.DynamoDB instead of AWS.DynamoDB.DocumentClient. Same error
Changing version for test_id in the query. Got ERROR ValidationException: Query condition missed key schema element: version
Sending both version and test_id on KeyConditionExpression. Got the following error:ERROR ValidationException: KeyConditionExpressions must only contain one condition per key
New Dynamo table, new GSI. Same errors
I wasn't expecting that at all. It's my first time using DynamoDB but as I understand it the idea behind GSI's (or one of them) is to be able to query a DynamoDB table by other attribute than it's main partition key without having to do a full scan.
Any help appreciated and if you need more details just ask! It's my first time asking on StackOverflow too so I'm sure I'll miss something.
EDIT
Tested the suggested solution and got an error suggesting that the schema was wrong, which was a new error and got me thinking, so I tried specifying the ExpressionAttributeNames and it WORKED!.
I created the request as follows using DocumentClient:
const dynamoDb = new AWS.DynamoDB.DocumentClient();
let params = {
TableName: 'CLASSIFIER_TESTS_DEV_us-east-1',
IndexName: 'version-createdAt-index',
KeyConditionExpression: '#versionAttr = :version AND #ca BETWEEN :tLower AND :tUpper',
ExpressionAttributeNames: {
"#versionAttr": "version",
"#ca": "createdAt"
},
ExpressionAttributeValues: {
":version": "v0_test",
":tUpper": Date.now(),
":tLower": 0
}
};
let result = await dynamoDb.query(params).promise();
Thanks everyone!
I still think that it should have worked the way I did it the first time as that's the way everyone does it on tutorials/examples/documentation, but oh well, got it working and that's what matters for now.
This should not be an issue. I would suggest the following:
Test locally, have your DynamoDB API call run from your local machine, this will ensure that no issues are being caused by Lambda invocations
Hard code your TableName and IndexName as strings in your parameters.
If the above is still causing an issue, try the same logic but this time a Query from the base table.
Let me know if you still face issue after that, happy to help.
I am always getting postgresql error on sfid field because of unique restrictions after mapping and a bad SF synchronisation (so that field stay blank).
After second SF synchronisation errors I can't save other row.
Please help
In order to save rows without getting UNIQUE errors on sfid,
try to create 18 length uid to avoid blank field (x2 blank fields in postgreSQL is considered equal)
node.js:
const { v4: uuidv4 } = require('uuid');
let uuid = uuidv4();
record.sfId = uuid.substring(1, 18);
When running tests using phpunit I'm getting a not so nice exception
SQLSTATE[HY000]: General error: 25 column index out of range (SQL: select count(*) as aggregate from "attachments" where "id" = f3bad3ad-a888-41bc-b6fd-9a5998f6b527
The Attachment.Id is a UUID and the column is defined as primary key. When running the tests I am using SQLite with an in-memory db.
When switching over to MySQL I do not get an error anymore. I don't really know why the query would cause an error.
Any tips?
1) Make sure the property public $incrementing = false; is set on the model.
2) The migration should be $table->uuid('id')->primary();
3) Add the below function to the model:
public function getKeyType()
{
return 'string';
}
4) To diagnose, create a database.sqlite and allow the test to run on that DB. Diagnose the size of the ID column. Does it fit the UUID string? If it actually doesn't, you may have to update your Laravel app or edit your migrations so if it's running on Sqlite, it will create a string column with the appropriate length.
In a Spring Boot project, Java8, with hibernate-spatial and PostgresDB 9.4
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-spatial</artifactId>
<version>5.2.10.Final</version>
</dependency>
application.properties
spring.datasource.driver-class-name=org.postgresql.Driver
spring.jpa.database-platform=org.hibernate.spatial.dialect.postgis.PostgisPG94Dialect
spring.jpa.properties.hibernate.dialect=org.hibernate.spatial.dialect.postgis.PostgisPG94Dialect
(I tried also PostgisPG9Dialect)
My Entity has a property
...
import com.vividsolutions.jts.geom.Point;
....
#Column(columnDefinition = "Point")
private Point cityLocation;
If I save with null value it's ok, but if I put a value
setCityLocation(new GeometryFactory().createPoint(new Coordinate(lng, lat));
I have:
PSQLException: ERROR: column "city_location" is of type point but expression is of type bytea You will need to rewrite or cast the expression.
In my db I can see the column definition as
type: point
column size: 2147483647
data type: 1111
num prec radix: 10
char octet length: 2147483647
I'M GOING CRAZY... Why It doesn't work?
UPDATE (It still don't work, I'm collecting new informations)
1) I'm thinking the problem could be the creation of the db.
In my application.properties I also have :
spring.jpa.properties.hibernate.hbm2ddl.auto=update
so the schema will update 'automatically' by hibernate.
2) I can run with success a query directly on the db (I use "Squirrel SQL" as client)
update my_table set city_location = POINT(-13,23) where id = 1
and if I
select city_location from my_table where id = 1
the answer is
<Other>
I can't see the value... I got the same answer for the record with null value inside the point type...
3) After set a value to the 'point' column with a query, I'm no more able to read from the table, I receive the exception:
org.geolatte.geom.codec.WktDecodeException : Wrong symbol at position: 1 in Wkt: (-13.0,23.0)
4) I look inside the hibernate-spatial-5.2.10.Final.jar and I found two "geolatte" named classes in the package org.hibernate.spatial :
GeolatteGeometryJavaTypeDescriptor.class
GeolatteGeometryType.class
5) And also (specific for Squirrel SQL client experts):
if I try to change a value of a column in "my_table" (not the 'point' city_location but anyone of the other columns) I recive an error similar to the one I recive in java when I try to insert a point value:
Exception seen during check on DB. Exception was:
ERROR: operator does not exist: point = character varying
Hint: No operator matches the given name and argument type(s). You might need to add explicit type casts.
Squirrel is made with java.. so I can accept this strange thing, may be it compose the query in a 'wrong' way, maybe it is connected to the value I see when I make a select...
Any ideas?
I found the solution!!
A fix to the code was needed and a magic trick I read in another stackoverflow question saved my life.
The problem was that the db column was created in a wrong way:
in the db the column type should be geometry NOT point
I removed the columnDefinition = "Point" from the #Column annotation and I ran the query
CREATE EXTENSION postgis;
on my db following these instructions:
Postgis installation: type "geometry" does not exist
Krishna Sapkota you are my new super hero!
Just remove columnDefinition = "POINT", from #Column annotation, and just use the Point object. (i.e. Use default column definition)
I use JOOQ-3.1.0 to generate and execute dynamic queries for Oracle and Postgresql with Spring-4. In a scenario I have a partitioned table, which I need to query using JOOQ. I use DSL.tableByName(vblTablename); where vblTablename is the string received as a string in the query generation method, ex, vbl_default partition(p_04-Dec-14). (The vblTablename pattern differs for different databases, and is configured in the external property file). The JOOQ generates the sql, but with the double-quote around the tablename. The query and error shown below
Query
SELECT COUNT(ID) COUNT FROM "vbl_default partition(p_04-Dec-14)"
where (rts between timestamp '2014-12-04 00:00:00.0' and timestamp '2014-12-05 00:00:00.0' and userid in (2))
Error
ORA-00972: identifier is too long
00972. 00000 - "identifier is too long"
*Cause: An identifier with more than 30 characters was specified.
*Action: Specify at most 30 characters.
Error at Line: 4 Column: 29
Though I have set the below settings on the DefaultDSLContext
Settings settings = new Settings();
settings.setRenderNameStyle(RenderNameStyle.AS_IS);
How do I remove the quote around the table? Any other settings have I missed?
The idea behind DSL.tableByName(String...) is that you provide a table ... by name :-)
What you're looking for is a plain SQL table, via DSL.table(String).
You can write:
// Assuming this import
import static org.jooq.impl.DSL.*;
DSL.using(configuration)
.select(count(VBL_DEFAULT.ID))
.from(table("vbl_default partition(p_04-Dec-14)"))
.where(...);
Or by using the convenient overload SelectFromStep.from(String)
DSL.using(configuration)
.select(count(VBL_DEFAULT.ID))
.from("vbl_default partition(p_04-Dec-14)")
.where(...);
More information about plain SQL in jOOQ can be obtained from this manual page:
http://www.jooq.org/doc/latest/manual/sql-building/plain-sql/
Partition support
Note that support for Oracle partitions is on the roadmap: #2775. If in the mean time you wish to use partitioned tables more often, you could also write your own function for that:
// Beware of the risk of SQL injection, though!
public <R extends Record> Table<R> partition(Table<R> table, String partition) {
return DSL.table("{0} partition(" + partition + ")", table);
}
... and then:
DSL.using(configuration)
.select(count(VBL_DEFAULT.ID))
.from(partition(VBL_DEFAULT, "p_04-Dec-14"))
.where(...);