I'm importing historical football (or soccer, if you're from the US) data into a Neo4j database using a spring boot application (2.1.6.RELEASE) with the spring-boot-starter-data-neo4j dependency and a standalone, locally running 3.5.6 Neo4j database server.
But for some reason searching for an entity by a simple property and an attached, referenced entity, does not work, althought the relation is present in the database.
This is the part of the model, that is currently giving me a headache:
#NodeEntity(label = "Season")
open class Season(
#Id
#GeneratedValue
var id: Long? = null,
#Index(unique = true)
var name: String,
var seasonNumber: Long,
#Relationship(type = "IN_LEAGUE", direction = Relationship.OUTGOING)
var league: League?,
var start: LocalDate,
var end: LocalDate
)
#NodeEntity(label = "League")
open class League(
#Id
#GeneratedValue
var id: Long? = null,
#Index(unique = true)
var name: String,
#Relationship(type = "BELONGS_TO", direction = Relationship.OUTGOING)
var country: Country?
)
(I left out the Country class, as I'm pretty sure that it is not part of the problem)
To allow running the import more than once, I want to check if the corresponding entity is already present in the database and only import newer ones. So I added the following method SeasonRepository:
open class SeasonRepository : CrudRepository<Season, Long> {
fun findBySeasonNumberAndLeague(number: Long, league: League): Season?
}
But it is giving me a null result instead of the existing entity on consecutive runs, hence I get duplicates in my database.
I would have expected spring-data-neo4j to reduce the passed League to its Id and then have a generated query that looks somewhat like this:
MATCH (s:Season)-[:IN_LEAGUE]->(l:League) WHERE id(l) = {leagueId} AND s.seasonNumber = {seasonNumber} WITH s MATCH (s)-[r]->(o) RETURN s,r,o
but when I turn on finer logging on the neo4j package I see this output in the log file:
MATCH (n:`Season`) WHERE n.`seasonNumber` = { `seasonNumber_0` } AND n.`league` = { `league_1` } WITH n RETURN n,[ [ (n)-[r_i1:`IN_LEAGUE`]->(l1:`League`) | [ r_i1, l1 ] ] ], ID(n) with params {league_1={id=30228, name=1. Bundesliga, country={id=29773, name=Deutschland}}, seasonNumber_0=1}
So for some reason, spring-data seems to think, that the league property is a simple / primitive property and not a full releation, that needs to be resolved by the id (n.league= {league_1}).
I only got it to work, by passing the id of the league, and providing a custom query using the #Query annotation but I actually thought, that it would work with spring-data-neo4j out of the box.
Any help appreciated. Let me know if you need more details.
Spring Data Neo4j does not support objects as parameters at the moment. It is possible to query for properties on related entities/nodes e.g. findBySeasonNumberAndLeagueName if this is a suitable solution.
Related
As stated in official documentation, it's preferable to use the Multimap return type for the Android Room database.
With the next very simple example, it's not working correctly!
#Entity
data class User(#PrimaryKey(autoGenerate = true) val _id: Long = 0, val name: String)
#Entity
data class Book(#PrimaryKey(autoGenerate = true) val _id: Long = 0, val bookName: String, val userId: Long)
(I believe a loooot of the developers have the _id primary key in their tables)
Now, in the Dao class:
#Query(
"SELECT * FROM user " +
"JOIN book ON user._id = book.userId"
)
fun allUserBooks(): Flow<Map<User, List<Book>>>
The database tables:
Finally, when I run the above query, here is what I get:
While it should have 2 entries, as there are 2 users in the corresponding table.
PS. I'm using the latest Room version at this point, Version 2.4.0-beta02.
PPS. The issue is in how UserDao_Impl.java is being generated:
all the _id columns have the same index there.
Is there a chance to do something here? (instead of switching to the intermediate data classes).
all the _id columns have the same index there.
Is there a chance to do something here?
Yes, use unique column names e.g.
#Entity
data class User(#PrimaryKey(autoGenerate = true) val userid: Long = 0, val name: String)
#Entity
data class Book(#PrimaryKey(autoGenerate = true) valbookid: Long = 0, val bookName: String, val useridmap: Long)
as used in the example below.
or
#Entity
data class User(#PrimaryKey(autoGenerate = true) #ColumnInfo(name="userid")val _id: Long = 0, val name: String)
#Entity
data class Book(#PrimaryKey(autoGenerate = true) #ColumnInfo(name="bookid")val _id: Long = 0, val bookName: String, val #ColumnInfo(name="userid_map")userId: Long)
Otherwise, as you may have noticed, Room uses the value of the last found column with the duplicated name and the User's _id is the value of the Book's _id column.
Using the above and replicating your data using :-
db = TheDatabase.getInstance(this)
dao = db.getAllDao()
var currentUserId = dao.insert(User(name = "Eugene"))
dao.insert(Book(bookName = "Eugene's book #1", useridmap = currentUserId))
dao.insert(Book(bookName = "Eugene's book #2", useridmap = currentUserId))
dao.insert(Book(bookName = "Eugene's book #3", useridmap = currentUserId))
currentUserId = dao.insert(User(name = "notEugene"))
dao.insert(Book(bookName = "not Eugene's book #4", useridmap = currentUserId))
dao.insert(Book(bookName = "not Eugene's book #5", useridmap = currentUserId))
var mapping = dao.allUserBooks() //<<<<<<<<<< BREAKPOINT HERE
for(m: Map.Entry<User,List<Book>> in mapping) {
}
for convenience and brevity a Flow hasn't been used and the above was run on the main thread.
Then the result is what I believe you are expecting :-
Additional
What if we already have the database structure with a lot of "_id" fields?
Then you have some decisions to make.
You could
do a migration to rename columns to avoid the ambiguous/duplicate column names.
use alternative POJO's in conjunction with changing the extract output column names accordingly
e.g. have :-
data class Alt_User(val userId: Long, val name: String)
and
data class Alt_Book (val bookId: Long, val bookName: String, val user_id: Long)
along with :-
#Query("SELECT user._id AS userId, user.name, book._id AS bookId, bookName, user_id " +
"FROM user JOIN book ON user._id = book.user_id")
fun allUserBooksAlt(): Map<Alt_User, List<Alt_Book>>
so user._id is output with the name as per the Alt_User POJO
other columns output specifically (although you could use * as per allUserBookAlt2)
:-
#Query("SELECT *, user._id AS userId, book._id AS bookId " +
"FROM user JOIN book ON user._id = book.user_id")
fun allUserBooksAlt2(): Map<Alt_User, List<Alt_Book>>
same as allUserBooksAlt but also has the extra columns
you would get a warning warning: The query returns some columns [_id, _id] which are not used by any of [a.a.so70190116kotlinroomambiguouscolumnsfromdocs.Alt_User, a.a.so70190116kotlinroomambiguouscolumnsfromdocs.Alt_Book]. You can use #ColumnInfo annotation on the fields to specify the mapping. You can annotate the method with #RewriteQueriesToDropUnusedColumns to direct Room to rewrite your query to avoid fetching unused columns. You can suppress this warning by annotating the method with #SuppressWarnings(RoomWarnings.CURSOR_MISMATCH). Columns returned by the query: _id, name, _id, bookName, user_id, userId, bookId. public abstract java.util.Map<a.a.so70190116kotlinroomambiguouscolumnsfromdocs.Alt_User, java.util.List<a.a.so70190116kotlinroomambiguouscolumnsfromdocs.Alt_Book>> allUserBooksAlt2();
Due to Note that Room will not rewrite the query if it has multiple columns that have the same name as it does not yet have a way to distinguish which one is necessary. the #RewriteQueriesToDropUnusedColumns doesn't do away with the warning.
if using :-
var mapping = dao.allUserBooksAlt() //<<<<<<<<<< BREAKPOINT HERE
for(m: Map.Entry<Alt_User,List<Alt_Book>> in mapping) {
}
Would result in :-
possibly other options.
However, I'd suggest fixing the issue once and for all by using a migration to rename columns to all have unique names. e.g.
I want to implement a projection for messages with a calculated distance using native query
#Entity
#Table(name= "messages")
data class Message(
#Id
#GeneratedValue
var id: Int?=null,
var message: String,
var userName: String,
var location: Point? = null
)
my projecting interface
interface FoundMessage {
var id: Int
var message: String
var location: Point
var userName: String
var distance: Float
}
my repository
interface MessageRepository : CrudRepository<Message, Int> {
#Query(
"SELECT id,message,location,user_name as \"userName\", st_distance(messages.location,ST_SetSRID(st_makepoint(:#{#currentPos.x},:#{#currentPos.y}),4326)) AS \"distance\"" +
"FROM messages WHERE location && st_makeenvelope(:#{#llb.x}, :#{#llb.y}, :#{#urt.x}, :#{#urt.y}, 4326) ORDER BY distance DESC", nativeQuery = true)
fun findByLocationWithinBoundingBoxOrderByDistance(#Param("llb") llb: Point,
#Param("urt") urt: Point,
#Param("currentPos") currentPos: Point): List<FoundMessage>
}
As soon I remove location from my native query everything works fine and location is null in my projection.
My native query also works fine if I return a List of my entity instead of the projection interface.
So the problem is that hibernate is not able to create "org.locationtech.jts.geom.Point" if I use the projecting interface.
Any ideas how to solve it?
Or different approaches?
BR
The "no dialect mapping" message usually means that you don't have the correct dialect configured. You need a SpatialDialect that register the JDBC mappings for the spatial types. Check the log messages. You should see a Dialect with the word "Spatial" in its name.
The problem with Spring-Boot is that it tries to automatically figure out a suitable Dialect for your database. But it won't select a SpatialDialect. You'll have to manually configure it.
I need to retrieve a list of Category from the DB on the basis of value of column called owner. Here is my Category -
#Entity
#Table(name = "categories")
class Category(#Column(name = "category_id", nullable = false)
#Id #GeneratedValue(strategyGenerationType.AUTO)
var id: Long = 0,
#Column(name = "category_owner", nullable = false)
#field:NotNull(message = "Please assign an owner")
var owner: Long?,
#Column(name = "category_name", nullable = false)
#field:NotEmpty(message = "Please assign a name")
var name: String?)
Here is my interface which defines the function findByOwner -
interface CategoryRepository: JpaRepository<Category, Long> {
fun findByOwner(categoryOwner: Long): List<Category>
}
However, when I call the method, I get no response. I have made sure that the DB has correct data and I'm providing the correct owner Id. Have even invalidated the cache etc. What could be going wrong?
EDIT:
After spring.jpa.show-sql=true -
findAll()
Hibernate: select category0_.category_id as category1_0_, category0_.category_name as category2_0_, category0_.category_owner as category3_0_ from categories category0_
findByOwner()
Hibernate: select category0_.category_id as category1_0_, category0_.category_name as category2_0_, category0_.category_owner as category3_0_ from categories category0_ where category0_.category_owner=?
EDIT 2:
Turns out that my implementation was fine all along. The bug was in my service.
Create your named method according with the name of the column.
fun findByCategoryOwner(categoryOwner: Long): List<Category>
Or use #Query
#Query("SELECT * FROM categories WHERE category_owner = ?1", nativeQuery = true)
fun findByOwner(cateogryOwner: Long): List<Category
Can you put a breakpoint in org.springframework.data.jpa.repository.query.JpaQueryExecution class and when you execute findByOwner, it will come here.
When it reaches this breakpoint, select the query.createQuery(accessor).getResultList() and evaluate to see what value is returned by hibernate for spring-data-jpa to use
This post should help you. It appears to be happeing because of the parameter name mismatch.
Use camelCase to name your variables in Entity class then jpa will auto recognise the column name
findByCategoryOwner(String categoryOwner)
If you still wish to have underscore in your column names then try this
findByCategory_Owner(String categoryOwner)
I haven't tried the second option though
At least in java you need to provide the id in the method name:
**fun findByOwner_Id(categoryOwner: Long): List<Category>**
So change it from findByOwner -> findByOwnerId.
I've an entity defined like below. If I use save() Hibernate does not create a new index for newly created entity. Updating/modifying an existing entity works well and as expected.
I'm using kotling with spring boot 2.
#Entity(name = "shipment")
#Indexed
data class Shipment(
#Id #GeneratedValue(strategy = GenerationType.IDENTITY) val id: Long = -1,
#JoinColumn(name = "user") #ManyToOne() var user: User?,
#IndexedEmbedded
#JoinColumn(name = "sender") #ManyToOne(cascade = [CascadeType.ALL]) val sender: Contact,
#IndexedEmbedded
#JoinColumn(name = "sender_information") #ManyToOne(cascade = [CascadeType.ALL]) val senderInformation: ShipmentInformation,
) {}
Save function, I'm using this same function to update my entity and index is updated if index exists.
#Transactional
fun save(user: User, shipment: Shipment): Shipment {
shipment.user = user;
return this.shipmentRepository.save(shipment)
}
application.properties
spring.jpa.properties.hibernate.search.default.directory_provider=filesystem
spring.jpa.properties.hibernate.search.default.indexBase=./lucene/
spring.jpa.open-in-view=false
If I restart the server, indexing manually works too.
#Transactional
override fun onApplicationEvent(event: ApplicationReadyEvent) {
val fullTextEntityManager: FullTextEntityManager = Search.getFullTextEntityManager(entityManager)
fullTextEntityManager.createIndexer().purgeAllOnStart(true)
fullTextEntityManager.createIndexer().optimizeAfterPurge(true)
fullTextEntityManager.createIndexer().batchSizeToLoadObjects(15)
fullTextEntityManager.createIndexer().cacheMode(CacheMode.IGNORE)
fullTextEntityManager.createIndexer().threadsToLoadObjects(2)
fullTextEntityManager.createIndexer().typesToIndexInParallel(2)
fullTextEntityManager.createIndexer().startAndWait()
return
}
I tried to force to use JPA transaction manager but It did not help me.
#Bean(name = arrayOf("transactionManager"))
#Primary
fun transactionManager(#Autowired entityManagerFactory: EntityManagerFactory): org.springframework.orm.jpa.JpaTransactionManager {
return JpaTransactionManager(entityManagerFactory)
}
Update
I think I found why I don't get the results of newly inserted entities.
My search query has a condition on "pid" field which is declared:
#Field(index = Index.YES, analyze = Analyze.NO, store = Store.NO)
#SortableField
#Column(name = "id", updatable = false, insertable = false)
#JsonIgnore
#NumericField val pid: Long,
and query:
query.must(queryBuilder.keyword().onField("customer.pid").matching(user.customer.id.toString()).createQuery())
pid is not stored and so newly inserted values are not visible. Can this be the cause?
BTW: How can I query/search by nested indexed document id? In my case it is customer.id which is DocumentId. I've tried to change the query like below but don't get any result, should I create a new field to query?
query.must(queryBuilder.keyword().onField("customer.id").matching(user.customer.id.toString()).createQuery())
Update 2
I found a solution and now getting the newly inserted datas too. There was an error with definition of "pid" field and I've defined my Fields as below and it works as expected.
#Fields(
Field(name = "pid", index = Index.YES, analyze = Analyze.YES, store = Store.NO)
)
#SortableField(forField = "pid")
#Id #GeneratedValue(strategy = GenerationType.IDENTITY) val id: Long?,
Can we search and sort by id in an easy way or is it the best practice? I know that we should use native JPA functions to get results by id but in my case I need to search by an embedded id to restrict search results. (depends on role of user) so therefore it is not an option for me.
And I don't understand why manual indexing works...
BTW: How can I query/search by nested indexed document id? In my case it is customer.id which is DocumentId. I've tried to change the query like below but don't get any result, should I create a new field to query?
Normally you don't need to create a separate field if all you want is to perform an exact match.
Can we search and sort by id in an easy way
Searching, yes, at least in Hibernate Search 5.
Sorting, no: you need a dedicated field.
or is it the best practice?
The best practice is to declare a field alongside your #DocumentId if you need anything more complex than an exact match on the ID.
I know that we should use native JPA functions to get results by id
I'm not sure I understand what you mean by "native JPA functions".
but in my case I need to search by an embedded id to restrict search results. (depends on role of user)
Yes, this should work. That is, it should work if the id is properly populated.
And I don't understand why manual indexing works...
Neither do I, but I suppose the explanation lies in the "error in the definition of "pid" field". Maybe the ID wasn't populated properly in some cases, leading to the entity being considered as deleted by Hibernate Search?
If you need me to give you a definitive answer, the best way to get it would be to create a reproducer. You can use this as a template: https://github.com/hibernate/hibernate-test-case-templates/tree/master/search
This looks odd:
#Id #GeneratedValue(strategy = GenerationType.IDENTITY) val id: Long = -1,
I'd expect a nullable long, initialized to null (or whatever is the Kotlin equivalent).
I'm not sure this is the problem, but I imagine it could be, as a non-null ID is generally only expected from an already persisted entity.
Other than that, I think you're on the right track: if mass indexing works but not automatic indexing, it may have something to do with your changes not being executed in database transactions.
This problem make me physically ill.
Joke aside, I've been trying to add an authentication layer to my web app using spring-boot with security plugin. Here is my data class.
#Document(collection = "user")
data class User (
var name : String,
var password : String,
var email : String,
var type : String,
var status : String,
var balance : Int
){
#Id val id : String = ObjectId.get().toHexString()
}
After some searching, Ctr+C, Ctr+V, I'm successfully set-up some custom authentication that will get user information from database, look like this:
override fun loadUserByUsername(name : String): UserDetails {
logger.info(name)
val user = repo.findByName(name)
return User(user!!.name,passwordEncoder.encode(user.password),AuthorityUtils.NO_AUTHORITIES)
}
Here where the fun begin, its seem that the code never run pass val user = repo.findByName(name). Worst thing is, there are no exception being thrown, the code run to that line and the rest just disappear.
Out of frustration, I decide to fake the return object so that I can get pass the authentication like this:
override fun loadUserByUsername(name : String): UserDetails {
logger.info(name)
//val user = repo.findByName(name)
logger.debug("asdkfhasdklfjhasdf")
return User("string",passwordEncoder.encode("you"),AuthorityUtils.NO_AUTHORITIES)
}
Now, finally I can get some exception:
{
"timestamp": "2018-11-08T18:08:29.541+0000",
"status": 500,
"error": "Internal Server Error",
"message": "No accessor to set property #org.springframework.data.annotation.Id()private final java.lang.String com.sonnbh.jwt.User.id!",
"path": "/user"
}
The exception state that spring cannot access property id so I change the type of id from val to var.
#Document(collection = "user")
data class User (
var name : String,
var password : String,
var email : String,
var type : String,
var status : String,
var balance : Int
){
#Id var id : String = ObjectId.get().toHexString()
}
Finally, my app work as expected. However, after some attempt trying to dig deeper to the problem, I found that this problem only occur to spring-boot v2.1.0. My old project which use spring-boot v2.0.5 actually run fine with val id. This led me to some question:
Did I my old implement of data class User properly? I just want to prevent any change to User.id after its being read from database or init. What can I do to improve?
Why spring-boot v2.1 can't access to the property like spring-boot v2.0.5 did?
Spring Data in 2.1. has changed the way in which it deals with final fields in entities. It no longer uses reflection to override the immutability of the fields, which in general is good. There are a few ways to cope with the problem.
They are described here: https://jira.spring.io/browse/DATACMNS-1374?focusedCommentId=182289&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-182289
Here's what the Spring guys recommend:
Add a #PersistenceConstructor to construct the entity that sets immutable fields.
Add wither methods (MyEntity withXxx(…)) to create a new instance that contains the changed property value.
Alternatively: Use Kotlin's data class feature. This will basically do the same as wither methods.
Can only answer the first part; you could try moving the declaration of ID to be apart of the constructor? That will satisfy your requirement of only initialising when the object is created and it will still be read only.