LocalParams is really just a Dictionary<string, string> behind the scenes.
However, I want to pass multiple Boost Queries, which use the key "bq". Obviously, any attempt to add my second "bq" key will fail with An item with the same key has already been added.
var lp = new LocalParams();
lp.Add("bq", "ContentType:Update^3.0");
lp.Add("bq", "ContentType:Comment^0.5"); // Error occurs here...
What's the trick to passing multiple Boost Queries (or multiple anything, really)...
The comment above set me onto ExtraParams.
I thought it wouldn't work since that was a Dictionary<string, string> (thus leaving me in the same situation), but the actual property definition is IEnumerable<KeyValuePair<string, string>>. It's just set to a Dictionary<string,string> in the constructor.
So I did this:
var extraParams = new List<KeyValuePair<string, string>>();
extraParams.Add(new KeyValuePair<string, string>("bq", "SomeQuery^10"));
extraParams.Add(new KeyValuePair<string, string>("bq", "SomeOtherQuery^10"));
var options new new QueryOptions();
options.ExtraParams = extraParams; //Since my List implements the right interface
solr.Query(myQuery, options)
My testing shows that it works as intended.
Related
I'm using a interface interceptor to cache all methods that starts with "Get" but i can't figure out how to generate a unique cache key for every unknown parameter it can be anything and using GetHashCode is not an option as i can't be 100% sure that they have overridden the GetHashCode.
some thoughts was someting in the line of How can I create a unique hashcode for a JObject?
where JSON is used for a JObject i was thinking on JSON serialize every parameter then get the hash code like explained in the link above:
var obj = JToken.Parse(jsonString);
var comparer = new JTokenEqualityComparer();
var hashCode = comparer.GetHashCode(obj);
However i think this will be a performence hit so how can this be solved ?
The code i have so far is this but it wont handle the complex type situation where .ToString won't generate the raw value type like int, string etc.
private string CreateCacheKey(IInvocation invocation)
{
string className = invocation.TargetType.FullName;
string methodName = invocation.Method.Name;
var builder = new StringBuilder(100);
builder.Append(className);
builder.Append(".");
builder.Append(methodName);
for (int i = 0; i < invocation.Arguments.Length; i++)
{
var argument = invocation.Arguments[i];
var argumentValue = invocation.GetArgumentValue(i);
builder.Append("_");
builder.Append(argument);
if (argument != argumentValue)
{
builder.Append(argumentValue);
}
}
return string.Format("{0}-{1}", this.provider.CacheKey, builder);
}
I ended up using GetHashCode as it is the only viable solution if thay dont override the method it will not cache.
New to Spring JDBC template but I'm wondering if I am able to pass a list of parameters and execute query once for each parameter in list. As I've seen many examples, the list of parameters being passed is for the execution of the query using all the parameters provided. Rather I am trying to execute query multiple times and for each time using new parameter in list.
For example:
Let's say I have a List of Ids - params (Strings)
List<String> params = new ArrayList<String>();
params.add("1234");
params.add("2345");
trying to do something like:
getJdbcTemplate().query(sql, params, new CustomResultSetExtractor());
which I know as per documentation is not allowed. I mean for one it has to be an array. I've seen simple examples where query is something like "select * from employee where id = ?" and they are passing new Object[]{"1234"} into method. And I'm trying to avoid the IN() condition. In my case each id will return multiple rows which is why I'm using ResultSetExtractor.
I know one option would be to iterate over list and include each id in list as a parameter, something like:
for(String id : params){
getJdbcTemplate().query(sql, new Object[]{id}, new CustomResultSetExtractor());
}
Just want to know if I can do this some other way. Sorry, I Should mention that I am trying to do a Select. Originally was hoping to return a List of custom objects for each resultset.
You do need to pass an array of params for the API, but you may also assume that your first param is an array. I believe this should work:
String sql = "select * from employee where id in (:ids)"; // or should there be '?'
getJdbcTemplate().query(sql, new Object[]{params}, new CustomResultSetExtractor());
Or you could explicitly specify, that the parameter is an array
getJdbcTemplate().query(sql, new Object[]{params}, new int[]{java.sql.Types.ARRAY}, new CustomResultSetExtractor());
You can use preparedStatement and do batch job:
eg. from http://docs.spring.io/spring/docs/current/spring-framework-reference/html/jdbc.html
public int[] batchUpdate(final List<Actor> actors) {
int[] updateCounts = jdbcTemplate.batchUpdate("update t_actor set first_name = ?, " +
"last_name = ? where id = ?",
new BatchPreparedStatementSetter() {
public void setValues(PreparedStatement ps, int i) throws SQLException {
ps.setString(1, actors.get(i).getFirstName());
ps.setString(2, actors.get(i).getLastName());
ps.setLong(3, actors.get(i).getId().longValue());
}
public int getBatchSize() {
return actors.size();
}
});
return updateCounts;
}
I know you don't want to use the in clause, but I think its the best solution for your problem.
If you use a for in this way, I think it's not optimal.
for(String id : params){
getJdbcTemplate().query(sql, new Object[]{id}, new CustomResultSetExtractor());
}
I think it's a better solution to use the in clause. And then use a ResultSetExtractor to iterate over the result data. Your extractor can return a Map instead of a List, actually a Map of List.
Map<Integer, List<MyObject>>
Here there is a simple tutorial explaining its use
http://pure-essence.net/2011/03/16/how-to-execute-in-sql-in-spring-jdbctemplate/
I think this is the best solution:
public List<TestUser> findUserByIds(int[] ids) {
String[] s = new String[ids.length];
Arrays.fill(s, "?");
String sql = StringUtils.join(s, ',');
return jdbcTemplate.query(String.format("select * from users where id in (%s)", sql),
ArrayUtils.toObject(ids), new BeanPropertyRowMapper<>(TestUser.class));
}
this one maybe what you want. BeanPropertyRowMapper is just for example, it will be very slow when there's a lot of records. you should change it to another more efficient RowMapper.
I'm using Spring data MongoTemplate to manage mongo operations. I'm trying to save & update json full documents (using String.class in java).
Example:
String content = "{MyId": "1","code":"UG","variables":[1,2,3,4,5]}";
String updatedContent = "{MyId": "1","code":"XX","variables":[6,7,8,9,10]}";
I know that I can update code & variables independently using:
Query query = new Query(where("MyId").is("1"));
Update update1 = new Update().set("code", "XX");
getMongoTemplate().upsert(query, update1, collectionId);
Update update2 = new Update().set("variables", "[6,7,8,9,10]");
getMongoTemplate().upsert(query, update2, collectionId);
But due to our application architecture, it could be more useful for us to directly replace the full object. As I know:
getMongoTemplate().save(content,collectionId)
getMongoTemplate().save(updatedContent,collectionId)
implements saveOrUpdate functionality, but this creates two objects, do not update anything.
I'm missing something? Any approach? Thanks
You can use Following Code :
Query query = new Query();
query.addCriteria(Criteria.where("MyId").is("1"));
Update update = new Update();
Iterator<String> iterator = json.keys();
while(iterator.hasNext()) {
String key = iterator.next();
if(!key.equals("MyId")) {
Object value = json.get(key);
update.set(key, value);
}
}
mongoTemplate.updateFirst(query, update, entityClass);
There may be some other way to get keyset from json, you can use according to your convenience.
You can use BasicDbObject to get keyset.
you can get BasicDbObject using mongoTemplate.getConverter().
What is the correct way, when using Elasticsearch with Spark, to update existing entities?
I wanted to something like the following:
Get existing data as a map.
Create a new map, and populate it with the updated fields.
Persist the new map.
However, there are several issues:
The list of returned fields cannot contain the _id, as it is not part of the source.
If, for testing, I hardcode an existing _id in the map of new values, the following exception is thrown:
org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest
How should the _id be retrieved, and how should it be passed back to Spark?
I include the following code below to better illustrate what I was trying to do:
JavaRDD<Map<String, Object>> esRDD = JavaEsSpark.esRDD(jsc, INDEX_NAME+"/"+TYPE_NAME,
"?source=,field1,field2).values();
Iterator<Map<String, Object>> iter = esRDD.toLocalIterator();
List<Map<String, Object>> listToPersist = new ArrayList<Map<String, Object>>();
while(iter.hasNext()){
Map<String, Object> map = iter.next();
// Get existing values, and do transformation logic
Map<String, Object> newMap = new HashMap<String, Object>();
newMap.put("_id", ??????);
newMap.put("field1", new_value);
listToPersist.add(newMap);
}
JavaRDD javaRDD = jsc.parallelize(ImmutableList.copyOf(listToPersist));
JavaEsSpark.saveToEs(javaRDD, INDEX_NAME+"/"+TYPE_NAME);
Ideally, I would want to update the existing map in place, rather than create a new one.
Does anyone have any example code to show, when using Spark, the correct way to update existing entities in elasticsearch?
Thanks
This is how I've done it (Scala/Spark 2.3/Elastic-Hadoop v6.5).
To read (id or other metadata):
spark
.read
.format("org.elasticsearch.spark.sql")
.option("es.read.metadata",true) // allow to read metadata
.load("yourindex/yourtype")
.select(col("_metadata._id").as("myId"),...)
To update particular columns in ES:
myDataFrame
.select("myId","columnToUpdate")
.saveToEs(
"yourindex/yourtype",
Map(
"es.mapping.id" -> "myId",
"es.write.operation" -> "update", // important to change operation to partial update
"es.mapping.exclude" -> "myId"
)
)
Try adding this upsert to your Spark:
.config("es.write.operation", "upsert")
that will let you add new fields to existing documents
According to Elasticsearch Configuration you can get document metadata like _id by set read metadata option to true:
.config("es.read.metadata", "true")
And i think you cannot use '_id' as field name.
But you can create new field with different name like:
newMap.put("idfield", yourId);
then set name of the new field as a value for mapping id option to inform elastic that this field has the document id:
.config("es.mapping.id", "idfield")
BTW don't forget to set write operation as update:
.config("es.write.operation", "update")
I have seen this question asked, but have not found answers that work so I am asking it again.
I have a restful web service with a POST method that returns a serialized List.
<ArrayOfstring xmlns:i="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://schemas.microsoft.com/2003/10/Serialization/Arrays">
<string>QoL5vA8OKgydWIn%2fdWiu70nobBrJo4N9iXeHmtM7Aj4%3d</string>
<string>vxHyJiSSSvDSZWXOdcfl5FMQC97xxGEWDO8Zy8Qp3X5CwADUbEE8ifACQHR1n7uamEaIUbf85ZuBDB8FFCNY2tJAMJ2jNw09SqGVTpMU5uI06DLtuYnJqsPIF735NOhlvRhBLPnpp62DFMCVsDNHy55UBF3Ggi06ZWTiJ8LTYIf3FYiFLPpXLZ1wWeE5chAWQGfz7zDYJa1OgSZ</string>
<string>OqGAT7Yqe6DfyVD29BeIXFyGtabVCloaC9FA1Fp20JkJ9T17ZzyqhnGxwWda7FqqyJUM8YK9OdcOCgTYrl4JxalECdtJm75TdSG8IAPQlFHp6Gidg4EwZaO9FKahlYMm5JrFpxTmLrdLgMAkYEV7gIR6zIyIByAGwYYDDwH3QCHrym3CuueRnFWAHCJu1LJbx0zRtt7g3yEaTiJ</string>
</ArrayOfstring>
The code performing the call is as follows below.
XDocument xDocResponse = RestPOSTToXDocument(szBaseUri, szInput);
string szNamespace = xDocResponse.Root.Name.Namespace.ToString();
IXmlUtils utility = new XmlUtils();
List<string> lst = utility.DeserializeList<string>(xDocResponse, szNamespace);
I have also attempted a different version of the call as shown below.
List<List<string>> lst = utility.DeserializeList<List<string>>(xDocResponse, szNamespace);
Both versions of the calls generate the error provided below.
An exception of type 'System.InvalidOperationException' occurred in System.Xml.dll but was not handled in user code
Additional information: There is an error in XML document (0, 0).
My Deserialize method is provided below.
public List<T> DeserializeList<T>(XDocument doc, string szNamespace)
{
List<T> result = null;
XmlSerializer serializer = new XmlSerializer(typeof(List<T>), szNamespace);
using (System.Xml.XmlReader reader = doc.CreateReader())
{
result = (List<T>)serializer.Deserialize(reader);
}
return result;
}
The XML within the XDocument is valid, so I do not understand why this error is being generated. Also, even if I attempt to get the elements via navigation in the XDocument, it does not work. If I look at the doc.Root.value, it appears that all of the strings are concatenated together into a single string.
Does anyone have any information on how I may deserialize this XDocument into a List?
Thanks to the assistance of mellamokb, I found the problem. When my List is serialized, it is serialized into an ArrayOfstring. If I take the XML and replace ArrayOfstring with ArrayOfString, the deserialization works.
This does not make much sense. I would welcome an explanation of why this is necessary though it may have something to do with String vs string.
Essentially, I had to do the following:
string szXml = xDocResponse.ToString();
int nEndRoot = szXml.IndexOf(">");
szXml = szXml.Replace("ArrayOfstring", "ArrayOfString");
xDocResponse = utility.LoadXDocument(szXml);
string szNamespace = xDocResponse.Root.Name.Namespace.ToString();
List<string> lst = utility.DeserializeList<string>(xDocResponse, szNamespace);
Simply get the response stream and deserealize into string array and store it to List
Code Snippet:
List<string> listNew=new List<string>();
using (Stream answer = webResponse.GetResponseStream())
{
DataContractSerializer xmlSer = new DataContractSerializer(typeof(string[]));
var stringArr= (string[])xmlSer.ReadObject(answer);
foreach (string item in GenreList)
{
listNew.Add(item);
}
}