How to get all Keys from Redis using redis template - spring

I have been stuck with this problem with quite some time.I want to get keys from redis using redis template.
I tried this.redistemplate.keys("*");
but this doesn't fetch anything. Even with the pattern it doesn't work.
Can you please advise on what is the best solution to this.

I just consolidated the answers, we have seen here.
Here are the two ways of getting keys from Redis, when we use RedisTemplate.
1. Directly from RedisTemplate
Set<String> redisKeys = template.keys("samplekey*"));
// Store the keys in a List
List<String> keysList = new ArrayList<>();
Iterator<String> it = redisKeys.iterator();
while (it.hasNext()) {
String data = it.next();
keysList.add(data);
}
Note: You should have configured redisTemplate with StringRedisSerializer in your bean
If you use java based bean configuration
redisTemplate.setDefaultSerializer(new StringRedisSerializer());
If you use spring.xml based bean configuration
<bean id="stringRedisSerializer" class="org.springframework.data.redis.serializer.StringRedisSerializer"/>
<!-- redis template definition -->
<bean
id="redisTemplate"
class="org.springframework.data.redis.core.RedisTemplate"
p:connection-factory-ref="jedisConnectionFactory"
p:keySerializer-ref="stringRedisSerializer"
/>
2. From JedisConnectionFactory
RedisConnection redisConnection = template.getConnectionFactory().getConnection();
Set<byte[]> redisKeys = redisConnection.keys("samplekey*".getBytes());
List<String> keysList = new ArrayList<>();
Iterator<byte[]> it = redisKeys.iterator();
while (it.hasNext()) {
byte[] data = (byte[]) it.next();
keysList.add(new String(data, 0, data.length));
}
redisConnection.close();
If you don't close this connection explicitly, you will run into an exhaustion of the underlying jedis connection pool as said in https://stackoverflow.com/a/36641934/3884173.

try:
Set<byte[]> keys = RedisTemplate.getConnectionFactory().getConnection().keys("*".getBytes());
Iterator<byte[]> it = keys.iterator();
while(it.hasNext()){
byte[] data = (byte[])it.next();
System.out.println(new String(data, 0, data.length));
}

Try redisTemplate.setKeySerializer(new StringRedisSerializer());

Avoid to use keys command. It may ruin performance when it is executed against large databases.
You should use scan command instead. Here is how you can do it:
RedisConnection redisConnection = null;
try {
redisConnection = redisTemplate.getConnectionFactory().getConnection();
ScanOptions options = ScanOptions.scanOptions().match("myKey*").count(100).build();
Cursor c = redisConnection.scan(options);
while (c.hasNext()) {
logger.info(new String((byte[]) c.next()));
}
} finally {
redisConnection.close(); //Ensure closing this connection.
}
or do it much simplier with Redisson Redis Java client:
Iterable<String> keysIterator = redisson.getKeys().getKeysByPattern("test*", 100);
for (String key : keysIterator) {
logger.info(key);
}

Try
import org.springframework.data.redis.core.RedisTemplate;
import org.apache.commons.collections.CollectionUtils;
String key = "example*";
Set keys = redisTemplate.keys(key);
if (CollectionUtils.isEmpty(keys)) return null;
List list = redisTemplate.opsForValue().multiGet(keys);

It did work, but seems not recommended? Because we can't use Keys command in production. I assume RedisTemplate.getConnectionFactory().getConnection().keys is calling redis Keys command. What are the alternatives?

I was using redisTemplate.keys(), but it was not working. So I used jedis, it worked. The following is the code that I used.
Jedis jedis = new Jedis("localhost", 6379);
Set<String> keys = jedis.keys("*".getBytes());
for (String key : keys) {
// do something
} // for

Solution can be like this
String pattern = "abc"+"*";
Set<String> keys = jedis.keys(pattern);
for (String key : keys) {
jedis.keys(key);
}
Or you can use jedis.hscan() and ScanParams instead.

Related

Check if redis key is hash or string using restTemplate

I have Redis DB consisting of hashes and strings.
I got all keys from DB using this code:
Set<byte[]> keys = redisTemplate.getConnectionFactory().getConnection().keys("*".getBytes());
Iterator<byte[]> it = keys.iterator();
while(it.hasNext()){
byte[] data = (byte[])it.next();
String key = (new String(data, 0, data.length));
System.out.println(key);
}
from here: How to get all Keys from Redis using redis template
Since the key can be hash or string, how to determine when I can use opsForHash and opsForValue i.e. how to check if it is hash or it is a string in spring boot using restTemplate.
The Redis command to get the type of a key is TYPE: https://redis.io/commands/type
You can use RedisTemplate's public DataType type(K key) method to execute this: https://docs.spring.io/spring-data/redis/docs/current/api/org/springframework/data/redis/core/RedisTemplate.html#type-K-
Here's an example:
Set<byte[]> keys = redisTemplate.getConnectionFactory().getConnection().keys("*".getBytes());
Iterator<byte[]> it = keys.iterator();
while(it.hasNext()){
byte[] data = (byte[])it.next();
String key = (new String(data, 0, data.length));
DataType type = redisTemplate.type(key);
if (type == DataType.HASH) {
// ...
} else if (type == DataType.STRING) {
// ...
}
}
Edit: one more piece of advice is that you may want to use SCAN instead of KEYS * (mentioned in one answer to the SO question you linked). Scan is generally better in production since it doesn't try to get and return all keys at once.

Convert ImmutableListMultimap to Map using Collectors.toMap

I would like to convert a ImmutableListMultimap<String, Character> to Map<String, List<Character>>.
I used to do it in the non-stream way as follows
void convertMultiMaptoList(ImmutableListMultimap<String, Character> reverseImmutableMultiMap) {
Map<String, List<Character>> result = new TreeMap<>();
for( Map.Entry<String, Character> entry: reverseImmutableMultiMap.entries()) {
String key = entry.getKey();
Character t = entry.getValue();
result.computeIfAbsent(key, x-> new ArrayList<>()).add(t);
}
//reverseImmutableMultiMap.entries().stream().collect(Collectors.toMap)
}
I was wondering how to write the above same logic using java8 stream way (Collectors.toMap).
Please share your thoughts
Well there is already a asMap that you can use to make this easier:
Builder<String, Character> builder = ImmutableListMultimap.builder();
builder.put("12", 'c');
builder.put("12", 'c');
ImmutableListMultimap<String, Character> map = builder.build();
Map<String, List<Character>> map2 = map.asMap()
.entrySet()
.stream()
.collect(Collectors.toMap(Entry::getKey, e -> new ArrayList<>(e.getValue())));
If on the other hand you are OK with the return type of the asMap than it's a simple method call:
ImmutableMap<String, Collection<Character>> asMap = map.asMap();
Map<String, List<Character>> result = reverseImmutableMultiMap.entries().stream()
.collect(groupingBy(Entry::getKey, TreeMap::new, mapping(Entry::getValue, toList())));
The important detail is mapping. It will convert the collector (toList) so that it collects List<Character> instead of List<Entry<String, Character>>. According to the mapping function Entry::getValue
groupingBy will group all entries by the String key
toList will collect all values with same key to a list
Also, passing TreeMap::new as an argument to groupingBy will make sure you get this specific type of Map instead of the default HashMap

OrientDB POJO Method proxy not working properly

I am using the OObjectDatabaseTx implementation of OrientDB to store my POJOs in the database. When I try to retrieve some POJOs with a SQL commant, I get the result set but the attributes of the POJOs seem to be empty (getters regurning null).
I register my classes properly with
db.getEntityManager().registerEntityClass(MyUser.class);
The following code describes my problem:
Map<String, String> params = new HashMap<String, String>();
params.put("name", username);
List<MyUser> users = db.command(
new OSQLSynchQuery<MyUser>(
"select * from MyUser where "
+ "name = :name"))
.execute(params);
for (MyUser founduser : users) {
ODocument doc = db.getRecordByUserObject(founduser, false);
String pass = doc.field("pwd");
assertEquals(pass != null, true); // passes
assertEquals(founduser.getPwd() != null, true); // fails
}
How can I get the method getPwd to return the proper value?
I am now using Version 1.3.0 and this has worked before (afaik in 1.1.0).
Can you see if the POJO has the "pwd" field set inside of it?

NHib 3 Configuration & Mapping returning empty results?

Note: I'm specifically not using Fluent NHibernate but am using 3.x's built-in mapping style. However, I am getting a blank recordset when I think I should be getting records returned.
I'm sure I'm doing something wrong and it's driving me up a wall. :)
Background / Setup
I have an Oracle 11g database for a product by IBM called Maximo
This product has a table called workorder which lists workorders; that table has a field called "wonum" which represents a unique work order number.
I have a "reporting" user which can access the table via the maximo schema
e.g. "select * from maximo.workorder"
I am using Oracle's Managed ODP.NET DLL to accomplish data tasks, and using it for the first time.
Things I've Tried
I created a basic console application to test this
I added the OracleManagedClientDriver.cs from the NHibernate.Driver on the master branch (it is not officially in the release I'm using).
I created a POCO called WorkorderBriefBrief, which only has a WorkorderNumber field.
I created a class map, WorkorderBriefBriefMap, which maps only that value as a read-only value.
I created a console application with console output to attempt to write the lines of work orders.
The session and transaction appear to open correct,
I tested a standard ODP.NET OracleConnection to my connection string
The Code
POCO: WorkorderBriefBrief.cs
namespace PEApps.Model.WorkorderQuery
{
public class WorkorderBriefBrief
{
public virtual string WorkorderNumber { get; set; }
}
}
Mapping: WorkorderBriefBriefMap.cs
using NHibernate.Mapping.ByCode;
using NHibernate.Mapping.ByCode.Conformist;
using PEApps.Model.WorkorderQuery;
namespace ConsoleTests
{
public class WorkorderBriefBriefMap : ClassMapping<WorkorderBriefBrief>
{
public WorkorderBriefBriefMap()
{
Schema("MAXIMO");
Table("WORKORDER");
Property(x=>x.WorkorderNumber, m =>
{
m.Access(Accessor.ReadOnly);
m.Column("WONUM");
});
}
}
}
Putting it Together: Program.cs
namespace ConsoleTests
{
class Program
{
static void Main(string[] args)
{
NHibernateProfiler.Initialize();
try
{
var cfg = new Configuration();
cfg
.DataBaseIntegration(db =>
{
db.ConnectionString = "[Redacted]";
db.Dialect<Oracle10gDialect>();
db.Driver<OracleManagedDataClientDriver>();
db.KeywordsAutoImport = Hbm2DDLKeyWords.AutoQuote;
db.BatchSize = 500;
db.LogSqlInConsole = true;
})
.AddAssembly(typeof(WorkorderBriefBriefMap).Assembly)
.SessionFactory().GenerateStatistics();
var factory = cfg.BuildSessionFactory();
List<WorkorderBriefBrief> query;
using (var session = factory.OpenSession())
{
Console.WriteLine("session opened");
Console.ReadLine();
using (var transaction = session.BeginTransaction())
{
Console.WriteLine("transaction opened");
Console.ReadLine();
query =
(from workorderbriefbrief in session.Query<WorkorderBriefBrief>() select workorderbriefbrief)
.ToList();
transaction.Commit();
Console.WriteLine("Transaction Committed");
}
}
Console.WriteLine("result length is {0}", query.Count);
Console.WriteLine("about to write WOs");
foreach (WorkorderBriefBrief wo in query)
{
Console.WriteLine("{0}", wo.WorkorderNumber);
}
Console.WriteLine("DONE!");
Console.ReadLine();
// Test a standard connection below
string constr = "[Redacted]";
OracleConnection con = new OracleConnection(constr);
con.Open();
Console.WriteLine("Connected to Oracle Database {0}, {1}", con.ServerVersion, con.DatabaseName.ToString());
con.Dispose();
Console.WriteLine("Press RETURN to exit.");
Console.ReadLine();
}
catch (Exception ex)
{
Console.WriteLine("Error : {0}", ex);
Console.ReadLine();
}
}
}
}
Thanks in advance for any help you can give!
Update
The following code (standard ADO.NET with OracleDataReader) works fine, returning the 16 workorder numbers that it should. To me, this points to my use of NHibernate more than the Oracle Managed ODP.NET. So I'm hoping it's just something stupid that I did above in the mapping or configuration.
// Test a standard connection below
string constr = "[Redacted]";
OracleConnection con = new Oracle.ManagedDataAccess.Client.OracleConnection(constr);
con.Open();
Console.WriteLine("Connected to Oracle Database {0}, {1}", con.ServerVersion, con.DatabaseName);
var cmd = new OracleCommand();
cmd.Connection = con;
cmd.CommandText = "select wonum from maximo.workorder where upper(reportedby) = 'MAXADMIN'";
cmd.CommandType = CommandType.Text;
Oracle.ManagedDataAccess.Client.OracleDataReader reader = cmd.ExecuteReader();
while (reader.Read())
{
Console.WriteLine(reader.GetString(0));
}
con.Dispose();
When configuring NHibernate, you need to tell it about your mappings.
I found the answer -- thanks to Oskar's initial suggestion, I realized it wasn't just that I hadn't added the assembly, I also needed to create a new mapper.
to do this, I added the following code to the configuration before building my session factory:
var mapper = new ModelMapper();
//define mappingType(s) -- could be an array; in my case it was just 1
var mappingType = typeof (WorkorderBriefBriefMap);
//use AddMappings instead if you're mapping an array
mapper.AddMapping(mappingType);
//add the compiled results of the mapper to the configuration
cfg.AddMapping(mapper.CompileMappingForAllExplicitlyAddedEntities());
var factory = cfg.BuildSessionFactory();

ODP.NET and parameters

I have built a common app that works with PostgreSQL and should work on Oracle.
However i'm getting strange errors when inserting records through a parametrized query.
My formatted query looks like this:
"INSERT INTO layer_mapping VALUES (#lm_id,#lm_layer_name,#lm_layer_file);"
Unlike Npgsql which documents how to use the parameters, i could not found how Oracle "prefers" them to be used. I could only find :1, :2, :3, for example.
I do not wanto use sequential parameters, i want to use them in a named way.
Is there a way to do it? Am i doing something wrong?
Thanks
You can use named parameters with ODP.NET like so:
using (var cx=new OracleConnection(connString)){
using(var cmd=cx.CreateCommand()){
cmd.CommandText="Select * from foo_table where bar=:bar";
cmd.BindByName=true;
cmd.Parameters.Add("bar",barValue);
///...
}
}
I made this lib https://github.com/pedro-muniz/ODPNetConnect/blob/master/ODPNetConnect.cs
so you can do parameterized write and read like this:
ODPNetConnect odp = new ODPNetConnect();
if (!String.IsNullOrWhiteSpace(odp.ERROR))
{
throw new Exception(odp.ERROR);
}
//Write:
string sql = #"INSERT INTO TABLE (D1, D2, D3) VALUES (:D1, :D2, :D3)";
Dictionary<string, object> params = new Dictionary<string, object>();
params["D1"] = "D1";
params["D2"] = "D2";
params["D3"] = "D3";
int affectedRows = odp.ParameterizedWrite(sql, params);
if (!String.IsNullOrWhiteSpace(odp.ERROR))
{
throw new Exception(odp.ERROR);
}
//read
string sql = #"SELECT * FROM TABLE WHERE D1 = :D1";
Dictionary<string, object> params = new Dictionary<string, object>();
params["D1"] = "D1";
DataTable dt = odp.ParameterizedRead(sql, params);
if (!String.IsNullOrWhiteSpace(odp.ERROR))
{
throw new Exception(odp.ERROR);
}
Notes: you have to change these lines in ODPNetConnect.cs to set connection string:
static private string devConnectionString = "SET YOUR DEV CONNECTION STRING";
static private string productionConnectionString = "SET YOUR PRODUCTION CONNECTION STRING";
And you need to change line 123 to set environment to dev or prod.
public OracleConnection GetConnection(string env = "dev", bool cacheOn = false)

Resources