if I do not set size, I can get 10 hits:
SearchResponse sr = client.prepareSearch("xxx").setTypes("xxx")
.setQuery(rangeQueryBuilder)
.setQuery(queryBuilder)
but when I set size more than 12:
SearchResponse sr = client.prepareSearch("xxx").setTypes("xxx")
.setSize(13)
.setQuery(rangeQueryBuilder)
.setQuery(queryBuilder)
I get this problem:
NoNodeAvailableException[None of the configured nodes were available: [{gw_172.28.236.85:40001}{oHcfPhqFQDSW4opwUuzCpA}{P1GbtDqrRda4nlbRRBmW1Q}{172.28.236.85}{172.28.236.85:40101}{xpack.installed=true},
my java connect code:
public static TransportClient client() throws UnknownHostException {
if (client != null) {
return client;
}
synchronized (esConnection_old.class) {
if (client == null) {
Settings settings = Settings.builder().put("cluster.name", ClusterName)
.put("client.transport.sniff", false)
.put(SecurityKey, basicAuthHeaderValue(SecurityUser, SecurityPassword))
.build();
client = new PreBuiltTransportClient(settings);
String[] oneInstance = GatewayIpPorts.split(",");
for (String item : oneInstance) {
String[] ipPort = item.split(":");
client.addTransportAddresses(new TransportAddress(InetAddress.getByName(ipPort[0]), Integer.parseInt(ipPort[1])));
}
return client;
}
return client;
}
}
Normally this exception comes when Elasticsearch needs to perform a certain action on a node (allocation of the shard, indexing, and searching data) and it does not find the nodes which can serve these requests.
You can have a look at NoNodeAvailableException Code and trace back to it, I looked this is the latest code and couldn't find None of the configured nodes were available: for search action which you are trying to perform.
Please provide your elasticsearch version and also confirm this exception comes just because of size param value more than 10?
Related
We have a spring webflux application which is querying Elasticsearch using ReactiveElasticsearchTemplate like this
final inline fun <reified R> getSearchMonoList(
indexName: String,
query: NativeSearchQuery,
metricName: String
): Mono<List<SearchHit<R>>> {
val startTime = getCurrentTime()
recordThroughput(metricName, THROUGHPUT)
return when (indexName == EMPTY_STRING) {
true -> getEsClientTemplate().search(query, R::class.java).collectList()
else -> getEsClientTemplate().search(query, R::class.java, IndexCoordinates.of(indexName)).collectList()
}.doOnError {
recordThroughput(metricName, FAILED_THROUGHPUT)
}.doFinally {
getEsClientTemplate()
recordTime(metricName, startTime)
}
}
Client Configuration is
public ReactiveElasticsearchClient reactiveElasticsearchClient() {
ClientConfiguration clientConfiguration = ClientConfiguration.builder()
.connectedTo(urls.toArray(String[]::new))
.withWebClientConfigurer(webClient -> {
final ExchangeStrategies exchangeStrategies = ExchangeStrategies.builder()
.codecs(configurer -> configurer.defaultCodecs()
.maxInMemorySize(-1))
.build();
return webClient.mutate().exchangeStrategies(exchangeStrategies).build();
})
.build();
return ReactiveRestClients.create(clientConfiguration);
}
Our Problem
whenever Template queries ES Request body: {"from":0,"size":10000}
is requesting ES for 10000 records which was way too much for a term query so we have fixed in query builder by using Pageable,
However we do not want to use scroll as this is exhausting Max scroll connections as this API is very heavily used.
NativeSearchQueryBuilder().withQuery(boolQueryBuilders).withPageable(PageRequest.of(0,1))
i am aware of this spring documentation where this was the suggested solution
SearchScrollHits<SampleEntity> scroll = template.searchScrollStart(1000, searchQuery, SampleEntity.class, index);
String scrollId = scroll.getScrollId();
List<SampleEntity> sampleEntities = new ArrayList<>();
while (scroll.hasSearchHits()) {
sampleEntities.addAll(scroll.getSearchHits());
scrollId = scroll.getScrollId();
scroll = template.searchScrollContinue(scrollId, 1000, SampleEntity.class);
}
template.searchScrollClear(scrollId);
However in current production we can not update the libraries because of DMZ restrictions, We are using spring data elasticsearch 4.1.1
How can i disable scroll or clear the scroll after use any help would be appreciable ?
I am currently new on Apache Nifi and still exploring it.
I made a custom processor where I will fetch data from server with pagination.
I pass the input file which will contains the attribute "url".
Finally transfer the response in output flow file, as I fetch data with pagination, so I made a new output flow file for each page and transferred it to Successful relationship.
Below is the code part:
#Override
public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
FlowFile incomingFlowFile = session.get();
String api = null;
if (incomingFlowFile == null) {
logger.info("empty input flow file");
session.commit();
return;
} else {
api=incomingFlowFile.getAttribute("url");
}
session.remove(incomingFlowFile);
if(api == null) {
logger.warn("API url is null");
session.commit();
return;
}
int page = Integer.parseInt(context.getProperty(PAGE).getValue());
while(page < 3) {
try {
String url = api + "&curpg=" + page;
logger.info("input url is: {}", url);
HttpResponse response = httpGetApiCall(url, 10000);
if(response == null || response.getEntity() == null) {
logger.warn("response null");
session.commit();
return;
}
String resp = EntityUtils.toString(response.getEntity());
InputStream is = new ByteArrayInputStream(StandardCharsets.UTF_16.encode(resp).array());
FlowFile outFlowFile = session.create();
outFlowFile = session.importFrom(is, outFlowFile);
session.transfer(outFlowFile, SUCCESSFUL);
} catch (IOException e) {
logger.warn("IOException :{}", e.getMessage());
return;
}
++page;
}
session.commit();
}
I am facing issue that for a single Input flow file, this processor get triggered twice and so it generates 4 flow files for a single input flow file.
I am not able to figure out this where I have done wrong.
Please help in this issue.
Thanks in advance.
======================================================================
processor group 1(Nifi_Parvin)
processor group 2 (News_Point_custom)
I have a cluster of rabbitmqs. And configure the spring.rabbitmq.address=xxx,yyy,cccc
Then I start two consumer clients. The question is that the clients only connect one node, there,s not connection to the other nodes. I trace the codes, and found that :
public Connection newConnection(ExecutorService executor, AddressResolver addressResolver, String clientProvidedName) throws IOException, TimeoutException {
if (this.metricsCollector == null) {
this.metricsCollector = new NoOpMetricsCollector();
}
FrameHandlerFactory fhFactory = this.createFrameHandlerFactory();
ConnectionParams params = this.params(executor);
if (clientProvidedName != null) {
Map<String, Object> properties = new
HashMap(params.getClientProperties());
properties.put("connection_name", clientProvidedName);
params.setClientProperties(properties);
}
if (this.isAutomaticRecoveryEnabled()) {
AutorecoveringConnection conn = new AutorecoveringConnection(params,
fhFactory, addressResolver, this.metricsCollector);
conn.init();
return conn;
} else {
List<Address> addrs = addressResolver.getAddresses();
Exception lastException = null;
Iterator var8 = addrs.iterator();
while(var8.hasNext()) {
Address addr = (Address)var8.next();
try {
**FrameHandler handler = fhFactory.create(addr);
AMQConnection conn = this.createConnection(params, handler, this.metricsCollector);
conn.start();
this.metricsCollector.newConnection(conn);
return conn;**
} catch (IOException var12) {
lastException = var12;
} catch (TimeoutException var13) {
lastException = var13;
}
}
if (lastException != null) {
if (lastException instanceof IOException) {
throw (IOException)lastException;
}
if (lastException instanceof TimeoutException) {
throw (TimeoutException)lastException;
}
}
throw new IOException("failed to connect");
}
}
We can see that it creates one connection then returned. But if I wanna the other consumer client can connect to the left nodes instead the same node, although both of the consumer clients have the same configuration:
spring:
rabbitmq:
username: aaaa
password: aaaa
virtual-host: /
addresses: xxxx:5672,yyy:5672,zzzzz:5672
listener:
simple:
concurrency: 4
max-concurrency: 4
prefetch: 4
What should I do ？ can someone give some suggestions?
The RabbitMQ team monitors this mailing list and only sometimes answers questions on StackOverflow.
Try rotating the list of addresses in each client's configuration:
First - xxxx:5672,yyy:5672,zzzzz:5672
Second - yyy:5672,zzzzz:5672,xxxx:5672
Third - zzzzz:5672,xxxx:5672,yyy:5672
I am learning about elastic search and I am following the next tutorial. In that tutorial it is used tweets of Twiter as example data. Method tweetJsonList return a example data. I am trying to save this in the index "tweets_juan" and type "tweet". The application run without problems, but when I search all documents using (http://localhost:9200/tweets_juan/tweet/_search?q=:) I do not found anything. Could you help me please to verify whats happens here?
public class App
{
#SuppressWarnings("unchecked")
public static void main( String[] args ) throws TwitterException, UnknownHostException
{
System.out.println( "Hello World!" );
List<String> tweetJsonList = searchForTweets();
Client client = TransportClient.builder().build()
.addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName("localhost"), 9300));
String index = "tweets_juan";
client.admin().indices()
.create(new CreateIndexRequest(index))
.actionGet();
save(client, tweetJsonList, index);
searchExample(client);
}
public static void save(Client client, List<String> tweetJsonList, String index) {
BulkRequestBuilder bulkRequestBuilder = client.prepareBulk().setRefresh(true);
for (String data : tweetJsonList) {
String indexName = index;
String typeName = "tweet";
String json = new Gson().toJson(data);
System.out.println("Juan Debug:" + data);
bulkRequestBuilder.add(client.prepareIndex(indexName, typeName).setSource(json));
}
bulkRequestBuilder.execute().actionGet();
}
public static void searchExample(Client client) {
BoolQueryBuilder queryBuilder = QueryBuilders
.boolQuery()
.must(termsQuery("text", "Baloncesto"));
SearchResponse searchResponse = client.prepareSearch("tweets_juan")
.setQuery(queryBuilder)
.setSize(25)
.execute()
.actionGet();
}
public static List searchForTweets() throws TwitterException {
Twitter twitter = new TwitterFactory().getInstance();
Query query = new Query("mundial baloncesto");
List tweetList = new ArrayList<>();
for (int i = 0; i < 10; i++) {
QueryResult queryResult = twitter.search(query);
tweetList.addAll(queryResult.getTweets());
if (!queryResult.hasNext()) {
break;
}
query = queryResult.nextQuery();
}
Gson gson = new Gson();
return (List) tweetList.stream().map(gson::toJson).collect(Collectors.toList());
}
}
You need to put more information before anyone can answer your question.
Since you are not using any explicit mapping your fields must be getting analyzed by default. So your text field will get tokenized into multiple terms.
Use "match all" query to see what data has been indexed.
Term query is used for exact match ( including exact case) and you are trying to run term query on an analyzed field "text" which will not work.
Try using match or match phrase query on the text field and see if you get back any result.
I am new to infinispan and I am trying to start from very basic.After going through the documentation about embedded cache that lives in the same JVM process as the running program, I am trying to see how it works.
Here is my code.
public class CacheClient {
public static void main(String[] args) {
CacheClient cc = new CacheClient();
cc.start();
}
public void start() {
boolean run = true;
EmbeddedCacheManager manager = **createCacheManagerProgrammatically**();
manager.start();
Cache<Object, Object> cache = manager.getCache("dist");
Scanner sc = new Scanner(System.in);
while (run) {
System.out.println("Enter the command:");
String command = sc.next();
switch (command) {
case "add":
System.out.println("Enter the key:");
int i = sc.nextInt();
cache.put(Integer.valueOf(i), Integer.valueOf(i));
break;
case "list":
System.out.println("The keys:");
Set<Object> keySet = cache.keySet();
Iterator<Object> iter = keySet.iterator();
while (iter.hasNext()) {
System.out.println((Integer) iter.next());
}
break;
default:
run = false;
break;
}
}
sc.close();
manager.stop();
}
private EmbeddedCacheManager **createCacheManagerProgrammatically**() {
System.out
.println("Starting a cache manager with a programmatic configuration");
EmbeddedCacheManager cacheManager = new DefaultCacheManager(
GlobalConfigurationBuilder
.defaultClusteredBuilder()
.transport()
.defaultTransport()
.clusterName("dist_cluster")
.addProperty("configurationFile",
"jgroups.xml")
.build(), new ConfigurationBuilder().clustering()
.cacheMode(CacheMode.REPL_SYNC).build());
cacheManager.defineConfiguration("dist", new ConfigurationBuilder()
.clustering().cacheMode(CacheMode.REPL_SYNC).hash()
.numOwners(2).build());
return cacheManager;
}
}
From the above code my expectation is that suppose I run this program first and start adding integers to cache using add command, then when I run the same program again (another process) then the moment I use the list command, I should see the contents immediately. But I am not able to see any content when I use list in the 2nd process.
1) Is this how Embedded cache supposed to work? Is my expectation correct? If so, then what am I missing?
Please correct my mistakes and point me to a tutorial that explains clearly how it works. I tried to go through the tutorial from Infinispan documentation. But I think it's not very clear there.
Any help is highly appreciated.
I finally figured out. Please set
System.setProperty("java.net.preferIPv4Stack", "true");
and the above code will work.