I'm using janusgraph with AWSKeyspace (cassandra) and elasticsearch as the storage and indexing backends respectively.
I have a requirement to delete all the edges older than 30days. Setting a TTL on edge property doesn't work as expected(error below).
ERROR org.apache.tinkerpop.gremlin.jsr223.DefaultGremlinScriptEngineManager - Could not create GremlinScriptEngine for gremlin-groovy
java.lang.IllegalStateException: javax.script.ScriptException: javax.script.ScriptException:
java.lang.UnsupportedOperationException: The storage engine does not support TTL
My script is below:
edgeId = mgmt.getPropertyKey("edge_id")
sourceId = mgmt.getPropertyKey("source_id")
severity = mgmt.getPropertyKey("severity")
source = mgmt.getPropertyKey("source")
createdAt = mgmt.getPropertyKey("created_at")
alertStatus = mgmt.getPropertyKey("alert_status")
investigationStatus = mgmt.getPropertyKey("investigation_status")
investigationPriority = mgmt.getPropertyKey("investigation_priority")
attributes = mgmt.getPropertyKey("attributes")
edgeType = mgmt.getPropertyKey("edge_type")
if (!mgmt.containsGraphIndex(edgeIndexName)) {
mgmt.buildIndex(edgeIndexName, Edge.class).
addKey(edgeType, Mapping.TEXT.asParameter()).
addKey(tenant, Mapping.TEXT.asParameter()).
addKey(severity, Mapping.DEFAULT.asParameter()).
addKey(source, Mapping.TEXT.asParameter()).
addKey(createdAt, Mapping.DEFAULT.asParameter()).
addKey(investigationPriority, Mapping.DEFAULT.asParameter()).
addKey(attributes, Mapping.TEXT.asParameter()).
buildMixedIndex(indexBackendName)
}
makeEdgeCompositeIndex(mgmt, edgeCompositeIndexName, edgeId)
makeEdgeCompositeIndex(mgmt, edgeCompositeIndexName, sourceId)
// ###########
// Add TTL
// ###########
mgmt.setTTL(sourceId, Duration.ofDays(30))
I did try replacing AWSKeyspace with cassandra and run the same script in a docker compose file and it works well(setting of TTL). Just like cassandra AWSKeyspace supports adding TTLs.
When using Amazon Keyspaces, TTL needs to be enabled using a statement such as
ALTER TABLE my_table WITH CUSTOM_PROPERTIES={'ttl':{'status': 'enabled'}};
See https://github.com/JanusGraph/janusgraph/blob/ba90c0eea2ed9df65fd72b3c56e5a1215a51[…]janusgraph/diskstorage/cql/builder/CQLStoreFeaturesBuilder.java
This was set false likely because keyspaces TTL support went GA after the PR for keyspaces support was done:
PR: June 2021 https://github.com/JanusGraph/janusgraph/pull/2644
TTL GA: Oct 2021 https://aws.amazon.com/blogs/database/announcing-amazon-keyspaces-time-to-live-ttl-general-availability/
This could be set to true now, if tested to work.
This issue has been fixed as of v1.0.0 https://github.com/JanusGraph/janusgraph/releases/tag/v1.0.0-rc1
Related
i am trying to upsert 10 documents using spring boot. It is failing to upsert "few documents" with TTL.There is no error or exception. If i do not provide ttl then it is working as expected.
In addition to that, if i increase the ttl to a different value then also all the documents are getting created.
On the other hand, if i reduce the ttl then failing to insert few more docuemnts.
I tried to insert the failed document(single document out of 10) from another poc with the same ttl the document is getting created.
public Flux<JsonDocument> upsertAll(final List<JsonDocument> jsonDocuments) {
return Flux
.from(keys())
.flatMap(key -> Flux
.fromIterable(jsonDocuments)
.parallel()
.runOn(Schedulers.parallel())
.flatMap(jsonDocument -> {
final String arg = String.format("upsertAll-%s", jsonDocument);
return Mono
.just(asyncBucket
.upsert(jsonDocument, 1000, TimeUnit.MILLISECONDS)
.doOnError(error -> log.error(jsonDocument.content(), error, "failed to upsert")))
.map(obs -> Tuples.of(obs, jsonDocument.content()))
.map(tuple2 -> log.observableHandler(tuple2))
.map(observable1 -> Tuples.of(observable1, jsonDocument.content()))
.flatMap(tuple2 -> log.monoHandler(tuple2))
;
})
.sequential())
;
}
List<JsonDocument> jsonDocuments = new LinkedList<>();
dbService.upsertAll(jsonDocuments)
.subscribe();
some one please suggest how to resolve this issue.
Due to an oddity in the Couchbase server API, TTL values less than 30 days are treated differently than values greater than 30 days.
In order to get consistent behavior with Couchbase Java SDK 2.x, you'll need to adjust the TTL value before passing it to the SDK:
// adjust TTL for Couchbase Java SDK 2.x
public static int adjustTtl(int ttlSeconds) {
return ttlSeconds < TimeUnit.DAYS.toSeconds(30)
? ttlSeconds
: (int) (ttlSeconds + (System.currentTimeMillis() / 1000));
}
In Couchbase Java SDK 3.0.6 this is no longer required; just pass a Duration and the SDK will adjust the value behind the scenes if necessary.
I tried enabling TTL for records in Ignite using 2 approaches, but didn't seems to be working. Need help to understand if I am missing something.
IgniteCache cache = ignite.getOrCreateCache(IgniteCfg.CACHE_NAME);
cache.query(new SqlFieldsQuery(
"CREATE TABLE IF NOT EXISTS City (id LONG primary key, name varchar, region varchar)"))
.getAll();
cache.withExpiryPolicy(new CreatedExpiryPolicy(new Duration(TimeUnit.SECONDS, 10)))
.query(new SqlFieldsQuery(
"INSERT INTO City (id, name, region) VALUES (?, ?, ?)").setArgs(1, "Forest Hill1", "GLB"))
.getAll();
So you see above I created table in Cache and inserted record mentioning expiry TTL for 10 seconds, but seems that it never expires.
I tried another approach of rather than setting TTL while inserting the record, I mentioned in CacheConfiguration while I initialize Ignite, below is the code sample
Ignition.setClientMode(true);
IgniteConfiguration cfg = new IgniteConfiguration();
// Disabling peer-class loading feature.
cfg.setPeerClassLoadingEnabled(false);
CacheConfiguration ccfg = createCacheConfiguration();
cfg.setCacheConfiguration(ccfg);
ccfg.setEagerTtl(true);
ccfg.setExpiryPolicyFactory(CreatedExpiryPolicy.factoryOf(new Duration(TimeUnit.SECONDS, 5)));
TcpCommunicationSpi commSpi = new TcpCommunicationSpi();
cfg.setCommunicationSpi(commSpi);
TcpDiscoveryVmIpFinder tcpDiscoveryFinder = new TcpDiscoveryVmIpFinder();
String[] addresses = { "127.0.0.1" };
tcpDiscoveryFinder.setAddresses(Arrays.asList(addresses));
TcpDiscoverySpi discoSpi = new TcpDiscoverySpi();
discoSpi.setIpFinder(tcpDiscoveryFinder);
cfg.setDiscoverySpi(discoSpi);
return Ignition.start(cfg);
Executing Ignite locally (not as in memory) as my final goal is to be able to connect to same Ignite from multiple instances of app or even multiple apps.
Ignite SQL currently doesn't interact with expiry policies and doesn't update TTL. There is a Feature Request for that: https://issues.apache.org/jira/browse/IGNITE-7687.
Using Terraform 0.9.6 I get this error when I try to create spot fleet using the code below, but when I use
v0.11.7, it works. Terraform doc is here. My question is how can I determine if this is due to lack of v0.9.6 features or if I'm using the syntax incorrectly. How can I find this out?
I need to make the sport fleet resource tags work with v0.9.6.
Error:
* aws_spot_fleet_request.cheap_compute: launch_specification.0: invalid or unknown key: tags
Code:
# Request a Spot fleet
resource "aws_spot_fleet_request" "cheap_compute" {
iam_fleet_role = "arn:aws:iam::xxxxxxxxxxxxx:role/aws-service-role/spotfleet.amazonaws.com/AWSServiceRoleForEC2SpotFleet"
spot_price = "0.03"
allocation_strategy = "diversified"
target_capacity = 2
valid_until = "2018-07-21T20:44:20Z"
launch_specification {
instance_type = "t2.micro"
ami = "ami-1853ac65"
spot_price = "0.777"
availability_zone = "us-east-1a"
key_name = "${var.key_name}"
tags {
Name = "spot-fleet-example"}
}
}
Support for tags in aws_spot_fleet_request was added in the terraform-provider-aws in 1.2.0. This was almost 5 months after the release of Terraform 0.9.6. At that time of 0.9.6 providers were shipped with Terraform. They later moved to shipping independently as part of 0.10.0 a few months later. You'll need a newer version to get tag support.
I'm trying to set-up jdbc read side processor in lagom service:
class ProjectEventsProcessor(readSide: JdbcReadSide)(implicit ec: ExecutionContext) extends ReadSideProcessor[ProjectEvent] {
def buildHandler = {
readSide.builder[ProjectEvent]("projectEventOffset")
.setEventHandler[ProjectCreated]((conn: Connection, e: EventStreamElement[ProjectCreated]) => insertProject(e.event))
.build
}
private def insertProject(e: ProjectCreated) = {
Logger.info(s"Got event $e")
}
override def aggregateTags: Set[AggregateEventTag[ProjectEvent]] = ProjectEvent.Tag.allTags
}
Services connects to database fine on startup
15:40:32.575 [info] play.api.db.DefaultDBApi [] - Database [default] connected at jdbc:postgresql://localhost/postgres?user=postgres
But right after this I'm getting exception.
com.typesafe.config.ConfigException$Missing: No configuration setting
found for key 'slick.profile'
First of all, why slick is involved here at all?
I'm using JdbcReadSide but not SlickReadSide.
Ok, let's say JdbcReadSide internally uses slick somehow.
I've added slick.profile in application.config of my service.
db.default.driver="org.postgresql.Driver"
db.default.url="jdbc:postgresql://localhost/postgres?user=postgres"
// Tried this way
slick.profile="slick.jdbc.PostgresProfile$"
// Also this fay (copied from play documentation).
slick.dbs.default.profile="slick.jdbc.PostgresProfile$"
slick.dbs.default.db.dataSourceClass = "slick.jdbc.DatabaseUrlDataSource"
slick.dbs.default.db.properties.driver = "org.postgresql.Driver"
But still getting this exception.
What is going on? How to solve this issue?
According to the docs, Lagom uses akka-persistence-jdbc, which under the hood:
uses Slick to map tables and manage asynchronous execution of JDBC calls.
A full configuration, using also the default connection pool (HikariCP), to set in the application.conf file, may be the following (mostly copied from the docs):
# Defaults to use for each Akka persistence plugin
jdbc-defaults.slick {
# The Slick profile to use
# set to one of: slick.jdbc.PostgresProfile$, slick.jdbc.MySQLProfile$, slick.jdbc.OracleProfile$ or slick.jdbc.H2Profile$
profile = "slick.jdbc.PostgresProfile$"
# The JNDI name for the Slick pre-configured DB
# By default, this value will be used by all akka-persistence-jdbc plugin components (journal, read-journal and snapshot).
# you may configure each plugin component to use different DB settings
jndiDbName=DefaultDB
}
db.default {
driver = "org.postgresql.Driver"
url = "jdbc:postgresql://localhost/postgres?user=postgres"
# The JNDI name for this DataSource
# Play, and therefore Lagom, will automatically register this DataSource as a JNDI resource using this name.
# This DataSource will be used to build a pre-configured Slick DB
jndiName=DefaultDS
# Lagom will configure a Slick Database, using the async-executor settings below
# and register it as a JNDI resource using this name.
# By default, all akka-persistence-jdbc plugin components will use this JDNI name
# to lookup for this pre-configured Slick DB
jndiDbName=DefaultDB
async-executor {
# number of objects that can be queued by the async executor
queueSize = 10000
# 5 * number of cores
numThreads = 20
# same as number of threads
minConnections = 20
# same as number of threads
maxConnections = 20
# if true, a Mbean for AsyncExecutor will be registered
registerMbeans = false
}
# Hikari is the default connection pool and it's fine-tuned to use the same
# values for minimum and maximum connections as defined for the async-executor above
hikaricp {
minimumIdle = ${db.default.async-executor.minConnections}
maximumPoolSize = ${db.default.async-executor.maxConnections}
}
}
lagom.persistence.jdbc {
# Configuration for creating tables
create-tables {
# Whether tables should be created automatically as needed
auto = true
# How long to wait for tables to be created, before failing
timeout = 20s
# The cluster role to create tables from
run-on-role = ""
# Exponential backoff for failures configuration for creating tables
failure-exponential-backoff {
# minimum (initial) duration until processor is started again
# after failure
min = 3s
# the exponential back-off is capped to this duration
max = 30s
# additional random delay is based on this factor
random-factor = 0.2
}
}
}
http://hadoop.apache.org/docs/r2.1.0-beta/hadoop-yarn/hadoop-yarn-site/WritingYarnApplications.html
I am try to make the example work well from the above link.but I can't compile the code below
Resource capability = Records.newRecord(Resource.class);
capability.setMemory(512);
amContainer.setResource(capability);
// Set the container launch content into the
// ApplicationSubmissionContext
appContext.setAMContainerSpec(amContainer);
amContainer is ContainerLaunchContext and my hadoop version is 2.1.0-beta.
I did some investigation. I found there's no method "setResource" in ContainerLaunchContext
I have 3 question about this
1) the method has been removed or something?
2) if the method has been removed, how can I do now?
3) is there any doc about yarn, because I found the doc in website is very easy, I hope I can get a manual or something. for example,
capability.setMemory(512);
I don't know it's 512k or 512M according comments in code.
This is actually proper solution to the question. Previous answer might cause incorrect execution !!!
#Dyin I couldn't fit it in the comment ;) Validated for 2.2.0 and 2.3.0
Driver setting up resources for AppMaster:
ApplicationSubmissionContext appContext = app.getApplicationSubmissionContext();
ApplicationId appId = appContext.getApplicationId();
appContext.setApplicationName(this.appName);
// Set up the container launch context for the application master
ContainerLaunchContext amContainer = Records.newRecord(ContainerLaunchContext.class);
Resource capability = Records.newRecord(Resource.class);
capability.setMemory(amMemory);
appContext.setResource(capability);
appContext.setAMContainerSpec(amContainer);
Priority pri = Records.newRecord(Priority.class);
pri.setPriority(amPriority);
appContext.setPriority(pri);
appContext.setQueue(amQueue);
// Submit the application to the applications manager
yarnClient.submitApplication(appContext); // this.yarnClient = YarnClient.createYarnClient();
In ApplicationMaster this is how you should specify resources for containers (workers).
private AMRMClient.ContainerRequest setupContainerAskForRM() {
// setup requirements for hosts
// using * as any host will do for the distributed shell app
// set the priority for the request
Priority pri = Records.newRecord(Priority.class);
pri.setPriority(requestPriority);
// Set up resource type requirements
// For now, only memory is supported so we set memory requirements
Resource capability = Records.newRecord(Resource.class);
capability.setMemory(containerMemory);
AMRMClient.ContainerRequest request = new AMRMClient.ContainerRequest(capability, null, null,
pri);
return request;
}
Some run() or main() method in your AppMaster
AMRMClientAsync.CallbackHandler allocListener = new RMCallbackHandler();
resourceManager = AMRMClientAsync.createAMRMClientAsync(1000, allocListener);
resourceManager.init(conf);
resourceManager.start();
for (int i = 0; i < numTotalContainers; ++i) {
AMRMClient.ContainerRequest containerAsk = setupContainerAskForRM();
resourceManager.addContainerRequest(containerAsk); //
}
Launching containers
You can use the original answer solution (java cmd), but it's just a cherry on top. It should work anyway.
You can set memory available to ApplicationMaster via commend. As such:
// Set the necessary command to execute the application master
Vector<CharSequence> vargs = new Vector<CharSequence>(30);
...
vargs.add("-Xmx" + amMemory + "m"); // notice "m" indicating megabytes, you can use also -Xms combined with -Xmx
... // transform vargs to String commands
amContainer.setCommands(commands);
This should solve your problem. As for the 3 questions. Yarn is rapidly evolving software. My advice forget documentation, get source code and read it. This will answer a lot of your questions.