Micronaut: How to get metrics in the Prometheus format? - metrics

How should I configure the Micronaut to get the /metrics in the Prometheus format ?
Used: micronaut 1.0.0.M3
Now:
micronaut:
...
metrics:
enabled: true
export:
prometheus:
enabled: true
and result: metrics name list
{"names":["jvm.memory.max","executor.pool.size"...]}
I need to get: metrics in the prometheus format(formats)

Micronaut Micrometer has an PrometheusEndpoint from version 1.1 that will
return in Prometheus format from /prometheus and
can be enabled in application.yml by:
endpoints:
prometheus:
sensitive: false
In combination with
micronaut:
metrics:
enabled: true
export:
prometheus:
enabled: true
step: PT1M
descriptions: true
(The documentation is missing the endpoint config but will be changed in the new release)

To piggyback on the other answers, here is an Micronaut endpoint that provides the Prometheus metrics in the format we needed:
package your.package.name
import io.micrometer.prometheus.PrometheusMeterRegistry
import io.micronaut.management.endpoint.annotation.Endpoint
import io.micronaut.management.endpoint.annotation.Read
#Endpoint(id = "prometheus", value = "/prometheus", defaultEnabled = true,
defaultSensitive = false)
class PrometheusController(val prometheusMeterRegistry: PrometheusMeterRegistry) {
#Read
fun scrape(): String {
return prometheusMeterRegistry.scrape()
}
}

At the moment, we solved the problem as follows.
Added a new endpoint. Or create a controller with a mapping on /metrics.
The new endpoint added a return of scrape().
Correlated endpoint with /prometheus (new endpoint can not be mapped on /metrics).
Disconnected endpoint metrics which by default.
Config:
micronaut:
...
metrics:
enabled: true
export:
prometheus:
enabled: true
...
endpoints:
...
metrics:
enabled: false
prometheus:
enabled: true

Haven't tested this out but based on the following:
https://github.com/micronaut-projects/micronaut-core/blob/master/configurations/micrometer-registry-prometheus/src/main/java/io/micronaut/configuration/metrics/micrometer/prometheus/PrometheusMeterRegistryFactory.java
Your yaml should look like
metrics:
prometheus:
enabled: true
don't believe the export comes into play.

Related

How to set the servers property of openapi with Apache Camel?

I am trying to setup a openapi specification and publish the API with Apache Camel and Spring. I tried using restConfiguration, adding the property in the application.yaml, and using the #OpenApiProperty on the app. Everytime the generated yaml reads:
- servers
- url: ""
platform:
# not used
urlrewrite:
enabled: false
token: ${LOCAL_TOKEN:}
apim:
token_ep: ${x/token}
client:
username: ${apim_client_username:}
password: ${apim_client_password:}
consumerKey: ${apim_client_id:}
consumerSecret: ${apim_client_secret:}
endpointsOrchestrated:
server: myServer
emr-endpoint: xxxxxxxx
triggerName: ${PLATFORM_MODULE_ID}
env: dev
domain: ${PLATFORM_MODULE_DOMAIN}
openapi:
title: My Sample api
version: 1.0.0
camel:
dataformat:
jackson:
auto-discover-object-mapper: true
springboot:
tracing: false
rest:
host: myhost.com
port: 8080
spring:
application:
name: aaa
profiles:
active: ${ENV:local}
server:
servlet:
context-path: /
port: ${TOMCAT_PORT:8080}
host: localhost
# MAX HTTP THREADS
tomcat:
threads:
max: ${MAIN_HTTP_THREADS:200}

Witnessing strange surge in RDS aurora mysql RDS

I have a springboot based microservice application.
id 'org.springframework.boot' version '2.5.4'
id 'io.spring.dependency-management' version '1.0.11.RELEASE'
id "com.palantir.docker" version "0.26.0"
id "com.palantir.docker-run" version "0.26.0"
id 'pl.allegro.tech.build.axion-release' version '1.13.2'
Database is mysql5.7 aurora rds
'mysql', name: 'mysql-connector-java', version: '8.0.28'
hikaricp - 4.0.3
I am witnessing one strange surge in cpu utilisation stats on RDS performance insight dashboard is that even when there are no requests on my app server, mysql still shows high cpu utilization.
Here are the screen shots:
we can observe from the logs that there are no requests on server but when the connection passes its max life, surge in cpu utilization on RDS aurora mysql:
"connection has passed maxLifetime" -> and the top sql shows set autocommit = 0 is the highest number of queries being fired.
Here are my configurations:
application.yml
spring:
application:
name: catalogue
profiles:
# The commented value for `active` can be replaced with valid Spring profiles to load.
# Otherwise, it will be filled in by gradle when building the JAR file
# Either way, it can be overridden by `--spring.profiles.active` value passed in the commandline or `-Dspring.profiles.active` set in `JAVA_OPTS`
active: dev
group:
dev:
- dev
- api-docs
# Uncomment to activate TLS for the dev profile
#- tls
prod:
- prod
- api-docs
# Uncomment to activate TLS for the dev profile
#- tls
stage:
- stage
jmx:
enabled: false
data:
web:
pageable:
default-page-size: 20
max-page-size: 20
jpa:
repositories:
bootstrap-mode: deferred
jpa:
open-in-view: false
properties:
hibernate.jdbc.time_zone: UTC
hibernate.id.new_generator_mappings: true
hibernate.connection.provider_disables_autocommit: true #https://vladmihalcea.com/why-you-should-always-use-hibernate-connection-provider_disables_autocommit-for-resource-local-jpa-transactions/
hibernate.cache.use_second_level_cache: true
hibernate.cache.region.factory_class: org.hibernate.cache.ehcache.EhCacheRegionFactory
hibernate.cache.use_query_cache: false
hibernate.javax.cache.missing_cache_strategy: create
# modify batch size as necessary
hibernate.jdbc.batch_size: 20
hibernate.order_inserts: true
hibernate.order_updates: true
hibernate.batch_versioned_data: true
hibernate.query.fail_on_pagination_over_collection_fetch: true
hibernate.query.in_clause_parameter_padding: true
hibernate.dialect: org.hibernate.dialect.MySQL5InnoDBDialect
javax.persistent.sharedCache.mode: ENABLE_SELECTIVE
hibernate:
ddl-auto: none
naming:
physical-strategy: org.springframework.boot.orm.jpa.hibernate.SpringPhysicalNamingStrategy
implicit-strategy: org.springframework.boot.orm.jpa.hibernate.SpringImplicitNamingStrategy
messages:
basename: i18n/messages
main:
allow-bean-definition-overriding: true
task:
execution:
thread-name-prefix: catalogue-task-
pool:
core-size: 2
max-size: 50
queue-capacity: 10000
scheduling:
thread-name-prefix: catalogue-scheduling-
pool:
size: 2
thymeleaf:
mode: HTML
output:
ansi:
console-available: true
server:
servlet:
session:
cookie:
http-only: true
tomcat:
mbeanregistry:
enabled: true
threads:
max: 100
compression:
enabled: true
mime-types: "text/html,text/xml,text/plain,text/css,text/javascript,application/javascript,application/json"
min-response-size: 1024
port: 8080
# Properties to be exposed on the /info management endpoint
info:
# Comma separated list of profiles that will trigger the ribbon to show
display-ribbon-on-profiles: 'dev'
management:
endpoints:
web:
exposure:
include: "health,info,metrics,prometheus"
endpoint:
health:
probes:
enabled: true
show-details: always
show-components: always
application-prod.yml
logging:
level:
ROOT: ERROR
org.hibernate.SQL: ERROR
com.pitstop.catalogue: ERROR
com.zaxxer.hikari: ERROR
config: classpath:logback-prod.xml
spring:
devtools:
restart:
enabled: true
additional-exclude: static/**
jackson:
serialization:
indent-output: true
datasource:
auto-commit: false
type: com.zaxxer.hikari.HikariDataSource
url: ${SPRING_DATASOURCE_URL}
username: ${SPRING_DATASOURCE_USERNAME}
password: ${SPRING_DATASOURCE_PASSWORD}
hikari:
poolName: CatalogJPAHikariCP
minimumIdle: 10
maximumPoolSize: 120
connectionTimeout: 30000
idleTimeout: 300000
maxLifetime: 600000
auto-commit: false
data-source-properties:
cachePrepStmts: true
prepStmtCacheSize: 250
prepStmtCacheSqlLimit: 2048
useServerPrepStmts: true
useLocalSessionState: true
rewriteBatchedStatements: true
cacheResultSetMetadata: true
cacheServerConfiguration: true
maintainTimeStats: true
servlet:
multipart:
location: /data/tmp
jpa:
hibernate:
ddl-auto: none
properties:
spring.jpa.show-sql: true
hibernate.generate_statistics: true
liquibase:
contexts: prod
messages:
cache-duration: PT1S # 1 second, see the ISO 8601 standard
thymeleaf:
cache: false
sleuth:
sampler:
probability: 1 # report 100% of traces

Unable to resolve values from spring boot application yaml files

Hi I am haveing a spring boot rest based application . I have added a couple of properties related to elastic search in my application-dev.yml and application-prod.yml file . These properties are injected into my PerformanceReportingRepository class . however they are set to null .
so following is my yml definition .
# ===================================================================
# Standard Spring Boot properties.
# Full reference is available at:
# http://docs.spring.io/spring-boot/docs/current/reference/html/common-application-properties.html
# ===================================================================
logging:
level:
ROOT: TRACE
com.openmind.primecast: DEBUG
io.github.jhipster: TRACE
spring:
profiles:
active: dev
include: swagger
devtools:
restart:
enabled: true
livereload:
enabled: false # we use gulp + BrowserSync for livereload
jackson:
serialization.indent_output: true
data:
cassandra:
contactPoints: localhost
protocolVersion: V4
compression: LZ4
keyspaceName: Openmind
mail:
host: localhost
port: 25
username:
password:
properties:
mail:
smtp:
auth: true
starttls.enable: true
messages:
cache-seconds: 1
thymeleaf:
cache: false
http:
multipart:
maxFileSize: 150MB
maxRequestSize: 150MB
# ===================================================================
# To enable SSL, generate a certificate using:
# keytool -genkey -alias primecast -storetype PKCS12 -keyalg RSA -keysize 2048 -keystore keystore.p12 -validity 3650
#
# You can also use Let's Encrypt:
# https://maximilian-boehm.com/hp2121/Create-a-Java-Keystore-JKS-from-Let-s-Encrypt-Certificates.htm
#
# Then, modify the server.ssl properties so your "server" configuration looks like:
#
# server:
# port: 8443
# ssl:
# key-store: keystore.p12
# key-store-password: <your-password>
# key-store-type: PKCS12
# key-alias: primecast
# ===================================================================
server:
port: 8080
# ===================================================================
# JHipster specific properties
#
# Full reference is available at: http://www.jhipster.tech/common-application-properties/
# ===================================================================
jhipster:
http:
version: V_1_1 # To use HTTP/2 you will need SSL support (see above the "server.ssl" configuration)
cache: # Cache configuration
hazelcast: # Hazelcast distributed cache
time-to-live-seconds: 3600
backup-count: 1
management-center: # Full reference is available at: http://docs.hazelcast.org/docs/management-center/3.9/manual/html/Deploying_and_Starting.html
enabled: false
update-interval: 3
url: http://localhost:8180/mancenter
# CORS is only enabled by default with the "dev" profile, so BrowserSync can access the API
cors:
allowed-origins: "*"
allowed-methods: "*"
allowed-headers: "*"
exposed-headers: "Authorization,Link,X-Total-Count"
allow-credentials: true
max-age: 1800
security:
authentication:
jwt:
secret: my-secret-token-to-change-in-production
# Token is valid 24 hours
token-validity-in-seconds: 86400
token-validity-in-seconds-for-remember-me: 2592000
mail: # specific JHipster mail property, for standard properties see MailProperties
from: primecast#localhost
base-url: http://127.0.0.1:8080
metrics: # DropWizard Metrics configuration, used by MetricsConfiguration
jmx.enabled: true
graphite: # Use the "graphite" Maven profile to have the Graphite dependencies
enabled: false
host: localhost
port: 2003
prefix: primecast
prometheus: # Use the "prometheus" Maven profile to have the Prometheus dependencies
enabled: false
endpoint: /prometheusMetrics
logs: # Reports Dropwizard metrics in the logs
enabled: false
report-frequency: 60 # in seconds
logging:
logstash: # Forward logs to logstash over a socket, used by LoggingConfiguration
enabled: false
host: localhost
port: 5000
queue-size: 512
google:
accessToken: AIzaSyBasWNyK7ELbLlopD3anXrJCgtXUPRZYN8
endpoint: https://www.googleapis.com/urlshortener/v1/url
urlshortener:
is.gd:
endpoint: https://is.gd
http:
proxyEnabled: false
proxyHost:
proxyPort:
password:
expiryInDays: 90
historyCount: 3
expiryEmailAlertInDays: 3
policy: '^(?!.*(.)\1\1)(?=.*?[A-Z])(?=.*?[a-z])(?=.*?[0-9])(?=.*?[#?!#$%^&*-]).{8,}.*$'
file:
size: 5242880
elasticsearch:
host: localhost
port: 9200
and my repository class is
#Repository
public class PerformanceReportingRepository {
#Value("${elasticsearch.host}")
private String host;
#Value("${elasticsearch.port}")
private Integer port;
#Value("${file.size}")
private long fileUploadSizeLimit;
private RestHighLevelClient elasticSearchClient;
public PerformanceReportingRepository() {
System.out.println(this.fileUploadSizeLimit);
this.elasticSearchClient = new RestHighLevelClient(RestClient.builder(new HttpHost(host, port)));
}
public void setElasticSearchClient(RestHighLevelClient elasticSearchClient) {
this.elasticSearchClient = elasticSearchClient;
}
}
However the bean creation fails because host and port are not resolved from the yaml file.
I am sharing the project in https://github.com/prasanthmp500/mep.git
please checkout the branch ENG-170-performance-reports
appreciate any help
thanks a lot

Spring Cloud Gateway+Consul configurations

We are using Spring Cloud Gateway before multiple microservices with consul as service discovery. There are several microservices developed in different languages.
Please find build.gradle for the application
buildscript {
ext {
springBootVersion = '2.1.2.RELEASE'
}
repositories {
mavenCentral()
}
dependencies {
classpath("org.springframework.boot:spring-boot-gradle-plugin:${springBootVersion}")
}
}
apply plugin: 'java'
apply plugin: 'org.springframework.boot'
apply plugin: 'io.spring.dependency-management'
group = 'com.demo'
version = '0.0.1-SNAPSHOT'
sourceCompatibility = '1.8'
repositories {
mavenCentral()
maven { url 'https://repo.spring.io/milestone' }
}
ext {
set('springCloudVersion', 'Greenwich.RELEASE')
}
dependencies {
implementation 'org.springframework.boot:spring-boot-starter-actuator'
implementation 'org.springframework.cloud:spring-cloud-starter-consul-config'
implementation 'org.springframework.cloud:spring-cloud-starter-consul-discovery'
implementation 'org.springframework.cloud:spring-cloud-starter-gateway'
implementation 'org.springframework.boot:spring-boot-starter-security'
// https://mvnrepository.com/artifact/io.netty/netty-tcnative-boringssl-static
compile group: 'io.netty', name: 'netty-tcnative-boringssl-static', version: '2.0.20.Final'
runtimeOnly 'org.springframework.boot:spring-boot-devtools'
compileOnly 'org.projectlombok:lombok'
testImplementation 'org.springframework.boot:spring-boot-starter-test'
}
dependencyManagement {
imports {
mavenBom "org.springframework.cloud:spring-cloud-dependencies:${springCloudVersion}"
}
}
Below is the example of API gateway configuration
application.yaml
server:
port: 10000
http:
port: 9000
# enable HTTP2
http2:
enabled: true
# enable compression
compression:
enabled: true
mime-types: text/html,text/xml,text/plain,text/css,text/javascript,application/javascript,application/json
ssl:
enabled: true
key-store: /var/.conf/self-signed.p12
key-store-type: PKCS12
key-store-password: "something"
key-alias: athenasowl
trust-store: /var/.conf/self-signe.p12
trust-store-password: "something"
spring:
application:
name: api-gateway
cloud:
gateway:
discovery:
locator:
enabled: true
predicates:
- Path="'/api/' + serviceId + '/**'"
filters:
- RewritePath="'/api/' + serviceId + '/(?<remaining>.*)'", "serviceId + '/${remaining}'"
management:
security:
enabled: false
server:
port: 10001
ssl:
enabled: false
endpoint:
gateway:
enabled: true
endpoints:
web:
exposure:
include: "*"
health:
sensitive: false
logging:
level:
root: DEBUG
org:
springframework:
web: INFO
pattern:
console: "%-5level %d{dd-MM-yyyy HH:mm:ss,SSS} [%F:%L] VTC : %msg%n"
file: "%-5level %d{dd-MM-yyyy HH:mm:ss,SSS} [%F:%L] VTC : %msg%n"
file: /tmp/log_files/apigateway.log
security:
basic:
enabled: false
There are a few configuration issues which we are facing, they are listed below:
Rewrite URL prefixed with /api/ to respective serviceId registered on consul: We tried to configure predicate to get path prefixed with api to rewrite path and remove api, but still it's not working. So there is another service /hello-service/ registered with consul server, but we want to do API call with /api/hello-service/
Redirect unmatched request to default path: We want to redirect all unmatched request to UI.
Redirecting HTTP to HTTPS on spring cloud gateway: We want to force all request coming to spring gateway to be https
Forwarding HTTPS request to HTTP serviceId registered with consul: Services registered with consul are on HTTP except for the API gateway, we want to be able to send HTTPS request to HTTP backend i.e. terminating HTTPS at API Gateway only.
Any help in solving the above issue would be good
Edit 1:
After some help from #spencergibb, we had setup the spring cloud gateway with https. But There are some additional issues which we faced
If HTTPS is enabled on both API gateway and service both, we received below error
javax.net.ssl.SSLException: handshake timed out at
io.netty.handler.ssl.SslHandler.handshake(...)(Unknown Source)
~[netty-handler-4.1.31.Final.jar:4.1.31.
If HTTPS is enabled on only API gateway, we received below error
There was an unexpected error (type=Not Found, status=404).
org.springframework.web.server.ResponseStatusException: 404 NOT_FOUND
and received
for path https://localhost:8443/api/hello-service/hello/message
Unable to Connect
for path http://localhost:8080/hello-service/hello/message
Please find the link for the sample applications
Instructions:
navigate to consul directory and Start consul server using command ./consul agent -dev
run api-gateway spring boot gradle project
run rest-demo spring boot gradle project
Edit 2
Thank You #spencergibb, We were able to successfully apply ssl on gateway and call the registered services on HTTP. Since Spring Webflux with Netty does not support listening on two ports, we created an additional tcp server bind to http port based on this answer.
There is still some issue we are facing with RewritePath for /api/ rule
predicates:
- name: Path
args:
pattern: "'/api/'+serviceId.toLowerCase()+'/**'"
filters:
- name: RewritePath
args:
regexp: "'/api/' + serviceId.toLowerCase() + '/(?<remaining>.*)'"
replacement: "'/${remaining}'"
below is the complete trace for the request
DEBUG 13-02-2019 03:32:01 [FilteringWebHandler.java:86] VTC : Sorted
gatewayFilterFactories:
[OrderedGatewayFilter{delegate=GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.AdaptCachedBodyGlobalFilter#257505fd},
order=-2147482648},
OrderedGatewayFilter{delegate=GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.GatewayMetricsFilter#400caab4},
order=-2147473648},
OrderedGatewayFilter{delegate=GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.NettyWriteResponseFilter#36e2c50b},
order=-1},
OrderedGatewayFilter{delegate=GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.ForwardPathFilter#66f0c66d}, order=0},
OrderedGatewayFilter{delegate=org.springframework.cloud.gateway.filter.factory.RewritePathGatewayFilterFactory$$Lambda$360/1720581802#5821f2e6,
order=0},
OrderedGatewayFilter{delegate=GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.RouteToRequestUrlFilter#27119239},
order=10000},
OrderedGatewayFilter{delegate=GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.LoadBalancerClientFilter#568a9d8f},
order=10100},
OrderedGatewayFilter{delegate=GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.WebsocketRoutingFilter#6ba77da3},
order=2147483646},
OrderedGatewayFilter{delegate=GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.NettyRoutingFilter#73c24516},
order=2147483647},
OrderedGatewayFilter{delegate=GatewayFilterAdapter{delegate=org.springframework.cloud.gateway.filter.ForwardRoutingFilter#461a9938},
order=2147483647}] TRACE 13-02-2019 03:32:01
[RouteToRequestUrlFilter.java:59] VTC : RouteToRequestUrlFilter start
TRACE 13-02-2019 03:32:02 [NettyWriteResponseFilter.java:68] VTC :
NettyWriteResponseFilter start TRACE 13-02-2019 03:32:02
[GatewayMetricsFilter.java:101] VTC : Stopping timer
'gateway.requests' with tags
[tag(outcome=CLIENT_ERROR),tag(routeId=rewrite_response_upper),tag(routeUri=http://httpbin.org:80),tag(status=NOT_FOUN
A number of things were needed
disable http2
Disable ssl configuration of httpclient
Update locator predicates and filters to use verbose configuration.
Here is the resulting portions of application.yml
server:
port: 8443
http:
port: 8080
servlet:
# enable HTTP2
# http2:
# enabled: true
# enable compression
# ... removed for brevity
spring:
application:
name: api-gateway
cloud:
consul:
enabled: true
gateway:
# httpclient:
# ssl:
# handshake-timeout-millis: 10000
# close-notify-flush-timeout-millis: 3000
# close-notify-read-timeout-millis: 0
# routes:
# - id: ui_path_route
# predicates:
# - Path="'/**'"
# filters:
# - RewritePath="'/**'", "/ui"
discovery:
instanceId: ${spring.application.name}:${vcap.application.instance_id:${spring.application.instance_id:${random.value}}}
locator:
enabled: true
predicates:
- name: Path
args:
pattern: "'/api/' + serviceId + '/**'"
filters:
- name: RewritePath
args:
regexp: "'/api/' + serviceId + '/(?<remaining>.*)'"
replacement: "'/${remaining}'"
#... removed for brevity

JHipster test: NoCacheRegionFactoryAvailableException when second level cache is disabled

When I use jhipster generate an app, I disabled the second level cache. However, when I run either "gradle test" or "run as junit test" to test the app, it is failed because the "NoCacheRegionFactoryAvailableException". I have checked the application.yml in directory "src/test/resources/config", and be sure that the second cache is disabled. I do not know why the app is still looking for second-cache. Is there any clue how this happen? or how to disable second level cache completely?
Except the test failure, everything else works well, the app can run successfully.
application.yml in src/test/resources/config
spring:
application:
name: EMS
datasource:
url: jdbc:h2:mem:EMS;DB_CLOSE_DELAY=-1
name:
username:
password:
jpa:
database-platform: com.espion.ems.domain.util.FixedH2Dialect
database: H2
open-in-view: false
show_sql: true
hibernate:
ddl-auto: none
naming-strategy: org.springframework.boot.orm.jpa.hibernate.SpringNamingStrategy
properties:
hibernate.cache.use_second_level_cache: false
hibernate.cache.use_query_cache: false
hibernate.generate_statistics: true
hibernate.hbm2ddl.auto: validate
data:
elasticsearch:
cluster-name:
cluster-nodes:
properties:
path:
logs: target/elasticsearch/log
data: target/elasticsearch/data
mail:
host: localhost
mvc:
favicon:
enabled: false
thymeleaf:
mode: XHTML
liquibase:
contexts: test
security:
basic:
enabled: false
server:
port: 10344
address: localhost
jhipster:
async:
corePoolSize: 2
maxPoolSize: 50
queueCapacity: 10000
security:
rememberMe:
# security key (this key should be unique for your application, and kept secret)
key: jhfasdhflasdhfasdkfhasdjkf
metrics: # DropWizard Metrics configuration, used by MetricsConfiguration
jmx.enabled: true
swagger:
title: EMS API
description: EMS API documentation
version: 0.0.1
termsOfServiceUrl:
contactName:
contactUrl:
contactEmail:
license:
licenseUrl:
enabled: false
Move src/test/resources/config/application.yml to src/test/resources directory.
You can find that solution from https://github.com/jhipster/generator-jhipster/issues/3730

Resources