I want to use logstash contrib plugin riemann in my config file. On running logstash error comes:
An unexpected error occurred. This is probably a bug. |
| You can find help with this problem in a few places: |
| |
| * chat: #logstash IRC channel on freenode irc. |
| IRC via the web: http://goo.gl/TI4Ro |
| * email: logstash-users#googlegroups.com |
| * bug system: https://logstash.jira.com/ |
| |
+---------------------------------------------------------+
The error reported is:
Couldn't find any output plugin named 'riemann'. Are you sure this is correct? Trying to load the riemann output plugin resulted in this error: no such file to load -- logstash/outputs/riemann
I have a folder inside which both the logstash and its contrib tar are present and extracted.
I am using logstash 1.4.1 and logstash-contrib-1.4.1.
I tried the manual installation for contrib too by :
./bin/plugin install contrib
but nothing appears on the console on running the command.
Any help?
EDIT
On ls the following is my directory structure:
ls
elasticsearch-1.1.1 kibana-3.1.0.tar.gz logstash-1.4.1.tar.gz logstash-contrib-1.4.1.tar.gz
elasticsearch-1.1.1.tar.gz logstash-1.4.1 logstash-contrib-1.4.1 riemann-0.2.5.tar.bz2
Thus I have untarred contrib in the same directory as logstash. Any IDEA??
You shall extract logstash-contrib-1.4.1.tar.gz inside logstash-1.4.1 directory with commands:
cd logstash-1.4.1
tar zxvf logstash-contrib-1.4.1.tar.gz --strip 1
Then check lib/logstash/outputs/riemann.rb is under logstash-1.4.1.
Related
I'm trying to invoke the update command of liquibase like follows:
liquibase update --changelog-file=./persistence/src/main/resources/changelog/db.changelog-dev.xml \
--url="jdbc:postgresql://localhost:5432/sigma"
This results in:
[...]
Starting Liquibase at 23:44:47 (version 4.17.2 #5255 built at 2022-11-01 18:07+0000)
Liquibase Version: 4.17.2
Liquibase Community 4.17.2 by Liquibase
Unexpected error running Liquibase: The file classpath:/changelog/db.changelog-master.xml was not found in the configured search path:
- /Users/ikaerom/Dev/sigma-backend
- /opt/homebrew/Cellar/liquibase/4.17.2/libexec/internal/lib/liquibase-core.jar
- /opt/homebrew/Cellar/liquibase/4.17.2/libexec/lib
- /opt/homebrew/Cellar/liquibase/4.17.2/libexec/internal/lib/jaybird.jar
- /opt/homebrew/Cellar/liquibase/4.17.2/libexec/internal/lib/ojdbc8.jar
- /opt/homebrew/Cellar/liquibase/4.17.2/libexec/internal/lib/snakeyaml.jar
- /opt/homebrew/Cellar/liquibase/4.17.2/libexec/internal/lib/snowflake-jdbc.jar
- /opt/homebrew/Cellar/liquibase/4.17.2/libexec/internal/lib/picocli.jar
- /opt/homebrew/Cellar/liquibase/4.17.2/libexec/internal/lib/jaxb-runtime.jar
- /opt/homebrew/Cellar/liquibase/4.17.2/libexec/internal/lib/jaxb-api.jar
- /opt/homebrew/Cellar/liquibase/4.17.2/libexec/internal/lib/jaxb-core.jar
- /opt/homebrew/Cellar/liquibase/4.17.2/libexec/internal/lib/hsqldb.jar
- /opt/homebrew/Cellar/liquibase/4.17.2/libexec/internal/lib/connector-api.jar
- /opt/homebrew/Cellar/liquibase/4.17.2/libexec/internal/lib/mssql-jdbc.jar
- /opt/homebrew/Cellar/liquibase/4.17.2/libexec/internal/lib/h2.jar
- /opt/homebrew/Cellar/liquibase/4.17.2/libexec/internal/lib/mariadb-java-client.jar
- /opt/homebrew/Cellar/liquibase/4.17.2/libexec/internal/lib/liquibase-commercial.jar
- /opt/homebrew/Cellar/liquibase/4.17.2/libexec/internal/lib/commons-lang3.jar
- /opt/homebrew/Cellar/liquibase/4.17.2/libexec/internal/lib/postgresql.jar
- /opt/homebrew/Cellar/liquibase/4.17.2/libexec/internal/lib/sqlite-jdbc.jar
- /opt/homebrew/Cellar/liquibase/4.17.2/libexec/internal/lib/opencsv.jar
- /opt/homebrew/Cellar/liquibase/4.17.2/libexec/internal/lib/commons-text.jar
- /opt/homebrew/Cellar/liquibase/4.17.2/libexec/internal/lib/commons-collections4.jar
- /opt/homebrew/Cellar/liquibase/4.17.2/libexec/internal/lib/jcc.jar
- /opt/homebrew/Cellar/liquibase/4.17.2/libexec/internal/lib
More locations can be added with the 'searchPath' parameter.
The db.changelog-dev.xml is essentially including db.changelog-master.xml, which then also references some SQL scripts. The two XML files lie in the same resource folder $PROJECT_ROOT/persistence/src/main/resources/changelog. The imported/included SQL files referenced within the changelog XML all lie in the resource folder's subfolders.
Any way of specifying this eluding searchPath or even --search-path parameter (as indicated in the documentation) seems to fail spectacularly:
$> liquibase update --searchPath="./persistence/src/main/resources/" --changelog-file=./persistence/src/main/resources/changelog/db.changelog-dev.xml --url="jdbc:postgresql://localhost:5432/sigma"
Unexpected argument(s): --searchPath=./persistence/src/main/resources/
So let's try the other indicated syntax:
$> liquibase update --search-path="./persistence/src/main/resources/" --changelog-file=./persistence/src/main/resources/changelog/db.changelog-dev.xml --url="jdbc:postgresql://localhost:5432/sigma"
Unexpected argument(s): --search-path=./persistence/src/main/resources/
If I attempt to use LIQUIBASE_SEARCH_PATH=, I end up with this:
[...]
Liquibase Version: 4.17.2
Liquibase Community 4.17.2 by Liquibase
Liquibase Community detected and ignored the following environment variables:
- LIQUIBASE_SEARCH_PATH
To configure Liquibase with environment variables requires a Liquibase Pro or Liquibase Labs license. Get a free trial at https://liquibase.com/trial. Options include the liquibase.licenseKey in the defaults file, adding a flag in the CLI, and more. Learn more at https://docs.liquibase.com.
[...]
I don't really want to buy a pro version just to get this feature working ;).
My question is: how do I specify the search path for liquibase to pick it up in my bash shell?
I find it hard to believe that this wouldn't work, given liquibase is so well documented, and it tries to always give you the correct hints and pointers, if you don't use it correctly. What did I miss?
Update: I have a suspicion that the order of invocation matters. So, the update command should be last in the list. However, no luck so far:
$> liquibase \
--changelog-file=./persistence/src/main/resources/changelog/db.changelog-dev.xml \
--url="jdbc:postgresql://localhost:5432/sigma" \
--searchpath="./persistence/src/main/resources/changelog/" \
update
[...]
Starting Liquibase at 14:29:51 (version 4.17.2 #5255 built at 2022-11-01 18:07+0000)
Liquibase Version: 4.17.2
Liquibase Community 4.17.2 by Liquibase
Unexpected error running Liquibase: The file ./persistence/src/main/resources/changelog/db.changelog-dev.xml was not found in the configured search path:
- /Users/ikaerom/Dev/sigma-backend/persistence/src/main/resources/changelog
More locations can be added with the 'searchPath' parameter.
For more information, please use the --log-level flag
Found the solution myself, after digging through the liquibase source code.
In my db.changelog-dev.xm I had a line which included db.changelog-master.xml as follows. That classpath:/ has to be removed:
- <include file="classpath:/changelog/db.changelog-master.xml"/>
+ <include file="changelog/db.changelog-master.xml"/>
Then, this invocation finally works (mind the adapted searchPath and the relative designation of the changelog parameter settings):
liquibase \
--hub-mode=off \
--headless=true \
--url="jdbc:postgresql://localhost:5432/sigma" \
--searchPath="./persistence/src/main/resources" \
--changelog-file=changelog/db.changelog-dev.xml \
update 2>&1 | grep -Ev -- "^##"
The --hub-mode=off will prevent liquibase from asking if you want to connect to the liquibase hub. The rest is sugar-coating.
The only problem open is that when liquibase is invoked from the shell CLI, the user ending up owning the changelog/lock tables is the user invoking the liquibase command:
ikaerom#/tmp:sigma> \dt databasechangeloglock
+--------+-----------------------+-------+---------+
| Schema | Name | Type | Owner |
|--------+-----------------------+-------+---------|
| public | databasechangeloglock | table | ikaerom |
+--------+-----------------------+-------+---------+
SELECT 1
Time: 0.011s
ikaerom#/tmp:sigma> \dt databasechangeloglock
+--------+-----------------------+-------+---------+
| Schema | Name | Type | Owner |
|--------+-----------------------+-------+---------|
| public | databasechangeloglock | table | ikaerom |
+--------+-----------------------+-------+---------+
SELECT 1
Time: 0.010s
However, when liquibase is updated by invoking the Spring boot application, then the table owner user is the one the application context is setting (in my case sigma):
ikaerom#/tmp:sigma> \dt databasechangeloglock
+--------+-----------------------+-------+-------+
| Schema | Name | Type | Owner |
|--------+-----------------------+-------+-------|
| public | databasechangeloglock | table | sigma |
+--------+-----------------------+-------+-------+
SELECT 1
Time: 0.010s
ikaerom#/tmp:sigma> \dt databasechangelog
+--------+-------------------+-------+-------+
| Schema | Name | Type | Owner |
|--------+-------------------+-------+-------|
| public | databasechangelog | table | sigma |
+--------+-------------------+-------+-------+
SELECT 1
Time: 0.009s
This clashes if you run your liquibase update first:
Caused by: liquibase.exception.DatabaseException: ERROR: relation "databasechangeloglock" already exists [Failed SQL: (0) CREATE TABLE public.databasechangeloglock (ID INTEGER NOT NULL, LOCKED BOOLEAN NOT NULL, LOCKGRANTED TIMESTAMP WITHOUT TIME ZONE, LOCKEDBY VARCHAR(255), CONSTRAINT databasechangeloglock_pkey PRIMARY KEY (ID))]
at liquibase.executor.jvm.JdbcExecutor$ExecuteStatementCallback.doInStatement(JdbcExecutor.java:397)
at liquibase.executor.jvm.JdbcExecutor.execute(JdbcExecutor.java:83)
at liquibase.executor.jvm.JdbcExecutor.execute(JdbcExecutor.java:151)
at liquibase.executor.jvm.JdbcExecutor.execute(JdbcExecutor.java:135)
at liquibase.lockservice.StandardLockService.init(StandardLockService.java:115)
at liquibase.lockservice.StandardLockService.acquireLock(StandardLockService.java:286)
... 94 common frames omitted
Caused by: org.postgresql.util.PSQLException: ERROR: relation "databasechangeloglock" already exists
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2675)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2365)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:355)
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:490)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:408)
at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:329)
at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:315)
at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:291)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:286)
at com.zaxxer.hikari.pool.ProxyStatement.execute(ProxyStatement.java:94)
at com.zaxxer.hikari.pool.HikariProxyStatement.execute(HikariProxyStatement.java)
at liquibase.executor.jvm.JdbcExecutor$ExecuteStatementCallback.doInStatement(JdbcExecutor.java:393)
This again can be solved by a proper GRANT for the sigma or a re-assignment of the owner to the rightful user. Or simply by adding the --username property to the name of the spring boot application context or database user owner:
liquibase \
--hub-mode=off \
--headless=true \
--username="sigma" \
--url="jdbc:postgresql://localhost:5432/sigma" \
--searchPath="./persistence/src/main/resources" \
--changelog-file=changelog/db.changelog-dev.xml \
update 2>&1 | grep -Ev -- "^##"
I have this non-descript CI error is it possible to get something more descript?
Code Snippet below:
$ semgrep-agent
| versions - semgrep 0.81.0 on Python 3.9.10
| environment - running in environment gitlab-ci, triggering event is 'pull_request'
| manage - not logged in
=== setting up agent configuration
| using semgrep rules from https://semgrep.dev/c/p/security-audit
| using semgrep rules from https://semgrep.dev/c/p/secrets
An unexpected error occurred:
<class 'sh.ErrorReturnCode_1'>
RAN: /usr/bin/git merge-base --all b7cb3f8e678d943b4cded2e55d45221fafbde030 FETCH_HEAD
STDOUT:
STDERR:
What could be happening is that you are setting the git_depth variable to something too low:
variables:
GIT_DEPTH: 1
Semgrep needs to fetch commit history to do a diff-aware scan between the source and target branch, so limiting the git_depth might be the cause of some merge-related errors.
I am running the command - curl https://cli-assets.heroku.com/install-ubuntu.sh | sh
Which throws the following error -
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1232 100 1232 0 0 5133 0 --:--:-- --:--:-- --:--:-- 5112
This script requires superuser access to install apt packages.
You will be prompted for your password by sudo.
+ dpkg -s apt-transport-https
+ echo ''
sh: line 4: /etc/apt/sources.list.d/heroku.list: No such file or directory
I also ran sudo snap install --classic heroku which returned sudo: snap: command not found.
Then I ran sudo apt install snapd which returned the following -
apt: invalid flag: install
Usage: apt <apt and javac options> <source files>
where apt options include:
-classpath <path> Specify where to find user class files and annotation processor factories
-cp <path> Specify where to find user class files and annotation processor factories
-d <path> Specify where to place processor and javac generated class files
-s <path> Specify where to place processor generated source files
-source <release> Provide source compatibility with specified release
-version Version information
-help Print a synopsis of standard options; use javac -help for more options
-X Print a synopsis of nonstandard options
-J<flag> Pass <flag> directly to the runtime system
-A[key[=value]] Options to pass to annotation processors
-nocompile Do not compile source files to class files
-print Print out textual representation of specified types
-factorypath <path> Specify where to find annotation processor factories
-factory <class> Name of AnnotationProcessorFactory to use; bypasses default discovery process
See javac -help for information on javac options.
warning: The apt tool and its associated API are planned to be
removed in the next major JDK release. These features have been
superseded by javac and the standardized annotation processing API,
javax.annotation.processing and javax.lang.model. Users are
recommended to migrate to the annotation processing features of
javac; see the javac man page for more information.
Finally, I ran wget 0- wget https://toolbelt.heroku.com/install-ubuntu.sh | sh which returned
--2020-09-10 18:15:53-- http://0-/
Resolving 0- (0-)... failed: Name or service not known.
wget: unable to resolve host address ‘0-’
--2020-09-10 18:15:54-- http://wget/
Resolving wget (wget)... failed: Name or service not known.
wget: unable to resolve host address ‘wget’
--2020-09-10 18:15:54-- https://toolbelt.heroku.com/install-ubuntu.sh
Resolving toolbelt.heroku.com (toolbelt.heroku.com)... 54.164.74.108, 107.23.162.152, 34.194.108.77, ...
Connecting to toolbelt.heroku.com (toolbelt.heroku.com)|54.164.74.108|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 719 [text/plain]
Saving to: ‘install-ubuntu.sh’
install-ubuntu.sh 100%[=============================================================>] 719 --.-KB/s in 0s
2020-09-10 18:15:54 (105 MB/s) - ‘install-ubuntu.sh’ saved [719/719]
FINISHED --2020-09-10 18:15:54--
Total wall clock time: 0.4s
Downloaded: 1 files, 719 in 0s (105 MB/s)
So then, I ran bash install-ubuntu.sh which returned -
This script requires superuser access to install apt packages.
You will be prompted for your password by sudo.
sh: line 3: /etc/apt/sources.list.d/heroku.list: No such file or directory
sh: line 6: apt-key: command not found
--2020-09-10 18:16:51-- https://toolbelt.heroku.com/apt/release.key
Resolving toolbelt.heroku.com (toolbelt.heroku.com)... 54.145.36.98, 54.164.74.108, 107.23.162.152, ...
Connecting to toolbelt.heroku.com (toolbelt.heroku.com)|54.145.36.98|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1737 (1.7K) [application/octet-stream]
Saving to: ‘STDOUT’
- 0%[ ] 0 --.-KB/s in 0s
Cannot write to ‘-’ (Success).
sh: line 9: apt-get: command not found
sh: line 12: apt-get: command not found
I am taking an online course on Upskill and am on video #125. Please advise how to progress.
Thanks for your time and help.
I have a gist I use specifically for setting up rails on Cloud9 (though I haven't updated it for Rails 6 yet). Recommend you read through it twice before you try to do it. You may need to go out of order in some cases, as exact approach depends on if you are cloning an existing repo or building a new app.
https://gist.github.com/MyklClason/791d6b14606bc56e72eba2995aab8e76
You probably don't need snap.
Also useful bash aliases for Cloud9:
https://gist.github.com/MyklClason/d71a39ace28b9ec9f0ad
As for your actual issue. Heroku toolbelt is obsolete, use this instead:
wget -qO- https://cli-assets.heroku.com/install-ubuntu.sh | sh
Also it's often best to just look online and check how to install the heroku cli (or anything really) for your OS using the offical documentation. Though that may not work for older setup. However, heroku is something where you basically have to use the newest version otherwise you are going to run into problems.
If you didn't figure this out...
nvm i v8
Followed by...
npm install -g heroku
It said
-> MarsTool (0.1.0)
- ERROR | [iOS] file patterns: The source_files pattern did not match any file.
- WARN | [iOS] license: Unable to find a license file
- NOTE | xcodebuild: note: Using new build system
- NOTE | [iOS] xcodebuild: note: Planning build
- NOTE | [iOS] xcodebuild: note: Constructing build description
Analyzed 1 podspec.
[!] The spec did not pass validation, due to 1 error and 1 warning.
but actually there are some files in the source_files in my local path.
The most common cause of this failure is not pushing a tag to the repo specified in the s.source attribute of the podspec that tags the desired source to validate.
I'm trying to use the 'cloudtrail' codec with logstash 1.4.2, which is in logstash-contrib. I've installed it on an ubuntu distro:
sudo /usr/share/elasticsearch/bin/plugin --install elasticsearch/logstash-contrib
I've verified the file is there:
ls /usr/share/elasticsearch/plugins/logstash-contrib/_site/lib/logstash/codecs/cloudtrail.rb
And I've included it in a basic configuration:
input {
s3 {
bucket => "bucketname"
delete => false
codec => "cloudtrail"
....
When I try to run it, I get a plugin error:
{:timestamp=>"2015-02-06T20:10:25.179000+0000", :message=>"
+---------------------------------------------------------+
| An unexpected error occurred. This is probably a bug. |
| You can find help with this problem in a few places: |
| |
| * chat: #logstash IRC channel on freenode irc. |
| IRC via the web: http://goo.gl/TI4Ro |
| * email: logstash-users#googlegroups.com |
| * bug system: https://logstash.jira.com/ |
| |
+---------------------------------------------------------+
The error reported is:
Couldn't find any codec plugin named 'cloudtrail'. Are you sure
this is correct? Trying to load the cloudtrail codec plugin resulted
in this error: no such file to load -- logstash/codecs/cloudtrail"}
Now, I see that most of the codecs are at /opt/logstash/lib/logstash/codecs. Do I need to manually copy it (that seems bad)? Is there a gem install I'm missing?
For the fun of it, I copied it to that /opt/logstash directory. It just seems wrong.
Per the docs, you should be able to install it with the following:
bin/plugin install logstash-codec-cloudtrail
EX: /opt/logstash/bin/plugin install logstash-codec-cloudtrail works for me with logstash 1.5.0.1.