I have installed Flume for the first time. I am using hadoop-1.2.1 and flume 1.6.0
I tried setting up a flume agent by following this guide.
I executed this command : $ bin/flume-ng agent -n $agent_name -c conf -f conf/flume-conf.properties.template
It says log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException: ./logs/flume.log (No such file or directory)
Isn't the flume.log file generated automatically? If not, how can I rectify this error ?
Try this:
mkdir ./logs
sudo chown `whoami` ./logs
bin/flume-ng agent -n $agent_name -c conf -f conf/flume-conf.properties.template
The first line creates the logs directory in the current directory if it does not already exist. The second one sets the owner of that directory to the current user (you) so that flume-ng running as your user can write to it.
Finally, please note that this is not the recommended way to run Flume, just a quick hack to try it.
You are getting this error probably because you are running command directly on console, you've to first go to the bin in flume and try running your command there over console.
As #Botond says, you need to set the right permissions.
However, if you run Flume within a program, like supervisor or with a custom script, you might want to change the default path, as it's relative to the launcher.
This path is defined in your /path/to/apache-flume-1.6.0-bin/conf/log4j.properties. There you can change the line
flume.log.dir=./logs
to use an absolute path that you would like to use - you still need the right permissions, though.
Related
I installed Nifi image docker in a VM Ubuntu. I've tried to get one xml file using GetFile processor, however when I set up the Input Directory field with specific directory created in the Ubuntu returns a error message like this:
Input Directory validated against '/home/john/nifi/inputs' is invalid because Directory does not exist
Is correct type path like this /home/john/nifi/inputs or between ${absolute path}?
Thank very much
Enter to the Docker container with "docker exec -it <name_container> /bin/bash"
Then "cd ~" to go to the your the nifi home folder: /home/nifi
So, you are set. Create the folder input with mkdir
I have my crontab set up as follows (this is inside a docker container).
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
SHELL=/bin/bash
5 * * * * bash /usr/local/bin/process-logs > /proc/1/fd/1 2>/proc/1/fd/
The /usr/local/bin/process-logs is designed to expose some MongoDB logs using mtools to a simple web server.
The problematic part of the script is fairly simple. raw_name is archive_name without the file extension.
aws s3 cp "s3://${s3_bucket}/${file_name}" "${archive_name}"
gunzip "${archive_name}"
mlogvis --no-browser "${raw_name}"
If I manually run the command as specified in the crontab config above
bash /usr/local/bin/process-logs > /proc/1/fd/1 2>/proc/1/fd/2
It all works as expected (this is the expected output from mlogvis)
...
copying /usr/local/lib/python3.5/dist-packages/mtools/data/index.html to /some/path/mongod.log-20190313-1552456862.html
...
When the script gets triggered via crontab it throws the following error
usage: mlogvis [-h] [--version] [--no-progressbar] [--no-browser] [--out OUT]
[--line-max LINE_MAX]
mlogvis: error: unrecognized arguments: mongod.log-20190313-1552460462
The mlogvis command that caused the following error (actual values not parameters)
mlogvis --no-browser "mongod.log-20190313-1552460462"
Again if I run this command myself it all works as expected.
mlogvis: http://blog.rueckstiess.com/mtools/mlogvis.html
I don't believe this to be an issue with the file not having correct permissions or not existing as mlogvis produces a different error in these conditions. I've also tested with removing '-' from the file name thinking it might be trying to parse these as arguments but it made no difference.
I know cron execution doesn't have the same execution environment as the user I tested the script as. I've set the PATH to be the same as the user and when the container starts up I execute env >> /etc/environment so all the environment vars and properly set.
Does anyone know of a way to debug this or has anyone encountered similar? All other components of the script are functioning except mlogvis which is core to the purpose of this job.
Summary of what I've tried as a fix:
Set environment and PATH for cron execution to be the same as the user I tested the script as
Replace - in file name(s) to see if it was parsing the parts as arguments
hardcode a filename with full permissions to see if it was permissions related
Manually run the script -> this works
Manually run the mlogvis command in isolation -> this works
try to load /home/user/.bash_profile before executing script and try again. I suspect that you have missing PATH or other environment variable which is not set.
source /home/user/.bash_profile
Please post your complete script, because while executing via crontab,
you have to be sure your raw_name variable was properly created. As
it seems to depend on archive_name, posting some more context can
help us to help you.
In any case, if you are using bash, you can try something like :
aws s3 cp "s3://${s3_bucket}/${file_name}" "${archive_name}"
gunzip "${archive_name}"
# here you have to be sure that archive_name is correct
raw_name_2=${archive_name%%.*}
mlogvis --no-browser "${raw_name_2}"
It is not going to solve your issue, but probably will take you closer to the right path.
I keep getting this error [2019-02-26T16:50:41,329][FATAL][logstash.runner ] Logstash could not be started because there is already another instance using the configured data directory. If you wish to run multiple instances, you must change the "path.data" setting.
when I launch logstash. I am using the cli to launch logstash. The command that I execute is:
screen -d -S logstash -m bash -c "cd;export JAVA_HOME=/nastools/jdk1.8.0_77/; export LS_JAVA_OPTS=-Djava.net.preferIPv4Stack=true; ~/monitoring/6.2.3/bin/logstash-6.2.3/bin/logstash -f ~/monitoring/6.2.3/config/logstash_forwarder/forwarder.conf"
I don't have any instance of logstash running. I tried running this:
ps xt | grep "logstash" and it didn't return any process. I tried the following as well: killall logstash but to no avail, it gives me the same error. I tried restarting my machine as well but still the same error.
Has anyone experienced something similar? Kibana and elastic search launch just fine.
Thanks in advance for your help!
The problem is solved now. I had to empty the contents of the data directory of logstash. I then restarted it and it generated the uuid and other files it needed.
To be more specific, you need to cd to the data folder of logstash (usually it is /usr/share/logstash/data) and delete the .lock file.
You can see if this file exists with:
ll -lah
In the data folder.
Learn it from http://www.programmersought.com/article/2009814657/;jsessionid=282FF6001AFE90D7D8609975B8222CE8
sudo /usr/share/logstash/bin/logstash --path.settings /etc/logstash/ --path.data sensor39 -f /etc/logstash/conf.d/company_dump.conf --config.reload.automatic
Try this cmd I hope it will work(but please check the .conf file path)
I installed Flume
and tried to run this command
flume-ng agent -n $agent_name -c conf -f /home/gautham/Downloads/apache-flume-1.5.0.1-bin/conf/flume-conf.properties.template
and I get this exception
ERROR node.Application: A fatal error occurred while running. Exception follows.
org.apache.commons.cli.MissingArgumentException: Missing argument for option: n
at org.apache.commons.cli.Parser.processArgs(Parser.java:343)
at org.apache.commons.cli.Parser.processOption(Parser.java:393)
at org.apache.commons.cli.Parser.parse(Parser.java:199)
at org.apache.commons.cli.Parser.parse(Parser.java:85)
at org.apache.flume.node.Application.main(Application.java:252)
Check if you are named your flume file with .conf extension.
And try to use the below command:
$ flume-ng agent \
--conf-file PathOfYourFlumeFile\
--name agentameInFlumeFile\
--conf $FLUME_HOME/conf \
Change $agent_name with the name of agent you have used in your flume file.
You have to mention path of flume file with .conf extension instead of this /home/gautham/Downloads/apache-flume-1.5.0.1-bin/conf/flume-conf.properties.template
Instead of $agent_name use the actual name of the agent in your conf file.
I suspect that you do not have an $agent_name environment variable so it's being replaced with empty string.
I had this similar issue. Later found out that by replacing all the hyphens(-) with hyphens again, it started working. Probably when i copied this command, the hyphens were replaced with minus(-) signs.
I'm installing a lighttpd server on a remote machine using a bash script. After installation, I need to configure the port for the server. The system says I don't have permission to modify the file /etc/lighttpd/lighttpd.conf even though I do
sudo echo "server.bind=2000" >> /etc/lighttpd/lighttpd.conf
How shall I modify this?
What you're doing is running echo as root, then trying to append its output to the config file as the normal user.
What you want is sudo sh -c 'echo "server.bind=2000" >> /etc/lighttpd/lighttpd.conf'
Try to change the file permission using chmod
$ sudo chmod a+x /etc/lighttpd/lighttpd.conf
If you don't have the right to change the file /etc/lighttpd/lighttpd.conf check the man page of lighthttpd. If you can start it with a different config file, then create a config file somewhere and start lighthttpd with it.
The problem is that the bit on the right of >> is not run under sudo. Either use sudo -i to bring up a root shell a superuser and run the command, or just use an editor as mentioned before.