rsyslog - how to send multipline messages using JSON template - rsyslog

I am shipping laravel logs using the rsyslog imfile module.
I use the startmsg.regex to deal with multiline messages. These work great.
input(type="imfile"
File="laravel.log"
Tag="laravel_app"
startmsg.regex="[[:digit:]]{4}-[[:digit:]]{1,2}-[[:digit:]]{1,2} [[:digit:]]{1,2}:[[:digit:]]{1,2}:[[:digit:]]{1,2}"
)
I then ship the logs out to Logstash (and then use them in Elasticsearch, and Kibana)
I ship the logs using a Rsyslog template
template(name="json-template"
type="list") {
constant(value="{")
constant(value="\"#timestamp\":\"") property(name="timereported" dateFormat="rfc3339")
constant(value="\",\"#version\":\"1")
constant(value="\",\"message\":\"") property(name="msg" format="json")
constant(value="\",\"sysloghost\":\"") property(name="hostname")
constant(value="\",\"severity\":\"") property(name="syslogseverity-text")
constant(value="\",\"facility\":\"") property(name="syslogfacility-text")
constant(value="\",\"programname\":\"") property(name="programname")
constant(value="\",\"procid\":\"") property(name="procid")
constant(value="\"}\n")
}
My logs look like this, where line-breaks are marked with \n
[2018-06-07 11:02:33] testing.ERROR: Class 'App\Http\Controllers\Deal'
not found {"exception":"[object]
(Symfony\Component\Debug\Exception\FatalThrowableError(code: 0): Class
'App\Http\Controllers\Deal' not found at
/app/Http/Controllers/TestingSyslogController.php:13)\n[stacktrace]\n#0
[internal function]:
App\Http\Controllers\TestingSyslogController->index()\n#1
/vendor/laravel/framework/src/Illuminate/Routing/Controller.php(54):
call_user_func_array(Array, Array)\n#2
/vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php(45):
Illuminate\Routing\Controller->callAction('index', Array)\n#3
/vendor/laravel/framework/src/Illuminate/Routing/Route.php(212):
Illuminate\Routing\ControllerDispatcher->dispatch(Object(Illuminate\Routing\Route),
Object(App\Http\Controllers\TestingSyslogController), 'index')\n#4
/vendor/laravel/framework/src/Illuminate/Routing/Route.php(169):
Illuminate\Routing\Route->runController()\n#5
So the entire message is in a single line {message}
How can I use the rsyslog template to split the message at '\n' into different objects to look something like this...
{message line 1},{message line 2},{message line x}

Related

Send logs to datadog with custom formatter shows only the first log of multiple ones

I am trying to send logs to DD (datadog) in such a way that the logs are being received as json and therefore shown properly in the portal through attributes.
My logger is a simple Logger.new(STDOUT, level: Logger::INFO).
If I stick to its standard output, it will in the form
I, [2022-07-30T22:43:35.216846 #1] INFO -- my-app: {"user":"1234"}
which is not really parsable by DD since not a proper JSON. In this case however all the logs appear at least on the DD portal.
Now.. I am trying to format the logs in a JSON manner in this way:
def self.logger
#logger ||= Logger.new(STDOUT, level: Logger::INFO)
#logger.progname = 'my-app'
#logger.formatter = proc do |severity, datetime, progname, msg|
{timestamp: datetime.to_s, progname: progname, severity: severity, correlation: Datadog::Tracing.log_correlation, message: msg}.to_json
end
#logger
end
This is my logger and thanks to this logs are seen properly in DD and parsed correctly because formatted in my app in a proper JSON.
The problem with this approach though seems to be that the logs are sent in 1 full block. Meaning that only the very first log is being visible. Let's say that I want to log this:
my_hash = {"message" => '1', "prop" => '1234'}.to_json
logger.info(my_hash)
my_hash = {"message" => '2', "prop" => '12345'}.to_json
logger.info(my_hash)
only the first log will be shown correctly on the DD portal. Parsed correctly with its message and prop attributes, but nothing about the second log.
Here is the thing, if I see the output of my app locally in the console I see this:
{"timestamp":"2022-07-31 01:15:39 +0200","progname":"my-app","severity":"INFO","correlation":"dd.service=my-app dd.trace_id=2976451780376429536 dd.span_id=0","message":"{"message":"1","prop":"1234"}"}{"timestamp":"2022-07-31 01:15:39 +0200","progname":"my-app","severity":"INFO","correlation":"dd.service=my-app dd.trace_id=2976451780376429536 dd.span_id=0","message":"{"message":"2","prop":"12345"}"}127.0.0.1 - - [31/Jul/2022:01:15:39 +0200] "GET /controller/test_controller HTTP/1.1" 200 - 0.0024
so the 2nd log gets actually outputted! But DD somehow sees only the first log..
(I know there is even a 3rd one shown in this message.. but that's just Sinatra automatic behavior for every http call reaching the api). What do you guys think is the problem?

Add text at the end of the logs

I actually use Rsyslog 8.24 and I configured my rsyslog to accept logs from multiples input/sources.
I want to add the syslog hostname at the end of every logs.
Example :
Old log : timestamps, header, message
New log : timestamps, header, message syslog.domain.local
I know that the variable $myhostname or $MYHOSTNAME should return the hostname of the syslog but I don't understand how to implement this and add the syslog hostname at the end of each log.
I managed to do what I wanted by adding the following template and binding it in the ruleset :
template (name="LogsFormat" type="string" string="%TIMESTAMP% %$year% %syslogtag% %msg% <SYSLOG_HOSTNAME>:%$myhostname%\n")
ruleset(name="RemoteLogPort") {
if (re_match($msg, "AP:aaa-bbbb-ccc-dddd-ap")) then {
action(type="omfile" dynaFile="ArubaNetworksPath" template="LogsFormat")
}
}
PS : ArubaNetworksPath is also a template defining the log path.

How to upload an image to a specific folder on cloudiary with laravel jrm2k6/cloudder

I want to crop the avatar image and upload that on a specific folder of cloudiary using Laravel jrm2k6/cloudder but faced to a error to upload an image.
Can anyone help me what's wrong or show how to solve?
Following is the error message stored on a log file.
[2019-10-20 10:39:38] local.ERROR: Invalid Signature 22800853dcbe6c9972b65d99946460f1965a6e06. String to sign - 'folder=media/users/1/avatar&timestamp=
1571535576'. {"userId":1,"exception":"[object] (Cloudinary\\Error(code: 401): Invalid Signature 22800853dcbe6c9972b65d99946460f1965a6e06. String to sig
n - 'folder=media/users/1/avatar&timestamp=1571535576'. at /Users/kondonator/Development/tiny-services/vendor/cloudinary/cloudinary_php/src/Uploader.ph
p:515)
[stacktrace]
#0 /Users/kondonator/Development/tiny-services/vendor/cloudinary/cloudinary_php/src/Uploader.php(374): Cloudinary\\Uploader::call_api('upload', Array,
Array, '/private/var/tm...')
#1 /Users/kondonator/Development/tiny-services/vendor/cloudinary/cloudinary_php/src/Uploader.php(99): Cloudinary\\Uploader::call_cacheable_api('upload'
, Array, Array, '/private/var/tm...')
#2 /Users/kondonator/Development/tiny-services/vendor/jrm2k6/cloudder/src/JD/Cloudder/CloudinaryWrapper.php(122): Cloudinary\\Uploader::upload('/privat
e/var/tm...', Array)
#3 /Users/kondonator/Development/tiny-services/vendor/laravel/framework/src/Illuminate/Support/Facades/Facade.php(239): JD\\Cloudder\\CloudinaryWrapper
->upload('/private/var/tm...', NULL, Array, Array)
#4 /Users/kondonator/Development/tiny-services/app/Http/Controllers/UserController.php(150): Illuminate\\Support\\Facades\\Facade::__callStatic('upload
', Array)
#5 [internal function]: App\\Http\\Controllers\\UserController->update(Object(Illuminate\\Http\\Request), '1')
And the following is the code of UserController.php.
$path = $request->file('avatar')
->getRealPath();
$image = Image::make($path);
$image->crop($request->width, $request->height, $request->x, $request->y)
->resize(200, 200)
->save();
$folder = 'media/users/'.$id.'/avatar';
Cloudder::upload($path, null, ['folder' => $folder], []);
$result = Cloudder::getResult();
$avatar = Cloudder::getPublicId();
The last is the part of .env those are from Cloudinary's Dashboard -> Account Details.
CLOUDINARY_API_KEY=XXXXX
CLOUDINARY_API_SECRET=YYYYY
CLOUDINARY_CLOUD_NAME=ZZZZZ
Thanks in advance.
The string (folder=media/users/1/avatar&timestamp=1571535576) prior to the api_secret being concatanated at the end is correct, but the signature it should've produced is 083dc25c1d1e4a90f84336450f21c9fb59b343f5.
I suggest double-checking the CLOUDINARY_API_SECRET in your configuration (.env) and if you've made any changes to it then you should restart your server so they take effect.
What you can also do is manually generate the correct signature on the command line just to double-check with the one that's being thrown as part of the exception. Replace <API_SECRET> with your actual one in the command below:
echo -n "folder=media/users/1/avatar&timestamp=1571535576<API_SECRET>" | sha1sum
Also, not related to the error, but I would recommend keeping your cloud name in all lowercase in your configuration. The API responses would return url and secure_url with it in all lowercase and also that's the case for using the URLs that are generated by the Media Library. Requesting resources with different casing would result in different cache-keys at the CDN.

Need some guidance regarding Elasticsearch php

I need some help in understanding the following things,
I am using composer for elasticsearch php integration. I have created a html page to post form data into elasticsearch. I am trying to access the html page from another pc by changing the localhost ip to the system ip where i have hosted the html page. But i'm getting error.
Fatal error: Uncaught Elasticsearch\Common\Exceptions\NoNodesAvailableException: No alive nodes found in your cluster in C:\xampp\htdocs\Client_portal\vendor\elasticsearch\elasticsearch\src\Elasticsearch\ConnectionPool\StaticNoPingConnectionPool.php:50 Stack trace: #0 C:\xampp\htdocs\Client_portal\vendor\elasticsearch\elasticsearch\src\Elasticsearch\Transport.php(77): Elasticsearch\ConnectionPool\StaticNoPingConnectionPool->nextConnection() #1 C:\xampp\htdocs\Client_portal\vendor\elasticsearch\elasticsearch\src\Elasticsearch\Transport.php(94): Elasticsearch\Transport->getConnection() #2 C:\xampp\htdocs\Client_portal\vendor\elasticsearch\elasticsearch\src\Elasticsearch\Connections\Connection.php(269): Elasticsearch\Transport->performRequest('POST', '/sib_api/_doc', Array, '{"requestid":"#...', Array) #3 C:\xampp\htdocs\Client_portal\vendor\react\promise\src\FulfilledPromise.php(25): Elasticsearch\Connections\Connection->Elasticsearch\Connections{closure}(Array) #4 C:\xampp\htdocs\Client_portal\vendor\guzzlehttp\ringphp\sr in C:\xampp\htdocs\Client_portal\vendor\elasticsearch\elasticsearch\src\Elasticsearch\ConnectionPool\StaticNoPingConnectionPool.php on line 50
To resolve the above issue i tried to call the autoload.php file using the complete path (i.e) http://localhost:8080/client_portal/vendor/autoload.php
But now I am getting another error
Fatal error: Uncaught Error: Class 'Elasticsearch\ClientBuilder' not found in C:\xampp\htdocs\Client_portal\Elastic_api.php:19 Stack trace: #0 {main} thrown in C:\xampp\htdocs\Client_portal\Elastic_api.php on line 19
Could someone help me to resolve this issue... and also guide me how to access my html page from another system without any isssues.
<?php
session_start();
$Engagement = $_SESSION["Engagement"];
$Application = $_SESSION["Application"];
$Priority = $_SESSION["Priority"];
$Requestid = $_SESSION["Requestid"];
$_SESSION["URL"] = $_POST['url'];
$_SESSION["REQ"] = $_POST['request'];
$_SESSION["RES"] = $_POST['response'];
$_SESSION["yn"] = $_POST['YN'];
$_SESSION["START"] = $_POST['start'];
$_SESSION["END"] = $_POST['end'];
if (isset($_POST['Submit2']))
{
require 'http://localhost:8080/client_portal/vendor/autoload.php';
$client = Elasticsearch\ClientBuilder::create()->build();
// the above line is where i'm getting the error...

Magento Error while trying to order / cancel an order that has more than one item

Hy
After the onepage checkout, in Magento, I received the following email error message
...
Payment transaction failed.
Reason
Mage registry key "_singleton/model/observer" already exists
Checkout Type
onepage
....
And on the admin page, when trying to delete this oreder, same error, again:
There has been an error processing your request
Mage registry key "_singleton/model/observer" already exists
Trace:
#0 /app/Mage.php(223): Mage::throwException('Mage registry k...')
#1 /app/Mage.php(478): Mage::register('_singleton/mode...', false)
#2 /app/code/core/Mage/Core/Model/App.php(1316): Mage::getSingleton('model/observer')
#3 /app/Mage.php(448): Mage_Core_Model_App->dispatchEvent('cataloginventor...', Array)
#4 /app/code/core/Mage/Core/Model/Abstract.php(466): Mage::dispatchEvent('cataloginventor...', Array)
#5 /app/code/core/Mage/CatalogInventory/Model/Stock/Item.php(787): Mage_Core_Model_Abstract->_afterSave()
#6 /app/code/core/Mage/Core/Model/Abstract.php(319): Mage_CatalogInventory_Model_Stock_Item->_afterSave()
#7 l/app/code/core/Mage/CatalogInventory/Model/Stock.php(210): Mage_Core_Model_Abstract->save()
#8 /app/code/core/Mage/CatalogInventory/Model/Observer.php(809): Mage_CatalogInventory_Model_Stock->backItemQty('542', 1)
#9 /app/code/core/Mage/Core/Model/App.php(1338): Mage_CatalogInventory_Model_Observer->cancelOrderItem(Object(Varien_Event_Observer))
#10 /app/code/core/Mage/Core/Model/App.php(1317): Mage_Core_Model_App->_callObserverMethod(Object(Mage_CatalogInventory_Model_Observer), 'cancelOrderItem', Object(Varien_Event_Observer))
#11 /app/Mage.php(448): Mage_Core_Model_App->dispatchEvent('sales_order_ite...', Array)
..and so on
This only happends to orders that have more than one item.
Any help will be appreciated, especially because the website is in production now.
Many thanks
Let me add a bit of Magento 'anatomy' to Alan's answer:
From your error log:
#10 /app/code/core/Mage/Core/Model/App.php(1317): `Mage_Core_Model_App->_callObserverMethod(Object(Mage_CatalogInventory_Model_Observer), 'cancelOrderItem', Object(Varien_Event_Observer))
class: Mage_CatalogInventory_Model_Observer
model: cataloginventory/observer
method: cancelOrderItem
This mean what #2 should be:
#2 /app/code/core/Mage/Core/Model/App.php(1316): Mage::getSingleton('cataloginventory/observer')
But you have:
#2 /app/code/core/Mage/Core/Model/App.php(1316): Mage::getSingleton('model/observer')
Why it failed:
1) when Magento try to execute Mage::getSingleton('model/observer') it call getSingleton() function from Alan's answer.
2) registry don't have '_singleton/model/observer' so it try to register it:
self::register($registryKey, self::getModel($modelClass, $arguments));
3) it call self::getModel('model/observer', array())) for registration and this method fails because you don't have such model.
Again, you have 'model/observer' model in one of your xml configuration files, but Magento can't find physical file with this model.
What to do?
This is occurs on 'sales_order_item_cancel' event, so try to find this string in config.xml files in third-party extension folders.
If in one of config.xml you find something like:
...
<events>
...
<sales_order_item_cancel>
<observers>
<inventory>
<class>model/observer</class>
<method>cancelOrderItem</method>
</inventory>
</observers>
</sales_order_item_cancel>
...
</events>
...
Just comment <sales_order_item_cancel> section.
If you want to make it working, set correct value in <class>model/observer</class> section.
That's a curious error -- my guess is you're dealing with a "core-hack"/edut (perhaps unintentionally) of your app/Mage.php file.
If you look at your stack trace, Magento calls
Mage::getSingleton('model/observer')
This is Magento's ways of saying "Instantiate a model/observer object, and make it a singleton instance", If you're not familiar with them, singletons are a "global" object that can only be instantiated once. If you try to instantiate a singleton again you get the original instance.
If you look at the implementation of getSingleton
#File: app/Mage.php
public static function getSingleton($modelClass='', array $arguments=array())
{
$registryKey = '_singleton/'.$modelClass;
if (!self::registry($registryKey)) {
self::register($registryKey, self::getModel($modelClass, $arguments));
}
return self::registry($registryKey);
}
You can see the call to self::registry from your stack trace that causes the error. However, you can also see that Magento checks to make sure the registry key doesn't exist via the if (!self::registry($registryKey)) { conditional.
My guess is someone changed the definition of getSingleton, or registry or register in your app/Mage.php file. In a sock Magento system it should not be possible to receive the exact error you've posted here.

Resources