I am using boost log and I choose the text_file_backend, but I got bad performance. No matter sync or async, the boost log has a low performance. About in 6 seconds, it wrote 30M data to log file.
Follow is my code snippet, anyone can help me?
typedef boost::log::sinks::asynchronous_sink<
boost::log::sinks::text_file_backend> TextSink;
boost::log::sources::severity_logger_mt<LogSev> logger_;
boost::shared_ptr<TextSink> report_sink_;
// initialize report_sink
boost::shared_ptr<sinks::text_file_backend> report_backend =
boost::make_shared<sinks::text_file_backend>(
keywords::file_name = target + "/" + file_name
+ ".report.log.%Y_%m_%d.%N",
keywords::rotation_size = file_size, keywords::time_based_rotation =
sinks::file::rotation_at_time_point(0, 0, 0),
keywords::auto_flush = false);
boost::shared_ptr<sinks::file::collector> report_collector = CreateCollector(
target, max_use_size / 2, min_free_size);
report_backend->set_file_collector(report_collector);
report_backend->scan_for_files();
// add sink: report_sink
report_sink_ = boost::make_shared<TextSink>(report_backend);
report_sink_->set_formatter(
expr::format("[%1%]" + sep + "[%2%]" + sep + "[%3%]" + sep + "%4%")
% expr::format_date_time<boost::posix_time::ptime>(
"TimeStamp", "%Y-%m-%d %H:%M:%S.%f")
% expr::attr<LogSev>("Severity")
% expr::attr<attrs::current_thread_id::value_type>("ThreadID")
% expr::message);
report_sink_->set_filter(expr::attr<LogSev>("Severity") >= report);
logging::core::get()->add_sink(report_sink_);
logging::add_common_attributes();
BOOST_LOG_SEV(logger_, info) << "blabal...";
I think one performance issue about your implementation is about Timestamp. It needs a system-call to find Time. I encountered the same problem. So I turned to use date library. It returns UTC time very fast. Also check first answer of this question, However if you want Timestamp based on timezone, date library is slow. you should better define your timezone and add to UTC.
see the example:
#include "date.h"
#define MY_TIME std::chrono::hours(4) + std::chrono::minutes(30)
string timestamp = date::format("%F %T", std::chrono::system_clock::now() +
MY_TIME);
Related
I get different answers depending on which watch the following code is run on
val prefs = getSharedPreferences(MY_PREFS_NAME, MODE_PRIVATE)
val appClosedTime = prefs.getLong(KEY_APP_CLOSE_TIME, System.currentTimeMillis()) // default to now
val elapsedTime = System.currentTimeMillis() - appClosedTime // in milliseconds
The above code gives the right answer on both the Samsung Galaxy Watch 4 (Wear OS 3) and a Google Emulator (Wear Round API 30)
Unfortunately when the same code is run on the TicWatch (Wear OS 2) the elapsedTime includes the local time offset from UTC.
The following code can be used to adjust for the difference
val prefs = getSharedPreferences(MY_PREFS_NAME, MODE_PRIVATE)
val appClosedTime = prefs.getLong(KEY_APP_CLOSE_TIME, System.currentTimeMillis()) // default to now
val currentSystemTime = GregorianCalendar()
val localTimeOffsetFromUTC = currentSystemTime.timeZone.getOffset(appClosedTime)
val elapsedTime = System.currentTimeMillis() - (appClosedTime-localTimeOffsetFromUTC)
My code currently checks which watch I am using and makes the adjustment however this 'hack' is clearly impractical since it must be tested on each and every watch!
I am hoping someone can suggest something I am missing / overlooking / doing wrong ?
Code snippets and further information
System.currentTimeMillis() says it returns the difference, measured in milliseconds, between the current time and midnight, January 1, 1970 UTC.
appCloseTime is stored in onStop()
val editor = getSharedPreferences(MY_PREFS_NAME, MODE_PRIVATE).edit()
editor.putLong(KEY_APP_CLOSE_TIME, System.currentTimeMillis())
editor.apply()
I've read through the available q and a on SO, but nothing I have found answers my question of how to format my time in 12hour format.
Following is my code that runs a query on a MySQL database and returns results, checking to see if an appointment is within 15 minutes of login so an alert can pop.
public void apptCheck(int userId) throws SQLException {
// this method checks for an appointment occurring within 15 minutes of login
Statement apptStatement = DBQuery.getStatement();
String apptQuery = "Select apt.start, cs.customerName from DBName.appointment apt "
+ "JOIN DBName.customer cs ON cs.customerId = apt.customerId WHERE "
+ "userId = " + userId + " AND start >= NOW() AND start < NOW() + interval 16 minute";
apptStatement.execute(apptQuery);
ResultSet apptRs = apptStatement.getResultSet();
while(apptRs.next()) {
Timestamp apptTime = apptRs.getTimestamp("start");
ResourceBundle languageRB = ResourceBundle.getBundle("wgucms/RB", Locale.getDefault());
Alert apptCheck = new Alert(AlertType.INFORMATION);
apptCheck.setHeaderText(null);
apptCheck.setContentText(languageRB.getString("apptSoon") + " " + apptTime.toInstant().atZone(ZoneId.systemDefault()));
apptCheck.showAndWait();
}
My result is:
I want the time to display 3:00, not the 19:00 - 06:00. How can I make that happen?
ZonedDateTime zonedDateTime=ZonedDateTime.of(apptTime.toLocalDateTime(),ZoneId.systemDefault());
You can use ZonedDateTime and format the time as you want.
docs.oracle.com/javase/8/docs/api/java/time/ZonedDateTime.html ZonedDateTime has a lot of features you can see all here and you can get the hour, minute, day etc.
DateTimeFormatter formatter = DateTimeFormatter.ofPattern("hh:mm:ss");
String formattedString = zonedDateTime.format(formatter);
if you only want time in 12hour format you can use this
I found the solution which will perform the UTC to local time conversion and then format the time so that the resulting alert is in 12 hour time format without the date or time zone info. Here is the full code:
while(apptRs.next()) {
Timestamp apptTime = apptRs.getTimestamp("start");
// perform time conversion from UTC to User Local Time
ZoneId zidApptTime = ZoneId.systemDefault();
ZonedDateTime newZDTApptTime = apptTime.toLocalDateTime().atZone(ZoneId.of("UTC"));
ZonedDateTime convertedApptTime = newZDTApptTime.withZoneSameInstant(zidApptTime);
ResourceBundle languageRB = ResourceBundle.getBundle("wgucms/RB", Locale.getDefault());
Alert apptCheck = new Alert(AlertType.INFORMATION);
apptCheck.setHeaderText(null);
// set the Alert text and format in 12 hour format
apptCheck.setContentText(languageRB.getString("apptSoon") +
convertedApptTime.toInstant().atZone(ZoneId.systemDefault())
.format(DateTimeFormatter.ofPattern("h:mm a")) + ".");
apptCheck.showAndWait();
}
The hackerrank challenge is in the following url: https://www.hackerrank.com/challenges/python-time-delta/problem
I got testcase 0 correct, but the website is saying that I have wrong answers for testcase 1 and 2, but in my pycharm, I copied the website expected output and compared with my output and they were exactly the same.
Please have a look at my code.
#!/bin/pyth
# Complete the time_delta function below.
from datetime import datetime
def time_delta(tmp1, tmp2):
dicto = {'Jan':1, 'Feb':2, 'Mar':3,
'Apr':4, 'May':5, 'Jun':6,
'Jul':7, 'Aug':8, 'Sep':9,
'Oct':10, 'Nov':11, 'Dec':12}
# extracting t1 from first timestamp without -xxxx
t1 = datetime(int(tmp1[2]), dicto[tmp1[1]], int(tmp1[0]), int(tmp1[3][:2]),int(tmp1[3][3:5]), int(tmp1[3][6:]))
# extracting t1 from second timestamp without -xxxx
t2 = datetime(int(tmp2[2]), dicto[tmp2[1]], int(tmp2[0]), int(tmp2[3][:2]), int(tmp2[3][3:5]), int(tmp2[3][6:]))
# converting -xxxx of timestamp 1
t1_utc = int(tmp1[4][:3])*3600 + int(tmp1[4][3:])*60
# converting -xxxx of timestamp 2
t2_utc = int(tmp2[4][:3])*3600 + int(tmp2[4][3:])*60
# absolute difference
return abs(int((t1-t2).total_seconds()-(t1_utc-t2_utc)))
if __name__ == '__main__':
# fptr = open(os.environ['OUTPUT_PATH'], 'w')
t = int(input())
for t_itr in range(t):
tmp1 = list(input().split(' '))[1:]
tmp2 = list(input().split(' '))[1:]
delta = time_delta(tmp1, tmp2)
print(delta)
t1_utc = int(tmp1[4][:3])*3600 + int(tmp1[4][3:])*60
For a time zone like +0715, you correctly add “7 hours of seconds” and “15 minutes of seconds”
For a timezone like -0715, you are adding “-7 hours of seconds” and “+15 minutes of seconds”, resulting in -6h45m, instead of -7h15m.
You need to either use the same “sign” for both parts, or apply the sign afterwards.
I am trying to find the time until the next day according to IST. I am using pytz. I am not getting why "tomorrow" is offset naive while tomorrow is defined using offset aware constants.
I've tried printing stuff like print tomorrow_date - now , but that gives 0 and i do not know how is that happening.
from datetime import datetime,timedelta
from pytz import timezone
ist = timezone('Asia/Kolkata')
def get_until_tomorrow():
now = datetime.now(ist)
#today = date.today()
tomorrow_date = now + timedelta(days=1)
tomorrow = datetime.combine(tomorrow_date,time=time(00,00))
seconds_left = tomorrow - now
return seconds_left.seconds
print(get_until_tomorrow())
I am getting the error that I cannot subtract an offset naive and an offset aware object. now is an offset aware object because i set it so directly, but tomorrow is, according to it, an offset naive variable. How is this possible when i have used only offset aware variables to define tomorrow?
Looks like you have to make it offset aware by using localise. See if this helps.
from datetime import datetime,timedelta, time
from pytz import timezone
ist = timezone('Asia/Kolkata')
def get_until_tomorrow():
now = datetime.now(ist)
tomorrow_date = now + timedelta(days=1)
tomorrow = datetime.combine(tomorrow_date,time=time(00,00))
tomorrow = ist.localize(tomorrow)
seconds_left = tomorrow - now
return seconds_left.seconds
print(get_until_tomorrow())
I am using Firebase realtime database and overtime there is a lot of stale data in it and I have written a script to delete the stale content.
My Node structure looks something like this:
store
- {store_name}
- products
- {product_name}
- data
- {date} e.g. 01_Sep_2017
- some_event
Scale of the data
#Stores: ~110K
#Products: ~25
Context
I want to cleanup all the data which is like 30 months old. I tried the following approach :-
For each store, traverse all the products and for each date, delete the node
I ran ~30 threads/script instances and each thread is responsible for deleting a particular date of data in that month. The whole script is running for ~12 hours to delete a month data with above structure.
I have placed a limit/cap on the number of pending calls in each script and it is evident from logging that each script reaches the limit very quickly and speed of firing the delete call is much faster than speed of deletion So here firebase becomes a bottleneck.
Pretty evident that I am running purge script at client side and to gain performance script should be executed close to the data to save network round trip time.
Questions
Q1. How to delete firebase old nodes efficiently ?
Q2. Is there a way we can set a TTL on each node so that it cleans up automatically ?
Q3. I have confirmed from multiple nodes that data has been deleted from the nodes but firebase console is not showing decrease in data. I also tried to take backup of data and it still is showing some data which is not there when I checked the nodes manually. I want to know the reason behind this inconsistency.
Does firebase make soft deletions So when we take backups, data is actually there but is not visible via firebase sdk or firebase console because they can process soft deletes but backups don't ?
Q4. For the whole duration my script is running, I have a continuous rise in bandwidth section. With below script I am only firing delete calls and I am not reading any data still I see a consistency with database read. Have a look at this screenshot ?
Is this because of callbacks of deleted nodes ?
Code
var stores = [];
var storeIndex = 0;
var products = [];
var productIndex = -1;
const month = 'Oct';
const year = 2017;
if (process.argv.length < 3) {
console.log("Usage: node purge.js $beginDate $endDate i.e. node purge 1 2 | Exiting..");
process.exit();
}
var beginDate = process.argv[2];
var endDate = process.argv[3];
var numPendingCalls = 0;
const maxPendingCalls = 500;
/**
* Url Pattern: /store/{domain}/products/{product_name}/data/{date}
* date Pattern: 01_Jan_2017
*/
function deleteNode() {
var storeName = stores[storeIndex],
productName = products[productIndex],
date = (beginDate < 10 ? '0' + beginDate : beginDate) + '_' + month + '_' + year;
numPendingCalls++;
db.ref('store')
.child(storeName)
.child('products')
.child(productName)
.child('data')
.child(date)
.remove(function() {
numPendingCalls--;
});
}
function deleteData() {
productIndex++;
// When all products for a particular store are complete, start for the new store for given date
if (productIndex === products.length) {
if (storeIndex % 1000 === 0) {
console.log('Script: ' + beginDate, 'PendingCalls: ' + numPendingCalls, 'StoreIndex: ' + storeIndex, 'Store: ' + stores[storeIndex], 'Time: ' + (new Date()).toString());
}
productIndex = 0;
storeIndex++;
}
// When all stores have been completed, start deleting for next date
if (storeIndex === stores.length) {
console.log('Script: ' + beginDate, 'Successfully deleted data for date: ' + beginDate + '_' + month + '_' + year + '. Time: ' + (new Date()).toString());
beginDate++;
storeIndex = 0;
}
// When you have reached endDate, all data has been deleted call the original callback
if (beginDate > endDate) {
console.log('Script: ' + beginDate, 'Deletion script finished successfully at: ' + (new Date()).toString());
process.exit();
return;
}
deleteNode();
}
function init() {
console.log('Script: ' + beginDate, 'Deletion script started at: ' + (new Date()).toString());
getStoreNames(function() {
getProductNames(function() {
setInterval(function() {
if (numPendingCalls < maxPendingCalls) {
deleteData();
}
}, 0);
});
});
}
PS: This is not the exact structure I have but it is very similar to what we have (I have changed the node names and tried to make the example a realistic example)
Whether the deletes can be done more efficiently depends on how you now do them. Since you didn't share the minimal code that reproduces your current behavior it's hard to say how to improve it.
There is no support for a time-to-live property on documents. Typically developers do the clean-up in a administrative program/script that runs periodically. The more frequently you run the cleanup script, the less work it has to do, and thus the faster it will be.
Also see:
Delete firebase data older than 2 hours
How to delete firebase data after "n" days
Firebase actually deletes the data from disk when you tell it to. There is no way through the API to retrieve it, since it is really gone. But if you have a backup from a previous day, the data will of course still be there.