I'm facing a huge performance issue, when execute an encryption method. I've written the below code to encrypt a text. When call the method in the code through loop, the execution time was huge as shown below.
Record(s) Time
----------------------
1 0.1 sec
5000 2.8 mins
50,000 28 mins
If I remove the remove the KEY and VECTOR(IV) then the execution time come down to 2 seconds for 50,000 records but, decryption triggers error for key and vector. Can anyone help me out on this.
Code:
public string EncrStng(string tmpText)
{
byte[] bytesBuff = Encoding.Unicode.GetBytes(tmpText);
using (Aes aes = Aes.Create())
{
Rfc2898DeriveBytes crypto = new Rfc2898DeriveBytes(keyF, salt1, 1000);
aes.Key = crypto.GetBytes(32);
aes.IV = crypto.GetBytes(16);
using (MemoryStream mStream = new MemoryStream())
{
using (CryptoStream cStream = new CryptoStream(mStream,aes.CreateEncryptor(), CryptoStreamMode.Write))
{
cStream.Write(bytesBuff, 0, bytesBuff.Length);
cStream.Close();<br> }<br> inText = Convert.ToBase64String(mStream.ToArray());
}
}
return inText;
}
Thanks,
anand
Related
I set up the TiDB, TiKV and PD cluster in order to benchmark them with YCSB tool, connected by the MySQL driver.
The cluster consists of 5 instances for each of TiDB, TiKV and PD.
Each node run a single TiDB, TiKV and PD instance.
However, when I play around the YCSB code in the update statement, I notice that if the value of the updated field is fixed and hardcoded, the total throughput is ~30K tps and the latency at ~30ms. If the updated field value is random, the total throughput is ~2k tps and the latency is around ~300ms.
The update statement creation code is as follow:
#Override
public String createUpdateStatement(StatementType updateType) {
String[] fieldKeys = updateType.getFieldString().split(",");
StringBuilder update = new StringBuilder("UPDATE ");
update.append(updateType.getTableName());
update.append(" SET ");
for (int i = 0; i < fieldKeys.length; i++) {
update.append(fieldKeys[i]);
String randStr = RandomCharStr(); // 1) 3K tps with 300ms latency
//String randStr = "Hardcode-Field-Value"; // 2) 20K tps with 20ms latency
update.append(" = '" + randStr + "'");
if (i < fieldKeys.length - 1) {
update.append(", ");
}
}
// update.append(fieldKey);
update.append(" WHERE ");
update.append(JdbcDBClient.PRIMARY_KEY);
update.append(" = ?");
return update.toString();
}
How do we account for this performance gap?
Is it due to the DistSQL query cache, as discussed in this post?
I manage to figure this out from this post (Same transaction returns different results when i ran multiply times) and pr (https://github.com/pingcap/tidb/issues/7644).
It is because TiDB will not perform the txn if the updated field is identical to the previous value.
I am trying to read and write the samples from a video file at a specific start point and end point for trimming a video. I am using AVAssetReader and AVAssetWriter.
The logic used here is -
STEP A:
Create asset reader instance with the specified asset.
Set the time range to the reader as per the start and end points. ( Say for example - start point = 5, end point = 15, file length = 55 sec )
Start reading the samples.
Get the sample's exact time stamp with respect to start point that we have passed in.
Store the time stamp of the sample that is accurate with the start point.( could be 5.13 or so ). Say ta = 5.13
Release the reader.
STEP B:
Create a new reader instance with the specified asset.
Set the time range to the reader as per the start and end points. ( Say for example - start point = 5, end point = 15, file length = 55 sec )
Start reading the samples.
Create a new sample buffer with sample timing info altered as
( sample buffer's time stamp t1- ta fetched from STEP A ) - This starts to write from 0
( sample buffer's time stamp t2- ta fetched from STEP A )
( sample buffer's time stamp t3- ta fetched from STEP A ) and so on till end point.
Release the reader.
The code sample for the same is:
STEP A:
while ([assetWriterInput isReadyForMoreMediaData] )
{
CMSampleBufferRef sampleBuffer = [assetReaderOutput copyNextSampleBuffer];
if (sampleBuffer != NULL)
{
CMTime originalTime = CMSampleBufferGetPresentationTimeStamp(sampleBuffer);
float time = CMTimeGetSeconds( originalTime );
// This is where we store the time stamp
fTimeTange = time;
[HelpMethods setCorrectEditTime:fTimeTange]; // This is stored globally
// This is to release the readers and writers and start a fresh call with the stored time stamp ta
[delegate resetTimeRange];
return;
}
}
STEP B:
while ([assetWriterInput isReadyForMoreMediaData] )
{
CMSampleBufferRef sampleBuffer = [assetReaderOutput copyNextSampleBuffer];
if (sampleBuffer != NULL)
{
CMSampleBufferRef finalBuffer = sampleBuffer;
CMSampleBufferRef newSampleBuffer;
CMSampleTimingInfo sampleTimingInfo;
CMTime cmm1 = CMSampleBufferGetOutputDuration(sampleBuffer);
CMTime originalTime = CMSampleBufferGetPresentationTimeStamp(sampleBuffer);
float time = CMTimeGetSeconds( originalTime );
// This is a helper method to get the stored ta at STEP A
fTimeTange = [HelpMethods getCorrectEditTime];
sampleTimingInfo.duration = cmm1;
float milliseconds = (time - fTimeTange) * 600;
NSLog( #"Timestamp in milliseconds = %f", milliseconds );
sampleTimingInfo.presentationTimeStamp = CMTimeMake(milliseconds, 600);
sampleTimingInfo.decodeTimeStamp = kCMTimeInvalid;
CMSampleBufferCreateCopyWithNewTiming(kCFAllocatorDefault,
sampleBuffer,
1,
&sampleTimingInfo,
&newSampleBuffer);
finalBuffer = newSampleBuffer;
BOOL success = YES;
success = [assetWriterInput appendSampleBuffer:finalBuffer];
CFRelease(sampleBuffer);
sampleBuffer = NULL;
}
Since the time stamps of the samples are read in out of order fashion, we end up getting an error saying
"(kFigFormatWriterError_InvalidTimestamp) (decode timestamp is less than previous sample's decode timestamp)"
and the values of time stamps are -
Timestamp in milliseconds = 0.000000
Timestamp in milliseconds = 79.999924
Timestamp in milliseconds = 39.999962
Timestamp in milliseconds = 119.999886
Timestamp in milliseconds = 200.000092
Timestamp in milliseconds = 160.000137
Timestamp in milliseconds = 280.000031
Timestamp in milliseconds = 240.000061
Timestamp in milliseconds = 319.999969
Timestamp in milliseconds = 399.999908
Timestamp in milliseconds = 359.999939
and so on
Any manipulation done to the presentation stamps results in out of order reading of samples.
Looking out a way to overcome this out of order reading of timestamps. Please let know.
Thanks in advance,
Champa
I am trying to perform incremental backup , I have already checked Export option but couldn't figure out start time option.Also please suggest on CopyTable , how can I restore.
Using CopyTable you just receive copy of given table on the same or another cluster (actually CopyTable MapReduce job). No miracle.
Its your own decision how to restore. Obvious options are:
Use the same tool to copy table back.
Just get / put selected rows (what I think you need here). Please pay attention you should keep timestamps while putting data back.
Actually for incremental backup it's enough for you to write job which scans table and gets/puts rows with given timestamps into table with the name calculated by date. Restore should work in reverse direction - read table with calculated name and put its record with the same timestamp.
I'd also recommend to you following technique: table snapshot (CDH 4.2.1 uses HBase 0.94.2). It looks not applicable for incremental backup but maybe you find something useful here like additional API. From the point of view of backup now it looks nice.
Hope this will help somehow.
The source code suggests
int versions = args.length > 2? Integer.parseInt(args[2]): 1;
long startTime = args.length > 3? Long.parseLong(args[3]): 0L;
long endTime = args.length > 4? Long.parseLong(args[4]): Long.MAX_VALUE;
The accepted answer doesn't pass version as a parameter. How did it work then?
hbase org.apache.hadoop.hbase.mapreduce.Export test /bkp_destination/test 1369060183200 1369063567260023219
From source code this boils down to -
1369060183200 - args[2] - version
1369063567260023219 - args[3] - starttime
Attaching source for ref:
private static Scan getConfiguredScanForJob(Configuration conf, String[] args) throws IOException {
Scan s = new Scan();
// Optional arguments.
// Set Scan Versions
int versions = args.length > 2? Integer.parseInt(args[2]): 1;
s.setMaxVersions(versions);
// Set Scan Range
long startTime = args.length > 3? Long.parseLong(args[3]): 0L;
long endTime = args.length > 4? Long.parseLong(args[4]): Long.MAX_VALUE;
s.setTimeRange(startTime, endTime);
// Set cache blocks
s.setCacheBlocks(false);
// set Start and Stop row
if (conf.get(TableInputFormat.SCAN_ROW_START) != null) {
s.setStartRow(Bytes.toBytesBinary(conf.get(TableInputFormat.SCAN_ROW_START)));
}
if (conf.get(TableInputFormat.SCAN_ROW_STOP) != null) {
s.setStopRow(Bytes.toBytesBinary(conf.get(TableInputFormat.SCAN_ROW_STOP)));
}
// Set Scan Column Family
boolean raw = Boolean.parseBoolean(conf.get(RAW_SCAN));
if (raw) {
s.setRaw(raw);
}
if (conf.get(TableInputFormat.SCAN_COLUMN_FAMILY) != null) {
s.addFamily(Bytes.toBytes(conf.get(TableInputFormat.SCAN_COLUMN_FAMILY)));
}
// Set RowFilter or Prefix Filter if applicable.
Filter exportFilter = getExportFilter(args);
if (exportFilter!= null) {
LOG.info("Setting Scan Filter for Export.");
s.setFilter(exportFilter);
}
int batching = conf.getInt(EXPORT_BATCHING, -1);
if (batching != -1){
try {
s.setBatch(batching);
} catch (IncompatibleFilterException e) {
LOG.error("Batching could not be set", e);
}
}
LOG.info("versions=" + versions + ", starttime=" + startTime +
", endtime=" + endTime + ", keepDeletedCells=" + raw);
return s;
}
Found out the issue here, the hbase documentation says
hbase org.apache.hadoop.hbase.mapreduce.Export <tablename> <outputdir> [<versions> [<starttime> [<endtime>]]]
so after trying a few of combinations, I found out that it is converted to a real example like below code
hbase org.apache.hadoop.hbase.mapreduce.Export test /bkp_destination/test 1369060183200 1369063567260023219
where
test is tablename,
/bkp_destination/test is backup destination folder,
1369060183200 is starttime,
1369063567260023219 is endtime
I'm not a groovy expert, just use it from time to time. One of the latest goals was to generate a very simple file containing some random data. I created the following script:
out = new File('sampledata.txt')
Random random = new Random();
java.util.Date dt = new java.util.Date();
for (int i=0; i<100000; ++i) {
dt = new java.util.Date();
out << dt.format('yyyMMdd HH:mm:ss.SSS') + '|box|process|||java.lang.Long|' + random.nextInt(100) + '|name\n'
}
Now, I'm really puzzled with its performance. It takes around 1.5 minutes to complete whilst the same code written in Java or Ruby takes less than a second.
Similar code in Ruby (takes around 1 second to execute):
require "time"
File.open("output.txt", "w") do |file|
100000.times do
line = Time.now.strftime("%Y%m%d %H:%M:%S.%L") + '|box|process|||java.lang.Long|' + rand(100).to_s + '|name'
file.puts line
end
end
Any ideas how groovy's processing speed could be improved?
The left shift operator opens the file, jumps to the end, appends the text, and closes the file again...
Instead, try:
Random random = new Random();
// Open the file and append to it.
// If you want a new file each time, use withWriter instead of withWriterAppend
new File('sampledata.txt').withWriterAppend { w ->
100000.times {
w.writeLine "${new Date().format('yyyMMdd HH:mm:ss.SSS')}|box|process|||java.lang.Long|${random.nextInt(100)}|name"
}
}
(this is also much more like what the Ruby code is doing)
I appreciate that this issue has been raised a couple of times before, but I can't find a definitive answer (maybe there isn't one!).
Anyway the title tells it all really. Create a new context, add a new entity, SaveChanges() takes 20 seconds. Add second entity in same context, SaveChanges() instant.
Any thoughts on this? :-)
============ UPDATE =============
I've created a very simple app running against my existing model to show the issue...
public void Go()
{
ModelContainer context = new ModelContainer(DbHelper.GenerateConnectionString());
for (int i = 1; i <= 5; i++)
{
DateTime start = DateTime.Now;
Order order = context.Orders.Single(c => c.Reference == "AA05056");
DateTime end = DateTime.Now;
double millisecs = (end - start).TotalMilliseconds;
Console.WriteLine("Query " + i + " = " + millisecs + "ms (" + millisecs / 1000 + "s)");
start = DateTime.Now;
order.Note = start.ToLongTimeString();
context.SaveChanges();
end = DateTime.Now;
millisecs = (end - start).TotalMilliseconds;
Console.WriteLine("SaveChanges " + i + " = " + millisecs + "ms (" + millisecs / 1000 + "s)");
Thread.Sleep(1000);
}
Console.ReadKey();
}
Please do not comment on my code - unless it is an invalid test ;)
The results are:
Query 1 = 3999.2288ms (3.9992288s)
SaveChanges 1 = 3391.194ms (3.391194s)
Query 2 = 18.001ms (0.018001s)
SaveChanges 2 = 4.0002ms (0.0040002s)
Query 3 = 14.0008ms (0.0140008s)
SaveChanges 3 = 3.0002ms (0.0030002s)
Query 4 = 13.0008ms (0.0130008s)
SaveChanges 4 = 3.0002ms (0.0030002s)
Query 5 = 10.0005ms (0.0100005s)
SaveChanges 5 = 3.0002ms (0.0030002s)
The first query takes time which I assume is the view generation? Or db connection?
The first save takes nearly 4 seconds which for the more complex save in my app takes over 20 seconds which is not acceptable.
Not sure where to go with this now :-(
UPDATE...
SQL Profiler shows first query and update are fast and are not different for first. So I know delay is Entity Framework as suspected.
It might not be the SaveChanges call - the first time you make any call to the database in EF, it has to do some initial code generation from the metadata. You can pre-generate this though at compile-time: http://msdn.microsoft.com/en-us/library/bb896240.aspx
I would be surprised if that's the only problem, but it might help.
Also have a look here: http://msdn.microsoft.com/en-us/library/cc853327.aspx
I would run the following code on app start up and see how long it takes and if after that the first SaveChanges is fast.
public static void UpdateDatabase()
{
//Note: Using SetInitializer is reconnended by Ladislav Mrnka with reputation 275k
//http://stackoverflow.com/questions/9281423/entity-framework-4-3-run-migrations-at-application-start
Database.SetInitializer<DAL.MyDbContext>(
new MigrateDatabaseToLatestVersion<DAL.MyDbContext,
Migrations.MyDbContext.Configuration>());
using (var db = new DAL.MyDbContext()) {
db.Database.Initialize(false);//Execute the migrations now, not at the first access
}
}