I found that the query is parallel, but in practical applications, it is found that the time consumption is the total serial time consumption. So I want to ask whether the query is parallel or serial.
enter image description here
Can anyone help? Thank you!
Related
I am trying to collect the following data using Veins towards each single lane, including throughput, density, mean speed, delay and collision. I know TraCI has the Simulation Value Retrieval, which can provide some information that I need. Also, the Lane Value Retrieval can help. But I have no clue where should I put the customized codes, so that the statistics can be recorded properly. For example, I want to collect the density and the mean speed of each lane every minute of the simulation time, which class should I put my codes to? TraCISenarioManager?
Any suggestion is appreciated.
I think putting the code in TraCIScenarioManager is entirely reasonable. If you want per-vehicle statistics I'd recommend putting them in the vehicles' application code, the way VEINS already collects some statistics out of the box.
Hello StackEx community.
I am implementing certain scenarios in Veins 3.0 and I wish to collect certain traffic statistics such as the Average Waiting Time, the Average Energy consumption, etc from my simulation.
Please help on how to generate and interpret these information.
Thanks
TraCIMobility already records some statistics that you can directly use or build on. See, for example totalCO2Emission. Other statistics you might have to implement yourself, e.g., after detecting that a car was stopped for a certain time. See the OMNeT++ user manual pages on result recording and analysis for general information on how to do that and the Tic Toc tutorial for a concrete example.
I want to find the expected run time of query in Hive. Using EXPLAIN gives the execution plan. Is there a way to find the expected time?
I need Hive equivalent of SQL query EXPLAIN COSTS .
There is no OOTB feature at this moment that facilitates this. One way to achieve this would be to learn from history. Gather patterns based on similar data and queries you have run previously and try to deduce some insights. You might find tools like Starfish helpful in the process.
I would not recommend you to decide anything based on a subset of your data, as running queries on a small dataset and on the actual dataset are very different. This is good to test the functionality but not for any kind of cost approximation. The reason behind this is that a lot of factors are involved in the process, like system resources(disk, CPU slots, N/W etc), system configuration, other running jobs etc. You might find smooth operation on a small dataset, but as the data size increases all these factors start playing much important role. Even a small configuration parameter may play an important role.(You might have noticed sometimes that a Hive query runs fast initially but starts getting slow gradually). Also, execution of a Hive query is much more involved than a simple MR job.
See this JIRA, to get some idea, where they are talking about developing a Cost Based Query optimization for Joins in Hive. You might also find this helpful.
I think that is not possible to because internally map reduce gets executed for any particular Hive query. Moreover map reduce job's execution time depends on the cluster load and its configuration. So it is tough to predict the execution time. May be you can do one thing you can use some timer before running the query and then after that finishes up you can calculate the exact execution time that was needed for execution.
May be you could sample a small % of records from your table using partitions , bucket features etc then run the query against the small dataset. Note the execution time and then multiply with the factor (total_size/sample_size).
For a business use case where we have to deal with minimum "2-3 terabyte" of data per day, I was doing analysis on "Hadoop & Storm".
Needless to say that “Storm” looks impressive because of its efficiency in processing incoming big data but I am not sure whether “Storm” will be capable enough to process “Terabyte” of data and at the same time providing me real-time results or not ?
Can anyone explain please?
Thanks,
Gajendra
Storm was developed by twitter. they process more than 8 TB per day with it. Sounds like this should be enough for your case. Afaik storm is the best streaming/realtime system for distributed computing. hadoop is not suitable for it due to job start up times and not native handling of streaming data.
a fact is, both can handle the data per day you wish when you have enough server power and storage etc.
Calling all database guys...
The situation is this:
I have a DB2 database that is being written to and read from. I need to do some performance testing on programmatically executed read/writes.
I know how to write a program to read/write to this database, but I am not sure as to what factors I should consider in my performance test.
Do I need to worry about the difference between one session reading/writing vs multiple sessions?
What is the best way to interact with DB2 itself to get the amount of time these executions take?
The process I am testing is basically like a continuous batch proccess, constantly taking messages and persisting them. There will probably only be one or two sessions max on the DB at any given time.
Is time it takes to read/write really the best metric?
I am sure there are plenty of tools for this sort of testing. Any advice is appreciated.
Further info:
One thing I am considering is to try is to run X number of reads/writes with my database API (homebrew) and try to "time" how long it takes. Unfortuneately DB2 will buffer these messages. Is there any way to get DB2 to do a callback when it is done with a read/write? Or some way to externally measure the time these operations take? (tool, etc)
What is the goal for your performance testing?. Is it to test the performance for concurrent users or is it to test the load for batch process. Based on this there are tools available to test this. You may want to look jmeter from Apache.
In that case, you may want to trigger couple of concurrent processes to simaltaneously CRUD the data and monitor the activity using performance expert or something similar to that. While you do that you may want to use larger output so that you would be able to find any bottlenecks with larger sets of data. search for performance tuning in IBM redbooks site and you will find some case studies for this.
One huge factor in DB2 performance is how Buffer Pools are configured. e.g. http://www.ibm.com/developerworks/data/library/techarticle/0212wieser/0212wieser.html