Consider the following toy example:
use https://data.princeton.edu/pop509/justices2.dta, clear
stset tenure, fail(event == 1)
stcrreg age year, compete (event == 2)
stcurve, cif
I want to plot a cumulative incidence curve as done above but then I want to store those values with their 95% confidence intervals. However, it is not clear to me how to access/store them as variables.
Cross-posted at Statalist.
Use the outfile() option of the stcurve command:
stcurve, cif outfile(stdata)
use stdata
list in 1/10
+---------------------+
| ci1 _t |
|---------------------|
1. | .0465373 5.691992 |
2. | 0 1.045859 |
3. | .2600816 20.6078 |
4. | .1169629 8.876112 |
5. | .0465373 5.724846 |
|---------------------|
6. | .1249585 9.440109 |
7. | 0 .4462697 |
8. | .1574731 13.49213 |
9. | .1991083 15.36756 |
10. | .0232038 4.769336 |
+---------------------+
Related
I have a table abcd in Oracle DB
+-------------+----------+
| abcd.speed | abcd.ab |
+-------------+----------+
| 4.0 | 2 |
| 4.0 | 2 |
| 7.0 | 2 |
| 7.0 | 2 |
| 8.0 | 1 |
+-------------+----------+
And I'm using a query like this:
select min(speed) keep (dense_rank last order by abcd.ab NULLS FIRST) MOD from abcd;
I'm trying to convert the code to Hive, but it looks like keep is not available in Hive.
Could you suggest an equivalent statement?
select -max(struct(ab,-speed)).col2 as mod
from abcd
;
+------+
| mod |
+------+
| 4.0 |
+------+
Let start by explaining min(speed) keep (dense_rank last order by abcd.ab NULLS FIRST):
Find the row(s) with the max value of ab.
For this/those row(s), find the min value of speed.
We are using 2 tricks here.
The 1st is based on the ability to get the max value of a struct.
max(struct(c1,c2,c3,...)) returns the same result as if you have sorted the structs by c1, then by c2, then by c3 etc. and then chose the last element.
The 2nd trick is to use -speed (which is the same of -1*speed).
Finding the max of -speed and then taking the minus of that value (which gives us speed), is the same of finding the min of speed.
If we would have ordered the structs, it would have looked like this (since 2 is bigger than 1 and -4 is bigger than -7):
+----+-------+
| ab | speed |
+----+-------+
| 1 | -8.0 |
| 2 | -7.0 |
| 2 | -7.0 |
| 2 | -4.0 |
| 2 | -4.0 |
+----+-------+
The last struct in this case in struct(2,-4.0), therefore this is the result of the max function.
The fields names for a struct are col1, col2, col3 etc., so
struct(2,-4.0).col2 is -4.0. and preceding it with minus (which is the same as multiple it by -1) as in -struct(2,-4.0).col2 is 4.0.
I am trying to expand a data set to include dates outside of the current range.
The data I have ranges from 1992q1 to 2017q1. Each observation exists within a portion of that larger window, for example from 1993q2 to 1997q1.
I need to create quarterly observations for each range to fill the missing time. I have already expanded the existing data into quarters.
What I cannot figure out how to do is add in those missing quarters. For example, country1 may have the dates 1993q2 to 1997q1. I need to add in the missing dates from 1992q1 to 1993q1 and 1997q2 to 2017q1.
A very simple analogue of I want I think is your question is shown by this sandbox dataset.
clear
set obs 10
gen id = cond(_n < 7, 1, 2)
gen qdate = yq(1992, 1) in 1
replace qdate = yq(1992, 3) in 7
bysort id (qdate) : replace qdate = qdate[_n-1] + 1 if missing(qdate)
format qdate %tq
list, sepby(id)
+-------------+
| id qdate |
|-------------|
1. | 1 1992q1 |
2. | 1 1992q2 |
3. | 1 1992q3 |
4. | 1 1992q4 |
5. | 1 1993q1 |
6. | 1 1993q2 |
|-------------|
7. | 2 1992q3 |
8. | 2 1992q4 |
9. | 2 1993q1 |
10. | 2 1993q2 |
+-------------+
fillin id qdate
list, sepby(id)
+-----------------------+
| id qdate _fillin |
|-----------------------|
1. | 1 1992q1 0 |
2. | 1 1992q2 0 |
3. | 1 1992q3 0 |
4. | 1 1992q4 0 |
5. | 1 1993q1 0 |
6. | 1 1993q2 0 |
|-----------------------|
7. | 2 1992q1 1 |
8. | 2 1992q2 1 |
9. | 2 1992q3 0 |
10. | 2 1992q4 0 |
11. | 2 1993q1 0 |
12. | 2 1993q2 0 |
+-----------------------+
So. fillin is a simple way of ensuring that all cross-combinations of identifier and time are present. However, to what benefit? Although not shown in this example, values of other variables spring into existence only as missing values. In some situations, proceeding with interpolation is justified, but usually, you just live with incomplete panels.
How to find solutions like these? One good strategy is to skim through the [D] manual to see what basic data management commands exist.
I have two tables one includes about 17K (NLIST) records while the other 57K (FNAMES).
I would like to join the both by comparing the records using levenshtein formula.
Here is the example for the content of tables:
Table NLIST:
+------+-------------+
| ID | S_NAME |
+------+-------------+
| 1 | Avi |
| 2 | Moshe |
| 3 | David |
....
Table FNAMES:
+------+-------------+
| ID | NICKNAMES |
+------+-------------+
| 1 | Avile |
| 2 | Dudi |
| 3 | Moshiko |
| 4 | Avi |
| 5 | DAVE |
....
The above tables are just examples. In the real case the names column can include more than one word.
The required result should be:
+------+-------------+--------+
| ID | NICKNAMES | S_NAME |
+------+-------------+--------+
| 1 | Avile | Avi |
| 2 | Dudi | David |
| 3 | Moshiko | Moshe |
| 4 | Avi | Avi |
| 5 | DAVE | David |
...
Here is the code I use:
select FNAMES.NICKNAMES, NLIST.S_NAME
from NICKNAMES
LEFT OUTER JOIN NLIST
ON(true)
WHERE levenshtein (FNAMES.NICKNAMES, NLIST.S_NAME) <=4
The above code runs for a very long time and I stopped its running.
How can I make it run in a reasonable time?
In addition, I think the levenshtein distance depends on the length of the words. How can I find the optimal value for the distance (in this case I chose 4 arbitrarily)?
Hive Table performance is depends upon various point .
Query enginee
File format
use VECTORIZATION set hive.vectorized.execution.enabled = true;set hive.vectorized.execution.reduce.enabled = true;
If you have good server you can try with Impala and definitely it is faster than Hive.
You can do the fine tuning of impala which will give you an edge to execute this query faster .Tuning Impala for Performance
Assuming I have a table with two columns CUSTTYPE and AMOUNT. I want to add a third column NTILE which I can then group on and use to get my averages, something like below:
CUSTTYPE | AMOUNT | NTILE
----------+---------+----------
RETAIL | 78.00 | 1
RETAIL | 234.00 | 1
RETAIL | 249.00 | 1
RETAIL | 278.00 | 2
RETAIL | 392.00 | 2
RETAIL | 498.00 | 2
RETAIL | 500.00 | 3
RETAIL | 738.00 | 3
RETAIL | 1250.00 | 3
RETAIL | 2029.00 | 4
RETAIL | 2393.00 | 4
RETAIL | 3933.00 | 4
Essentially, I am trying to take the average of every n terms (here, n=3):
CUSTTYPE | AMOUNT | NTILE
----------+---------+----------
RETAIL | 187.00 | 1
RETAIL | 389.33 | 2
RETAIL | 829.33 | 3
RETAIL | 2785.0 | 4
From the Pig reference here, it seems this could be achieved using Over() but I could not find an example of how this could be done. Thoughts?
You can rank every record of your data using RANK operator:
http://pig.apache.org/docs/r0.14.0/basic.html#rank
like this:
A = LOAD 'path' AS (schema);
B = RANK A;
and then divide each rank by 3:
C = FOREACH B generate ($0 + 1) / 3 as NTILE, CUSTTYPE, AMOUNT;
i need help with a query, can't get it to work correctly. What i'm trying to achieve is to have a select box displaying the number of records associated with a particular theme, for some theme it works well for some it displays (0) when infact there are 2 records, I'm wondering if someone could help me on this, your help would be greatly appreciated, please see below my actual query + table structure :
SELECT theme.id_theme, theme.theme, calender.start_date,
calender.id_theme1,calender.id_theme2, calender.id_theme3, COUNT(*) AS total
FROM theme, calender
WHERE (YEAR(calender.start_date) = YEAR(CURDATE())
AND MONTH(calender.start_date) > MONTH(CURDATE()) )
AND (theme.id_theme=calender.id_theme1)
OR (theme.id_theme=calender.id_theme2)
OR (theme.id_theme=calender.id_theme3)
GROUP BY theme.id_theme
ORDER BY theme.theme ASC
THEME table
|---------------------|
| id_theme | theme |
|----------|----------|
| 1 | Yoga |
| 2 | Music |
| 3 | Taichi |
| 4 | Dance |
| 5 | Coaching |
|---------------------|
CALENDAR table
|---------------------------------------------------------------------------|
| id_calender | id_theme1 | id_theme2 | id_theme3 | start_date | end_date |
|-------------|-----------|-----------|-----------|------------|------------|
| 1 | 2 | 4 | | 2015-07-24 | 2015-08-02 |
| 2 | 4 | 1 | 5 | 2015-08-06 | 2015-08-22 |
| 3 | 1 | 3 | 2 | 2014-10-11 | 2015-10-28 |
|---------------------------------------------------------------------------|
LISTBOX
|----------------|
| |
| Yoga (1) |
| Music (1) |
| Taichi (0) |
| Dance (2) |
| Coaching (1) |
|----------------|
Thanking you in advance
I think that themes conditions should be into brackets
((theme.id_theme=calender.id_theme1)
OR (theme.id_theme=calender.id_theme2)
OR (theme.id_theme=calender.id_theme3))
Hope this help