Cumulative counting values from files - bash

I need to extract numbers from several files into a cumulative variable and print that variable for each parent directory as shown
├───Parent1
│ ├───20210824_2000
│ │ ├───200000_child1
│ │ │ report.md
│ │ │ log.log
│ │ │ file.xml
│ │ │ input.json
│ │ │
│ │ ├───200030_child2
│ │ │
│ │ ├───200034_child3
│ │ │
│ │ ├───200039_child4
│ │ ...
│ ├───20210825_0800
│ │ ├───200000_child1
│ │ │ report.md
│ │ │ log.log
│ │ │ file.xml
│ │ │ input.json
│ │ │
│ │ ├───200030_child2
│ │ │
│ │ ├───200034_child3
│ │ │
│ │ ├───200039_child4
│ │ ...
│ ...
├───Parent2
│ ├───20210824_2000
│ │ ├───200000_child1
│ │ │ report.md
│ │ │ log.log
│ │ │ file.xml
│ │ │ input.json
│ │ │
│ │ ├───200030_child2
│ │ │
│ │ ├───200034_child3
│ │ │
│ │ ├───200039_child4
│ │ ...
│ ├───20210825_0800
│ │ ├───200000_child1
│ │ │ report.md
│ │ │ log.log
│ │ │ file.xml
│ │ │ input.json
│ │ │
│ │ ├───200030_child2
│ │ │
│ │ ├───200034_child3
│ │ │
│ │ ├───200039_child4
│ │ ...
│ ...
...
I can't seem to extract the grep output into a numeric variable. The child folder has a timestamp attached so I sort since I only want the latest files.
Here's what I have so far:
#!/bin/bash
find . -type d -iname 'parent*' | while read -r dir; do
sum=0;
find "$dir" -maxdepth 1 -type d | sort -r | head -1 |
(
while read -r subdir; do
count="$(find "$subdir" -type f -iname '*report.md' -exec grep -ohP '(?<=\*)\d+(?=\*+ number of things)' {} \+)"
sum=$((sum + count))
done
basename "$dir" "$sum"
)
done
But this doesn't seem to want to add count to sum it rather just prints count to the console for each file.

There is a problem due to the sub shell that is created by each while, so variable are not global, you can find useful information in "Shell variables set inside while loop not visible outside of it"
Also count variable could have problem because find should returns multiple line.
Try with this:
#!/bin/bash
find . -type d -iname 'parent*' | while read -r dir; do
sum=0;
while read -r subdir; do
while read report; do
count="$(grep -ohP '(?<=\*)\d+(?=\*+ number of things)' $report)"
sum=$((sum + count))
done < <(find "$subdir" -type f -iname '*report.md')
done < <(find "$dir" -maxdepth 1 -type d | sort -r | head -1)
folder=$(basename $dir)
echo "$folder $sum"
done
Pay attention to the spaces in done < <(.
I have not tested, sorry.

Related

Include a static library in Visual Studio Code debugger in C

I’m trying to debug some C code, and I’ve got an issue to include my static library. Here is the error message:
Here is a tree of my working directory:
.
├── Makefile
├── include
│ ├── my.h
│ └── script.h
├── lib
│ └── my
│ ├── Makefile
│ ├── include
│ │ ├── my.h
│ │ └── my_printf.h
│ ├── libmy.a
│ └── src
│ ├── my.h
│ ├── my_compute_power_rec.c
│ ├── my_compute_power_rec.o
│ ├── my_compute_square_root.c
│ ├── my_compute_square_root.o
│ ├── my_find_prime_sup.c
│ ├── my_find_prime_sup.o
│ ├── my_getnbr.c
│ ├── my_getnbr.o
│ ├── my_is_prime.c
│ ├── my_is_prime.o
│ ├── my_isneg.c
│ ├── my_isneg.o
│ ├── my_printf.c
│ ├── my_printf.h
│ ├── my_printf.o
│ ├── my_printf_base_process.c
│ ├── my_printf_base_process.o
│ ├── my_printf_flags.c
│ ├── my_printf_flags.o
│ ├── my_printf_identifiers.c
│ ├── my_printf_identifiers.o
│ ├── my_put_error.c
│ ├── my_put_error.o
│ ├── my_put_long_nbr.c
│ ├── my_put_long_nbr.o
│ ├── my_put_nbr.c
│ ├── my_put_nbr.o
│ ├── my_putchar.c
│ ├── my_putchar.o
│ ├── my_putstr.c
│ ├── my_putstr.o
│ ├── my_revstr.c
│ ├── my_revstr.o
│ ├── my_showmem.c
│ ├── my_showmem.o
│ ├── my_showstr.c
│ ├── my_showstr.o
│ ├── my_sort_int_array.c
│ ├── my_sort_int_array.o
│ ├── my_str_append_index.c
│ ├── my_str_append_index.o
│ ├── my_str_is_letter.c
│ ├── my_str_is_letter.o
│ ├── my_str_isalpha.c
│ ├── my_str_isalpha.o
│ ├── my_str_islower.c
│ ├── my_str_islower.o
│ ├── my_str_isnum.c
│ ├── my_str_isnum.o
│ ├── my_str_isprintable.c
│ ├── my_str_isprintable.o
│ ├── my_str_isupper.c
│ ├── my_str_isupper.o
│ ├── my_str_to_word_array.c
│ ├── my_str_to_word_array.o
│ ├── my_strcapitalize.c
│ ├── my_strcapitalize.o
│ ├── my_strcat.c
│ ├── my_strcat.o
│ ├── my_strcmp.c
│ ├── my_strcmp.o
│ ├── my_strcpy.c
│ ├── my_strcpy.o
│ ├── my_strlen.c
│ ├── my_strlen.o
│ ├── my_strlowcase.c
│ ├── my_strlowcase.o
│ ├── my_strncat.c
│ ├── my_strncat.o
│ ├── my_strncmp.c
│ ├── my_strncmp.o
│ ├── my_strncpy.c
│ ├── my_strncpy.o
│ ├── my_strstr.c
│ ├── my_strstr.o
│ ├── my_strupcase.c
│ ├── my_strupcase.o
│ ├── my_swap.c
│ └── my_swap.o
├── lib_file
├── new_lib_file
├── script
├── src
│ ├── main.c
│ ├── main.o
│ ├── script.c
│ └── script.o
└── tests
└── test.c
And here is my tasks.json file:
{
"tasks": [
{
"type": "cppbuild",
"label": "C/C++: gcc.exe générer le fichier actif",
"command": "C:\\MinGW\\bin\\gcc.exe",
"args": [
"-fdiagnostics-color=always",
"-g",
"${workspaceFolder}\\lib\\my\\libmy.a",
"${workspaceFolder}\\src\\main.c",
"${workspaceFolder}\\src\\script.c",
"-o",
"${fileDirname}\\${fileBasenameNoExtension}.exe",
"-I",
"${workspaceFolder}\\include",
],
"options": {
"cwd": "${fileDirname}"
},
"problemMatcher": [
"$gcc"
],
"group": {
"kind": "build",
"isDefault": true
},
"detail": "Tâche générée par le débogueur."
}
],
"version": "2.0.0"
}
I don't really understand why isn't it working.
I run my code on Windows WSL and I’m trying to debug it from Windows Visual Studio Code, so maybe there is a point here.
Is there something wrong?

select group by dimension and accumulate column percent

I've using ClickHouse and I have this table
| URL | visits |
| ------------- |:------------- |
| URL1 | 5 |
| URL2 | 30 |
| URL3 | 1 |
| URL4 | 30 |
| URL5 | 9 |
| URL1 | 5 |
| URL2 | 20 |
I can group by url,
select
url,
sum(visits) as visits
from
database.tableVistis
group by
url
| URL | visits |
| ------------- |:------------- |
| URL1 | 10 |
| URL2 | 50 |
| URL3 | 1 |
| URL4 | 30 |
| URL5 | 9 |
And I want this result (group by url, % over total of visits, and % sum accumulate)
| URL | visits | % | Accumulate |
| ------------- |:------------- |----------|------------|
| URL2 | 50 |50% | 50% |
| URL4 | 30 |30% | 80% |
| URL1 | 10 |10% | 90% |
| URL5 | 9 |9% | 99% |
| URL3 | 1 |1% | 100% |
Any idea?
Thanks!!
Try this query:
SELECT result.1 AS URL, result.2 AS visits, round(result.3,2) AS "%", round(result.4, 2) AS Accumulate
FROM (
SELECT
groupArray((URL, visits)) url_visits,
arraySum(x -> x.2, url_visits) total_visits,
arrayMap(x -> (100 / total_visits) * x.2, url_visits) percent_visits,
arrayCumSum(percent_visits) acc_percent_visits,
arrayJoin(arrayMap((x, y, z) -> (x.1, x.2, y, z), url_visits, percent_visits, acc_percent_visits)) result
FROM (
SELECT URL, sum(visits) visits
FROM (
/* test data */
SELECT data.1 URL, data.2 visits
FROM (
SELECT arrayJoin([
('URL1', 5 ),
('URL2', 30),
('URL3', 1 ),
('URL4', 30),
('URL5', 9 ),
('URL1', 5 ),
('URL2', 20)]) data))
GROUP BY URL
ORDER BY visits DESC))
/* result
┌─URL──┬─visits─┬──%─┬─Accumulate─┐
│ URL2 │ 50 │ 50 │ 50 │
│ URL4 │ 30 │ 30 │ 80 │
│ URL1 │ 10 │ 10 │ 90 │
│ URL5 │ 9 │ 9 │ 99 │
│ URL3 │ 1 │ 1 │ 100 │
└──────┴────────┴────┴────────────┘
*/
Yea! Thanks!!
SELECT
result.1 AS URL,
result.2 AS visits,
round(result.3,
2) AS "%",
round(result.4,
2) AS Accumulate
FROM
(
SELECT
groupArray((URL,
visits)) url_visits,
arraySum(x -> x.2,
url_visits) total_visits,
arrayMap(x -> (100 / total_visits) * x.2,
url_visits) percent_visits,
arrayCumSum(percent_visits) acc_percent_visits,
arrayJoin(arrayMap((x,
y,
z) -> (x.1,
x.2,
y,
z),
url_visits,
percent_visits,
acc_percent_visits)) result
FROM
(
SELECT
URL,
sum(visits) visits
FROM
(/* test data */
SELECT
URL,
visits
FROM
(
select
landing as URL,
visitas as visits
from
Analytics
where
fecha > '2019-05-01'))
GROUP BY
URL
ORDER BY
visits DESC))
This query is more compact:
SELECT
result.1 AS URL,
result.2 AS visits,
round(result.3, 2) AS `%`,
round(result.4, 2) AS Accumulate
FROM
(
SELECT
groupArray((URL, visits)) AS url_visits,
arraySum(x -> (x.2), url_visits) AS total_visits,
arrayMap(x -> ((100 / total_visits) * (x.2)), url_visits) AS percent_visits,
arrayCumSum(percent_visits) AS acc_percent_visits,
arrayJoin(arrayMap((x, y, z) -> (x.1, x.2, y, z), url_visits, percent_visits, acc_percent_visits)) AS result
FROM
(
SELECT
landing AS URL,
sum(visitas) AS visits
FROM Analytics
WHERE fecha > '2019-05-01'
GROUP BY URL
ORDER BY visits DESC
)
)

How to fill dates in between in subquery?

I have got a table with purchases:
player_id | date | registration_date | price
pl1 | 2019-01-21 | 2019-01-20 | 20
pl1 | 2019-01-23 | 2019-01-20 | 10
pl1 | 2019-01-24 | 2019-01-20 | 15
After calculations with groupArray on 'date' and arrayCumSum on 'price' and using ArrayJoin I got the table with cumulative sum on each day:
player_id | date | registration_date | sum_price
pl1 | 2019-01-21 | 2019-01-20 | 20
pl1 | 2019-01-23 | 2019-01-20 | 30
pl1 | 2019-01-24 | 2019-01-20 | 45
But I need to add missing dates since registration until today (today is '2019-01-25'):
player_id | date | registration_date | sum_price
pl1 | 2019-01-20 | 2019-01-20 | 0
pl1 | 2019-01-21 | 2019-01-20 | 20
pl1 | 2019-01-22 | 2019-01-20 | 20
pl1 | 2019-01-23 | 2019-01-20 | 30
pl1 | 2019-01-24 | 2019-01-20 | 45
pl1 | 2019-01-25 | 2019-01-20 | 45
How can I do it?
Try this one:
SELECT player_id, result.1 as date, registrationDate as registration_date, result.2 as sum_price
FROM
(
SELECT
player_id,
groupArray((date, price)) AS purchases,
min(registration_date) AS registrationDate,
arrayMap(x -> registrationDate + x, range(toUInt32(toDate('2019-01-25') - registrationDate + 1))) dates,
arrayFilter(x -> arrayFirstIndex(p -> p.1 = x, purchases) = 0, dates) AS missed_dates,
arrayMap(x -> (x, 0), missed_dates) AS dummy_purchases,
arraySort(x -> x.1, arrayConcat(purchases, dummy_purchases)) all_purchases,
arrayCumSum(x -> x.2, all_purchases) cum_prices,
arrayMap(index -> (all_purchases[index].1, cum_prices[index]), arrayEnumerate(all_purchases)) flat_result,
arrayJoin(flat_result) result
FROM test.purchases01
GROUP BY player_id
)
/* result
┌─player_id─┬───────date─┬─registration_date─┬─sum_price─┐
│ pl1 │ 2019-01-20 │ 2019-01-20 │ 0 │
│ pl1 │ 2019-01-21 │ 2019-01-20 │ 20 │
│ pl1 │ 2019-01-22 │ 2019-01-20 │ 20 │
│ pl1 │ 2019-01-23 │ 2019-01-20 │ 30 │
│ pl1 │ 2019-01-24 │ 2019-01-20 │ 45 │
│ pl1 │ 2019-01-25 │ 2019-01-20 │ 45 │
└───────────┴────────────┴───────────────────┴───────────┘
*/
/* Prepare test data */
CREATE TABLE test.purchases01
(
`player_id` String,
`date` Date,
`registration_date` Date,
`price` int
)
ENGINE = Memory;
INSERT INTO test.purchases01
VALUES ('pl1', '2019-01-21', '2019-01-20', 20),
('pl1', '2019-01-23', '2019-01-20', 10),
('pl1', '2019-01-24', '2019-01-20', 15);

Fail of bash script with flock

I have a task: Calling a script by curl every ten seconds.
For that I do:
Create bash-script: admin_cron.sh
#!/bin/bash
while :; do
sleep 10
flock -n /var/www/admin/data/cron_lock -c \
'curl -m 3 "http://url.com/?mod=kasdim_robot_cron&server=true"' &
done
Add rule in crontab:
*/1 * * * * flock -n /var/www/admin/data/admin_lock -c /var/www/admin/data/admin_cron.sh
So, cron will start admin_cron.sh every minute if the file admin_lock is free. Then, if admin_cron.sh is terminated or dies, admin_cron.sh starts again. It's ok.
admin_cron.sh tries to call every 10 sec my URL through curl. Max time for curl is three seconds.
Problem: when I start, everything works fine for about two to three days; after, I see in the htop util that admin_cron.sh is hover process:
at start time:
CPU[ 0.0%] Tasks: 61, 53 thr; 1 running
Mem[||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||461M/996M] Load average: 0.44 0.83 0.94
Swp[ 0K/0K] Uptime: 86 days, 18:43:58
PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command
707 root 20 0 29600 2828 2512 S 0.0 0.3 1:27.18 ├─ /usr/sbin/cron -f
22150 root 20 0 48868 2864 2416 S 0.0 0.3 0:00.00 │ └─ /usr/sbin/CRON -f
22151 admin 20 0 4288 720 648 S 0.0 0.1 0:00.00 │ └─ /bin/sh -c flock -n /var/www/admin/data/admin_lock -c /var/www/admin/data/admin_cron.sh
22156 admin 20 0 10044 816 728 S 0.0 0.1 0:00.00 │ └─ flock -n /var/www/admin/data/admin_lock -c /var/www/admin/data/admin_cron.sh
22161 admin 20 0 4288 764 692 S 0.0 0.1 0:00.00 │ └─ /bin/sh -c /var/www/admin/data/admin_cron.sh
22167 admin 20 0 11132 2896 2648 S 0.0 0.3 0:00.00 │ └+ /bin/bash /var/www/admin/data/admin_cron.sh
After some time:
CPU[|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||100.0%] Tasks: 64, 53 thr; 2 running
Mem[||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||466M/996M] Load average: 1.00 1.00 1.00
Swp[ 0K/0K] Uptime: 86 days, 18:42:30
PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command
707 root 20 0 29600 2828 2512 S 0.0 0.3 1:27.18 ├─ /usr/sbin/cron -f
25818 root 20 0 48868 2864 2416 S 0.0 0.3 0:00.00 │ └─ /usr/sbin/CRON -f
25819 admin 20 0 4288 752 676 S 0.0 0.1 0:00.00 │ └─ /bin/sh -c flock -n /var/www/admin/data/admin_lock -c /var/www/admin/data/admin_cron.sh
25821 admin 20 0 10044 800 712 S 0.0 0.1 0:00.00 │ └─ flock -n /var/www/admin/data/admin_lock -c /var/www/admin/data/admin_cron.sh
25824 admin 20 0 4288 788 716 S 0.0 0.1 0:00.00 │ └─ /bin/sh -c /var/www/admin/data/admin_cron.sh
25825 admin 20 0 12636 4260 2552 R 98.7 0.4 60h56:25 │ └─ /bin/bash /var/www/admin/data/admin_cron.sh
Where is my problem?

specify Kotlin class as the Main-Class entry point in a MANIFEST.MF?

How do I explicitly tell shadowJar to use a Kotlin file as the entry point for the resulting uberJar? (Or, fat JAR as you prefer.)
clumsily editing META-INF/MANIFEST.MF and the replacing it in the zip file (used the GUI to put it back):
thufir#dur:~/NetBeansProjects/kotlin/build/libs$
thufir#dur:~/NetBeansProjects/kotlin/build/libs$ ll
total 900
drwxrwxr-x 2 thufir thufir 4096 Nov 5 03:42 ./
drwxrwxr-x 8 thufir thufir 4096 Nov 5 03:42 ../
-rw-rw-r-- 1 thufir thufir 903184 Nov 5 03:42 kotlinAdder.jar
thufir#dur:~/NetBeansProjects/kotlin/build/libs$
thufir#dur:~/NetBeansProjects/kotlin/build/libs$ jar xf kotlinAdder.jar
thufir#dur:~/NetBeansProjects/kotlin/build/libs$
thufir#dur:~/NetBeansProjects/kotlin/build/libs$ ll
total 928
drwxrwxr-x 7 thufir thufir 4096 Nov 5 03:42 ./
drwxrwxr-x 8 thufir thufir 4096 Nov 5 03:42 ../
drwxrwxr-x 2 thufir thufir 4096 Nov 5 03:42 demo/
drwxrwxr-x 17 thufir thufir 12288 Nov 5 03:42 kotlin/
-rw-rw-r-- 1 thufir thufir 903184 Nov 5 03:42 kotlinAdder.jar
drwxrwxr-x 3 thufir thufir 4096 Nov 5 03:42 META-INF/
drwxrwxr-x 3 thufir thufir 4096 Nov 5 03:42 net/
drwxrwxr-x 4 thufir thufir 4096 Nov 5 03:42 org/
thufir#dur:~/NetBeansProjects/kotlin/build/libs$
thufir#dur:~/NetBeansProjects/kotlin/build/libs$ nano META-INF/MANIFEST.MF
thufir#dur:~/NetBeansProjects/kotlin/build/libs$
thufir#dur:~/NetBeansProjects/kotlin/build/libs$ cat META-INF/MANIFEST.MF
Manifest-Version: 1.0
Main-Class: net.bounceme.dur.kotlin.App
thufir#dur:~/NetBeansProjects/kotlin/build/libs$
thufir#dur:~/NetBeansProjects/kotlin/build/libs$ nano META-INF/MANIFEST.MF
thufir#dur:~/NetBeansProjects/kotlin/build/libs$
thufir#dur:~/NetBeansProjects/kotlin/build/libs$ cat META-INF/MANIFEST.MF
Manifest-Version: 1.0
Main-Class: demo.MainKt
thufir#dur:~/NetBeansProjects/kotlin/build/libs$
thufir#dur:~/NetBeansProjects/kotlin/build/libs$ java -jar kotlinAdder.jar
Hello, world!
thufir#dur:~/NetBeansProjects/kotlin/build/libs$
Success of sorts. It ran the Kotlin class as intended.
project:
.
├── build.gradle
├── gradle
│   └── wrapper
│   ├── gradle-wrapper.jar
│   └── gradle-wrapper.properties
├── gradlew
├── gradlew.bat
├── settings.gradle
└── src
├── main
│   ├── java
│   │   └── net
│   │   └── bounceme
│   │   └── dur
│   │   └── kotlin
│   │   └── App.java
│   └── kotlin
│   └── demo
│   ├── example.kt
│   └── main.kt
└── test
└── java
13 directories, 9 files
build file:
plugins {
id 'com.gradle.build-scan' version '1.8'
// id 'java'
id 'application'
id "org.jetbrains.kotlin.jvm" version "1.1.51"
id 'com.github.johnrengelman.shadow' version '2.0.1'
}
buildScan {
licenseAgreementUrl = 'https://gradle.com/terms-of-service'
licenseAgree = 'yes'
//publishAlways()
}
configurations {
provided
}
shadowJar {
baseName = 'kotlinAdder'
classifier = null
version = null
}
sourceCompatibility = 1.8
targetCompatibility = 1.8
//mainClassName = 'net.bounceme.dur.kotlin.App'
mainClassName = 'demo.MainKt'
repositories {
jcenter()
}
configurations {
provided
}
dependencies {
compile "org.jetbrains.kotlin:kotlin-stdlib:1.1.51"
compile 'com.google.firebase:firebase-auth:11.4.2'
}
How do I tell gradle to keep using shadowJar but to build the MANIFEST.MF as manually edited, to set Main-Class: demo.MainKt explicitly as the entry point.
Else, the Java code runs by default even when I've explicitly set the entry point as above in the build.gradle file.

Resources