I have the following data containing a subset of record numbers formatting like so:
>head pilot.dat
AnalogPoint,206407
AnalogPoint,2584
AnalogPoint,206292
AnalogPoint,206278
AnalogPoint,206409
AnalogPoint,206410
AnalogPoint,206254
AnalogPoint,206266
AnalogPoint,206408
AnalogPoint,206284
I want to compare the list of entries to another subset file called "disps.dat" to find duplicates, which is formatted in the same way:
>head disps.dat
StatusPoint,280264
StatusPoint,280266
StatusPoint,280267
StatusPoint,280268
StatusPoint,280269
StatusPoint,280335
StatusPoint,280336
StatusPoint,280334
StatusPoint,280124
I used the command:
grep -f pilot.dat disps.dat > duplicate.dat
However, the output file "duplicate.dat" is listing records that exist in the second file "disps.dat", but do not exist in the first file.
(Note, both files are big, so the sample shown above don't have duplicates, but I do expect and have confirmed at least 10-12k duplicates to show up in total)
> head duplicate.dat
AnalogPoint,208106
AnalogPoint,208107
StatusPoint,1235220
AnalogPoint,217270
AnalogPoint,217271
AnalogPoint,217272
AnalogPoint,217273
AnalogPoint,217274
AnalogPoint,217275
AnalogPoint,217277
> grep "AnalogPoint,208106" pilot.dat
>
I tested the above command with a smaller sample of data (10 records), also formatted the same, and the results work fine, so I'm a little bit confused on why it is failing on the larger execution.
I also tried feeding it in as a string with -F thinking that the "," comma might be the source of issue. Right now, I am feeding the data through a 'for' loop and echoing each line, which is executing very, very slowly but at least it will help me cross out the regex possibility.
the -x or -w option is needed to do an exact match.
-x will match exact string, and -w will match exact substring and block non-word characters which works in my case to handle trailing numbers.
The issue is that a record in the first file such as:
"AnalogPoint,1"
Would end up flagging records in the second file like:
"AnalogPoint,10"
"AnalogPoint,123"
"AnalogPoint,100200"
And so on.
Thanks to #Barmar for pointing out my issue.
I got a list of files that I would have to iterate over and read each json as text from a file, see if there are duplicates key value pair and remove the duplicates.
I have written the script to iterate over each file and read line by line, and need help to search for duplicates. I know the duplicates key for now - "user_context and User_Context".
My sample json:
{"context_id”:”8fba96a626d2-1234“,”context_1”:”00016”,”User_Context”:”ctx1”,”context_Id":"CountRows”,”date":"2020-08-17”,”user_context”:”ctx1”,"name":"n1"}
JSON is big and i have removed many keys for testing. And there are also many files.
Tried the below jq command:
jq -c -M 'del(.user_context)' ${str1} > tmp
inside #!/bin/bash script and get "jq: command not found"
Any help please.
Installed Jp and used to remove duplicates
My environment created a variable that looks like this:
SM_TRAINING_ENV={"additional_framework_parameters":{},"channel_input_dirs":{"training":"/opt/ml/input/data/training"},"current_host":"algo-1","framework_module":"sagemaker_tensorflow_container.training:main","hosts":["algo-1"],"hyperparameters":{"bool_param":true,"float_param":1.25,"int_param":5,"model_dir":"s3://bucket/detection/prefix/testing-2019-04-06-02-24-20-194/model","str_param":"bla"},"input_config_dir":"/opt/ml/input/config","input_data_config":{"training":{"RecordWrapperType":"None","S3DistributionType":"FullyReplicated","TrainingInputMode":"File"}},"input_dir":"/opt/ml/input","is_master":true,"job_name":"testing-2019-04-06-02-24-20-194","log_level":20,"master_hostname":"algo-1","model_dir":"/opt/ml/model","module_dir":"s3://bucket/prefix/testing-2019-04-06-02-24-20-194/source/sourcedir.tar.gz","module_name":"launcher.sh","network_interface_name":"ethwe","num_cpus":8,"num_gpus":1,"output_data_dir":"/opt/ml/output/data","output_dir":"/opt/ml/output","output_intermediate_dir":"/opt/ml/output/intermediate","resource_config":{"current_host":"algo-1","hosts":["algo-1"],"network_interface_name":"ethwe"},"user_entry_point":"launcher.sh"}
EDIT by Ed Morton: per the OPs comment below, this is what (s)he is trying to describe above as the sample input:
$ SM_TRAINING_ENV='{"additional_framework_parameters":{},"channel_input_dirs":{"training":"/opt/ml/input/data/training"},"current_host":"algo-1","framework_module":"sagemaker_tensorflow_container.training:main","hosts":["algo-1"],"hyperparameters":{"bool_param":true,"float_param":1.25,"int_param":5,"model_dir":"s3://bucket/detection/prefix/testing-2019-04-06-02-24-20-194/model","str_param":"bla"},"input_config_dir":"/opt/ml/input/config","input_data_config":{"training":{"RecordWrapperType":"None","S3DistributionType":"FullyReplicated","TrainingInputMode":"File"}},"input_dir":"/opt/ml/input","is_master":true,"job_name":"testing-2019-04-06-02-24-20-194","log_level":20,"master_hostname":"algo-1","model_dir":"/opt/ml/model","module_dir":"s3://bucket/prefix/testing-2019-04-06-02-24-20-194/source/sourcedir.tar.gz","module_name":"launcher.sh","network_interface_name":"ethwe","num_cpus":8,"num_gpus":1,"output_data_dir":"/opt/ml/output/data","output_dir":"/opt/ml/output","output_intermediate_dir":"/opt/ml/output/intermediate","resource_config":{"current_host":"algo-1","hosts":["algo-1"],"network_interface_name":"ethwe"},"user_entry_point":"launcher.sh"}'
$ echo "$SM_TRAINING_ENV"
{"additional_framework_parameters":{},"channel_input_dirs":{"training":"/opt/ml/input/data/training"},"current_host":"algo-1","framework_module":"sagemaker_tensorflow_container.training:main","hosts":["algo-1"],"hyperparameters":{"bool_param":true,"float_param":1.25,"int_param":5,"model_dir":"s3://bucket/detection/prefix/testing-2019-04-06-02-24-20-194/model","str_param":"bla"},"input_config_dir":"/opt/ml/input/config","input_data_config":{"training":{"RecordWrapperType":"None","S3DistributionType":"FullyReplicated","TrainingInputMode":"File"}},"input_dir":"/opt/ml/input","is_master":true,"job_name":"testing-2019-04-06-02-24-20-194","log_level":20,"master_hostname":"algo-1","model_dir":"/opt/ml/model","module_dir":"s3://bucket/prefix/testing-2019-04-06-02-24-20-194/source/sourcedir.tar.gz","module_name":"launcher.sh","network_interface_name":"ethwe","num_cpus":8,"num_gpus":1,"output_data_dir":"/opt/ml/output/data","output_dir":"/opt/ml/output","output_intermediate_dir":"/opt/ml/output/intermediate","resource_config":{"current_host":"algo-1","hosts":["algo-1"],"network_interface_name":"ethwe"},"user_entry_point":"launcher.sh"}
How can I create a new bash variable that is equal to the value of SM_TRAINING_ENV["hyperparameters"]["model_dir"]?
For completeness, I was trying simple things like echo ${SM_TRAINING_ENV} | jq . and kept getting errors with everything I tried.
Edit: I've been informed that this value isn't a proper json, so rewording the question. I think the environment sets it to the value of a python dictionary, so jq seems not usable. Removed json tag. Maybe this is a job for awk?
It looks like I can match the value I want if I assume the structure doesn't change with the regex pattern s3.*?model, but not sure how to set a regex pattern to a new variable.
First, you need to quote the JSON value so that the double quotes will be included in the value.
SM_TRAINING_ENV='{"additional_framework_parameters":{},"channel_input_dirs":{"training":"/opt/ml/input/data/training"},"current_host":"algo-1","framework_module":"sagemaker_tensorflow_container.training:main","hosts":["algo-1"],"hyperparameters":{"bool_param":true,"float_param":1.25,"int_param":5,"model_dir":"s3://bucket/detection/prefix/testing-2019-04-06-02-24-20-194/model","str_param":"bla"},"input_config_dir":"/opt/ml/input/config","input_data_config":{"training":{"RecordWrapperType":"None","S3DistributionType":"FullyReplicated","TrainingInputMode":"File"}},"input_dir":"/opt/ml/input","is_master":true,"job_name":"testing-2019-04-06-02-24-20-194","log_level":20,"master_hostname":"algo-1","model_dir":"/opt/ml/model","module_dir":"s3://bucket/prefix/testing-2019-04-06-02-24-20-194/source/sourcedir.tar.gz","module_name":"launcher.sh","network_interface_name":"ethwe","num_cpus":8,"num_gpus":1,"output_data_dir":"/opt/ml/output/data","output_dir":"/opt/ml/output","output_intermediate_dir":"/opt/ml/output/intermediate","resource_config":{"current_host":"algo-1","hosts":["algo-1"],"network_interface_name":"ethwe"},"user_entry_point":"launcher.sh"}'
Then you can use the jq utility to extract the value you want.
new_var=$(echo "$SM_TRAINING_ENV" | jq '.hyperparameters.model_dir')
This doesn't really index but it works if order is always the same:
NEW_VAR=$(echo $SM_TRAINING_ENV | egrep -o s3.*?model | head -1)
Would much prefer something not dependent on order though.
If I use wbemcli to enumerate all the instances I get something similar to this:
wbemcli -nl -t -noverify ei 'https://aaa/aaa:aaa'
https://aaa/aaa:aaa.Version="",Vendor="",Name=""
-Version#=""
-Vendor#=""
-Name#=""
-Description=""
How can I call wbemcli to get only one item (i.e. the Name)? and not everything.
The -t option says:
-t
Append array ([]), reference (&) and key property (#) indicators to property names
but I wasn't able to utilize this in my favor.
Is there a way to retrieve this information in a key/value pair format?
Or maybe pipe the output into an array or something from which I can grab only what I need?
When I drop the output into an array all the data is stored in the first element ${a[0]}.
EDIT
Here's an output example:
$ wbemcli -nl -t -noverify ei 'https://user:pw#000.000.000.000:0000/root/aaa:AA_AaaAaaaAaaaa'
000.000.000.000:0000/root/aaa:AA_AaaAaaaAaaaa.ClassName="AA_AaaAaaaAaaaa",Name="123456a7ff890123"
-ClassName#="AA_AaaAaaaAaaaa"
-Name#="123456a7ff890123"
-Caption="aa aaa"
-Description="aa aa"
-ElementName="aa aaa aaaa"
-OperationalStatus[]=2
-HealthState=5
-CommunicationStatus=2
-DetailedStatus=1
-OperatingStatus=0
-PrimaryStatus=1
-EnabledState=5
-RequestedState=12
-EnabledDefault=2
-TransitioningToState=12
-PrimaryOwnerName="Uninitialized Contact"
-PrimaryOwnerContact="Uninitialized Contact"
The output is usually in this format.
If the query returns multiple objects they will be grouped and all will have the same members with their appropriate values.
http://linux.dell.com/files/whitepapers/WBEM_based_management_in_Linux.pdf has a number of examples which simply suggest to use grep to obtain the specific key and value you are looking for. There does not seem to be a way to directly query for a specific key within a result set.
Expanding on the comment by Etan Reisner, you could use something like
wbemcli <<query>> | grep -oP "^-$key=\K.*"
to obtain the value for the key named in $key, provided you have GNU grep which provides the -P option for Perl-compatible regular expressions (here, the \K "forget up through here" operator is useful). So for your specific example,
wbemcli -nl -t -noverify ei 'https://aaa/aaa:aaa' |
grep -oP '^-Name#=\K.*'
There is also a -dx option which produces XML output, which may be more robust if you are planning to write a major application on top of this protocol (but then perhaps you should be looking at a dedicated WBEM library such as the C or Java libraries listed in their wiki). It would not seem to be implausible to write a simple (e.g.) Python client to retrieve (part of?) the result tree and let you query or manipulate it locally, either.
So I am trying to make a script that references other files. I want to be able to keep track of the file even if it moves. So I was thinking if I could assign a file a unique value then I could find the location of the file by searching by the unique value I assigned it.
Is there a better way to do this?
Basically I'd like to be able to find a file from a value it has as an extended attribute. But I don't know if this is possible.
Any help would be great. Thanks.
You could use the inode number (show it with ls -i /some/file) which will be unique per file and which does not change when the file is changed or moved, UNLESS you move the file to a different partition. If you don't need to track files over multiple partitions than this would be a very easy solution.
To find a file by inode number you can use find -inum <inode number>
If you need to move the file across filesystems, you can use the command setfattr to set extended attributes to file, note: your kernel must be able to do this.
There aren't one-shot command to find the file but you can create a simple script to search the file by your unique key.
To set to yourfile a key similar to user.unique and a string value as you like, e.g.: n1234
setfattr -n user.unique -v n1234 yourfile
to retrieve the value you can use the command getfattr
getfattr -n user.unique yourfile
or to get all extended file attributes:
getfattr -d yourfile
To test if your kernel is able to handle extended attributes on type of your filesystems:
zcat /proc/config.gz | grep FS_XATTR
To write a simple script to search using extended attributes you can refer to this Hack; on the same page you can read more about extended attributes.