Pipelining listing and copying files - terminal

I have a server which is hosting my files which I can list with the following command:
xrdfs servername ls path/to/file
Similarly, I can copy file using the following command:
xrdcp server/path/to/file .
For, some reason the server doesn't support copying an entire folder(of course with -r option). So, I am trying to pipeline these two commands such a way that xrdfs will list the files and xrdcp will copy it to my destination. I tried the following line:
xrdfs servername ls path/to/file | xrdcp server/$() .
I get the following message:
Prepare: [ERROR] Invalid arguments
This is not very enlightening. Can somebody help with this?

Ok, I found an answer and I am posting here for reference
xrdfs servername ls path/to/file | while read -r out; do xrdcp server$out .; done

Related

schema file at path is invalid because no version is specified

I run my dbt command in a loop and save its contents to a .yml file. This works well and generates a schema in my .yml file accurately:
for file in models/l30_mart/*.sql; do
table=$(basename "$file" .sql)
dbt run-operation generate_model_yaml --args "{\"model_name\": \"$table\"}" > test.yml
done
However, in the example above, I am just saving the test.yml file in the root directory. When I try to save the file in another path for example models/l30_mart/test.yml, it doesn't work:
for file in models/l30_mart/*.sql; do
table=$(basename "$file" .sql)
dbt run-operation generate_model_yaml --args "{\"model_name\": \"$table\"}" > models/l30_mart/test.yml
done
In this case, when I open the test.ymlfile, I see this:
12:06:42 Running with dbt=1.0.1
12:06:43 Encountered an error:
Compilation Error
The schema file at models/l30_mart/test.yml is invalid because no version is specified. Please consult the documentation for more information on schema.yml syntax:
https://docs.getdbt.com/docs/schemayml-files
It's probably a syntax mistake. Do I need any quotes to specify the path? or brackets? Why can't I save the .yml file in the same folder? Similarly, something like this and try to save different files with the tablename, it also doesn't work:
for file in models/l30_mart/*.sql; do
table=$(basename "$file" .sql)
dbt run-operation generate_model_yaml --args "{\"model_name\": \"$table\"}" > models/l30_mart/$table.yml
done
In this case, the files either have this output:
20:39:44 Running with dbt=1.0.1
20:39:45 Encountered an error:
Compilation Error
The schema file at models/l30_mart/firsttable.yml is invalid because no version is specified. Please consult the documentation for more information on schema.yml syntax:
https://docs.getdbt.com/docs/schemayml-files
or this (eg in the second tablename.yml file):
20:39:48 Running with dbt=1.0.1
20:39:49 Encountered an error:
Parsing Error
Error reading dbt_4flow: l30_mart/firstablename.yml - Runtime Error
Syntax error near line 2
------------------------------
1 | 20:39:44 Running with dbt=1.0.1
2 | 20:39:45 Encountered an error:
3 | Compilation Error
4 | The schema file at models/l30_mart/firsttablename.yml is invalid because no version is specified. Please consult the documentation for more information on schema.yml syntax:
5 |
Raw Error:
------------------------------
mapping values are not allowed in this context
in "<unicode string>", line 2, column 31
Note that the secondtablename.yml mentions the firsttablename.yml.

Gitlab CI not displaying complete output in terminal

I have created a pipeline which performs ansible-lint on $CI_PROJECT_DIR. The problem is the complete output is not shown in UI as compared to running on my local machine.
You can notice the difference in output for both.
Below is the output from my local machine (Ubuntu with ansible-lint installed)
**ansible-lint create-dir.yaml -v**
WARNING Overriding detected file kind 'yaml' with 'playbook' for given positional argument: create-dir.yaml
INFO Executing syntax check on create-dir.yaml (0.31s)
WARNING Listing 1 violation(s) that are fatal
syntax-check: 'file' is not a valid attribute for a Play
create-dir.yaml:4:3 [WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'
ERROR! 'file' is not a valid attribute for a Play
The error appears to be in '/tmp/create-dir.yaml': line 4, column 3, but may be elsewhere in the file depending on the exact syntax problem.
The offending line appears to be:
---
- name: Create a directory if it does not exist
^ here
Finished with 1 failure(s), 0 warning(s) on 1 files.
Below is output from Gitlab CI :
$ find ./ -not \( -name "*.ansible-lint" -o -name ".gitlab-ci.yml" \) \( -name "*yml" -o -name "*yaml" \) | xargs ansible-lint -v
WARNING Overriding detected file kind 'yaml' with 'playbook' for given positional argument: ./ansible/create-dir.yaml
INFO Executing syntax check on ansible/create-dir.yaml (0.39s)
Cleaning up project directory and file based variables 00:01
Job succeeded
I would like to know why there is difference in output and how to print the complete message on Gitlab CI
Job logs in GitLab CI/CD have a limited length. It's designed for operational reasons, as the jobs can create arbitrary outputs so one could - by mistake or by purpose - output even gigabytes or terabytes of text.
But if you scroll up in the output, you have the abillity to view the full logs by pressing 'Complete Raw'.

how to download file VIA wget while target path include Wildcards

here is elegant example how to download file and copy it to
/etc/yum.repo.d folder
example
REPOSITORY_SERVER=master_machine01
wget -nd -r -P /etc/yum.repos.d/ -A ".repo" "http://$REPOSITORY_SERVER/ambari/centos7/2.6.2.2-1/ambari.repo"
after above command ambari.repo file will copied to /etc/yum.repos.d/
note: the file amabri.rep path is
ls -ltr /var/www/html/ambari/centos7/2.6.2.2-1/ambari.repo
-rw-r--r-- 1 root users 304 Jun 11 2018 /var/www/html/ambari/centos7/2.6.2.2-1/ambari.repo
so this is the simple case
now what about path could be as ( with diff path's )
$REPOSITORY_SERVER/ambari/centos7/2.6.2.3-1/ambari.repo
or
$REPOSITORY_SERVER/ambari/centos7/2.6.2.2-4/ambari.repo
then how to use the cli with Wildcards
we try the following
wget -nd -r -P /etc/yum.repos.d/ -A ".repo" "http://$REPOSITORY_SERVER/ambari/centos7/*/ambari.repo"
but we get
HTTP request sent, awaiting response... 404 Not Found
2021-11-28 18:40:07 ERROR 404: Not Found.
or even with backslash
wget -nd -r -P /etc/yum.repos.d/ -A ".repo" "http://$REPOSITORY_SERVER/ambari/centos7/\*/ambari.repo"
BUT WITH THE SAME ERROR
any idea how to resolve this issue?
how to use the cli with Wildcards
It is not possible to perform a glob expansion with HTTP protocol. These are very unrelated technologies.
how to resolve this issue?
Devise and implement a method of getting the available files under certain path from an HTTP server. For example, contact the server administrator and ask him about it. Potentially, if the HTTP server supports serving a directory listing, recursively filter the listing to find all matching paths. Or find and query some other site that contains all the links and filter the obtained answer to extract all links, for example. Etc.

Backup whole F5 bigIP configuration and save as file

I'm trying to export my current F5 bigIP configuration into a file and later create a replica of the same bigIP with that configuration.
This reference mentioned following this:
Create the UCS archive file by using the following command syntax, replacing with the full path to the UCS archive file:
tmsh save /sys ucs <path/to/UCS>
I logged in to my bigIP, but when trying this, it gives me error. What is the problem? How to fix it?
Syntax Error: "ucs" unexpected argument
You can verify the syntax within the TMSH shell just like on a Cisco. Just enter tmsh and then type save and hit ? from there you can also use tab to complete the command.
This command should save a dated ucs file to your /var/local/ucs directory.
[yourbigip] # tmsh save sys ucs "$(date | tr -d ':' | tr -d )".ucs
You can always find additional help at https://devcentral.f5.com.

set additional folder for snmp MIBs

I am rebuilding an Icinga server that has been left behind by a previous employee. I have everything up and running, except for a bunch of MIB files for 3com switches that I cannot get to work.
The server is a CentOS 6 OpenVZ container.
In the original server there is a bunch of mib files in the default location at /usr/share/snmp/mibs/ and the 3com ones at /usr/share/snmp/mibs/3Com_4500/MIBs. The 3Com mibs work fine:
/usr/lib/nagios/plugins/check_snmp -H 10.10.111.11 -P 2c -C public -o hwDevMFanStatus.65536 -s "active(1)" -m A3COM-HUAWEI-LswDEVM-MIBSNMP OK - active(1) |
In the new server, the MIBs in the 3com folder do not get acknowledged and I get errors like the following:
/usr/lib/nagios/plugins/check_snmp -H 10.10.111.11 -P2c -C someuser -o hwDevMFanStatus.65536 -s "active(1)" -m A3COM-HUAWEI-LswDEVM-MIB
External command error: No log handling enabled - turning on stderr logging
Cannot find module (A3COM-HUAWEI-LswDEVM-MIB): At line 0 in (none)
hwDevMFanStatus.65536: Unknown Object Identifier (Sub-id not found: (top) -> hwDevMFanStatus)
/etc/snmp/snmpd.conf is identical for both servers and so is /etc/sysconfig/snmp.
set does not show any ENV variable related to snmp or mib.
Thanks
You are confusing snmpd.conf and snmp.conf the former being the configuration file for the SNMP daemon whereas Net-SNMP applications use snmp.conf.
The mibs/mibdirs directives you are interested in would be specified in snmp.conf (see also man snmp.conf.

Resources