I'm just stepping into the world of PASE and Qshell on the IBM i, and am still learning the "shell way" of solving problems.
I'm trying to write a shell script that will delete all logical files in a given library that are associated with a given physical file. The OS is IBM i 7.2. In QSYS-land, I'd probably DSPDBR on the physical file to an outfile, then read through the outfile and delete each dependent file. How would you do this in PASE or Qshell? I had a couple ideas, but they all seem overly-complicated, and the more I learn about shell scripting, the more shortcuts I'm finding.
My first idea was to basically replicate the above process, doing something like this and then somehow using the output of the SELECT:
system "DSPDBR FILE(MYLIB/MYFILE) OUTPUT(*OUTFILE) OUTFILE(QTEMP/DSPDBR)"
db2 "select WHREFI from QTEMP/DSPDBR where WHRELI = 'MYLIB'"
(I see now that QTEMP doesn't really work as a temporary library, but maybe there's a way around this.)
My second idea was to do something like pipe the output of the DSPDBR statement into something like awk to pick out the logical file names, and redirect the output of that to a stream file (or shell variable?). And from there, somehow use this list to delete the logical files.
Is there a more straightforward approach? It seems like whatever the answer is, it will be a pattern that is often repeated when writing shell scripts to interact with QSYS commands and objects.
First, you might review Running SQL queries from PASE instead of QSH. The db2 utility is part of Qshell and not PASE. Unfortunately, the processing behind it is based in ILE, so it's not directly usable within the PASE (AIX run-time) environment. The linked question provides a method of bridging between the two.
However, directly in QShell, you can experiment with something like this:
db2 "SELECT substr(VIEW_NAME,1,18), substr(OBJECT_NAME,1,10),
substr(OBJECT_SCHEMA,1,10), substr(VIEW_SCHEMA,1,10),
substr(TABLE_NAME,1,18)
FROM qsys2.sysviewdep
WHERE OBJECT_SCHEMA = '<yourSchemaName>'"
The SUBSTR() functions might or might not be useful. It depends on your name lengths and whether you want them limited or not. The output can be redirected to an outfile or perhaps piped into sed or another utility for manipulation.
A QTEMP library is specific to [is scoped to] the process that implements the request in the shell. Ensure all of the requests that depend on the same QTEMP library will run in the same process; the system utility runs in a separate process, as does the db2 utility. For example, in the following command-line invocation of the QSHell, all of the requests dependent on QTEMP will run in the same process in which the db2 utility runs; the db2 utility runs a script [or could instead run a dynamic compound statement] that in this scenario was generated by the scripted requests for\within the shell utility:
qsh cmd('
echo "call qsys2.qcmdexc
(''dspdbr mylib/myfile output(*outfile) outfile(qtemp/dspdbr)'')
" >> mydb2.script
; echo "select WHREFI from QTEMP.DSPDBR where WHRELI =''MYLIB''
" >> mydb2.script
; db2 -f mydb2.script
; rm mydb2.script
')
There is an open source equivalent of the QShell db2 command: https://bitbucket.org/litmis/db2util
Related
I've been wondering how I could do some of these cool customization options for the terminal on a Mac and I came across lolcat.
However, I can't seem to find an answer as to how to add this into my bashrc (FYI: I'm using zsh now just in case that makes a difference as to which file to add my customizations in) I have tried what many others have suggested, which was just typing zsh | lolcat into the terminal to get rainbow output in the current session, but I was wondering how I can have this every time I start a terminal session.
Also, I'm not sure if this is a bug or if there's something wrong with my terminal settings, but when I use a command with lolcat, I get an output like this:
karenlee#Karens-MBP ~ % Documents
Downloads
Library
Movies
Music
Pictures
Postman
Public
38;5;48m
karenlee#Karens-MBP ~ % 38;5;48m
The colors look right, but as you can see, when I type the ls command on the command line, it disappears and the output also gets messy. It also seems like there's extra lines of 38;5;48m which are appearing. And it also seemed like many of the gems that are installed with lolcat have deprecated; is there another alternative to lolcat that plays nicely with macOS Catalina?
I made a shell extension for the world's fastest website generator that I make called Nift. It has an easter egg where you can turn on lolcat output for most things with lolcat.on (after starting the shell with eg. nift sh). You will even get rainbow output when pressing tab to get possible completion options, I doubt you get that with any other suggested solutions.
The shell extension is for f++ which is the in-built scripting language, which has these functions and these types available. But anything it doesn't recognise is run as a system call using the (probably primary/default) shell on your machine (hence calling it a shell extension in REPL shell mode).
Nift will look for a version of lolcat installed on your machine and otherwise use an in-built version of it which should be the world's fastest (it's near identical to my c++ implementation lolcat-cc which is the world's fastest). I highly recommend installing lolcat-cc (or another version of lolcat) on top of Nift though as otherwise you are frequently running the ~5mb Nift binary for basically all system calls, instead of a <1mb binary for lolcat.
f++ is somewhat of an interesting scripting language as well. Take this script for example which creates and deletes 100k empty text files. On my machine this runs in ~3.5 seconds whereas this Bash script doing essentially the same thing takes more like 3 minutes!! Some of the cool things you might already notice from that f++ script is you can expand variables inside strings, you can define variables with types, you can expand variables in to multiple parameters with function calls, you can have LOTS more than 10k input parameters for function calls (should be able to have millions!).
You can find some more information about the Nift REPLs (including shortcuts for different platforms) here.
If you need to define shell variables (not through f++ but the underlying shell) then you will need to do blocks of code using the sys/system function. You can also do blocks of code for Lua(JIT) and ExprTk similarly as they are both embedded in to Nift. You can use both Lua and ExprTk with accessing/modifying f++ variables as well..
So to get this shell (extension). Install Nift through a package manager or clone, make and install off GitHub. Enter nift sh in to your terminal to start the Nift f++ shell extension, then enter lolcat.on to turn on rainbow output.
ssh to an IP supplied via user input
Inject show running-config command
Search the output of the command and look for specific parameters like ports, QoS, VTY lines, SMTP settings, IP helpers etc.
Output only if the predefined parameters are not in place
If I store the output of the ssh run in a variable, is there a method I can use for parsing through it and just go with endless if, elsif, and else statements? How would that look?
It can be in Python bash or Perl, doesn't matter to me really but I don't know how to structure the thing and then fill in the gaps and expand the script.
Here's where I imagine starting from
use Net::SSH2
use File::Slurp;
print "\nEnter command list filename: ";
my $commandlist = <STDIN>;
chomp($commandlist);
my #commandfile = read_file($commandlist, chomp => 1);
my $commandsize = #commandfile;
How would I store the output of the commands in a variable or a temporary file and parse through it with conditions?
This is a very broad question, and it is unclear exactly what is puzzling you
The documentation for
Net::SSH2
and
Net::SSH2::Channel
describes clearly how to open a channel, send commands to it, and receive the response
You ask how you could store the results of the command in a variable, but it would be very awkward to do anything else. Again, the documentation describes this clearly
I suggest that you try writing some working code and experiment with it. It will be much easier to help you when you have a specific question
Is there any way of adding "helper" methods to the mongo shell that loads each time you use it?
Basically whenever you want to query by _id, you have to do something like this.
db.collectionName.findOne({_id: ObjectId('THIS-IS-AN-OBJECTID')})
Whenever I'm going to be doing a lot of command line commands, I alias the ObjectId function to make it easier to type.
var ob = ObjectId;
db.collectionName.findOne({_id: ob('AN-OBJECTID')})
db.collectionName.findOne({_id: ob('ANOTHER-ONE')})
db.collectionName.findOne({_id: ob('ANOTHER')})
It would be pretty chill if there was a way of either running a specified piece of JS / add a chunk of code that runs each time mongo is pulled up from the shell. I checked out MongoDB's CLI documentation, but didn't see anything like that available, so I figured I would ask here.
I know there is a possibility of using this nefariously, so this might be a situation where it might be unsupported by the mongo shell by default. This might be a situation where we can create a helper bash script of some sort to launch the shell, then inject keyboard input to create the helper ob function? Not sure how this could be tackled personally, but would love some insight on how to do something like this, either natively or through a helper script of some sort.
If you want code to execute every time you launch the shell, then whatever you place in .mongorc.js will be run on launch:
.mongorc.js File
When starting, mongo checks the user’s HOME directory for a JavaScript file named .mongorc.js. If found, mongo interprets the content of .mongorc.js before displaying the prompt for the first time. If you use the shell to evaluate a JavaScript file or expression, either by using the --eval option on the command line or by specifying a .js file to mongo, mongo will read the .mongorc.js file after the JavaScript has finished processing. You can prevent .mongorc.js from being loaded by using the --norc option.
So simply define your variable association there.
You could also supply a file of your choice along with the --shell option to let the command know you want the shell opened on completion of any instructions contained:
mongo --shell file_with_javascript.js
But as mentioned, the .mongorc.js file would still be called (if present) unless the --norc option was also specified.
UPDATE:
Right now I am trying to take in to an array around 120M of rows. Reason I'm not doing this over UTL_file is because in our production server it requires oracle user access to write and move the written flat-file into our desired directory. Add in to the problem is that there are still a lot of text manipulations needed after extraction which I think is a job for Perl.
What I want to do right now is to write my own implementation of the DBI::Iterator and all other dependencies it may need; to be clear though, not necessarily all. (like some methods only of DBI, then just idb_rows... just to get this running since I cannot install modules)
Original Question:
Good day,
I'm relatively new to perl programming, A week ago I started receiving out of memory messages again in perl. I was able to get around this before by switching to 64 bit perl.
I just found yesterday that the kernel of my production machine is not allowing me to use more than 4GB of memory (in my other production server I am able to load large amounts of data to memory)
Here are my specific restrictions in my production server
I am not allowed to install new modules of perl
I am allowed, in a way, to install them locally but I am having trouble
What I intend to do now is to recreate this module.
Iterator::DBI
I have no background of iterators. for the longest time I develop database extractions and ETL processes through the function below. It's my 1st time to encounter an out of memory error again after a year and a half of using the function below.
sub myDBI
{
my ($filename) = #_;
my $query = "";
unless(open(FILE,$filename))
{
Logger("[ ERR ] unable to open $SETTINGS{SQL_TRIGGER}\n");
print
die;
}
my #result=`sqlplus -S $SETTINGS{USER_NAME}/$SETTINGS{PASSWORD}\#$SETTINGS{DB_NAME} <<END
SET HEADING OFF
SET FEEDBACK OFF
SET SERVEROUTPUT ON
SET LINES 5000
SET COLSEP "||"
$query
/
`
;
#result;
}
You have several options:
If you have local::lib installed, you can install CPAN modules like Iterator::DBI to a user directory. You'll just need to set some environment variables to specify which directory to use.
export PERL_MB_OPT='--install_base /home/username/perl5'
export PERL_MM_OPT='INSTALL_BASE=/home/username/perl5'
export PERL5LIB='/home/username/perl5/lib/perl5/i386-linux:/home/username/perl5/lib/perl5'
export PATH="/home/username/perl5/bin:$PATH"
You actually don't need Iterator::DBI. That module just wraps an Iterator object around a DBI statement handle, which is an iterator itself. So you can just use DBI directly to connect to the database. (Note that either way, you'll connect to the database directly rather than going through sqlplus.)
use DBI;
my $dbh = DBI->connect(...);
my $sth = $dbh->prepare($sql_query);
$sth->execute(#params);
# iterate
while (my $row = $sth->fetchrow_arrayref) {
...
}
If you really want to use Iterator::DBI and you can't install the module, you can just copy the source code directly and put it in ./Iterator/DBI.pm relative to your application. But the problem with this is you'll need to get around the dependencies. To do that I would replace exceptions with a simple die or croak, and replace Iterator with a closure (see chapter 5 of Higher Order Perl for how to do this). This option looks quite difficult for a beginner Perl programmer.
If you really can't get DBI working, you can pipe the sqlplus output to a file and iterate through the file.
I am trying to write a script that automates other perl scripts. Essentially, I have a few scripts that rollup data for me and need to be run weekly. I also have a couple that need to be run on the weekend to check things and email me if there is a problem. I have the email worked out and everything but the automation. Judging by an internet search, it seems as though using Proc::Background is the way to go. I tried writing a very basic script to test it and can't quite figure it out. I am pretty new to Perl and have never automated anything before (other than through windows task scheduler), so I really don't understand what the code is saying.
My code:
use Proc::Background;
$command = "C:/strawberry/runDir/SendMail.pl";
my $proc1 = Proc::Background -> new($command);
I receive an error that says no executable program located at C:... Can someone explain to me what exactly the code (Proc::Background) is doing? I will then at least have a better idea of how to accomplish my task and debug in the future. Thanks.
I did notice on Proc::Background's documentation the following:
The Win32::Process module is always used to spawn background processes
on the Win32 platform. This module always takes a single string
argument containing the executable's name and any option arguments.
In addition, it requires that the absolute path to the executable is
also passed to it. If only a single argument is passed to new, then
it is split on whitespace into an array and the first element of the
split array is used at the executable's name. If multiple arguments
are passed to new, then the first element is used as the executable's
name.
So, it looks like it requires an executable, which a Perl script would not be, but "perl.exe" would be.
I typically specify the "perl.exe" in my Windows tasks as well:
C:\dwimperl\perl\bin\perl.exe "C:\Dropbox\Programming\Perl\mccabe.pl"