RHEL/CentOS 8.x, Ruby 2.5.5, irb 0.9.6
We have a specialized set up that keeps user and group information in /usr/local/etc/[user|group] and a custom Makefile to add that information to /var/db/[group|passwd].db accordingly.
I am trying to get the list of groups a user belongs to, in Ruby, after login.
I'm relatively new to the language, and have just read the documentation for the Etc module but that seems to exclusively work with the /etc/ filesystem. Not outrageous tbh.
Is there an easy way to access the Berkley DB info in /var/db or am I going to have to iterate through the /usr/local/etc/group file to get this information?
I suspect that the documentation of that module is heavily simplified, heavily outdated, or both. I am almost 100% sure that Etc will use the OS-provided standard library functions instead of going off and parsing system files by itself. (Why would the Ruby developers write parsers for the configuration files of every single OS they support, instead of just calling a POSIX function?)
You can confirm this suspicion using strace.
If you look at how the Ruby Etc module is structured, it maps 1:1 to the POSIX functions:
Etc::gegtrent: gets the next group entry.
Etc::endgtrent: stops iterating the groups.
Etc::settrent: resets the iteration.
Here are the POSIX functions for comparison:
endgrent, getgrent, setgrent – group database entry functions
In my case, I am testing this on macOS (which already has specialized user handling), but furthermore, I tested it on my company laptop, which is macOS joined to a Microsoft ActiveDirectory Domain. My user account and its group aren't even mentioned in either /etc/passwd or /etc/group, and yet, I can easily read them with Etc. I am very sure that the Ruby developers did not implement ActiveDirector just to accommodate my personal weird use case, so it must use system functions.
So, if you have e.g. a uid and want to know what groups it belongs to, you need to first get its name, and then search the groups. Unfortunately, it looks like the provided iterator wrappers do not support the common idiom that not supplying the block returns an Enumerator, so you have to create one yourself.
require 'etc'
uid = 0 # for example
username = Etc.getpwuid(uid).name
groups = Etc.enum_for(:group).select {|group| group.mem.include?(username) }
p groups
Related
Note: Please see the #### UPDATE ### section below. I've heavily modified the question for clarity on what I'm trying to achieve, but added it as an addendum rather than rewrite the question.
As my infrastructure grows, adding input variables in my variables.tf files and then syncing those values to output variables in my outputs.tf file is now impossible to do manually. Not only is it taking up a lot of unnecessary time, probably more time is spent going back and fixing the ones that terraform validate told me that I missed by human error. This is especially true when building / using modules whose arguments add an additional layer to manage.
There has to be a better way? Here is what I want to achieve.
Let's say I'm creating an Azure AKS Kubernetes cluster. The Terraform resource is azurerm_kubernetes_cluster.
Only 8 arguments are required to create a base install, but there are almost 250 additional ones. They all have default values. Per the documentation page, they also already have fantastic descriptions. (I'm tired of copying and pasting into my variables { description = "this"} block.)
The information is there in the documentation. terraform plan also has knowledge of every single additional one because it of course comes up in the pre-apply plan. (known after apply) means its optional, but will have a default value.
In my dream world, I'd run this hypothetical command sequence:
terraform plan
terraform document <- Here it auto generates every argument as a variables block and inserts it into variables.tf. It also auto generates every possible output "out_putable" {} block and inserts it into outputs.tf.
terraform apply -update-inputs -update-outputs <- Here everything that was optional (known after apply) is now known and it should auto update variables.tf and outputs.tf accordingly. Adding a -update-modules flag lets it take care of that additional layer introduced by using modules.
This feels like a problem that has been addressed before. Before I write a custom tool that parses Terraform web docs and the output of terraform show, is there already a way to do this? Terraform-docs is the closest I've come to finding a solution for README.md. If it can do what I need, I haven't figure it out yet.
How can I automate all this?
############
UPDATE
############
This article and video is spot-on when it comes to Terraform's evolution in an organization. My organization is somewhere between late-stage pattern 3 and early 5. As we decompose our "Terralith" we have inconsistencies among teams (patterns, naming conventions, variable and argument choices etc). These are starting to cause errors in CI/CD forcing a ticket-review process that is slowing things down.
All resources have required and optional arguments. But in my organization, we have, for example, additional optional arguments that are required for us.
Scenario: Dev A in Japan creates a resource, forgets an optional variable or two or names them something obscure, etc. Dev B in America is blocked until they can convene and discuss. Given time zones, language differences, ticket review, this one issue is now a week or more delayed.
I need to automate this and create exact consistency so that Dev A starts out with exactly what Dev B would start with or is expecting; and, what CI/CD tests are expecting - templating the initial process, if you will. In other words, I need to remove the human element of manually creating main.tf, variables.tf, outputs.tf, etc.
Here are thoughts on how to achieve this:
Use Golang to autogenerate the files by querying the API
How can I query the API to get a list of all required arguments for a specific resource?
I found that I can query for provider information, but I can't find info to retrieve resource information. My thinking is when a developer wants to create a new resource, He'll run a go or typescript to generate the manifest files along with expected naming conventions, and populate main.tf, variables.tf, outputs.tf, etc, with exactly what data that everyone is expecting. I'm looking form something like curl registry.terraform.io/providers/hashicorp/azurerm/v2.99/resource_group?required=yes This should show me all required arguments along with descriptions and other info I can use straight from the API.
Use CDKTF to generate an HCL manifest.tf file from JSON
How can I use CDKTF to generate an HCL .tf file?
CDKTF is EXACTLY what I'm looking for - except in reverse. HCL is seamlessly compatible with JSON. Running cdktf synth creates ./out/cdk.tf.out I'm so close! How do I turn that file into main.tf?!?
The goal here is to have a master file from which all future manifest files are derived. Whether we use azurerm_kubernetes_cluster 1 time or 1000 times, I know for certain that every argument, every variable name, every desired output is exactly the same. If a chance is needed in our desired structure, it will be updated at the JSON level, and CI/CD can ensure those changes are propagated across instances of its use.
I know that I can use the cdk.out.tf file as a drop in replacement for a module, but I don't want my team members to have to learn typescript or how to read json. If I can create a templatized JSON file containing exactly what I'm expecting users to start with, and if they can run some command like cdktf convert cdk.tf.out --HCL output-file.tf then I've accomplished my goal.
If cdktf synth can create an HCL JSON file, and cdktf convert can take a manifest.tf file and turn it into HCL JSON, can't it do the exact opposite? Turn the HCL JSON file into the human-readable, declarative, manifest.tf file?
Perhaps think of it this way. Terraform has a required file structure for a module if it's to be allowed into the module registry. I'm trying to create a similar required structure for each of the resources our organization uses regardless of when and where it's used.
If your goal is to derive input variables and output values from resource type schemas then Terraform can provide you with the information to do so.
In the working directory of a configuration that already uses the provider whose resource type you want to use, run the following command:
terraform providers schema -json
The result contains a JSON description of all of the resource types available in the providers for the current configuration, and for each one the metadata about its attributes, including the type constraint information and descriptions for each one.
From that you can generate whatever other files you need based on that information.
Note that if you are intending to build modules which export the entire surface area (all inputs and all outputs) of a particular resource type the Terraform documentation explicitly recommends against this, suggesting to just use the resource type directly instead since such a module would often not offer sufficient benefit to outweigh the additional complexity and maintenance overhead it implies:
In principle any combination of resources and other constructs can be factored out into a module, but over-using modules can make your overall Terraform configuration harder to understand and maintain, so we recommend moderation.
A good module should raise the level of abstraction by describing a new concept in your architecture that is constructed from resource types offered by providers.
For example, aws_instance and aws_elb are both resource types belonging to the AWS provider. You might use a module to represent the higher-level concept "HashiCorp Consul cluster running in AWS" which happens to be constructed from these and other AWS provider resources.
We do not recommend writing modules that are just thin wrappers around single other resource types. If you have trouble finding a name for your module that isn't the same as the main resource type inside it, that may be a sign that your module is not creating any new abstraction and so the module is adding unnecessary complexity. Just use the resource type directly in the calling module instead.
I've got the same question and develop a small bash script to create output definitions based on module code
This code required the hcledit tool to extract blocks from hcl code
#!/usr/bin/env bash
set -o pipefail
_hcledit=$(which hcledit)
for tf_file in $(ls *.tf); do
cat $tf_file | $_hcledit block list | while read line; do
block_type="${line%%.*}"
line="${line#*.}"
case $block_type in
locals|output|variable|data) continue; break ;;
module)
output_name=$line
output_description="Module '$output_name' attributes"
output_value="$block_type.$output_name"
;;
resource)
label_kind="${line%.*}"
label_name="${line#*.}"
output_name="${label_kind}_${label_name//[\-]/_}"
output_description="Resource '$label_kind.$label_name' attributes"
output_value="$label_kind.$label_name"
;;
esac
cat <<-EOT
output "$output_name" {
description = "$output_description"
value = $output_value
}
EOT
done
done
I have a standalone ruby script that is intended to be run from a commandline.
Currently I have a section to define stuff like paths to files and such.
key_path = "C:\\OpenSSL-Win64\\keys\\"
consumer_file = "henrb028.consumer"
key_file = "henrb028.key"
I'd love to pull all of that out of the .rb file and into some external config file. Ideally it would be as lightweight as possible. For example, just copy-paste that into another ruby file and then cram it in at runtime.
I've tried both require and include and gotten varying errors. Without resorting to something like YAML, what's an easy and robust way to do this?
There are a few standard ways to realize what you describe.
Read from environmental variables
Each of those environmental stores a particular parameter. Then users can control, if they want to change some or all of the parameters.
Pros: Standard and handy way for both developers and users, providing the parameters are seldom changed. The default parameters can be easily shared in the same environment (like the same account or even platform-wide), as well as with other programs.
Cons: Somewhat obscured. For example, if a user's work with the tool is logged in a file, other people (or even the user her/himself later) cannot tell what the exact set of the parameters used was from the logfile. Or, if a user wants to change a parameter frequently, it is awkward.
An example:
key_file = ENV['KEY_FILE'] || "henrb028.key"
Read from a configuration file
Then either or both of each user and the system administrator can control and change it, when needed. How to determine the filename of the configuration file varies.
Pros: Suitable to store a large set of parameters.
Cons: Somewhat obscured. For example, if a user's work with the tool is logged in a file, other people (or even the user her/himself later) cannot tell what the exact set of the parameters used was. If a user wants to change a parameter frequently, it is very awkward.
A (crude!) example:
Suppose /etc/OUR_CONFIG.txt is as follows,
CONSUMER_FILE: henrb028.consumer
KEY_FILE: henrb028.key
Then, read them like,
opts = { 'KEY_FILE' => 'default.key' }
IO.readlines("/etc/OUR_CONFIG.txt").each do |ec|
a = ec.strip.split(/\s*:\s*/)
opts[a[0]] = a[1]
end
p opts
Specify with command-line options
If some (or all) options are not specified at the run time, it should fall back to the default value. OptionParser (as #tadaman suggested in his/her comment) is a powerful and versatile library to handle command-line options.
Pros: Standard and handy way for users, especially for a quick change at the run time. If you see the log file, you know what exactly the user did.
Cons: The default parameters cannot be shared with other tools on its own. To circumvent the problem, there are a few ways to make a fall-back routine:
Simply hard-code the default value in the main code.
Store the default value in an environmental variable.
Store the default value in a configuration file. The developer must decide how to determine the filename of the configuration file (hard-coding, environmental variable, etc).
A crude example (without using OptionParser):
opts = { 'key_file' => 'default.key' }
%w(consumer_file key_file).each do |ec|
cands = ARGV.grep(/^--#{Regexp.quote ec}=.+/)
opts[ec] = cands[0].split('=')[1] unless cands.empty?
end
p opts
I think these are the most typical ways, though there are other ways!
I want to write a shell script to bulk add users to a group in eDirectory. The trouble is, I'm not sure where to begin regarding LDAP calls and whatnot for this task. Can anyone point me to a helpful resource or demonstrate a method for accomplishing this task?
Using straight-up LDAP would be fine, but I imagine that there must exist faster utilities/constructs that can be used in shell.
NetIQ's LDAPConfig or NDSConfig utilities seemed promising, but I don't see anything about adding a user to a group in the documentation.
Turns out, I don't have to code a solution to this. Instead of selecting each item and adding them to a group, there's a tool buried deep inside iManager that allows you to select all users matching a set of conditions and add them to a group.
I'd like to be able to write tests or specs that state that user soandso can read or edit file such_and_such. Although I would ultimately like this to be in RSpec, I'm assuming it will either all boil down to Bash commands (neatly wrapped in Ruby methods, of course), since it doesn't seem to be available in Ruby's File class, or anything else I've looked at. I've been hunting around for something in Bash to do this, and can't find anything (and nothing in the File class looks useful either). (Note: File.owned? does not do what I want, as it only tests if the process running the Ruby code is the owner, not if any arbitrary user has rights to edit or read, so it's significantly different on two counts.)
Is there any way to do this, built into Bash? (Or Ruby, or some Ruby gem, that I missed?) Or do I need to build a system myself, by getting owner and group info for the file, as well as read, write, and execute bits for each, and then looking up members of the group (in case so_and_so is not the owner), and seeing if the desired permissions are available to soandso either through ownership or group membership?
I'm currently only concerned about doing this on Unix-like systems, although something not dependent on a Unix shell that would also run on Windows would be a nice bonus.
Also, I'm interested in testing actual files, so something like FakeFS is not (as far as I can see) useful to me. I'm not trying to test how my Ruby code will interact with a file system, I'm trying to verify that all the necessary operations can be performed on actual files and directories. (Again,) I want to be able to specify that soandso can edit (or read) file such_and_such, NOT specify that file such_and_such is owned by soandso. The whole point of this is not specifying ownership (since that's an implementation detail and may need to change to accommodate other requirements, such as security) but only specify what my application and users actually need to be able to do to/in the file system.
If the user running your Ruby script has the necessary permissions, you could try running a test with the specific user with sudo.
sudo -u soandso test -r such_and_such
-r tests readability, -w writability. The return code is 0 if the test is passed.
By the way, I think that since the feature you are looking for could only be available for the super-user (since figuring out whether someone else has enough permissions would need you to have read permissions at least), it makes sense that it is not readily available in Ruby File class.
I haven't tried it, but I wonder if the FileTest methods would work as they say they operate on the "effective user id".
File.readable?(file_name) → true or false Link
Returns true if the named file is readable by the effective user id of this process.
File.writable?(file_name) → true or false Link
Returns true if the named file is writable by the effective user id of this process.
That in conjunction with seteuid might do the trick.
Process::Sys.seteuid(integer) → nil Link
Set the effective user ID of the calling process to integer. Not available on all platforms.
There is also setegid for group stuff...
Parse Permissions with File::Stat and Etc Modules
While this isn't a pre-packaged solution, you should be able to inspect a given file where you have sufficient access for stat in order to get mode, uid, and gid information. You can then parse /etc/passwd and /etc/group in order to find out whether a given user would have sufficient permissions.
Some building blocks include:
# Get the file's octal mode.
File.stat('/etc/passwd').mode.to_s(8)
#=> "100644"
# Get the group ID assigned to the file.
File.stat('/etc/passwd').gid
#=> 0
# Get the username associated with the given GID.
Etc.getpwuid(0).name
#=> "root"
You could also use the id utility to get information about a given user. For example:
%x{id root}
#=> "uid=0(root) gid=0(root) groups=0(root)\n"
but I think it would be easier to use the standard libraries rather than parse the output from id. YMMV.
I'm currently writing an rails application that generates some PDFs. The generated documents are things like proposals, invoices, order confirmations etc.
My intention is to provide 2 default layouts and to allow the user to request a custom layout. The user will never even see the generator code, this is all handled by trusted persons -> security is no concern at all.
My requirements:
Easy to change / add new generators (for developers, users never see those generators)
No redeployment
Works well with any number of generators
The way I see it, implementing those generators as regular classes in .rb files in the source code falls flat (redeployment required, file clutter with enough of them).
I'm currently thinking about the following and would like some input into the viability / better ways to do it:
The generator code is stored in the database and whenever a document is to be printed, the code is evaled in a scope where all the relevant variables are already set (document (-> line items | customer ..), pdf, user etc.) and the generator code just uses them.
My questions is basically: Is this the way to do it and if that's the case, are there some things I should be aware of?