I'm currently using ansible 2.3.3 and trying to upgrade it to a newever version but since 2.4+ the ansible.inventory.Inventory class has been removed.
Does anybody know what is the equivalent recommended replacement of ansible.inventory.Inventory in Ansible 2.4+?
from ansible.inventory import Inventory
from ansible.parsing.dataloader import DataLoader
from ansible.vars import VariableManager
inv = Inventory(
loader = DataLoader(),
variable_manager = VariableManager(),
host_list = my_path,
)
There isn't a one to one replacement in the newer code but the following code can be used but note that the inventory now has a different data structure
from ansible.inventory.manager import InventoryManager
from ansible.parsing.dataloader import DataLoader
m = InventoryManager(loader=DataLoader(), sources='/path/to/inventory')
m.hosts
m.groups
Related
When I run this simple pipeline (in GCP's Vertex AI Workbench) I get an error:
ModuleNotFoundError: No module named 'sklearn'
Here is my code:
from kfp.v2 import compiler
from kfp.v2.dsl import pipeline, component
from google.cloud import aiplatform
#component(
packages_to_install=["sklearn"],
base_image="python:3.9",
)
def test_sklearn():
import sklearn
#pipeline(
pipeline_root=PIPELINE_ROOT,
name="sklearn-pipeline",
)
def pipeline():
test_sklearn()
compiler.Compiler().compile(pipeline_func=pipeline, package_path="sklearn_pipeline.json")
job = aiplatform.PipelineJob(
display_name=PIPELINE_DISPLAY_NAME,
template_path="sklearn_pipeline.json",
pipeline_root=PIPELINE_ROOT,
location=REGION
)
job.run(service_account=SERVICE_ACCOUNT)
What do I do wrong? :)
It seems that the package name sklearn does not work after a version upgrade.You need to change the value of packages_to_install from "sklearn" to "scikit-learn" in the #component block.
After testing the logger library locally, I uploaded it to pypi.
Afterwards, when I proceeded with pip install, there was an error saying that the module inside the library could not be found.
So, as a temporary measure, I added a syntax to register all .py in init.py in the library package folder, and I want to improve this. This is because you have to install all dependencies for features that users may not be using
What improvements can I take in this situation?
If possible, I would like to know how to lazy use only the modules used by the user instead of registering all .py in init.py .
Or is there something structurally I'm overlooking?
Here is the project structure I used
project_name
- pacakge_name
- __init__.py. <- all loggers were registered
- file_logger.py
- console_logger.py
- ...
- fluent_logger.py <- used external library
- scribe_logger.py <- used external library
init.py
"""
Description for Package
"""
from .composite_logger import CompositeLogger
from .console_logger import ConsoleLogger
from .file_logger import FileLogger
from .fluent_logger import FluentLogger
from .jandi_logger import JandiLogger
from .line_logger import LineLogger
from .logger_impl import LoggerImpl
from .logger_interface import LoggerInterface
from .logger import Logger
from .memory_logger import MemoryLogger
from .null_logger import NullLogger
from .scribe_logger import ScribeLogger
from .telegram_logger import TelegramLogger
from .retry import Retry
__all__ = [
'CompositeLogger',
'ConsoleLogger',
'FileLogger',
'FluentLogger',
'JandiLogger',
'LineLogger',
'LoggerImpl',
'LoggerInterface',
'Logger',
'MemoryLogger',
'NullLogger',
'ScribeLogger',
'TelegramLogger',
'Retry',
]
setup.py
import setuptools
from distutils.core import setup
with open("README.md", "r", encoding="utf-8") as f:
long_descriprion = f.read()
setuptools.setup(
name = 'project_name',
version = '0.0.1',
description = 'python logger libary',
long_description = long_descriprion,
long_description_content_type = "text/markdown",
author = 'abc',
author_email = 'abc#gmail.com',
url = "https://github.com/##/##",
packages = ["pacakge_name"],
install_requires=[ <- contains too many external libraries
'requests>=2.0.0',
'thrift>=0.16.0',
'facebook-scribe>=2.0.post1',
'fluent-logger>=0.10.0'
],
keywords = ['logger'],
python_requires = '>=3.7',
classifiers = [
'Programming Language :: Python :: 3.7',
"License :: OSI Approved :: MIT License",
"Operating System :: OS Independent"
],
)
I am trying to use the AWS-SDK and Kinesis Client library with JMeter to produce records to put into my Kinesis stream for performance testing purposes. My issue is that JMeter will not recognize the libraries to be able to call them from my code. I am at a loss as to what to troubleshoot next.
Installed JMeter 5.0 (I tried with version 4.0 with same issues) and Java 8.
$ java -version
java version "1.8.0_202-ea"
Java(TM) SE Runtime Environment (build 1.8.0_202-ea-b03)
Java HotSpot(TM) 64-Bit Server VM (build 25.202-b03, mixed mode)
Placed JAR files for the AWS libraries in the lib folder of my JMeter installation (also tried lib/ext):
- amazon-kinesis-client-2.1.0.jar
- aws-java-sdk-1.11.490.jar
- aws-sdk-java-2.3.10-SNAPSHOT.jar
- aws-sdk-java-aws-core-2.3.10-SNAPSHOT.jar
- aws-sdk-java-kinesis-2.3.10-SNAPSHOT.jar
- aws-sdk-java-sdk-core-2.3.10-SNAPSHOT.jar
(also added these three based on another post - not sure if they are really needed or not)
- jackson-annotations-2.9.8.jar
- jackson-core-2.9.8.jar
- jackson-databind-2.9.8.jar
JSR223 sampler using Groovy code (based on other examples):
import groovy.json.*
import groovy.json.JsonBuilder
import groovy.json.JsonOutput
import com.amazonaws.AmazonClientException
import com.amazonaws.AmazonServiceException
import com.amazonaws.auth.AWSCredentials
import com.amazonaws.auth.profile.ProfileCredentialsProvider
import com.amazonaws.services.kinesis.AmazonKinesisClient
import com.amazonaws.services.kinesis.model.CreateStreamRequest
import com.amazonaws.services.kinesis.model.DescribeStreamRequest
import com.amazonaws.services.kinesis.model.DescribeStreamResult
import com.amazonaws.services.kinesis.model.ListStreamsRequest
import com.amazonaws.services.kinesis.model.ListStreamsResult
import com.amazonaws.services.kinesis.model.PutRecordRequest
import com.amazonaws.services.kinesis.model.PutRecordResult
import com.amazonaws.services.kinesis.model.ResourceNotFoundException
import com.amazonaws.services.kinesis.model.StreamDescription
class AmazonKinesisRecordProducerSample {
def kinesis
def init() {
AWSCredentials credentials = null
credentials = new ProfileCredentialsProvider().getCredentials()
kinesis = new AmazonKinesisClient(credentials)
}
}
def amazonKinesisRecordProducerSample= new AmazonKinesisRecordProducerSample()
amazonKinesisRecordProducerSample.init()
def Environment = vars.get("Environment") //as String
def AWS_Region = vars.get("AWS_Region") //as String
def myStreamName="stream123-" + Environment + "-" + AWS_Region
def jsondata = new JsonBuilder()
jsondata {
<VARIOUS JSON DATA ADDED>
}
def data=jsondata.toPrettyString()
log.info("jsondata: " + data)
def partitionkey="<PARTITION KEY>"
def putRecordRequest = new PutRecordRequest()
putRecordRequest.setStreamName(myStreamName)
putRecordRequest.setData(ByteBuffer.wrap(String.valueOf(data).getBytes()))
putRecordRequest.setPartitionKey(partitionkey)
def putRecordResult = new PutRecordResult()
putRecordResult = amazonKinesisRecordProducerSample.kinesis.putRecord(putRecordRequest)
log.info("Successfully put record, partition key : %s, ShardID : %s, SequenceNumber : %s.\n",
putRecordRequest.getPartitionKey(),
putRecordResult.getShardId(),
putRecordResult.getSequenceNumber())
I get this error for every com.amazonaws.* I try to import:
Script1.groovy: 14: unable to resolve class com.amazonaws.services.kinesis.AmazonKinesisClient
# line 14, column 1.
import com.amazonaws.services.kinesis.AmazonKinesisClient
for reference
How to put 25k record to kinesis stream and Test tool to acknowledge it and Using HTTP Request of Jmeter to put records into Amazon Kinesis
To resolve com.amazonaws.services.kinesis classes you may:
drop aws-java-sdk-1.11.490.jar to JMeter\lib from AWS SDK for Java 1.11.0
Restart JMeter
Also, note
JMETER_HOME/lib/ext used for JMeter components and plugins
Other jars (such as JDBC, JMS implementations and any other support libraries needed by the JMeter code) should be placed in the JMETER_HOME/lib directory - not the lib/ext directory, or added to user.classpath.
This a section in a schema file
imports:
- path: configs/folder1/resources/gcpresource/test/*
I am trying to import all the files in a folder using the template's schema file.
I know that this does not work.
My question is,
what is a better way to import all the files using * or something that's suitable so that deployment manager can import all of the files in a folder without having to explicitly specify them ?
You could do so in python if they are all yaml config files
# my-template.yaml
imports:
- path: omni-importer.py
And in the middleware-like python template do something like:
# omni-importer.py
import yaml
from os import listdir
from os.path import isfile, join
def generate_config(ctx):
mypath = ctx.properties['path']
files = [f for f in listdir(mypath) if isfile(join(mypath, f)) and f.endswith('yaml')]
yamls = map(lambda f: yaml.load(open('test.txt')['resources']), files)
return {
'resources': yamls,
}
This is not working code, but rather a proof of concept
I know this is obvious to everybody but me.
But where and how do I define a custom jinja2 test?
I tried specifying in my flask:
import flask
import jinja2
from jinja2 import environment as env
from jinja2 import *
app = Flask(__name__)
app.config.from_object(__name__)
app = Flask(__name__)
# jinja2 filter
def isList(value):
return isinstance(value, list)
env.tests['isList'] = isList
The resulting error is:
AttributeError: 'module' object has no attribute 'tests'
The configured Jinja2 environment for Flask applications is app.jinja_env. If you change the last line in your code to
app.jinja_env.tests['isList'] = isList
it should work properly.