I created a custom MergeField jar file and copied it to debezium/connect:1.6 image to create a new image "connect-test".
FROM debezium/connect:1.6
COPY MergeField /kafka/plugins/MergeField
Then run my yaml file and specified the plugin path
CONNECT_PLUGIN_PATH=/kafka/plugins/,/kafka/connect
connect:
image: connect-test
container_name: connect
ports:
- 8083:8083
- 5005:5005
links:
- kafka
- elastic
- zookeeper
- schema-registry
environment:
- BOOTSTRAP_SERVERS=kafka:9092
- GROUP_ID=1
- CONFIG_STORAGE_TOPIC=my_connect_configs
- OFFSET_STORAGE_TOPIC=my_connect_offsets
- STATUS_STORAGE_TOPIC=my_source_connect_statuses
- KEY_CONVERTER=io.confluent.connect.avro.AvroConverter
- VALUE_CONVERTER=io.confluent.connect.avro.AvroConverter
- INTERNAL_KEY_CONVERTER=org.apache.kafka.connect.json.JsonConverter
- INTERNAL_VALUE_CONVERTER=org.apache.kafka.connect.json.JsonConverter
- CONNECT_KEY_CONVERTER_SCHEMA_REGISTRY_URL=http://schema-registry:8081
- CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL=http://schema-registry:8081
- CONNECT_PLUGIN_PATH=/kafka/plugins/,/kafka/connect
From the log file, it looks like the custom transformation was loaded.
Kafka-connect log file
"transforms.MergeFields.type":"org.apache.kafka.connect.transforms.MergeField$Value",
"transforms.MergeFields.field.list":"__table,Id",
"transforms.MergeFields.field.root":"uniqueId"
Adding the connector with the custom transformation gave this error
{"error_code":400,"message":"Connector configuration is invalid and contains the following 2 error(s):\nInvalid value org.apache.kafka.connect.transforms.MergeField$Value for configuration transforms.MergeFields.type: Class org.apache.kafka.connect.transforms.MergeField$Value could not be found.\nInvalid value null for configuration transforms.MergeFields.type: Not a Transformation\nYou can also find the above list of errors at the endpoint `/{connectorType}/config/validate`"}
Which I think is because kafka connect could not find the custom transformation.
What am I doing wrong?
Related
The Log entry looks like this
2022-05-12 10:32:19,659 - root:loadrequestheaders:33 - INFO: otds=XYZ, domain=XYZ, username=user, origin=10.98.76.67
I tried to add the promtail configuration to scrap log entry to get new labels :
Updated ConfigMap - loki-promtail with below content
- job_name: kubernetes-pods-static
pipeline_stages:
- match:
selector: '{container="model"}'
stages:
- regex:
expression: '\.*username=(?P<username>\S+).*origin=(?P<origin>\S+)'
- labels:
username:
origin:
I could see loki pods are running successfully with above config map update but I don't see the entries "username and origin" - in the labels list
Any suggestion on how to add new labels in loki datasource.
enter image description here
Im trying to use Docker-Compose on Microsoft Windows to create a stack for Seafile.
The error message after creating is:
Deployment error
failed to deploy a stack: Named volume “C:/Users/Administrator/Docker/Volumes/Seafile/Mysql:/var/lib/mysql:rw” is used in service “db” but no declaration was found in the volumes section. : exit status 1
Here's my problematic docker-compose.yaml file :
version: '2'
services:
db:
image: mariadb:10.5
container_name: seafile-mysql
environment:
- MYSQL_ROOT_PASSWORD=db_dev # Requested, set the root's password of MySQL service.
- MYSQL_LOG_CONSOLE=true
volumes:
- C:/Users/Administrator/Docker/Volumes/Seafile/Mysql:/var/lib/mysql # Requested, specifies the path to MySQL data persistent store.
networks:
- seafile-net
memcached:
image: memcached:1.5.6
container_name: seafile-memcached
entrypoint: memcached -m 256
networks:
- seafile-net
seafile:
image: seafileltd/seafile-mc:latest
container_name: seafile
ports:
- "9000:80"
# - "443:443" # If https is enabled, cancel the comment.
volumes:
- C:/Users/Administrator/Docker/Volumes/Seafile/Seafile:/shared # Requested, specifies the path to Seafile data persistent store.
environment:
- DB_HOST=db
- DB_ROOT_PASSWD=db_dev # Requested, the value shuold be root's password of MySQL service.
- TIME_ZONE=Etc/UTC # Optional, default is UTC. Should be uncomment and set to your local time zone.
- SEAFILE_ADMIN_EMAIL=me#example.com # Specifies Seafile admin user, default is 'me#example.com'.
- SEAFILE_ADMIN_PASSWORD=asecret # Specifies Seafile admin password, default is 'asecret'.
- SEAFILE_SERVER_LETSENCRYPT=false # Whether to use https or not.
- SEAFILE_SERVER_HOSTNAME=docs.seafile.com # Specifies your host name if https is enabled.
depends_on:
- db
- memcached
networks:
- seafile-net
networks:
seafile-net:
If you see the error "no declaration was found in the volumes section" - probably you are not declaring the volumes from the root section.
The error message can cause confusion. Here how to do it correctly:
...
services:
...
volumes:
- a:/path1
- b:/path2
...
volumes:
a:
b:
...
I know that this could be somehow scattered and I know Docker could handle it differently in another universe, but at the current version it does it in this way: the root section declares the volume, while the services section just use them.
Let me know if this was your problem.
More info:
https://docs.docker.com/storage/volumes/#use-a-volume-with-docker-compose
I have set an Oracle docker image (https://github.com/oracle/docker-images/tree/main/OracleDatabase/SingleInstance/dockerfiles) which by default is running on port 1521.
I would like to change the port in the Image to 1531.
I know that in the docker-compose I can set "1531:1521" BUT the other container still searching for port 1521 in the created network.
I tried to modify the port referenced in the Dockerfile of the version I want to use (19.3.0) and also in the createDB.sh but when I try to connect with the SID it fails, the listener is not working as expected.
Anybody already succeeded?
Update 1:
Here is the error message when I try to connect to the database after I changed the port.
SQL> CONNECT sys/HyperSecuredPassword#ORCLCDB AS sysdba; ERROR: ORA-12514: TNS:listener does not currently know of service requested in connect descriptor
Update 2:
I have the following docker-compose.yaml to set up the other containers for my project.
version: "3.8"
services:
hadea-database:
image: hadea_oracle_1521:19.3.0
container_name: hadea_oracle_1930
ports:
- "1521:1521"
environment:
- ORACLE_SID=ORCLCDB
- ORACLE_PDB=ORCLPDB
- ORACLE_PWD=Oracle4System
- ORACLE_MEM=2000
volumes:
- ./database/OracleDB/oradata:/opt/oracle/oradata
- ./database/OracleDB/setup:/opt/oracle/scripts/setup
- ./database/OracleDB/startup:/opt/oracle/scripts/startup
networks:
- hadea-network
hadea-maildev:
image: maildev/maildev
container_name: hadea_maildev
command: bin/maildev --web 80 --smtp 25 --hide-extensions STARTTLS
ports:
- "8081:80"
networks:
- hadea-network
hadea-server:
build:
context: ./server
dockerfile: Dockerfile
container_name: hadea_back
environment:
- HTTP_PORT=3000
- HTTP_HOST=0.0.0.0
- DATABASE_HOST=hadea-database
- DATABASE_PORT=1521 # CONTAINER port NOT the HOST port
- DATABASE_SID=ORCLCDB
- MAIL_HOST=hadea-maildev
- MAIL_PORT=25 # CONTAINER port NOT the HOST port
ports:
- "3000:3000"
working_dir: /usr/src/app
volumes:
- ./server:/usr/src/app
networks:
- hadea-network
depends_on:
- hadea-database
- hadea-maildev
hadea-front:
build:
context: ./front
dockerfile: Dockerfile
container_name: hadea_front
ports:
- "4200:4200"
- "3001:3001"
volumes:
- ./front:/usr/src/app
networks:
- hadea-network
depends_on:
- hadea-database
- hadea-maildev
- hadea-server
networks:
hadea-network:
If you want to change the port used WITHIN the container (I think this is the question), you could try building a new image after modifying the conf file, e.g. (for the 18c image). The other images hard code the 1521 port in various files in that repo depending on the oracle version you are using, so those would have to be changed prior to building the image.
I have been using this image: container-registry.oracle.com/database/express:latest. This is version 18c and it has a conf file within the image located at /etc/sysconfig/oracle-xe-18c.conf, I would just build a new Dockerfile and overwrite that file with a new one that has the port you require. Or, you could extract the entire contents of that directory, dump it to a host directory, modify the file as needed, and map a volume to etc/sysconfig (make sure the permissions are correct). This way you could tweak the file from the host. It might be possible to set the variable in that conf file from an environment variable within a docker-compose.yaml file or on the docker command line. This variable is named LISTENER_PORT. Some of the variables in these scripts are defined locally and do not pull their values from environment variable though.
We have a Ruby project where we are using Wercker as Continuous Integration.
We need to start an Elastic Search service in order to run some integration tests.
Locally, we added the Elastic configuration to the docker file and everything runs smoothly:
services:
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:6.5.1
container_name: elasticsearch
environment:
- discovery.type=single-node
ports:
- "9200:9200"
- "9300:9300"
In The Wercker.yml file, we tried several things, but we cannot reach the elastic service.
Our wercker.yml contains:
services:
- id: elasticsearch:6.5.1
env:
ports:
- "9200:9200"
- "9300:9300"
We have this king of error when trying to use Elastic in our tests:
Errno::EADDRNOTAVAIL: Failed to open TCP connection to localhost:9200 (Cannot assign requested address - connect(2) for "localhost" port 9200)
Do you have any idea of what we are missing?
So, we found a solution:
In wercker.yml
services:
- id: elasticsearch:6.5.1
cmd: "/elasticsearch/bin/elasticsearch -Ediscovery.type=single-node"
And we added a step to check the connection:
build:
steps:
- script:
name: Test elasticsearch connection
code: curl http://elasticsearch:9200
I am working on IdentityServer in Microservice. I am having issued with exception:
System.InvalidOperationException: 'HTTPS endpoints can only be configured using KestrelServerOptions.Listen().'
I placed UseKestrel() inside the BuildWebHost in Program.cs file. What other thing do I need to configure. I can see a Certificate folder under eShopOnContainer project. How do I generate this files.
This is the identity server docker.compose config:
services:
identitywithdocker:
environment:
- ASPNETCORE_ENVIRONMENT=Development
- ASPNETCORE_URLS=https://+:443;http://+:80
- ASPNETCORE_HTTPS_PORT=44301
ports:
- "64636:80"
- "44301:443"
volumes:
- ${APPDATA}/ASP.NET/Https:/root/.aspnet/https:ro
- ${APPDATA}/Microsoft/UserSecrets:/root/.microsoft/usersecrets:ro