Failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB) in quic golang appengine - go

I am using a google cloud app engine to deploy my quic-go server. But getting the error:
failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB).
I am using app.yaml file to build a docker file which is as follows:
FROM golang:1.18.3
RUN mkdir /app
ADD . /app
WORKDIR /app
RUN apt-get update && apt-get install -y ffmpeg
CMD sudo --sysctl net.core.rmem_default=15000000
CMD sudo --sysctl net.core.rmem_max=15000000
RUN go build -x server.go
ENV GCS_BUCKETNAME xyz
ENV AI_CLIENT_SSL_CERT /path to cert
ENV AI_CLIENT_SSL_KEY /path to key
ENV GCP_BUCKET_SERVICE_ACCOUNT_CREDS /path to google cloud service account credential
CMD [ "./server" ]
This is my app.yaml
runtime: custom
env: flex
env_variables:
GCS_BUCKETNAME : "xyz"
AI_CLIENT_SSL_CERT : "./path to cert"
AI_CLIENT_SSL_KEY : "./path to key"
GCP_BUCKET_SERVICE_ACCOUNT_CREDS : "./path to google cloud credential.json file"
service: streaming-app
automatic_scaling:
min_num_instances: 1
max_num_instances: 20
cpu_utilization:
target_utilization: 0.85
target_concurrent_requests: 100
Any sort of help will be appreciated.

Since sysctl is an OS-level config that doesn't fit in line with App Engine's principle use case. App Engine does not currently have any way of configuring the underlying sysctl config files. I believe that Google Kubernetes engine may be a better use case for running that server, as App Engine environments have a limited set of configurable settings.
can you tell me the scenarios when this file is not present in the kernel?
I’m not sure about the scenarios as I have least experience with kernel. For me it seems a different question rather than original post. you can raise a new StackOverflow question regarding this.

Related

ERROR: (gcloud.auth.activate-service-account) The .json key file is not in a valid format -- via impersonate-service-account

Is it possible to use short-lived credentials, with docker-compose, to run a bash scripted gcloud command?
Related posts that I attempted to use but they are 5+ years old and I've been led to believe that the gcloud auth command has changed during this time:
ERROR: (gcloud.auth.activate-service-account) Failed to activate the given service account. Please ensure provided key file is valid
gcloud auth activate-service-account [ERROR] Please ensure provided key file is valid
Setup
there is a lot going on but I've attempted to abbreviate to the relevant parts
Makefile
auth: ## commands for short lived auth
#gcloud config set project ${GCP_PROJECT}
#gcloud auth application-default login --impersonate-service-account="inst-dataflow-svc#${GCP_PROJECT}.iam.gserviceaccount.com"
#gcloud auth configure-docker $(REGION)-docker.pkg.dev
gcloud-flex-build: ## build & push base docker image
docker-compose build gcloud-build-flex-local
docker-compose run gcloud-build-flex-local
docker-compose.yaml
version: '3.4'
services:
gcloud-build-flex-local:
build:
dockerfile: docker/gcloud-build-flex-template.dockerfile
context: .
image: us-central1-docker.pkg.dev/gcp-project/dataflow-docker-registry/local-build/pubsub-to-gbq-build-flex-template
volumes:
- type: bind
source: ${HOME}/.config/gcloud/
target: /tmp
docker/gcloud-build-flex-template.dockerfile
FROM gcr.io/google.com/cloudsdktool/cloud-sdk:408.0.1
COPY docker/scripts/gcloud-build-flex-template.sh /app/gcloud-build-flex-template.sh
COPY dataflow/pubsub-to-gbq/pubsub-to-gbq-metadata /app/pubsub-to-gbq-metadata
WORKDIR /app
ENTRYPOINT "/app/gcloud-build-flex-template.sh"
/app/gcloud-build-flex-template.sh
#!/bin/bash
set -euo pipefail
SERVICE_ACCOUNT_EMAIL=inst-dataflow-svc#gcp-project.iam.gserviceaccount.com
GCP_PROJECT=gcp-project
export GOOGLE_APPLICATION_CREDENTIALS=/tmp/application_default_credentials.json
# debugging
echo $GOOGLE_APPLICATION_CREDENTIALS
ls -lah /tmp/
cat $GOOGLE_APPLICATION_CREDENTIALS
gcloud auth activate-service-account $SERVICE_ACCOUNT_EMAIL --project=$GCP_PROJECT --key-file=$GOOGLE_APPLICATION_CREDENTIALS
Execution
make auth
make gcloud-flex-build
Error
ERROR: (gcloud.auth.activate-service-account) The .json key file is not in a valid format.
make: *** [gcloud-flex-build] Error 1
stdout (abbreviated)
docker-compose build gcloud-build-flex-local
[+] Building 0.4s (9/9) FINISHED
...
docker-compose run gcloud-build-flex-local
drwxr-xr-x 17 root root 544 Dec 30 10:36 .
drwxr-xr-x 1 root root 4.0K Dec 30 10:40 ..
-rw------- 1 root root 591 Dec 30 10:36 application_default_credentials.json
{
"delegates": [],
"service_account_impersonation_url": "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/inst-dataflow-svc#gcp-project.iam.gserviceaccount.com:generateAccessToken",
"source_credentials": {
"client_id": "alphanumeric string .apps.googleusercontent.com",
"client_secret": "alphanumeric string",
"refresh_token": "alphanumeric string",
"type": "authorized_user"
},
"type": "impersonated_service_account"
}
I can make it work via docker run by spoofing the credentials to include only the "source_credentials" object, passed in as a volume, but this same trick doesn't seem to work with docker-compose running a script inside a container...
There is a similar type of configuration mentioned in this document. This involves three major steps:
Create short-lived credentials for your service account and download
your service account keys.
Create the configuration files for making your docker environment up. Use
the above cred files for granting required permissions.
Once you have all the configuration files in place use your docker-compose
commands for making your environment up.
Follow this documentation for more details.

Docker Image on AWS Lambda is executing the RUN/Entrypoint Twice on Testing

I am currently learning and playing around with Lambda and Docker. I have currently the Docker File:
FROM amazonlinux:2.0.20191016.0
RUN yum install jq -y
COPY . ./
CMD chmod 755 ./random.sh ; chmod 755 ./discord.sh
ENTRYPOINT "./random.sh"
Pretty Basic and the File random.sh is sending via discord.sh a message to my Discord Server.
When i am doing a Test Run its seems like he is calling the ./random.sh twice
2022-04-05T13:24:23.537+02:00 9
2022-04-05T13:24:23.537+02:00 https://www.oetker.at/dr-oetker-cms/oetker.de/image/image-thumb__47425__auto_23393e4cf279157878cad04620baa711/Paula-am-kochen_02.png
2022-04-05T13:24:23.971+02:00 START RequestId: c3dca9f8-1a3f-415b-8a0c-b41cd441fb84 Version: $LATEST
2022-04-05T13:24:24.023+02:00 3
2022-04-05T13:24:24.023+02:00 https://www.sueddeutsche.de/image/sz.1.937584/640x360?v=1528418182
2022-04-05T13:24:24.726+02:00 END RequestId: c3dca9f8-1a3f-415b-8a0c-b41cd441fb84
2022-04-05T13:24:24.726+02:00 REPORT RequestId: c3dca9f8-1a3f-415b-8a0c-b41cd441fb84 Duration: 752.69 ms Billed Duration: 753 ms Memory Size: 128 MB Max Memory Used: 6 MB
2022-04-05T13:24:24.726+02:00 RequestId: c3dca9f8-1a3f-415b-8a0c-b41cd441fb84 Error: Runtime exited without providing a reason Runtime.ExitError
taht is the log of the execution of the File and it seems like he is running it twice the code of my random.sh as you can see with the numbers and the Link which get logged.
And a other Problem ist how do i fix the Runtime error. because my Random.sh is executing always with a exit 0 which should return a succesfull.
I hope you can help me out, i could fix it simply by writing in python and use simple lamdba function but i wanted to try this out and normaly the ENTRYPOINT should be exceuted once. I made Asynchronous invocation aswell to 0 so it dont retry at failed
The lambda deployed as Image isn't expected to work as containerized App (such as Amazon ECS)
You need to install lambda runtime interface client and integrate your code with it, check the following guide (Creating images from alternative base images)

Unable to get heroku started on cloud9 ide

I am running the command - curl https://cli-assets.heroku.com/install-ubuntu.sh | sh
Which throws the following error -
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1232 100 1232 0 0 5133 0 --:--:-- --:--:-- --:--:-- 5112
This script requires superuser access to install apt packages.
You will be prompted for your password by sudo.
+ dpkg -s apt-transport-https
+ echo ''
sh: line 4: /etc/apt/sources.list.d/heroku.list: No such file or directory
I also ran sudo snap install --classic heroku which returned sudo: snap: command not found.
Then I ran sudo apt install snapd which returned the following -
apt: invalid flag: install
Usage: apt <apt and javac options> <source files>
where apt options include:
-classpath <path> Specify where to find user class files and annotation processor factories
-cp <path> Specify where to find user class files and annotation processor factories
-d <path> Specify where to place processor and javac generated class files
-s <path> Specify where to place processor generated source files
-source <release> Provide source compatibility with specified release
-version Version information
-help Print a synopsis of standard options; use javac -help for more options
-X Print a synopsis of nonstandard options
-J<flag> Pass <flag> directly to the runtime system
-A[key[=value]] Options to pass to annotation processors
-nocompile Do not compile source files to class files
-print Print out textual representation of specified types
-factorypath <path> Specify where to find annotation processor factories
-factory <class> Name of AnnotationProcessorFactory to use; bypasses default discovery process
See javac -help for information on javac options.
warning: The apt tool and its associated API are planned to be
removed in the next major JDK release. These features have been
superseded by javac and the standardized annotation processing API,
javax.annotation.processing and javax.lang.model. Users are
recommended to migrate to the annotation processing features of
javac; see the javac man page for more information.
Finally, I ran wget 0- wget https://toolbelt.heroku.com/install-ubuntu.sh | sh which returned
--2020-09-10 18:15:53-- http://0-/
Resolving 0- (0-)... failed: Name or service not known.
wget: unable to resolve host address ‘0-’
--2020-09-10 18:15:54-- http://wget/
Resolving wget (wget)... failed: Name or service not known.
wget: unable to resolve host address ‘wget’
--2020-09-10 18:15:54-- https://toolbelt.heroku.com/install-ubuntu.sh
Resolving toolbelt.heroku.com (toolbelt.heroku.com)... 54.164.74.108, 107.23.162.152, 34.194.108.77, ...
Connecting to toolbelt.heroku.com (toolbelt.heroku.com)|54.164.74.108|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 719 [text/plain]
Saving to: ‘install-ubuntu.sh’
install-ubuntu.sh 100%[=============================================================>] 719 --.-KB/s in 0s
2020-09-10 18:15:54 (105 MB/s) - ‘install-ubuntu.sh’ saved [719/719]
FINISHED --2020-09-10 18:15:54--
Total wall clock time: 0.4s
Downloaded: 1 files, 719 in 0s (105 MB/s)
So then, I ran bash install-ubuntu.sh which returned -
This script requires superuser access to install apt packages.
You will be prompted for your password by sudo.
sh: line 3: /etc/apt/sources.list.d/heroku.list: No such file or directory
sh: line 6: apt-key: command not found
--2020-09-10 18:16:51-- https://toolbelt.heroku.com/apt/release.key
Resolving toolbelt.heroku.com (toolbelt.heroku.com)... 54.145.36.98, 54.164.74.108, 107.23.162.152, ...
Connecting to toolbelt.heroku.com (toolbelt.heroku.com)|54.145.36.98|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1737 (1.7K) [application/octet-stream]
Saving to: ‘STDOUT’
- 0%[ ] 0 --.-KB/s in 0s
Cannot write to ‘-’ (Success).
sh: line 9: apt-get: command not found
sh: line 12: apt-get: command not found
I am taking an online course on Upskill and am on video #125. Please advise how to progress.
Thanks for your time and help.
I have a gist I use specifically for setting up rails on Cloud9 (though I haven't updated it for Rails 6 yet). Recommend you read through it twice before you try to do it. You may need to go out of order in some cases, as exact approach depends on if you are cloning an existing repo or building a new app.
https://gist.github.com/MyklClason/791d6b14606bc56e72eba2995aab8e76
You probably don't need snap.
Also useful bash aliases for Cloud9:
https://gist.github.com/MyklClason/d71a39ace28b9ec9f0ad
As for your actual issue. Heroku toolbelt is obsolete, use this instead:
wget -qO- https://cli-assets.heroku.com/install-ubuntu.sh | sh
Also it's often best to just look online and check how to install the heroku cli (or anything really) for your OS using the offical documentation. Though that may not work for older setup. However, heroku is something where you basically have to use the newest version otherwise you are going to run into problems.
If you didn't figure this out...
nvm i v8
Followed by...
npm install -g heroku

Envoy and Evans cli running issue

I'm writing some envoy control plane based on
https://github.com/envoyproxy/go-control-plane
And trying to use evans cli for debugging
There is some issue I can't making it work with envoy data plane.
I've downloaded data-plane
https://github.com/envoyproxy/data-plane-api
Running evans
evans -p 5678 envoy/api/v2/*.proto
evans: failed to run REPL mode: failed to instantiate a new spec: failed to instantiate the spec from proto files: envoy/api/v2/core/http_uri.proto:11:8: open validate/validate.proto: no such file or dir$
ctory
Ok install https://github.com/envoyproxy/protoc-gen-validate
And run again
evans -p 5678 --path $GOPATH/src/github.com/envoyproxy/protoc-gen-validate envoy/api/v2/*.proto
evans: failed to run REPL mode: failed to instantiate a new spec: failed to instantiate the spec from proto files: envoy/api/v2/discovery.proto:12:8: open google/rpc/status.proto: no such file or directory
Is any right way to use data-plane?
And you know correct how-to to generate *.go files with protoc from envoy data-plane *.proto

Error starting hyperledger-composer network after Fabric and Composer version upgrade

I've come across an error starting the hyperledger-composer network that isn't answered in the composer-wiki.
✖ Starting business network definition. This may take a minute...
Error: Error trying to start business network. Error: No valid responses from any peers.
Response from attempted peer comms was an error: Error: transaction returned with failure: can't find PEM header: undefined
Command failed
Checking pre-requisites,
Fabric 1.2
Composer 0.20.4
Node 8.12.0
Docker 18.01.1
"composer network install" was successful, with file appearing in the docker peer at /var/hyperleder/production/chaincodes
After running the "composer network start" command, a "docker ps" shows new docker instance with name:
dev-peer0.org1.example.com-<<business-network-name>>-0.0.7
But any attempt to ping this results in a failure like this:
Error: Error trying to ping. Error: make sure the chaincode <<business-network-name>> has been successfully instantiated and try again: getccdata composerchannel/<<business-network-name>> responded with error: could not find chaincode with name '<<business-network-name>>'
Checking the log of the dev-peer0, it ends with the following:
2018-11-05T05:03:18.227Z [4264161f] ERROR :Composer :Init() can't find PEM header: undefined
2018-11-05T05:03:18.227Z [4264161f] VERBOSE :Composer :#PERF Init() Total (ms) duration for txnID [4264161fc30a61c70884d4c7efb460fea6a755d07bc4852875c393346795227a]: 929.00
2018-11-05T05:03:18.228Z ERROR [lib/handler.js] [composerchannel-4264161f]Calling chaincode Init() returned error response [can't find PEM header: undefined]. Sending ERROR message back to peer
The corresponding error in the peer0 log is a big larger:
2018-11-05 05:03:18.229 UTC [endorser] SimulateProposal -> ERRO 439d [composerchannel][4264161f] failed to invoke chaincode name:"lscc" , error: transaction returned with failure: can't find PEM header: undefined
github.com/hyperledger/fabric/core/chaincode.(*ChaincodeSupport).Execute
/opt/gopath/src/github.com/hyperledger/fabric/core/chaincode/chaincode_support.go:202
github.com/hyperledger/fabric/core/endorser.(*SupportImpl).Execute
/opt/gopath/src/github.com/hyperledger/fabric/core/endorser/support.go:131
github.com/hyperledger/fabric/core/endorser.(*Endorser).callChaincode
/opt/gopath/src/github.com/hyperledger/fabric/core/endorser/endorser.go:173
github.com/hyperledger/fabric/core/endorser.(*Endorser).SimulateProposal
/opt/gopath/src/github.com/hyperledger/fabric/core/endorser/endorser.go:287
github.com/hyperledger/fabric/core/endorser.(*Endorser).ProcessProposal
/opt/gopath/src/github.com/hyperledger/fabric/core/endorser/endorser.go:501
github.com/hyperledger/fabric/core/handlers/auth/filter.(*expirationCheckFilter).ProcessProposal
/opt/gopath/src/github.com/hyperledger/fabric/core/handlers/auth/filter/expiration.go:61
github.com/hyperledger/fabric/core/handlers/auth/filter.(*filter).ProcessProposal
/opt/gopath/src/github.com/hyperledger/fabric/core/handlers/auth/filter/filter.go:31
github.com/hyperledger/fabric/protos/peer._Endorser_ProcessProposal_Handler
/opt/gopath/src/github.com/hyperledger/fabric/protos/peer/peer.pb.go:112
github.com/hyperledger/fabric/vendor/google.golang.org/grpc.(*Server).processUnaryRPC
/opt/gopath/src/github.com/hyperledger/fabric/vendor/google.golang.org/grpc/server.go:923
github.com/hyperledger/fabric/vendor/google.golang.org/grpc.(*Server).handleStream
/opt/gopath/src/github.com/hyperledger/fabric/vendor/google.golang.org/grpc/server.go:1148
github.com/hyperledger/fabric/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1
/opt/gopath/src/github.com/hyperledger/fabric/vendor/google.golang.org/grpc/server.go:637
runtime.goexit
/opt/go/src/runtime/asm_amd64.s:2361
2018-11-05 05:03:18.229 UTC [endorser] SimulateProposal -> DEBU 439e [composerchannel][4264161f] Exit
Since this last worked I have updated composer from 0.19 to 0.20.4, and taken Fabric from 1.1 to 1.2.
Googling suggests that this kind of error "can't find PEM header: undefined" is associated with a change in key signing. After tearing down Fabric I re-ran ./createPeerAdminCard.sh - is there another card or similar that needs to be re-created to accomodate the latest versions?
Thanks to #R Thatcher for putting me onto the right direction. This was all down to mismatching cards, and was resolved by clearing out everything and starting again.
Specifically, in /fabric-dev-servers:
./stopFabric.sh
./teardownFabric.sh
composer card list
composer card delete -c admin#<business-network-name>
composer card delete -c PeerAdmin#hlfv1
./startFabric.sh
./createPeerAdminCard.sh
Then changing into the composer/business-network-name directory:
composer network install --card PeerAdmin#hlfv1 --archiveFile business-network-name\#0.0.7.bna
composer network start -c PeerAdmin#hlfv1 -n business-network-name -V 0.0.7 -A admin -S adminpw --file networkadmin.card
composer card import --file networkadmin.card --card admin#business-network-name
composer network ping -c admin#business-network-name
So yes, it was about mismatching cards and not cleaning these up as part of a new deployment.
Although not part of the original problem, it's also worth noting that the -A and -S parameters of the composer network start command HAD to be set to admin and adminpw respectively. See composer issue #3781.
Answering the the last remark from #Capn Sparrow
"the -A and -S parameters of the composer network start command HAD to be set to admin and adminpw respectively."
This is the correct and expected behaviour :-)
with the composer network start command the -A and -S are specifying an existing user in the CA that we want a new set of Credentials (certificate and keys) for which is then bound to a Composer System participant.
When you use the 'standard development fabric' this has a CA configured with a user called 'admin' with a secret of 'adminpw'. If you had build your own Fabric from scratch you could choose the name and secret of your first default user. Alternatively you could work with the fabric-ca client software to create additional users in the CA.

Resources