I'm trying to set up proxy for scrapy-playwright but always get the error
playwright._impl._api_types.Error: net::ERR_TIMED_OUT at http://whatismyip.com/
=========================== logs ===========================
navigating to "http://whatismyip.com/", waiting until "load"
when executing the code:
from scrapy import Spider, Request
from scrapy_playwright.page import PageMethod
class ProxySpider(Spider):
name = "check_proxy_ip"
custom_settings = {
"PLAYWRIGHT_LAUNCH_OPTIONS": {
"proxy": {
"server": "http://host:port",
"username": "user",
"password": "pass",
},
},
"PLAYWRIGHT_DEFAULT_NAVIGATION_TIMEOUT": "300000",
}
def start_requests(self):
yield Request("http://whatismyip.com",
meta=dict(
playwright=True,
playwright_include_page=True,
playwright_page_methods=[PageMethod('wait_for_selector', 'span.ipv4-hero')]
),
callback=self.parse,
)
def parse(self, response):
print(response.text)
The proxies tried are paid and working as checked, and the DOWNLOAD_DELAY in settings.py is set to DOWNLOAD_DELAY=30. This happens whether PLAYWRIGHT_DEFAULT_NAVIGATION_TIMEOUT is set to 0, 10000, or 300000 (as copied in the code above). What is the problem here?
Related
It does not give errors, but the statute simply does not appear in the discord.
from pypresence import Presence
from time import time
RPC = Presence("848861630226104320")
btns = [
{
"label": "YouTube",
"url": "https://www.youtube.com/watch?v=dQw4w9WgXcQ"
}
]
RPC.connect()
# noinspection PyTypeChecker
RPC.update(
start=time(),
buttons=btns,
)
input()
I reinstalled the library many times and wrote that everything is there, but nothing works. Any suggestions?
My Pods are running on AKS Cluster. Whenever my pods restarted, I had to get a notification on my team's channel, are there any articles or commands to configure the notification?
For that same, you can use tools or application like botkube : https://www.botkube.io/
Also check the Kubewatch : https://github.com/bitnami-labs/kubewatch
You can also implement the Grafana with the Prometheus and Alert manager for monitoring and getting the alert system. : https://github.com/grafana-operator/grafana-operator
However if you can not looking for any tools or applications you can write down the custom script of python, node or any language you are good with and monitor any pod restart event and send the slack hook event.
Sharing one example python code with check the POD running or crashing and send a notification to slack you can update the logic as per need.
from kubernetes import client, config, watch
import json
import requests
import time
logger = logging.getLogger('k8s_events')
logger.setLevel(logging.DEBUG)
# If running inside pod
#config.load_incluster_config()
# If running locally
config.load_kube_config()
v1 = client.CoreV1Api()
v1ext = client.ExtensionsV1beta1Api()
w = watch.Watch()
mydict={}
webhook_url = '';
while True:
pod_list= v1.list_namespaced_pod("default");
for i in pod_list.items:
for c in i.status.container_statuses:
if(c.ready == True):
if i.metadata.name in mydict:
print("Inside mydict If");
print("Pod updated : ",i.metadata.name);
print("My dict value : ",mydict);
mydict[i.metadata.name]['end_time'] = i.status.conditions[1].last_transition_time;
dt_started = mydict[i.metadata.name]['start_time'].replace(tzinfo=None);
dt_ended = mydict[i.metadata.name]['end_time'].replace(tzinfo=None);
duration = str((dt_ended - dt_started).total_seconds()) + ' Sec';
fields = [{"title": "Status", "value": "READY", "short": False }, {"title": "Pod name", "value": i.metadata.name, "short": False }, {"title": "Duration", "value": duration, "short": False }, {"title": "Service name", "value": c.name, "short": False } ]
if c.name not in ('conversation-auto-close-service-scheduler','admin-service-trail-fllow-up-scheduler','bot-trial-email-scheduler','conversation-service-scheduler','faq-service-scheduler','nlp-service-scheduler','refresh-add-on-scheduler','response-sheet-scheduler'):
text = c.name + " Pod is started";
data = {"text": text, "mrkdwn": True, "attachments" : [{"color": "#FBBC05", "title": "Pod Details", "fields" : fields, "footer": "Manvar", "footer_icon": "https://cdn.test.manvar.com/assets/manvar-icon.png"}, ], }
print("Final data to post: ",data);
response = requests.post(webhook_url, data=json.dumps(data),headers={'Content-Type': 'application/json'});
del mydict[i.metadata.name]
if response.status_code != 200:
raise ValueError('Request to slack returned an error %s, the response is:\n%s' % (response.status_code, response.text));
time.sleep(1);
else:
mydict[i.metadata.name] = {"start_time": i.status.conditions[0].last_transition_time,"end_time": i.status.conditions[1].last_transition_time};
I tried out Botkube but I did not want to publicly expose my cluster endpoint, so I wrote the following script based on the code from #Harsh Manvar. You can connect this to Teams using the Incoming Webhook Teams app from Microsoft.
from kubernetes import client, config
import json
import requests
import time
def monitorNamespace(namespace: str, webhookUrl: str):
v1 = client.CoreV1Api()
pod_list= v1.list_namespaced_pod(namespace);
podsNotRunning = {"Namespace": namespace, "Pods": []}
for pod in pod_list.items:
status = getPodStatus(pod)
if status != "Running":
podsNotRunning["Pods"].append({"Podname": pod.metadata.name, "status": status})
if len(podsNotRunning)>0:
sendAlert(podsNotRunning, webhookUrl)
def sendAlert(podsNotRunning, webhookUrl):
print(podsNotRunning)
response = requests.post(webhookUrl, data=json.dumps(podsNotRunning),headers={'Content-Type': 'application/json'});
if response.status_code != 200:
print('Response error:', response)
def getPodStatus(pod: client.models.v1_pod.V1Pod) -> str:
status = pod.status.phase
containerStatus = pod.status.container_statuses[0]
if containerStatus.started is False or containerStatus.ready is False:
waitingState = containerStatus.state.waiting
if waitingState.message is not None:
status = waitingState.reason
return status
if __name__ == "__main__":
# If running inside pod:
#config.load_incluster_config()
# If running locally:
config.load_kube_config()
webhookUrl = 'http://webhookurl'
namespace='default
interval = 10
while True:
monitorNamespace(namespace, webhookUrl)
time.sleep(interval)
After upgrading grails 2.5.2 to 3.3.11, I am getting Method Not Allowed response while hitting POST request. The GET method works fine.
Controller:
package omapi
import grails.rest.RestfulController
import org.springframework.http.HttpStatus
import static org.springframework.http.HttpStatus.NOT_FOUND
import static org.springframework.http.HttpStatus.NO_CONTENT
class PhaseController extends RestfulController {
PhaseController(){
super(Phase)
}
static responseFormats = ['json', 'xml']
static allowedMethods = [show: 'GET', index:'GET',productAcronym:'GET', phaseByName: 'GET',save:'POST',update:'PUT',delete:'DELETE', deleteGroup: 'DELETE', deletePhase: 'DELETE']
def phaseService
def index(){
def phases = phaseService.getAllPhases()
respond phases
}
def show(){
def phase = phaseService.getPhaseById(params.id)
respond phase
}
def phaseByName(){
def phase = phaseService.getPhaseByName(params?.name)
respond phase
}
def productAcronym() {
def phase = phaseService.getPhaseByProductAcronym(params?.acronym)
log.info("==== phase by product Acronym =====$params.acronym==");
respond phase;
}
}
URL Mapping:
package omapi
import grails.core.GrailsApplication
import java.nio.file.AccessDeniedException
class UrlMappings {
static mappings = {
"/applications/$id/processingGroup"(controller: "application",action: "processingGroup")
"/accounts/$id/applications/"(controller: "accounts", action: "applications")
"/accounts/$aId/application/$id/products/"(controller: "application", action: "products")
"/ftpServer/connection"(controller: 'ftpServer', action: 'testConnection')
"/application/$id/jobs/"(controller: "job",action: "jobByApplication")
"/application/cycleDates"(controller: "cycleDates",action: "getCycleDatesByApplication")
"/ProcessingBranch/branch/"(controller: "processingTicketBranch",action: "branch")
"/application/$appCode/Offset"(controller: "ReportClientOffset",action: "offsetValue")
"/$controller/$action?/$id?(.$format)?"{
constraints {
// apply constraints here
}
}
"/$controller/$id"(parseRequest: true){
action = [GET:"show",PUT:"update",POST:"404",DELETE:"delete"]
constraints {
id matches: /\d+/
}
}
"/$controller"{
action = [GET:"index",POST: "save",PUT:"update",DELETE:"delete"]
constraints {
}
}
"/"(view:"/index")
"403"(controller: "error", action: "error403")
"404"(controller: "error", action: "error404")
"409"(controller: "error", action: "error409")
"500"(controller: "error", action: "error500")
"500"(controller: "error", action: "error403", exception: AccessDeniedException)
}
}
Request: [POST] localhost:5555/OMApi/phase
Response:
{
"timestamp": 1594295030496,
"status": 405,
"error": "Method Not Allowed",
"message": "No message available",
"path": "/OMApi/phase"
}
For grails 2.5.2, everything works fine. It looks like a Spring related issue. All the searches on this matter provided no results. Any idea? Is it due to some error in UrlMapping or other problems like CORS Interceptor not working?
After upgrading grails 2.5.2 to 3.3.11, I am getting Method Not
Allowed response while hitting POST request. The GET method works
fine.
It looks like you have /OMApi/phase mapped to the index action in PhaseController which is configured with index: 'GET' in allowedMethods which means the index action is only accessible via a 'GET' request. Any other verb should result in a 405 for that action. If you want to allow both GET and POST for the index action (unclear why you want to do that) then change your allowedMethods to include index: ['GET', 'POST'] instead of index: 'GET'.
I hope that helps.
Trying to get hints to work with Google Cloud Speech API. However, I can not get Google's own example to work for me. I get the same result with and without hints. I believe the following code is what the documentation is recommending:
Here is my script:
#!/usr/bin/python
import os
import base64
import googleapiclient.discovery
speech_file = os.path.join(
os.path.dirname(__file__),
'resources',
'shwazil_hoful.flac')
with open(speech_file, 'rb') as speech:
b64speech = base64.urlsafe_b64encode(speech.read())
service = googleapiclient.discovery.build('speech', 'v1')
service_request = service.speech().recognize(
body={
"config": {
"encoding": "FLAC", # raw 16-bit signed LE samples
"sampleRateHertz": 16000, # 16 khz
"languageCode": "en-US", # a BCP-47 language tag
"speechContexts": [{
"phrases": ["hoful","shwazil"]
}]
},
"audio": {
"content": b64speech
#"uri":"gs://cloud-samples-tests/speech/brooklyn.flac"
}
})
response = service_request.execute()
recognized_text = 'Transcribed Text: \n'
for i in range(len(response['results'])):
recognized_text += response['results'][i]['alternatives'][0]['transcript']
print(recognized_text)
Output:
it's a swazzle Hopple day
I was expecting:
it's a swazzle hoful day
Is there anything I am doing wrong?
I've tried both Python2 and Python3
I'm trying to set proxy headers for angularcli. Here's what I have so far in my proxy.config.json file:
"/api": {
"target": "https://applications.str.coni.com/api",
"secure": false,
"logLevel": "debug"
But I haven't had any luck so far, perhaps I'm missing something (probably in another file). Any suggestions would be much appreciated.
Angular-cli uses http-proxy-middleware https://github.com/chimurai/http-proxy-middleware
there is an option called headers that you can use: https://github.com/chimurai/http-proxy-middleware#http-proxy-option
example:
"/api":
{
"target": "https://applications.str.coni.com/api",
"secure": false,
"logLevel": "debug",
"headers": {"host":"www.example.org"}
}
You can modify both request and response headers and request is easy with the code from #AhmedMusallam answer but for response headers you'll need to change file to proxy.config.js and modify proxy config file to look like this:
const PROXY_CONFIG = {
"/api": {
"target": "https://applications.str.coni.com/api",
"secure": false,
"logLevel": "debug",
"onProxyRes": function (proxyRes, req, res) {
proxyRes.headers['Access-Control-Allow-Headers'] = 'Authorization';
},
},
};
module.exports = PROXY_CONFIG;
I am using Angular CLI v13 and its uses bypass interceptor for modifying the headers.
sample code of proxy config put it in a file and name it proxy.conf.js (you can choose any suitable name of the file as you like)
const PROXY_CONFIG = {
"/api/proxy": {
"target": "http://localhost:80",
"secure": false,
"bypass": function (req, res, proxyOptions) {
if (req.headers.accept.indexOf("html") !== -1) {
console.log("Skipping proxy for browser request.");
return "/index.html";
}
req.headers["X-Custom-Header"] = "yes"; // adding oe setting header
res.removeHeader('X-Header-Name'); //removing header
}
}
}
module.exports = PROXY_CONFIG;
and then simply run serve command with proxy.
ng serve -proxy-config proxy.conf.js