I have the following code refered from here in my middlewares.py which i'm trying to change my ip in TOR with every request
def _set_new_ip():
with Controller.from_port(port=9051) as controller:
controller.authenticate(password='tor_password')
controller.signal(Signal.NEWNYM)
class RandomUserAgentMiddleware(object):
def process_request(self, request, spider):
ua = random.choice(settings.get('USER_AGENT_LIST'))
if ua:
request.headers.setdefault('User-Agent', ua)
class ProxyMiddleware(object):
def process_request(self, request, spider):
_set_new_ip()
request.meta['proxy'] = 'http://127.0.0.1:8118'
spider.log('Proxy : %s' % request.meta['proxy'])
But when i try to start the crawling in scrapy it keeps returning me the following:
2017-09-10 22:36:44 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2017-09-10 22:36:44 [stem] DEBUG: GETCONF __owningcontrollerprocess (runtime: 0.0004)
2017-09-10 22:36:44 [stem] INFO: Error while receiving a control message (SocketClosed): empty socket content
2017-09-10 22:36:44 [IT] DEBUG: Proxy : http://127.0.0.1:8118
2017-09-10 22:36:44 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://www.jobstreet.com.sg/en/job-search/job-vacancy.php?ojs=10&key=information%20technology> (failed 1 times): Connection was refused by other side: 61: Connection refused.
2017-09-10 22:36:44 [stem] DEBUG: GETCONF __owningcontrollerprocess (runtime: 0.0003)
2017-09-10 22:36:44 [stem] INFO: Error while receiving a control message (SocketClosed): empty socket content
2017-09-10 22:36:44 [IT] DEBUG: Proxy : http://127.0.0.1:8118
2017-09-10 22:36:52 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://www.jobstreet.com.sg/en/job-search/job-vacancy.php?ojs=10&key=information%20technology> (failed 2 times): Connection was refused by other side: 61: Connection refused.
2017-09-10 22:36:52 [stem] DEBUG: GETCONF __owningcontrollerprocess (runtime: 0.0004)
2017-09-10 22:36:52 [stem] INFO: Error while receiving a control message (SocketClosed): empty socket content
2017-09-10 22:36:52 [IT] DEBUG: Proxy : http://127.0.0.1:8118
2017-09-10 22:36:56 [scrapy.downloadermiddlewares.retry] DEBUG: Gave up retrying <GET https://www.jobstreet.com.sg/en/job-search/job-vacancy.php?ojs=10&key=information%20technology> (failed 3 times): Connection was refused by other side: 61: Connection refused.
2017-09-10 22:36:56 [scrapy.core.scraper] ERROR: Error downloading <GET https://www.jobstreet.com.sg/en/job-search/job-vacancy.php?ojs=10&key=information%20technology>: Connection was refused by other side: 61: Connection refused.
2017-09-10 22:36:56 [scrapy.core.engine] INFO: Closing spider (finished)
Related
i'm trying to start filebeat and send the output to ELK server, but getting the following error message when run command /usr/share/filebeat/bin/filebeat -e -c /etc/filebeat/filebeat.yml
error message:
2021-10-18T11:46:18.575Z ERROR [logstash] logstash/async.go:280 Failed to publish events caused by: write tcp 172.31.20.157:48724->MyPublicIP:5044: write: connection reset by peer
2021-10-18T11:46:18.575Z INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2021-10-18T11:46:18.575Z INFO [publisher] pipeline/retry.go:223 done
2021-10-18T11:46:20.215Z ERROR [publisher_pipeline_output] pipeline/output.go:180 failed to publish events: write tcp 172.31.20.157:48724->MyPublicIP:5044: write: connection reset by peer
2021-10-18T11:46:20.215Z INFO [publisher_pipeline_output] pipeline/output.go:143 Connecting to backoff(async(tcp://MyPublicIP:5044))
2021-10-18T11:46:20.215Z INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2021-10-18T11:46:20.215Z INFO [publisher] pipeline/retry.go:223 done
2021-10-18T11:46:20.216Z INFO [publisher_pipeline_output] pipeline/output.go:151 Connection to backoff(async(tcp://MyPublicIP:5044)) established
2021-10-18T11:46:20.273Z ERROR [logstash] logstash/async.go:280 Failed to publish events caused by: write tcp 172.31.20.157:48726->MyPublicIP:5044: write: connection reset by peer
here is the filbeat.yml config
# ------------------------------ Logstash Output -------------------------------
output.logstash:
# The Logstash hosts
hosts: ["MyPublicIP:5044"]
and here is the logstash-sample.conf in the ELK server
input {
beats {
port => 5044
type => "logs"
ssl => true
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
}
file {
path => "/var/log/*.log"
type => "syslog"
}
file {
path => "/tmp/logstash.txt"
}
}
output {
elasticsearch {
hosts => ["MyPublicIP:9200"]
index => "%{[#metadata][beat]}-%{[#metadata][version]}-%{+YYYY.MM.dd}"
#user => "elastic"
#password => "changeme"
}
}
i'm able to telnet the logstash from filbeat client server
[root#ip-172-x-x-x-system]# telnet MyPublicIP 5044
Trying MyPublicIP ...
Connected to MyPublicIP .
Escape character is '^]'.
Hi I have a implemented oauth2 service in springboot. Auth-server and resource servers are enabled with eureka client they have successfully registered to eureka server as well. URI for auth-server in resource server is configured this way
security:
oauth2:
resource:
token-info-uri: http://auth-server/oauth/check_token
Now auth-server in the above URI is the app name of authserver. I am expecting resource server should contact auth server using token-info-uri after it gets url resolved from eureka server. But with this setup I am getting an error nested exception is java.net.UnknownHostException: auth-server
I modified my configuration to this
security:
oauth2:
resource:
prefer-token-info: false
service-id: auth-server
token-info-uri: http://${security.oauth2.resource.service-id}/oauth/check_token
loadBalanced: true
but I am getting response
{
"error": "invalid_token",
"error_description": "e2f95093-085c-4b59-90a5-c89fb5d1eccb"
}
When I debug I have this log
asset-mgmt-v1.1_1 | 2019-11-16 17:52:08.048 INFO 1 --- [nio-6001-exec-2] o.s.web.servlet.DispatcherServlet : Completed initialization in 21 ms
asset-mgmt-v1.1_1 | 2019-11-16 17:52:08.086 DEBUG 1 --- [nio-6001-exec-2] o.s.b.a.s.o.r.UserInfoTokenServices : Getting user info from: null
asset-mgmt-v1.1_1 | 2019-11-16 17:52:08.109 DEBUG 1 --- [nio-6001-exec-2] org.springframework.web.HttpLogging : HTTP GET
asset-mgmt-v1.1_1 | 2019-11-16 17:52:08.117 DEBUG 1 --- [nio-6001-exec-2] org.springframework.web.HttpLogging : Accept=[application/json, application/*+json]
asset-mgmt-v1.1_1 | 2019-11-16 17:52:08.119 WARN 1 --- [nio-6001-exec-2] o.s.b.a.s.o.r.UserInfoTokenServices : Could not fetch user details: class java.lang.IllegalStateException, Request URI does not contain a valid hostname:
asset-mgmt-v1.1_1 | 2019-11-16 17:52:08.120 DEBUG 1 --- [nio-6001-exec-2] o.s.b.a.s.o.r.UserInfoTokenServices : userinfo returned error: Could not fetch user details
asset-mgmt-v1.1_1 | 2019-11-16 17:52:08.127 DEBUG 1 --- [nio-6001-exec-2] o.s.b.a.audit.listener.AuditListener : AuditEvent [timestamp=2019-11-16T17:52:08.125Z, principal=access-token, type=AUTHENTICATION_FAILURE, data={type=org.springframework.security.authentication.BadCredentialsException, message=e2f95093-085c-4b59-90a5-c89fb5d1eccb}]
Basically URI is picked up is what I see
you cannot use just name in property files. it should be
security:
oauth2:
resource:
service-id: {Service ID as at eureka server registered}
token-info-uri: http://${security.oauth2.resource.service-id}/oauth/check_token
loadBalanced=true
prefer-token-info=false
P.S I just typed by hand. make sure keep proper spaces
In the error log presented by Darshu, there is the following message:" asset-mgmt-v1.1_1 | 2019-11-16 17:52:08.120 DEBUG 1 --- [nio-6001-exec-2] o.s.b.a.s.o.r.UserInfoTokenServices : userinfo returned error: Could not fetch user details".
The prefer-token-info must be set true to determine that token-info-uri is preferred over user-info-uri.
security:
oauth2:
resource:
prefer-token-info: true
See more in https://docs.spring.io/spring-security-oauth2-boot/docs/2.0.0.RC2/reference/htmlsingle/#boot-features-security-oauth2-resource-server
I deployed my spring cloud application in docker,include eureka server,zuul,eureka client. I want to access eureka client via zuul.
Zuul and eureka client are registered at eureka server.I access each application ,it is work. When I access eureka client via zuul, zuul console infomation show java.net.NoRouteToHostException. I don't know why and how to deal with this problem.
Eureka server config is like this.
server:
port: 1020
spring:
application:
name: eureka-server
security:
basic:
enabled: true
user:
name: admin
password: admin
eureka:
client:
fetch-registry: true
register-with-eureka: true
serviceUrl:
defaultZone: http://${eureka.instance.hostname}:${server.port}/eureka/
instance:
hostname: 192.168.90.183
prefer-ip-address: true
ip-address: 192.168.90.183
server:
enable-self-preservation: false
eviction-interval-timer-in-ms: 5000
management:
endpoints:
web:
exposure:
include: "*"
endpoint:
shutdown:
enabled: true
Zuul config is like this.
server:
port: 8088
spring:
application:
name: gateway
security:
oauth2:
management:
security:
enabled: false
endpoints:
web:
exposure:
exclude: refresh,health,info
ribbon:
ReadTimeout: 20000
SocketTimeout: 20000
zuul:
# sensitiveHeaders: "*"
routes:
tdcm-linyi:
path: /371300/**
serviceId: tdcm
ratelimit:
key-prefix: your-prefix
enabled: true
behind-proxy: true
default-policy:
limit: 100
quota: 1000
refresh-interval: 60
type:
- user
- origin
- url
host:
connect-timeout-millis: 20000
socket-timeout-millis: 20000
#================================eureka setting==============================
eureka:
instance:
instance-id: ${eureka.instance.hostname}:${server.port}
hostname: 192.168.90.183
prefer-ip-address: true
ip-address: 192.168.90.183
lease-expiration-duration-in-seconds: 10
lease-renewal-interval-in-seconds: 5
client:
serviceUrl:
defaultZone: http://admin:admin#${EUREKA_HOST:192.168.90.183}:${EUREKA_PORT:1020}/eureka
fetch-registry: true
register-with-eureka: true
Eureka client config is like this.
spring:
application:
name: tdcm
banner:
charset: UTF-8
http:
encoding:
charset: UTF-8
enabled: true
force: true
messages:
encoding: UTF-8
mvc:
throw-exception-if-no-handler-found: true
# Server
server:
port: 8926
tomcat:
uri-encoding: UTF-8
#================================eureka settinig==============================
eureka:
instance:
instance-id: ${eureka.instance.hostname}:${server.port}
hostname: 192.168.90.183
prefer-ip-address: true
ip-address: 192.168.90.183
lease-expiration-duration-in-seconds: 10
lease-renewal-interval-in-seconds: 5
client:
serviceUrl:
defaultZone: http://admin:admin#${EUREKA_HOST:192.168.90.183}:${EUREKA_PORT:1020}/eureka
fetch-registry: true
register-with-eureka: true
My test operate is like this.
I access the zuul by http://192.168.90.183:8088 ,it works well.
I access the eureka client by http://192.168.90.183:8926/getCityCenter , it works well.
When I access the eureka client via zuul by
http://192.168.90.183:8088/371300/getCityCenter , it doesn't work.
The console show the information like this.
03-29 01:55:27.229 INFO [c.n.loadbalancer.DynamicServerListLoadBalancer] - DynamicServerListLoadBalancer for client tdcm initialized: DynamicServerListLoadBalancer:{NFLoadBalancer:name=tdcm,current list of Servers=[192.168.90.183:8926],Load balancer stats=Zone stats: {defaultzone=[Zone:defaultzone; Instance count:1; Active connections count: 0; Circuit breaker tripped count: 0; Active connections per server: 0.0;]
},Server stats: [[Server:192.168.90.183:8926; Zone:defaultZone; Total Requests:0; Successive connection failure:0; Total blackout seconds:0; Last connection made:Thu Jan 01 00:00:00 UTC 1970; First connection made: Thu Jan 01 00:00:00 UTC 1970; Active Connections:0; total failure count in last (1000) msecs:0; average resp time:0.0; 90 percentile resp time:0.0; 95 percentile resp time:0.0; min resp time:0.0; max resp time:0.0; stddev resp time:0.0]
]}ServerList:org.springframework.cloud.netflix.ribbon.eureka.DomainExtractingServerList#3275110f
03-29 01:55:28.201 INFO [com.netflix.config.ChainedDynamicProperty] - Flipping property: tdcm.ribbon.ActiveConnectionsLimit to use NEXT property: niws.loadbalancer.availabilityFilteringRule.activeConnectionsLimit = 2147483647
03-29 01:55:28.545 INFO [org.apache.http.impl.execchain.RetryExec] - I/O exception (java.net.NoRouteToHostException) caught when processing request to {}->http://192.168.90.183:8926: No route to host (Host unreachable)
03-29 01:55:28.546 INFO [org.apache.http.impl.execchain.RetryExec] - I/O exception (java.net.NoRouteToHostException) caught when processing request to {}->http://192.168.90.183:8926: No route to host (Host unreachable)
03-29 01:55:28.546 INFO [org.apache.http.impl.execchain.RetryExec] - Retrying request to {}->http://192.168.90.183:8926
03-29 01:55:28.546 INFO [org.apache.http.impl.execchain.RetryExec] - Retrying request to {}->http://192.168.90.183:8926
03-29 01:55:28.547 INFO [org.apache.http.impl.execchain.RetryExec] - I/O exception (java.net.NoRouteToHostException) caught when processing request to {}->http://192.168.90.183:8926: No route to host (Host unreachable)
03-29 01:55:28.548 INFO [org.apache.http.impl.execchain.RetryExec] - Retrying request to {}->http://192.168.90.183:8926
03-29 01:55:28.555 ERROR [c.t.gateway.component.exception.ProducerFallback] - s:tdcm
03-29 01:55:28.556 ERROR [c.t.gateway.component.exception.ProducerFallback] - exception: null
03-29 01:55:29.549 ERROR [c.t.gateway.component.exception.ProducerFallback] - s:tdcm
03-29 01:55:29.550 ERROR [c.t.gateway.component.exception.ProducerFallback] - exception: null
03-29 01:55:29.550 ERROR [c.t.gateway.component.exception.ProducerFallback] - s:tdcm
03-29 01:55:29.551 ERROR [c.t.gateway.component.exception.ProducerFallback] - exception: null
03-29 01:55:29.549 ERROR [c.t.gateway.component.exception.ProducerFallback] - s:tdcm
03-29 01:55:29.552 ERROR [c.t.gateway.component.exception.ProducerFallback] - exception: null
03-29 01:55:37.508 ERROR [c.t.gateway.component.exception.ProducerFallback] - s:tdcm
03-29 01:55:37.510 ERROR [c.t.gateway.component.exception.ProducerFallback] - exception: null
03-29 01:55:39.031 ERROR [c.t.gateway.component.exception.ProducerFallback] - s:tdcm
03-29 01:55:39.033 ERROR [c.t.gateway.component.exception.ProducerFallback] - exception: null
It seems the zuul can't find the router to eureka client of tdcm.
I tried to deployed all application on computer,include eureka server,zuul,eureka client,not in docker. The same config as this article descript,it works well. I don't know why it isn't work when access the eureka client via zuul in docker deployed.
I use the host computer IP address for spring cloud appliction.
My docker version is 17.12.1-ce.
My spring cloud version is Finchley.SR1.
My Spring boot version is 2.0.3.RELEASE.
My host computer is cent-os 7.
How can I deal with the problem?
I know the problem how to dealing.Eureka client config delete the yml value of ip-address.
eureka:
instance:
ip-address: 192.168.90.183
The reason is eureka client config in the inner network of docker.It can access from zuul through inner network of docker.
I am new to Ansible and trying to connect to windows machine using Ansible.
I am getting following error
xxx.xxx.xxx.com | UNREACHABLE! => {
"changed": false,
"msg": "plaintext: HTTPConnectionPool(host='xxx.xxx.xxx.com', port=5985): Max retries exceeded with url: /wsman (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x3d76050>: Failed to establish a new connection: [Errno 111] Connection refused',))",
"unreachable": true
}
I have made changes in these three files.
Inventory:
[windows]
xxx.xxx.xxx.com
Krb5.conf
[realms]
XXX.XXX.COM = {
kdc = xxx.xxx.xxx.com
admin_server = xxx.xxx.xxx.com
}
[domain_realm]
.xxx.xxx.com = XXX.XXX.COM
group_vars/windows.yml
ansible_ssh_user: user
ansible_ssh_pass: password
ansible_ssh_port: 5985
ansible_connection: winrm
I am new to this and may have made mistake in the process of connectivity.
Any help would be appreciated. Thanks in advance.
The msg key says "plaintext", you need to enable winrm over ssl on the windows hosts with this PowerShell script provided by ansible:
https://github.com/ansible/ansible/blob/devel/examples/scripts/ConfigureRemotingForAnsible.ps1
Since that will add a self signed certificate be sure to add ansible_winrm_server_cert_validation: ignore to your group_vars/windows.yml
The request fetching in on wensocket server is :
Upgrade: websocket
Connection: Upgrade
Host: 10.1.5.20:5555
Origin: http://localhost:8080
Sec-WebSocket-Protocol: sip
Pragma: no-cache
Cache-Control: no-cache
Sec-WebSocket-Key: T3jkd1s0pRceQbgdTLoaiQ==
Sec-WebSocket-Version: 13
Sec-WebSocket-Extensions: x-webkit-deflate-frame
The response I am trying to send back for the handshake is :
HTTP/1.1 101 WebSocket Protocol Handshake
Upgrade: WebSocket
Connection: Upgrade
Sec-WebSocket-Origin: http://localhost:8080
Sec-WebSocket-Location: ws://10.1.5.20:5555/
But the error I receive is :
**Exception in thread "main" java.io.IOException: Handshake failed
at websocket4j.AbstractWebSocket.<init>(AbstractWebSocket.java:123)
at websocket4j.server.WebSocket.<init>(WebSocket.java:73)
at websocket4j.server.WebServerSocket.accept(WebServerSocket.java:119)
at websocket4j.examples.EchoServer.main(EchoServer.java:51)
Caused by: java.io.IOException: End of stream
at websocket4j.AbstractWebSocket.readBytes(AbstractWebSocket.java:230)
at websocket4j.server.WebSocket.handshake(WebSocket.java:203)
at websocket4j.AbstractWebSocket$HandshakeRunner.run(AbstractWebSocket.java:79)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)**
shouldn't you respond with the key?
try to add Sec-WebSocket-Accept: sha1(Sec-WebSocket-Key + magic string ) on your response