I need to use different proxies to avoid being banned.
i read this question and this one
i put my proxies in proxylist.txt
http://xx.xx.xxx.xxx:port
http://xxx.xxx.xx.xxx:port
http://xx.xx.xxx.xxx:port
http://xxx.xxx.xx.xxx:port
and i set it in custom_settings
class MyClass(scrapy.Spider):
reload(sys)
sys.setdefaultencoding('utf8')
start_time = time.time()
name = 'mySpider'
custom_settings = {
'ROBOTSTXT_OBEY' : 'False',
'RETRY_TIMES' : 10,
'PROXY_LIST' : 'proxylist.txt',
'PROXY_MODE' : 0,
'DOWNLOADER_MIDDLEWARES' : {
'scrapy.downloadermiddlewares.retry.RetryMiddleware': 90,
'scrapy_proxies.RandomProxy': 100,
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 110,
'scrapy_crawlera.CrawleraMiddleware': 600,
'scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware': None,
'random_useragent.RandomUserAgentMiddleware': 400
},
}
this version worked when my proxies had no Passwords
Now i have the password for each proxy, how can i change it for working with password ?
I answer my own question.
I use this and it works:
http://USERNAME:PASSWORD#PROXYIP:PROXYPORT
Related
Can't get content from method upload.getFile using inputPhotoFileLocation, getting exeption FILE_REFERENCE_EXPIRED, readed lots of forums but cant get answer
I'm using MTProto client on js
this.call('upload.getFile', {
location: {
_: 'inputPhotoFileLocation',
id: message.media.photo.id,
access_hash: message.media.photo.access_hash,
file_reference: message.media.photo.file_reference, //tried
//Buffer.from(message.media.photo.file_reference.toString('hex'), 'hex'),
//[...message.media.photo.file_reference] and others
thumb_size: JSON.stringify(message.media.photo.sizes.find(size => size._ == 'photoSizeProgressive'))
},
offset: 0,
limit: 1024 * 1024
})
https://core.telegram.org/constructor/inputPhotoFileLocation
You must pass type from message.media.photo.sizes in the thumb_size field
That is, instead of JSON.stringify(message.media.photo.sizes.find(size => size._ == 'photoSizeProgressive')) you need to specify message.media.photo.sizes.find(size => size._ == 'photoSizeProgressive').type
I deployed my nextjs app to Vercel and it was a success.
I can see a preview of the website and even the log saying it works well.
But i try to access the website via the default Vercel domain :
https://tly-nextjs.vercel.app/
I get an ERR_TOO_MANY_REDIRECTS.
I do not have this problem locally.
I tried :
Disabling a language redirect that I use (/en for english folks, and / for french people).
Disabling the language detector of i18next.
But none of these solutions changed anything.
Any ideas ?
i18n.js file
import i18n from 'i18next';
import { initReactI18next } from 'react-i18next';
import Cache from 'i18next-localstorage-cache';
import LanguageDetector from 'i18next-browser-languagedetector';
const fallbackLng = ['fr'];
const availableLanguages = ['fr', 'en'];
const en = require('./locales/en/common.json');
const fr = require('./locales/fr/common.json');
const options = {
order: ['querystring', 'navigator'],
lookupQuerystring: 'lng'
}
const cacheOptions = {
// turn on or off
enabled: true,
// prefix for stored languages
prefix: 'i18next_res_',
// expiration
expirationTime: 365*24*60*60*1000,
// language versions
//versions: {}
}
i18n
.use(Cache)
.use(initReactI18next)
.use(LanguageDetector)
.init({
cache: cacheOptions,
fallbackLng: fallbackLng,
debug: true,
detection: options,
supportedLngs: availableLanguages,
nonExplicitSupportedLngs: true,
resources: {
en: {translation: en},
fr: {translation: fr},
},
interpolation: {
escapeValue: false,
},
react: {
wait: true,
useSuspense: true,
},
});
export default i18n;
My change Language function :
const changeLanguageInHouse = (lang, bool) => {
setLoading(true);
i18next.changeLanguage(lang).then((t) => {
setLanguage(lang);
bake_cookie("langChoice", lang);
setLoading(false);
if (bool === true) {
var newUrl2 = (lang === "fr" ? "" : "/en") + asPath;
window.location.replace(newUrl2);
}
});
};
What happend at your server is following:
You enter https://tly-nextjs.vercel.app/ and it is redirected to /en with HTTP-Status-Code 307 (Temporary Redirect).
And /en redirect with 301 (Permanent Redirect) to /.
You can reproduce this by open the Browser-Dev-Tools and have a look at the Network Tab.
It might be, that you have some kind of url-rewriting activated at your server, which redirect everything to your domain-root.
Is there a public repo available for this? Here is how it worked for me.
Try changing the order of the locales and the default locale (not sure this helps, but it changed something for me. Undocumented if that is the case!)
So I put the default locale first (which is nl for me) in both the locales array and the domains array.
Let me know if that helps!
module.exports = {
i18n: {
localeDetection: false,
// These are all the locales you want to support in
// your application
locales: ['nl', 'en'],
// This is the default locale you want to be used when visiting
// a non-locale prefixed path e.g. `/hello`
defaultLocale: 'nl',
// This is a list of locale domains and the default locale they
// should handle (these are only required when setting up domain routing)
domains: [
{
domain: 'example.be',
defaultLocale: 'nl',
},
{
domain: 'example.com',
defaultLocale: 'en',
},
],
},
trailingSlash: true,
};
I changed all my getInitialProps to getServerSideProps
and realised I was doing a redirect on response :
res.writeHead(301, { Location: "/" })
I just delete it.
And now I don't have this endless redirect.
Doing this worked for me...
https://ardasevinc.tech/cloudflare-vercel-too-many-redirects
I think it's the actual solution to the cause of the problem rather than a bandage!
I'm able to establish a firm connection over rosbridge_server with roslibjs, however, whenever I call .publish() the Websocket disconnects and returns
I've tried calling JSON.stringify() to shorten the message size.
I've ensure the message is of the right format.
<script type="text/javascript" src="http://static.robotwebtools.org/EventEmitter2/current/eventemitter2.min.js"></script>
<script type="text/javascript" src="http://static.robotwebtools.org/roslibjs/current/roslib.min.js"></script>
var ros = new ROSLIB.Ros({
url : 'ws://192.168.1.122:9090'
});
var rosbridge_obj = new ROSLIB.Topic({
ros : ros,
name : '/topicname',
messageType : 'topicname/messagetype'
});
var message_obj = new ROSLIB.Message({
header : {
seq : 0,
stamp : 8768,
frame_id : 'Redacted',
},
mobility : {
ForwardY : 0,
TurningX : 0,
},
arm : {
J1 : 0,
J2 : 0,
J3 : 0,
J4 : 0,
J51 : 0,
J52 : 0,
},
mode : {
mode : 0,
},
});
rosbridge_obj.publish(message_obj);
Expected result is that the Websocket will publish and not close however I receive the following:
CloseEvent
bubbles: false
code: 1009
composed: false
currentTarget: WebSocket {url: "ws://192.168.1.122:9090/", readyState: 3, bufferedAmount: 0, onopen: ƒ, onerror: ƒ, …}
defaultPrevented: false
eventPhase: 0
isTrusted: true
path: []
reason: "message too big"
returnValue: true
I encountered the same problem using roslibpy, I was running basic examples of publisher and the connection was closed immediately after publishing the message with error:
WebSocket connection closed: Code=1009, Reason=message too big
It turns out, rosbridge_websocket.launch has a parameter max_message_size which is by default set to None. When I changed it to Inf it started working.
If you have your own launch file:
<include file="$(find rosbridge_server)/launch/rosbridge_websocket.launch">
<arg name="max_message_size" value="Inf" />
</include>
or just (from the command line):
roslaunch rosbridge_server rosbridge_websocket.launch max_message_size:=Inf
Hope this will help, it helped me.
This turned out to be a hardware specific issue. The issue was completely resolved after switching to new hardware.
I'm creating a self signed certificate using CertCreateSelfSignCertificate. This works and I can encrypt/sign/decrypt/verify data with it.
I would like to limit the intended purposes of the certificate, but I always end up with a certificate that has "<All>" intended purposes enabled. This is the code I'm using to prepare the pExtensions parameter to the CertCreateSelfSignCertificate call:
BYTE key_usage_value = CERT_DATA_ENCIPHERMENT_KEY_USAGE |
CERT_DIGITAL_SIGNATURE_KEY_USAGE;
CERT_KEY_USAGE_RESTRICTION_INFO key_usage = {
0, NULL,
{ sizeof(key_usage_value), &key_usage_value }
};
auto key_usage_data = EncodeObject(szOID_KEY_USAGE_RESTRICTION, &key_usage);
CERT_EXTENSION extension[] = {
{ szOID_KEY_USAGE_RESTRICTION, TRUE, {
key_usage_data.size(), key_usage_data.data()
} }
};
CERT_EXTENSIONS extensions = {
elemsof(extension),
extension
};
EncodeObject simply calls CryptEncodeObject and returns the result as a std::vector.
I have not found much documentation on this so I'm not actually sure this is what I'm supposed to do. Can anyone point out to me what I'm doing wrong?
I guess the Extended Key Usage of your certificate is beeing build empty, that means that all purposes are allowed, if you want to limit those, you will need to define them including the specific OIDs of each one, for instance, A certificate capable only for:
Smartcardlogon, Digital Signature and Non-Repudiation
will have Extended Key Usage field filled with
1.3.6.1.4.1.311.20.2.2
2.5.29.37.3
2.5.29.37
Hope it helps
After looking into szOID_ENHANCED_KEY_USAGE according to srbob's answer I managed to change the key usage field.
Here is the (simplified) code I'm using to create the extensions on the certificate, again, this is the code I'm using to prepare the pExtensions parameter to the CertCreateSelfSignCertificate call:
BYTE key_usage_value = CERT_DATA_ENCIPHERMENT_KEY_USAGE |
CERT_DIGITAL_SIGNATURE_KEY_USAGE;
CERT_KEY_USAGE_RESTRICTION_INFO key_usage = {
0, NULL,
{ sizeof(key_usage_value), &key_usage_value }
};
auto key_usage_data = EncodeObject(szOID_KEY_USAGE_RESTRICTION, &key_usage);
LPSTR enh_usage_value[] = { szOID_KP_DOCUMENT_SIGNING };
CERT_ENHKEY_USAGE enh_usage = {
elemsof(enh_usage_value),
enh_usage_value
};
auto enh_usage_data = EncodeObject(szOID_ENHANCED_KEY_USAGE, &enh_usage);
CERT_EXTENSION extension[] = {
{ szOID_KEY_USAGE_RESTRICTION, TRUE, {
key_usage_data.size(), key_usage_data.data() } },
{ szOID_ENHANCED_KEY_USAGE, TRUE, {
enh_usage_data.size(), enh_usage_data.data() } },
};
CERT_EXTENSIONS extensions = {
elemsof(extension),
extension
};
Note that the code above still adds the szOID_KEY_USAGE_RESTRICTION extension as well.
I installed the Spring Security core plug-in 1.2.7.3 on Grails 2.1.1, ran the s2-quickstart command, and then initialized the initial user and roles in the bootstrap.groovy, but I still cannot login. Text of the relevant piece of BootStrap.groovy follows:
if (SecRole.count == 0) {
def fUserRole = SecRole.findByAuthority('ROLE_FlowUser') ?: new SecRole(authority: 'ROLE_FlowUser').save(failOnError: true, flush: true)
def fAdminRole = SecRole.findByAuthority('ROLE_FlowAdmin') ?: new SecRole(authority: 'ROLE_FlowAdmin').save(failOnError: true, flush: true)
def bf = SecUser.findByUsername('bill') ?: new SecUser(
username: 'bill',
password: 'eagle',
firstName: 'bill',
lastName: 'fly',
email: 'bill.fly#baylorhealth.edu',
accountExpired: false,
accountLocked: false,
passwordExpired: false,
enabled: true
).save(failOnError: true, flush: true)
if (!bf.authorities.contains(fAdminRole)) {
SecUserSecRole.create bf, fAdminRole, true
}
if (!bf.authorities.contains(fUserRole)) {
SecUserSecRole.create bf, fUserRole, true
}
}
I am not encrypting the password in bootstrap, as seems to be the answer to most of the questions of this type. All four records are getting written to the database tables, but of course, I cannot tell if the password is encrypted correctly. My initial controller has the following annotation ahead of the class statement:
#Secured(['IS_AUTHENTICATED_FULLY'])
Also, I added the following to the config.groovy:
// Added by the Spring Security Core plugin:
grails.plugins.springsecurity.userLookup.userDomainClassName = 'cocktail.SecUser'
grails.plugins.springsecurity.userLookup.authorityJoinClassName = 'cocktail.SecUserSecRole'
grails.plugins.springsecurity.authority.className = 'cocktail.SecRole'
grails.plugins.springsecurity.password.algorithm = 'SHA-256'
Your password may be encoded two times (problem may occure if you are using multi datasources).
Try this :
class User {
...
transient bEncoded = false
...
protected void encodePassword() {
if (!bEncoded ) {
password = springSecurityService.encodePassword(password);
bEncoded = true;
}
}
}
My guess is the authorities.contains check is failing because of missing hashCode and equals methods in your role class. But if there are no roles (your 1st check) then the user wouldn't have any granted, so you can just remove those checks:
SecUserSecRole.create bf, fAdminRole, true
SecUserSecRole.create bf, fUserRole, true
If that doesn't fix it, it's most likely a password encoding issue - add debug logging for Spring Security and it should show you why it's failing; add debug 'org.springframework.security' in your log4j block in Config.groovy
p.s. if (SecRole.count == 0) { should be if (SecRole.count() == 0) { or just if (!SecRole.count()) {