Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 1 year ago.
Improve this question
How to create a simple throttle control to stop your API receiving to many requests. or effectively getting DDOSed? Since sometimes you might have a forward facing API passing all connections to your service. There is as mentioned some other solutions that build throttle into the actual connection, but do over complicate what is effectively a simple solution, looking at the recommended solutions, they pretty much do what is suggested below by adding id to a map. For those still learning, this is not a bad option, but as go is so great, you can attempt simple, and then refine to a better solution as you begin to understand the mechanisms better.
Though this was flagged as promoting some Book or something, it is simply my attempt to help others. if that's bad, then Ill stay bad. :D
Here is some simple throttle control code, use this as an IF call with the unique identifier of said service, in this case an IP, and the time you want to wait. as you can see by the code, you can change seconds to minutes or milliseconds. and you would be better to use a service like cloudflare, but as a last stand option, placing this in your API and putting an IF statement around the handler code, you can throttle control the connections. this is to keep it simple, I am sure there are other elegant solutions out their, and my dumb attempt might be ridiculed, but I am sure someone will learn from this, and suggestions for improvement will also be included if they make sense.
/******************************************************************************
* _ _ _ _ _ _ _
* | | | | | | | | | | /\ | | |
* | |_| |__ _ __ ___ | |_| |_| | ___ / \ | | | _____ __
* | __| '_ \| '__/ _ \| __| __| |/ _ \ / /\ \ | | |/ _ \ \ /\ / /
* | |_| | | | | | (_) | |_| |_| | __// ____ \| | | (_) \ V V /
* \__|_| |_|_| \___/ \__|\__|_|\___/_/ \_\_|_|\___/ \_/\_/
* ----------------------------------------------------------------------------
* This function will temp store the value in a map and then remove it, it will
* return true or false if the item is in the map, Now sets delay on second response
* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
var throttle = make(map[string]bool)
func throttleAllow(ip string, timeout int) (retVal bool) {
if throttle[ip] == true {
fmt.Println("WARM","-=Throttle=-To frequent calls from:",ip)
time.Sleep(time.Duration(timeout)*time.Second) //Random next cycle.
retVal = true // false will result is receiging to frequent message
} else {
throttle[ip] = true
go func(){
time.Sleep(time.Duration(timeout)*time.Second) //Random next cycle.
delete(throttle, ip)
}()
retVal = true
}
return
}
Related
I have a use case where we have a strict need that all of the queues have a dedicated thread and consumer, which means each thread will only serve one queue.
I have read this accepted answer and the comment by Gary Russell. He mentioned creating a child application context for having all the features of #RabbitListener for achieving what I am trying. Still I didn't get how to do this when the queues are dynamically added during runtime.
If possible please point me to the relevant article for solving this in both the application context way(also how can I create the child contexts) and the MessageListenerAdapter way as advised.
Here is some solution based on Spring Boot and child application context declaration.
I have #RabbitListener in the component which will be declared in the child context:
#EnableRabbit
public class DynamicRabbitListener {
#Bean
public AmqpAdmin amqpAdmin(ConnectionFactory connectionFactory) {
return new RabbitAdmin(connectionFactory);
}
#RabbitListener(queuesToDeclare = #Queue(name = "${dynamic.queue}", autoDelete = "true"))
public void listen(String payload, #Header(AmqpHeaders.CONSUMER_QUEUE) String queueName) {
System.out.printf("Received %s from queue %s%n", payload, queueName);
}
}
We need #EnableRabbit to register respective annotation processor to the child application context and be able to trigger the proper lifecycle.
W need AmqpAdmin in this context to be able to declare dynamic queues.
Both of those aspects give us some isolation in the processing and logic.
Now this is how I declare those contexts:
#Bean
ApplicationRunner applicationRunner(ConfigurableApplicationContext parent, RabbitTemplate rabbitTemplate) {
return args -> {
for (int i = 0; i < 10; i++) {
String dynamicQueue = "dynamicQueue#" + i;
AnnotationConfigApplicationContext childApplicationContext =
new AnnotationConfigApplicationContext();
childApplicationContext.setParent(parent);
childApplicationContext.register(DynamicRabbitListener.class);
ConfigurableEnvironment environment = parent.getEnvironment();
MapPropertySource propertySource =
new MapPropertySource("dynamic.queues", Map.of("dynamic.queue", dynamicQueue));
environment.getPropertySources().addLast(propertySource);
childApplicationContext.setEnvironment(environment);
childApplicationContext.refresh();
rabbitTemplate.convertAndSend(dynamicQueue, "test data #" + i);
}
};
}
Pay attention to the propertySource to achieve a dynamic requirements for every child application context with its specific #RabbitListener.
The output of my program is like this:
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v2.7.0)
Received test data #0 from queue dynamicQueue#0
Received test data #1 from queue dynamicQueue#1
Received test data #2 from queue dynamicQueue#2
Received test data #3 from queue dynamicQueue#3
Received test data #4 from queue dynamicQueue#4
Received test data #5 from queue dynamicQueue#5
Received test data #6 from queue dynamicQueue#6
Received test data #7 from queue dynamicQueue#7
Received test data #8 from queue dynamicQueue#8
Received test data #9 from queue dynamicQueue#9
In Rust 1.29.0 one of my tests has started failing. I managed to get the strange bug down to this example:
#[derive(Clone, Debug)]
struct CountDrop<'a>(&'a std::cell::RefCell<usize>);
struct MayContainValue<T> {
value: std::mem::ManuallyDrop<T>,
has_value: u32,
}
impl<T: Clone> Clone for MayContainValue<T> {
fn clone(&self) -> Self {
Self {
value: if self.has_value > 0 {
self.value.clone()
} else {
unsafe { std::mem::uninitialized() }
},
has_value: self.has_value,
}
}
}
impl<T> Drop for MayContainValue<T> {
fn drop(&mut self) {
if self.has_value > 0 {
unsafe {
std::mem::ManuallyDrop::drop(&mut self.value);
}
}
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn check_drops() {
let n = 2000;
let drops = std::cell::RefCell::new(0usize);
let mut slots = Vec::new();
for _ in 0..n {
slots.push(MayContainValue {
value: std::mem::ManuallyDrop::new(CountDrop(&drops)),
has_value: 1,
});
}
unsafe { std::mem::ManuallyDrop::drop(&mut slots[0].value); }
slots[0].has_value = 0;
assert_eq!(slots.len(), slots.clone().len());
}
}
I know the code looks strange; it is all ripped out of context. I reproduced this problem with cargo test on 64-bit Ubuntu on Rust 1.29.0. A friend could not reproduce on Windows with the same Rust version.
Other things that stop reproduction:
Lowering n below ~900.
Not running the example from within cargo test.
Replacing CountDrop's member with u64.
Using a Rust version before 1.29.0.
What's going on here? Yes, MayContainValue can have an uninitialized member, but this is never used in any way.
I also managed to reproduce this on play.rust-lang.org.
I'm not interested in 'solutions' that involve re-engineering MayContainValue in some safe way with Option or enum, I'm using manual storage and occupied/vacant discrimination for a good reason.
TL;DR: Yes, creating an uninitialized reference is always undefined behavior. You cannot use mem::uninitialized safely with generics. There is not currently a good workaround for your specific case.
Running your code in valgrind reports 3 errors, each with the same stack trace:
==741== Conditional jump or move depends on uninitialised value(s)
==741== at 0x11907F: <alloc::vec::Vec<T> as alloc::vec::SpecExtend<T, I>>::spec_extend (vec.rs:1892)
==741== by 0x11861C: <alloc::vec::Vec<T> as alloc::vec::SpecExtend<&'a T, I>>::spec_extend (vec.rs:1942)
==741== by 0x11895C: <alloc::vec::Vec<T>>::extend_from_slice (vec.rs:1396)
==741== by 0x11C1A2: alloc::slice::hack::to_vec (slice.rs:168)
==741== by 0x11C643: alloc::slice::<impl [T]>::to_vec (slice.rs:369)
==741== by 0x118C1E: <alloc::vec::Vec<T> as core::clone::Clone>::clone (vec.rs:1676)
==741== by 0x11AF89: md::tests::check_drops (main.rs:51)
==741== by 0x119D39: md::__test::TESTS::{{closure}} (main.rs:36)
==741== by 0x11935D: core::ops::function::FnOnce::call_once (function.rs:223)
==741== by 0x11F09E: {{closure}} (lib.rs:1451)
==741== by 0x11F09E: call_once<closure,()> (function.rs:223)
==741== by 0x11F09E: <F as alloc::boxed::FnBox<A>>::call_box (boxed.rs:642)
==741== by 0x17B469: __rust_maybe_catch_panic (lib.rs:105)
==741== by 0x14044F: try<(),std::panic::AssertUnwindSafe<alloc::boxed::Box<FnBox<()>>>> (panicking.rs:289)
==741== by 0x14044F: catch_unwind<std::panic::AssertUnwindSafe<alloc::boxed::Box<FnBox<()>>>,()> (panic.rs:392)
==741== by 0x14044F: {{closure}} (lib.rs:1406)
==741== by 0x14044F: std::sys_common::backtrace::__rust_begin_short_backtrace (backtrace.rs:136)
Reducing while keeping the Valgrind error (or one extremely similar) leads to
use std::{iter, mem};
fn main() {
let a = unsafe { mem::uninitialized::<&()>() };
let mut b = iter::once(a);
let c = b.next();
let _d = match c {
Some(_) => 1,
None => 2,
};
}
Running this smaller reproduction in Miri in the playground leads to this error:
error[E0080]: constant evaluation error: attempted to read undefined bytes
--> src/main.rs:7:20
|
7 | let _d = match c {
| ^ attempted to read undefined bytes
|
note: inside call to `main`
--> /root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/rt.rs:74:34
|
74| lang_start_internal(&move || main().report(), argc, argv)
| ^^^^^^
note: inside call to `closure`
--> /root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/rt.rs:59:75
|
59| ::sys_common::backtrace::__rust_begin_short_backtrace(move || main())
| ^^^^^^
note: inside call to `closure`
--> /root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/sys_common/backtrace.rs:136:5
|
13| f()
| ^^^
note: inside call to `std::sys_common::backtrace::__rust_begin_short_backtrace::<[closure#DefId(1/1:1823 ~ std[82ff]::rt[0]::lang_start_internal[0]::{{closure}}[0]::{{closure}}[0]) 0:&dyn std::ops::Fn() -> i32 + std::marker::Sync + std::panic::RefUnwindSafe], i32>`
--> /root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/rt.rs:59:13
|
59| ::sys_common::backtrace::__rust_begin_short_backtrace(move || main())
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: inside call to `closure`
--> /root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/panicking.rs:310:40
|
31| ptr::write(&mut (*data).r, f());
| ^^^
note: inside call to `std::panicking::try::do_call::<[closure#DefId(1/1:1822 ~ std[82ff]::rt[0]::lang_start_internal[0]::{{closure}}[0]) 0:&&dyn std::ops::Fn() -> i32 + std::marker::Sync + std::panic::RefUnwindSafe], i32>`
--> /root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/panicking.rs:306:5
|
30| / fn do_call<F: FnOnce() -> R, R>(data: *mut u8) {
30| | unsafe {
30| | let data = data as *mut Data<F, R>;
30| | let f = ptr::read(&mut (*data).f);
31| | ptr::write(&mut (*data).r, f());
31| | }
31| | }
| |_____^
note: inside call to `std::panicking::try::<i32, [closure#DefId(1/1:1822 ~ std[82ff]::rt[0]::lang_start_internal[0]::{{closure}}[0]) 0:&&dyn std::ops::Fn() -> i32 + std::marker::Sync + std::panic::RefUnwindSafe]>`
--> /root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/panic.rs:392:9
|
39| panicking::try(f)
| ^^^^^^^^^^^^^^^^^
note: inside call to `std::panic::catch_unwind::<[closure#DefId(1/1:1822 ~ std[82ff]::rt[0]::lang_start_internal[0]::{{closure}}[0]) 0:&&dyn std::ops::Fn() -> i32 + std::marker::Sync + std::panic::RefUnwindSafe], i32>`
--> /root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/rt.rs:58:25
|
58| let exit_code = panic::catch_unwind(|| {
| _________________________^
59| | ::sys_common::backtrace::__rust_begin_short_backtrace(move || main())
60| | });
| |__________^
note: inside call to `std::rt::lang_start_internal`
--> /root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/rt.rs:74:5
|
74| lang_start_internal(&move || main().report(), argc, argv)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The short version is that mem::uninitialized creates a null pointer, which is being treated as a reference. That's the undefined behavior.
In your original code, the Vec::clone is implemented by iterating over an iterator. Iterator::next returns an Option<T>, so you have an option of a reference, which causes the null pointer optimization to kick in. This counts as a None, which terminates the iteration early, resulting in your empty second vector.
It turns out that having mem::uninitialized, a piece of code that gives you C-like semantics, is a giant footgun and is frequently misused (surprise!), so you aren't alone here. The main things you should follow as replacements are:
Tracking issue for RFC 1892, "Deprecate uninitialized in favor of a new MaybeUninit type"
Implement MaybeUninit
Rust 1.29.0 changed the definition of ManuallyDrop. It used to be a union (with a single member), but now it's a struct and a lang item. The role of the lang item in the compiler is to force the type to not have a destructor, even if it wraps a type that has once.
I tried copying the old definition of ManuallyDrop (which requires nightly, unless a T: Copy bound is added) and using that instead of the one from std, and it avoids the issue (at least on the Playground). I also tried dropping the second slot (slots[1]) instead of the first (slots[0]) and that also happens to work.
Although I haven't been able to reproduce the problem natively on my system (running Arch Linux x86_64), I found something interesting by using miri:
francis#francis-arch /data/git/miri master
$ MIRI_SYSROOT=~/.xargo/HOST cargo run -- /data/src/rust/so-manually-drop-1_29/src/main.rs
Finished dev [unoptimized + debuginfo] target(s) in 0.03s
Running `target/debug/miri /data/src/rust/so-manually-drop-1_29/src/main.rs`
error[E0080]: constant evaluation error: attempted to read undefined bytes
--> /home/francis/.rustup/toolchains/nightly-2018-09-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/liballoc/vec.rs:1903:32
|
1903 | for element in iterator {
| ^^^^^^^^ attempted to read undefined bytes
|
note: inside call to `<std::vec::Vec<T> as std::vec::SpecExtend<T, I>><MayContainValue<CountDrop>, std::iter::Cloned<std::slice::Iter<MayContainValue<CountDrop>>>>::spec_extend`
--> /home/francis/.rustup/toolchains/nightly-2018-09-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/liballoc/vec.rs:1953:9
|
1953 | self.spec_extend(iterator.cloned())
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: inside call to `<std::vec::Vec<T> as std::vec::SpecExtend<&'a T, I>><MayContainValue<CountDrop>, std::slice::Iter<MayContainValue<CountDrop>>>::spec_extend`
--> /home/francis/.rustup/toolchains/nightly-2018-09-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/liballoc/vec.rs:1402:9
|
1402 | self.spec_extend(other.iter())
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: inside call to `<std::vec::Vec<T>><MayContainValue<CountDrop>>::extend_from_slice`
--> /home/francis/.rustup/toolchains/nightly-2018-09-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/liballoc/slice.rs:168:9
|
168 | vector.extend_from_slice(s);
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: inside call to `std::slice::hack::to_vec::<MayContainValue<CountDrop>>`
--> /home/francis/.rustup/toolchains/nightly-2018-09-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/liballoc/slice.rs:369:9
|
369 | hack::to_vec(self)
| ^^^^^^^^^^^^^^^^^^
note: inside call to `std::slice::<impl [T]><MayContainValue<CountDrop>>::to_vec`
--> /home/francis/.rustup/toolchains/nightly-2018-09-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/liballoc/vec.rs:1687:9
|
1687 | <[T]>::to_vec(&**self)
| ^^^^^^^^^^^^^^^^^^^^^^
note: inside call to `<std::vec::Vec<T> as std::clone::Clone><MayContainValue<CountDrop>>::clone`
--> /data/src/rust/so-manually-drop-1_29/src/main.rs:54:33
|
54 | assert_eq!(slots.len(), slots.clone().len());
| ^^^^^^^^^^^^^
note: inside call to `tests::check_drops`
--> /data/src/rust/so-manually-drop-1_29/src/main.rs:33:5
|
33 | tests::check_drops();
| ^^^^^^^^^^^^^^^^^^^^
error: aborting due to previous error
For more information about this error, try `rustc --explain E0080`.
(Note: I can get the same error without using Xargo, but then miri doesn't show the source code for the stack frames in std.)
If I do this again with the original definition of ManuallyDrop, then miri doesn't report any issue. This confirms that the new definition of ManuallyDrop causes your program to have undefined behavior.
When I change std::mem::uninitialized() to std::mem::zeroed(), I can reliably reproduce the issue. When running natively, if it happens that the uninitialized memory is all zeroes, then you'll get the issue, otherwise you won't.
By calling std::mem::zeroed(), I've made the program generate null references, which are documented as undefined behavior in Rust. When the vector is cloned, an iterator is used (as shown in miri's output above). Iterator::next returns an Option<T>; that T here has a reference in it (coming from CountDrops), which causes Option's memory layout to be optimized: instead of having a discrete discriminant, it uses a null reference to represent its None value. Since I am generating null references, the iterator returns None on the first item and thus the vector ends up empty.
What's interesting is that when ManuallyDrop was defined as a union, Option's memory layout was not optimized.
println!("{}", std::mem::size_of::<Option<std::mem::ManuallyDrop<CountDrop<'static>>>>());
// prints 16 in Rust 1.28, but 8 in Rust 1.29
There is a discussion about this situation in #52898.
I want to handle ANY changes in 3 folders (user programs, common programs, downloads), so I'm subscribing to FS events like this:
LPITEMIDLIST pidlDownloads = ::ILCreateFromPath(env::FOLDER_Downloads().c_str());
DWORD dwFlags = SHCNE_CREATE | SHCNE_DELETE | SHCNE_MKDIR | SHCNE_RENAMEFOLDER |
SHCNE_RENAMEITEM | SHCNE_RMDIR | SHCNE_UPDATEITEM | SHCNE_UPDATEDIR;
SHChangeNotifyEntry monitoredFolders[3];
monitoredFolders[0].fRecursive = TRUE;
monitoredFolders[0].pidl = pidlCommonPrograms;
monitoredFolders[1].fRecursive = TRUE;
monitoredFolders[1].pidl = pidlPrograms;
monitoredFolders[2].fRecursive = TRUE;
monitoredFolders[2].pidl = pidlDownloads;
dwFSSubscriptionID_ = ::SHChangeNotifyRegister(hWnd_,
SHCNRF_ShellLevel | SHCNRF_InterruptLevel | SHCNRF_NewDelivery,
dwFlags, WM_FOLDER_UPDATED, _countof(monitoredFolders), monitoredFolders);
if (dwFSSubscriptionID_ == 0)
{
LOGERROR << L"FS watchdog failed to start.";
}
Unfortunately, on some machines no message is sent, when I rename files. When I tried to create/delete folders, it was sent, but not when I renamed them. On other machines everything seems to be working. Why?
I am already working long on this without success.
Imagine you have your main function some sort like this:
bool running = true;
int i = 0;
//waitHandler();
while(running)
i++;
Now I would like to add and call a timer, which sets running to false,
when it expires.
void waitHandler(){
boost::asio::io_service timerService;
//create and bind the timer
boost::asio::deadline_timer timer(timerService,
boost::posix_time::milliseconds(2000));
timer.wait();
running = true;
cout<<"WaitHandler triggered"<<endl;
}
Of course this does not work (when u uncomment the comment above),
since the timer will block the main thread.
What to do, if I would want to have this functionality without blocking the main function.
EDIT:
//transfer some error message
void set_result(boost::system::error_code* a, boost::system::error_code b,deadline_timer &timer)
{
a->assign(b.value(),b.category());
}
template<class SOCKET>
void read_with_timeout(SOCKET & sock, unsigned int delay,
const asio::mutable_buffers_1& buffers)
{
//create error messages
boost::system::error_code timer_result;
boost::system::error_code read_result;
//initialize timer
deadline_timer timer(sock.get_io_service());
timer.expires_from_now(boost::posix_time::milliseconds(delay));
timer.async_wait(boost::bind(set_result, &timer_result, _1,boost::ref(timer)));
//initialize receive mechanism
sock.async_receive(buffers, boost::bind(set_result, &read_result, _1,boost::ref(timer)));
sock.get_io_service().reset();
//should run for one handler
while (sock.get_io_service().run_one())
{
if (read_result.value()==0){ //zero stands for, that the message was received properly.
timer.cancel();
//cout<<"Message received: => Timer cancelled => RETURN!"<<endl;
return;
}
if(timer.expires_from_now().total_milliseconds() <=0){
sock.cancel();
//cout<<"Timeout => Socket cancelled => RETURN!"<<endl;
return;
}
}
}
As said this almost shows the wished behavior, but there are some questions to it:
Why by even using run_one, both the handler for the timer and the one for the receive can be fired
Why does receive also fire, when 0 bytes are received. For me that sounds like nothing is received and the function is supposed to wait?
Is this the right way to do it - as I said I want to receive or timeout. (like pinging)
Actually the pakets are received in wrong order as they appeared in Wireshark - I guess it has something to do with async_receive, which does not really wait for a incoming message, but just takes what is in the buffer before the function call.
What to do?
You are making this much more complex than it needs to be. There are piles of questions on this site dealing with timeouts, and a fantastic example on the Boost.Asio website. The comment from the async_tcp_client example has an excellent ASCII diagram explaining this scenario
// This class manages socket timeouts by applying the concept of a deadline.
// Some asynchronous operations are given deadlines by which they must complete.
// Deadlines are enforced by an "actor" that persists for the lifetime of the
// client object:
//
// +----------------+
// | |
// | check_deadline |<---+
// | | |
// +----------------+ | async_wait()
// | |
// +---------+
//
// If the deadline actor determines that the deadline has expired, the socket
// is closed and any outstanding operations are consequently cancelled.
//
// Connection establishment involves trying each endpoint in turn until a
// connection is successful, or the available endpoints are exhausted. If the
// deadline actor closes the socket, the connect actor is woken up and moves to
// the next endpoint.
//
// +---------------+
// | |
// | start_connect |<---+
// | | |
// +---------------+ |
// | |
// async_- | +----------------+
// connect() | | |
// +--->| handle_connect |
// | |
// +----------------+
// :
// Once a connection is :
// made, the connect :
// actor forks in two - :
// :
// an actor for reading : and an actor for
// inbound messages: : sending heartbeats:
// :
// +------------+ : +-------------+
// | |<- - - - -+- - - - ->| |
// | start_read | | start_write |<---+
// | |<---+ | | |
// +------------+ | +-------------+ | async_wait()
// | | | |
// async_- | +-------------+ async_- | +--------------+
// read_- | | | write() | | |
// until() +--->| handle_read | +--->| handle_write |
// | | | |
// +-------------+ +--------------+
//
// The input actor reads messages from the socket, where messages are delimited
// by the newline character. The deadline for a complete message is 30 seconds.
//
// The heartbeat actor sends a heartbeat (a message that consists of a single
// newline character) every 10 seconds. In this example, no deadline is applied
// message sending.
//
You should strive to achieve a similar design in your application. There is no need to bumble around by writing a read_with_timeout() function like you have posted in your question. Using async_read(), async_write(), and async_wait() will be enough to give you the desired functionality.
I think part of your confusion arises over threading. Don't think about it, understand the basic concepts first. You will want to use a single thread (the one invoking main()) and a single io_service to start. After that, you can explore more advanced concepts. If you're trying to integrate this code into a larger application, that is a different question entirely.
Studying the proactor design pattern may be helpful to you as well.
You can either execute io_service::run in a separate thread (and somehow synchronize the access to running) or pump the io_service loop manually within your while loop, using run_one()/poll()/poll_one() - whatever is appropriate in your case.
I have found out some sort of solution. I am ok with it even though there are things I do not understand.
//transfer some error message
void set_result(boost::system::error_code* a, boost::system::error_code b,deadline_timer &timer)
{
a->assign(b.value(),b.category());
}
template<class SOCKET>
void read_with_timeout(SOCKET & sock, unsigned int delay,
const asio::mutable_buffers_1& buffers)
{
//create error messages
boost::system::error_code timer_result;
boost::system::error_code read_result;
//initialize timer
deadline_timer timer(sock.get_io_service());
timer.expires_from_now(boost::posix_time::milliseconds(delay));
timer.async_wait(boost::bind(set_result, &timer_result, _1,boost::ref(timer)));
//initialize receive mechanism
sock.async_receive(buffers, boost::bind(set_result, &read_result, _1,boost::ref(timer)));
sock.get_io_service().reset();
//should run for one handler
while (sock.get_io_service().run_one())
{
if (read_result.value()==0){ //zero stands for, that the message was received properly.
timer.cancel();
//cout<<"Message received: => Timer cancelled => RETURN!"<<endl;
return;
}
if(timer.expires_from_now().total_milliseconds() <=0){
sock.cancel();
//cout<<"Timeout => Socket cancelled => RETURN!"<<endl;
return;
}
}
}
This actually works for my case and was taken from http://lists.boost.org/Archives/boost/2007/04/120339.php
referenced by this thread: How to set a timeout on blocking sockets in boost asio?
I just adapted it to Boost 1.51.
A few things are still obscured to me like e.g.
io_service.run_one actually still fires more event handlers even though it is supposed to only fire one.
Also there are events from the timer which do not at all interest me. I just want to catch the timeout and not other stuff. (I don know why there is other stuff)
In any case my problem was solved so far.
You have to spawn the timer on its own thread and then make sure that you protect the running variable from concurrent access.
In my project, I have a function in postgres (plpgsql) that determines country from a given ip address:
CREATE OR REPLACE FUNCTION get_country_for_ip(character varying)
RETURNS character varying AS
$BODY$
declare
ip ALIAS for $1;
ccode varchar;
cparts varchar[];
nparts bigint[];
addr bigint;
begin
cparts := string_to_array(ip, '.');
if array_upper(cparts, 1) <> 4 then
raise exception 'gcfi01: Invalid IP address: %', ip;
end if;
nparts := array[a2i(cparts[1])::bigint, a2i(cparts[2])::bigint, a2i(cparts[3])::bigint, a2i(cparts[4])::bigint];
if(nparts[1] is null or nparts[1] < 0 or nparts[1] > 255 or
nparts[2] is null or nparts[2] < 0 or nparts[2] > 255 or
nparts[3] is null or nparts[3] < 0 or nparts[3] > 255 or
nparts[4] is null or nparts[4] < 0 or nparts[4] > 255) then
raise exception 'gcfi02: Invalid IP address: %', ip;
end if;
addr := (nparts[1] << 24) | (nparts[2] << 16) | (nparts[3] << 8) | nparts[4];
addr := nparts[1] * 256 * 65536 + nparts[2] * 65536 + nparts[3] * 256 + nparts[4];
select into ccode t_country_code from ip_to_country where addr between n_from and n_to limit 1;
if ccode is null then
ccode := '';
end if;
return ccode;
end;$BODY$
LANGUAGE plpgsql VOLATILE
COST 100;
This may not be the most efficient, but it does the job. Note that it uses an internal table (ip_to_country), which contains data as below (the numbers n_from and n_to are the long values of the start and end of address ranges:
n_from | n_to | t_country_code
----------+----------+----------------
0 | 16777215 | ZZ
16777216 | 16777471 | AU
...
Now we are starting to look at the IPv6 addressing as well - and I need to add similar functionality for IPv6 addresses. I have a similar set of data for IPv6, which looks like this:
t_start | t_end | t_country_code
-------------+-----------------------------------------+----------------
:: | ff:ffff:ffff:ffff:ffff:ffff:ffff:ffff | ZZ
100:: | 1ff:ffff:ffff:ffff:ffff:ffff:ffff:ffff | ZZ
...
2000:: | 2000:ffff:ffff:ffff:ffff:ffff:ffff:ffff | ZZ
...
2001:1200:: | 2001:1200:ffff:ffff:ffff:ffff:ffff:ffff | MX
...
Now, given an IP address ::1, how do I (1) check that it's a valid IPv6 address and (2) get the corresponding country mapping?
I believe I found the solution. It involves modifying the data first and then some massaging of the input. Here's what worked.
First, the data needs to be converted so that all addresses are full, without shortening, with semicolon separators removed. The sample data shown in my question is converted to:
t_start | t_end | t_country_code
----------------------------------+----------------------------------+----------------
00000000000000000000000000000000 | 00ffffffffffffffffffffffffffffff | ZZ
01000000000000000000000000000000 | 01ffffffffffffffffffffffffffffff | ZZ
...
20000000000000000000000000000000 | 2000ffffffffffffffffffffffffffff | ZZ
...
20011200000000000000000000000000 | 20011200ffffffffffffffffffffffff | MX
...
This is what is stored in the database.
The next step was to convert the IP address received in the code to be in the same format. This is done in PHP with the following code (assume that $ip_address is the incoming IPv6 address):
$addr_bin = inet_pton($ip_address);
$bytes = unpack('n*', $addr_bin);
$ip_address = implode('', array_map(function ($b) {return sprintf("%04x", $b); }, $bytes));
Now variable $ip_adress wil contain the full IPv6 address, for example
:: => 00000000000000000000000000000000
2001:1200::ab => 200112000000000000000000000000ab
and so on.
Now you can simply compare this full address with the ranges in the database. I added a second function to the database to deal with IPv6 addresses, which looks like this:
CREATE OR REPLACE FUNCTION get_country_for_ipv6(character varying)
RETURNS character varying AS
$BODY$
declare
ip ALIAS for $1;
ccode varchar;
begin
select into ccode t_country_code from ipv6_to_country where addr between n_from and n_to limit 1;
if ccode is null then
ccode := '';
end if;
return ccode;
end;$BODY$
LANGUAGE plpgsql VOLATILE
COST 100;
Finally, in my php code I added the code that calls one or the other Postgres function based on the input ip_address.
First, I see a couple things you are doing that will pose problems. The first is this use of varchar and long to represent IP addresses when PostgreSQL has perfectly valid INET and CIDR types that will do what you want only better and faster. Note these do not support GIN indexing properly at present so you can't do exclude constraints on them. If you need that, look at the ip4r extension which does support this.
Note as a patch for now you can cast your varchar to inet. Inet also supports both ipv4 and ipv6 addresses as does cidr, and similar types exist on ip4r.
This will solve the ipv6 validation issue for you, and likely cut down on your storage as well as provide better operational checks and better performance.
As for countries, I am also thinking that the mappings may not be so straight-forward.