I'm working on an IMAP client using Ruby and Rails. I can successfully import messages, mailboxes, and more... However, after the initial import, how can I detect any changes that have occurred since my last sync?
Currently I am storing the UIDs and UID validity values in the database, comparing them, and searching appropriately. This works, but it doesn't detect deleted messages or changes to message flags, etc.
Do I have to pull all messages every time to detect these changes? How do other IMAP clients do it so quickly (i.e. Apple Mail and Postbox). My script is already taking 10+ seconds per account with very few email addresses:
# select ourself as the current mailbox
#imap_connection.examine(self.location)
# grab all new messages and update them in the database
# if the uid's are still valid, we will just fetch the newest UIDs
# otherwise, we need to search when we last synced, which is slower :(
if self.uid_validity.nil? || uid_validity == self.uid_validity
# for some IMAP servers, if a mailbox is empty, a uid_fetch will fail, so then
begin
messages = #imap_connection.uid_fetch(uid_range, ['UID', 'RFC822', 'FLAGS'])
rescue
# gmail cries if the folder is empty
uids = #imap_connection.uid_search(['ALL'])
messages = #imap_connection.uid_fetch(uids, ['UID', 'RFC822', 'FLAGS']) unless uids.empty?
end
messages.each do |imap_message|
Message.create_from_imap!(imap_message, self.id)
end unless messages.nil?
else
query = self.last_synced.nil? ? ['All'] : ['SINCE', Net::IMAP.format_datetime(self.last_synced)]
#imap_connection.search(query).each do |message_id|
imap_message = #imap_connection.fetch(message_id, ['RFC822', 'FLAGS', 'UID'])[0]
# don't mark the messages as read
##imap_connection.store(message_id, '-FLAGS', [:Seen])
Message.create_from_imap!(imap_message, self.id)
end
end
# now assume all UIDs are valid
self.uid_validity = uid_validity
# now remember that we just fetched all those messages
self.last_synced = Time.now
self.save!
There is an IMAP extension for Quick Flag Changes Resynchronization (RFC-4551). With this extension it is possible to search for all messages that have been changed since the last synchronization (based on some kind of timestamp). However, as far as I know this extension is not widely supported.
There is an informational RFC that describes how IMAP clients should do synchronization (RFC-4549, section 4.3). The text recommends issuing the following two commands:
tag1 UID FETCH <lastseenuid+1>:* <descriptors>
tag2 UID FETCH 1:<lastseenuid> FLAGS
The first command is used to fetch the required information for all unknown mails (without knowing how many mails there are). The second command is used to synchronize the flags for the already seen mails.
AFAIK this method is widely used. Therefore, many IMAP servers contain optimizations in order to provide this information quickly. Typically, the network bandwidth is the limiting factor.
The IMAP protocol is brain dead this way, unfortunately. IDLE really should be able to return this kind of stuff while connected, for example. The FETCH FLAGS suggestion above is the only way to do it.
One thing to be careful of, however, is that UIDs are only valid for a given session per the spec. You should not store them, even if some servers persist them.
Related
We are using amavisd-new (amavisd-new/oldstable,now 1:2.10.1-4) to filter both incomming and outgoing e-mails.
The thing is we receive a log of spam with fake sender from our domain. Since we have less than 100 accounts in our system, is there a plugin that can take the sender address then check it against a valid list of senders?
Thank you a lot.
our system configuration is:
debian stretch
amavisd-new/oldstable,now 1:2.10.1-4
spamassassin/oldstable,oldstable,now 3.4.2-1
If spamassassin checks outgoing email then perhaps a local rule that checks for allowed senders such as:
header LOCAL_WHITELIST From =~ /(me)|(you)|(etc)#mydomain.org/
meta LOCAL_WHITELIST_MATCH ((LOCAL_WHITELIST) =1)
score LOCAL_WHITELIST_MATCH -1.0
meta LOCAL_WHITELIST_MISS ((LOCAL_WHITELIST) =0)
score LOCAL_WHITELIST_MISS 1.0
Unfortunately, I have no idea how to do this just for outgoing email.
It should be straightforward to write a shell script that automatically generates the white list for you and creates the above as a whitelist.cf for spamassassin. That would be cool. Especially if you could get it to run automatically after the creation or deletion of an email account and then amavisd-new reload && service amavisd-new restart.
I have an EA set in place that loops history trades and builds one large string with trade information. I then send this string every second from MT4 to the python backend using a plain PUSH/PULL pattern.
For whatever reason, the data isn't received on the pull side when the string transferred becomes too long. The backend PULL-socket slices each string and further processes it.
Any chance that the PULL-side is too slow to grab and process all the data which then causes an overflow (so that a delay arises due to the processing part)?
Talking about file sizes we are well below 5kb per second.
This is the PULL-socket, which manipulates the data after receiving it:
while True:
# check 24/7 for available data in the pull socket
try:
msg = zmq_socket.recv_string()
data = msg.split("|")
print(data)
# if data is available and msg is account info, handle as follows
if data[0] == "account_info":
[...]
except zmq.error.Again:
print("\nResource timeout.. please try again.")
sleep(0.000001)
I am a bit curious now since the pull socket seems to not even be able to process a string containing 40 trades with their according information on a single MT4 client - Python connection. I actually planned to set it up to handle more than 5.000 MT4 clients - python backend connections at once.
Q : Any chance that the pull side is too slow to grab and process all the data which then causes an overflow (so that a delay arises due to the processing part)?
Zero chance.
Sending 640 B each second is definitely no showstopper ( 5kb per second - is nowhere near a performance ceiling... )
The posted problem formulation is otherwise undecidable.
Step 1) POSACK/NACK prove whether a PUSH side accepts the payload for sending error-free.
Step 2) prove the PULL side is not to be blamed - [PUSH.send(640*chr(64+i)) for i in range( 10 )] via a python-2-python tcp://-transport-class solo-channel crossing host-to-host hop, over at least your local physical network ( no VMCI/emulated vLAN, no other localhost colocation )
Step 3) if either steps above got POSACK-ed, your next chances are the ZeroMQ configuration space and/or the MT4-based PUSH-side incompatibility, most probably "hidden" inside a (not mentioned) third party ZeroMQ wrapper used / first-party issues with string handling / processing ( which you must have already read about, as it has been so many times observed and mentioned in the past posts about this trouble with well "hidden" MQL4 internal eco-system changes ).
Anyway, stay tuned. ZeroMQ is a sure bet and a truly horsepower for professional and low-latency designs in distributed-system's domain.
We are using Kaltura to notify our CMS about changes in the videos. In the KMC under Settings->Integrations Settings we have checked all the checkboxes under "Sent by Server".
Some times these checkmarks disappear? IT happens maybe once a week or once a month. How can we find the reason to these boxes being deactivated?
Those notifications are being stored on the partner object in partner table. The actual data is stored in the custom_data field, which holds large amount of PHP-serialized data.
I can suspect cases that due to updates of other fields in the custom_data object, the notifications section will be erased.
Your best shot would be first check the value of that field when the config got erased. If it was actually erased in the database, try to find the following log messages in api_v3.log (which can lead you to the actual API request that modified the field):
[2124167851][propel] */ UPDATE partner SET
`UPDATED_AT`='2017-10-04 14:11:36',
`NOTIFY`='1',
`CUSTOM_DATA`='a:79:{s:9:"firstName";s:5:"Roman";s:12:"isFirstLogin";b:0;
... tons of PHP serialized data ...
i:1;s:19:"notificationsConfig";s:42:"*=0;1=1;2=1;3=1;4=0;21=0;6=0;7=0;26=0;5=0;";
... tons of PHP serialized data ...
}' WHERE partner.ID='101' AND MD5(cast(partner.CUSTOM_DATA as char character set latin1)) = '7eb7781cc04c7f98077efc2e3c1e9426'
The key that stores the notifications config is notificationsConfig (Each number represents the notification type, then 0 / 1 for off / no).
As a side note, which CE version are you using? There might be a more reliable way to integrate with your CMS.
I want to check provided string is a valid IPFS or IPNS path or not.
For example when I do:
ipfs ls <invalid-hash>
//waits keep searching ...
This will not return anything but halts within the program, so I would need to wait not knowing given hash is valid or not.
I have used: https://github.com/xicombd/is-ipfs but an invalid-hash string that I am giving into functions returns true, so it does not work properly on my side, any advice?
I could wait N-seconds as a threshold for ipfs to return results for ipfs cat valid-hash, ipfs ls valid-hash but results of ipfs cat valid-hash or ipfs ls valid-hash may take longer than N-seconds, which is not trustable to rely on.
For example: (I am running commands inside node app.)
[$] node
const isIPFS = require('is-ipfs')
> isIPFS.multihash('QmYooooooooooooooooooooooooooooooooaoooooooooo')
true //returns true but it is an invalid ipfs hash, should have returned false.
Thank you for your valuable time and help.
It seems like the isIPFS library only verifies the validity of a hash on a syntaxical level (Does it start with Qm ? Is it the expected length ?), but does not check wether or not a file can be adressed using this hash.
In fact, I don't think it's technically feasible to assert that no file can be retrieved using a certain hash, apart from waiting N seconds for an answer.
Think about IPFS the same way you'd think about HTTP. Sometimes an HTTP request takes minutes to be completed for some reasons, but it doesn't mean the IP+port you're trying to connect leads nowhere. Your browser or app will decide to timeout after an arbitrary amount of time so as not to wait forever, but you can't have a scientific proof that nothing can be reached using a certain HTTP address/IPFS hash.
I was using the Hipchat API (v2) a bit today and ran into an odd issue where I was not able to really pull up all of the history for a room. It seemed as though when I queried a specific date, for example, it would only retrieve a fraction of the history for that date given. I had had plans to simply iterate across all of the dates for a Room to extract the history in a format that I could use, but ended up hitting this and am now unsure if it is really possible to pull out the history fully.
I realize that this is a bit clunky. It is pulling the JSON as a string and then I have to form it into a hash so I know I'm not doing this as good as it could be done, but here is roughly what I quickly did just to test out the history method for the API:
api_token = "MY_TOKEN"
client = HipChat::Client.new(api_token, :api_version => 'v2')
history = client['ROOM_NAME'].history
history = JSON.parse(history)
history.each do |key, history|
if history.is_a? Array
history.each do |message|
if message.is_a? Hash
puts "#{message['from']['name']}: #{message['message']}"
end
end
end
end
Obviously then the extension to that was to just curse through the dates in the desired range (using: client['ROOM_NAME'].history(:date => '2010-11-19', :timezone => 'PST')), but again, I was only getting a fraction of the history for the room. Are there some additional parameters that I'm missing for this to make it work as expected?
I got this working but it was a big pain.
Start by sending a query with the current time, in UTC, but without including the time zone, as the start date:
https://internal-hipchat-server/v2/room/2/history?reverse=false&date=2015-06-25T20:42:18.658439&max-results=1000&auth_token=XXX
This is very fiddly:
If you specify just the current date, without a timezone, as documented in the API, it is interpreted as midnight last night and you only get messages from yesterday or older.
If you try specifying tomorrow’s date instead, the response is 400 Bad Request This day has not yet come to pass.
If you specify the time as 2015-06-25T20:42:18.658439+00:00, which is the format that times come in HipChat API responses, HipChat’s parser seems to fail and interpret it as midnight last night.
When you get the response back, take the oldest items.date property, strip the timezone, and resubmit the above URL with an updated date parameter:
https://internal-hipchat-server/v2/room/2/history?reverse=false&date=2015-06-17T19:56:34.533182&max-results=1000&auth_token=XXX
Be sure to include the microseconds, in case a notification posted multiple messages to the same room in the same second.
This will get you the next page of messages. Keep doing this until you get fewer than max-results messages back.
There is a start-index parameter I tried passing before I got the above working, and it will give you a few pages of results, with responses lacking a links.next property, but it won’t give you the full history. On a chatroom with 9166 messages in the history according to statistics.messages_sent, it only returned 3217 messages. So don’t use it. You can use statistics.messages_sent as a sanity check for whether you get all messages.
Oh yeah, and the last_active property in the /v2/room call cannot be trusted because it doesn’t update when notification messages are posted to the room.