Fetching and deleting a message over IMAP using cURL - bash

I want to fetch and delete the top message (with UID=1) through IMAP using curl. I can do it in multiple commands, however when I send the delete command, I have no guarantee that the uid has not changed. Is there a way to persist the imap (or pop3) session in curl?

You actually do have a guarantee that the UID has not changed: this is what the IMAP UIDVALIDITY field is all about. As long as the UIDVALIDITY value for a mailbox has not changed, any message UIDs you have will continue to point to the same message. Read about this in more detail in the IMAP RFC.
Using curl, you can assert a UIDVALIDITY value in a request by including that in the IMAP URL, like this:
curl 'imaps://imap.example.com/inbox;UIDVALIDITY=6' ...
If the UIDVALIDITY has changed, curl will report:
curl: (78) Mailbox UIDVALIDITY has changed
You can see the UIDVALIDITY value for a mailbox by passing the -v option to curl when doing, e.g., an EXAMINE:
$ curl -v imaps://imap.example.com/ -X 'examine inbox'
[...]
> A003 examine inbox
< * FLAGS (\Answered \Flagged \Draft \Deleted \Seen $NotPhishing $Phishing Old SEEN)
< * OK [PERMANENTFLAGS ()] Flags permitted.
< * OK [UIDVALIDITY 7] UIDs valid.
< * 9 EXISTS
< * 0 RECENT
< * OK [UIDNEXT 7422] Predicted next UID.
< * OK [HIGHESTMODSEQ 14264365]
< A003 OK [READ-ONLY] inbox selected. (Success)
A necessary prerequisite to using the UIDVALIDITY value is to request message uids. You can ask for the UID attribute when fetching information about messages:
$ curl -sv --netrc 'imaps://imap.example.com/inbox' -X 'fetch 1:* (UID FLAGS)'
[...]
< * 1 FETCH (UID 7186 FLAGS (\Seen))
< * 2 FETCH (UID 7188 FLAGS (\Seen))
< * 3 FETCH (UID 7278 FLAGS (\Seen))
< * 4 FETCH (UID 7288 FLAGS (Old \Seen))
< * 5 FETCH (UID 7364 FLAGS (\Seen))
< * 6 FETCH (UID 7413 FLAGS (\Seen))
< * 7 FETCH (UID 7417 FLAGS (\Seen))
< * 8 FETCH (UID 7419 FLAGS (\Seen))
< * 9 FETCH (UID 7421 FLAGS (\Seen))
Once you have a UID, you can use that to refer to a message by prefixing commands with the UID keyword. For example, instead of asking for message 9 like this:
FETCH 9 (ENVELOPE)
We would instead need to send the following IMAP command:
UID FETCH 7421 (ENVELOPE)
Curl doesn't appear to have native support for UID commands, but we can pass custom commands using the -X option. For example, to fetch message with UID 7421, we would run:
curl 'imaps://imap.example.com/inbox;uidvalidity=7' -X 'UID FETCH 7421 (ENVELOPE)'
This would assert that the mailbox UIDVALIDITY must still be 7 (because if it has changed, our UID is no longer valid), and retrieve the message envelope.
To delete that message:
curl 'imaps://imap.example.com/inbox;uidvalidity=7' -X 'UID STORE 7421 +Flags \Deleted'

Related

Virtualmin mails not receiving. Two mailfolders in home directory

I've installed a vps on ubuntu 18.04 with virtualmin as control panel. Once I create a user in a virtual server, the server automatically creates a 'mail' directory besides the 'Maildir' folder (which is normally used for storing mails via IMAP). The server makes use of postfix.
My folder structure in the user folder is now like this:
root#server01:/home/tuppr/homes/dprins# ls -al
total 28
drwxr-x--- 7 dprins#tuppr.nl tuppr 4096 Jan 11 21:02 .
drwxr-xr-x 3 tuppr tuppr 4096 Jan 11 21:01 ..
drwx------ 3 dprins#tuppr.nl tuppr 4096 Jan 11 21:02 mail
drwx------ 9 dprins#tuppr.nl tuppr 4096 Jan 11 21:01 Maildir
drwx------ 2 dprins#tuppr.nl tuppr 4096 Jan 11 21:02 .spamassassin
drwxr-xr-x 2 dprins#tuppr.nl tuppr 4096 Jan 11 21:04 .tmp
drwx------ 7 dprins#tuppr.nl tuppr 4096 Jan 11 21:02 .usermin
The problem is that incoming mails are strangely enough stored in the 'Maildir' directory, but email clients and webmail don't see it.
My postfix main.cf looks like this:
root#server01:/home/tuppr/homes/dprins# cat /etc/postfix/main.cf
# See /usr/share/postfix/main.cf.dist for a commented, more complete version
# Debian specific: Specifying a file name will cause the first
# line of that file to be used as the name. The Debian default
# is /etc/mailname.
#myorigin = /etc/mailname
smtpd_banner = $myhostname ESMTP $mail_name (Ubuntu)
biff = no
# appending .domain is the MUA's job.
append_dot_mydomain = no
# Uncomment the next line to generate "delayed mail" warnings
#delay_warning_time = 4h
readme_directory = no
# See http://www.postfix.org/COMPATIBILITY_README.html -- default to 2 on
# fresh installs.
compatibility_level = 2
# TLS parameters
smtpd_tls_cert_file=/etc/ssl/certs/ssl-cert-snakeoil.pem
smtpd_tls_key_file=/etc/ssl/private/ssl-cert-snakeoil.key
smtpd_use_tls=yes
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
# See /usr/share/doc/postfix/TLS_README.gz in the postfix-doc package for
# information on enabling SSL in the smtp client.
smtpd_relay_restrictions = permit_mynetworks permit_sasl_authenticated defer_unauth_destination
myhostname = server01.dennisprins.nl
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
myorigin = /etc/mailname
mydestination = $myhostname, server01.dennisprins.nl, localhost.dennisprins.nl, , localhost
#relayhost =
relayhost = vps.transip.email:587
smtp_sasl_auth_enable = yes
smtp_sasl_security_options = noanonymous
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_use_tls = yes
smtp_tls_security_level = encrypt
smtp_tls_note_starttls_offer = yes
mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128
mailbox_size_limit = 0
recipient_delimiter = +
inet_interfaces = all
inet_protocols = all
virtual_alias_maps = hash:/etc/postfix/virtual
sender_bcc_maps = hash:/etc/postfix/bcc
mailbox_command = /usr/bin/procmail-wrapper -o -a $DOMAIN -d $LOGNAME
home_mailbox = Maildir/
smtpd_sasl_auth_enable = yes
smtpd_sasl_security_options = noanonymous
broken_sasl_auth_clients = yes
smtpd_recipient_restrictions = permit_mynetworks permit_sasl_authenticated reject_unauth_destination
#smtp_tls_security_level = may
allow_percent_hack = no
In the 'mail' directory I usually only find some 'dovecot' log files, but no real e-mails. Those are stored in the 'Maildir', but for some reason every e-mailclient I tried doesn't seem to discover the mails stored in that folder. Like it's ignoring those. The webmail client also doesn't see any incoming mails. But they are physically stored in 'Maildir'.
Sending mails, however, is going well. Every mail I sent using a third party email client (on my Mac for example) are stored on the server and the webmail client also sees them in the 'Sent' folder.
My dovecot configuration is also pointing towards 'Maildir'. Executing a grep command for '/mail' doesn't show up anything rather than outcommented rules in dovecot.
My usermin mailbox configuration looks like this:
root#server01:/etc/dovecot# cat /etc/usermin/mailbox/config
spam_always=0
folder_types=local,ext,pop3,imap
no_mailer=0
server_attach=0
top_buttons=1
mail_dir=/var/spool/mail
mail_dir_qmail=Maildir
mail_file=mbox
shortindex=0
mail_system=4
from_format=1
smtp_ssl=0
pop_locks=1
nologout=1
wrap_width=80
spam_report=
perpage=20
folder_virts=virt,comp
show_to=0
mail_style=0
no_orig_ip=1
html_base64=0
edit_from=0
no_crlf=0
sendmail_path=/usr/lib/sendmail
noindex_hostname=1
from_map=/etc/postfix/virtual
mail_qmail=
pop3_server=localhost
send_mode=localhost
server_name=
smtp_pass=
global_address=
ldap_login=
ldap_quotas=
ldap_base=
smtp_user=
max_attach=
global_address_group=
smtp_port=
ldap_host=
ldap_tls=
max_quota=
ldap_pass=
smtp_auth=
ldap_port=
I also read and tried this article:
https://www.virtualmin.com/node/35887
https://www.virtualmin.com/node/36027
I checked literally everything but I can't find out why it's not working. All DNS records are perfectly fine. SPF records as well. Those are already checked. Whenever I delete the 'mail' directory, it's coming back as soon as I use the e-mailaccount (whether it's via a mail client or webmail)
Do you guys have any idea why it's behaving like this?
Hello your configuration is wrong when you do the email configuration, there is no local host, but the DNS you configured for POP3 or SMTP.
they must have SSL configuring.
  Also, the type of sending is not localhost. You must also have a configured port. Whether for SMTP or POP3.
Well basically this file you sent is missing a lot of information
I checked your server to see if there was any wrong configuration. But in reality there is no configuration in this domain that you have posted.
Below are the main settings you should make in your DNS
http tuppr.nl The remote name could not be resolved: 'tuppr.nl' (http://tuppr.nl)
spf tuppr.nl DNS Record not found
dmarc tuppr.nl DNS Record not found
mx tuppr.nl No DMARC Record found
mx tuppr.nl DNS Record not found
dns tuppr.nl DNS Record not found
mx tuppr.nl DMARC Quarantine / Reject policy not enabled
send your feedback

Trouble With Post Request to Slack Endpoint Ruby

I'm having trouble posting to the slack endpoint through a ruby script I'm writing:
#!/usr/bin/env ruby
#Notification Script Test
def send_slack_message
slack_rooms = [ '#test_channel_notify' ]
slack_token_file = (File.join(ENV['HOME'], '.slack_api_token'))
slack_api_token = (File.open(slack_token_file).readlines)[0].chomp
msg = 'This is a test message send'
slack_url = "https://slack.com/api/chat.postMessage"
%x{curl -X POST -d 'token=#{slack_api_token}\&channel=#{slack_rooms}\&text=#{msg}' '#{slack_url}'}
end
send_slack_message
The curl command with all the parameters hardcoded in the command line works properly and my message gets posted:
curl -X POST -d 'token=xxxxx-11111111111&channel=#test_channel_notify&text=This is a test' 'https://slack.com/api/chat.postMessage'
However from the script, I just get the following returned:
Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 154 100 40 100 114 289 825 --:--:-- --:--:-- --:--:-- 832
If anyone knows what I'm doing wrong would be much appreciated! Also I do not want to have to require any modules in order to do this, nor should I have to.
You're specifying your channel as an array, so if you don't specify which value, or index it'll be added as an array in your request. If you have more than one try iterating over every value within your slack_rooms array or just specifying the first element "slack_rooms.first".
%x{curl -X POST -d 'token=#{slack_api_token}&channel=#{slack_rooms.first}&text=#{msg}' #{slack_url}}
Note the slack_url is also a variable but in a curl request it doesn't need to be specified with quotes.
And you don't need the backslash-es to join the parameters, just the &.
The problem was that I was specifying an array for my channel, but only passing in a single channel into my command. When I removed the array and tested as:
slack_rooms = '#test-notification-channel'
It worked as expected.

Bash query sqlite

sorry for the noob question. I'm trying to write a simple bash script, based on newsbeuter. Basically I'm trying to get the first 5 articles I haven't read yet, once I got them, I send them to my phone with pushover and I need so set them as read on newsbeuter.
#!/bin/bash --
urls=$( sqlite3 /home/pi/.newsbeuter/cache.db <<END
select url from rss_item where unread = 1 limit 5;
END
)
This is the first query. I send the message variable through the pushover api.
Now I need to get how to update the table and set the articles as read.
Any ideas? (I'm totally new to bash syntax).
I tried both to recreate a query like
UPDATE rss_item set unread = 0 where url = '$url'
I looped it but it didn't work, then I tried to make
`UPDATE rss_item set unread = 0 where url in ($urls)`
but I keep getting errors I can't even understand! I really need a syntax lecture!
Try this:
#!/bin/bash --
urls="$(
sqlite3 /home/pi/.newsbeuter/cache.db \
'select url from rss_item where unread = 1 limit 5' \
)"
for url in $urls; do
sqlite3 /home/pi/.newsbeuter/cache.db \
"UPDATE rss_item set unread = 0 where url = '$url'"
done

Can curl default to using https?

I have a script which acts as a wrapper around curl: it accepts all of curl's arguments but also adds some of its own (like -H 'Content-Type: application/json'), and then it does some parsing of the output.
The problem is that curl accepts curl google.com as meaning curl http://google.com. I want to force an HTTPS connection, but I don't want to parse curl's command line to find and edit the hostname. (The user might have typed curlwrapper -H "foo: bar" -XPOST google.com -d '{"hello":"world"}')
Is there any way to tell curl "use an HTTPS connection when you're not given a URL scheme"?
It does not appear to be possible due to how libcurl determines the protocol to use when no scheme is given. An excerpt from the code:
/*
* Since there was no protocol part specified, we guess what protocol it
* is based on the first letters of the server name.
*/
/* Note: if you add a new protocol, please update the list in
* lib/version.c too! */
if(checkprefix("FTP.", conn->host.name))
protop = "ftp";
else if(checkprefix("DICT.", conn->host.name))
protop = "DICT";
else if(checkprefix("LDAP.", conn->host.name))
protop = "LDAP";
else if(checkprefix("IMAP.", conn->host.name))
protop = "IMAP";
else if(checkprefix("SMTP.", conn->host.name))
protop = "smtp";
else if(checkprefix("POP3.", conn->host.name))
protop = "pop3";
else {
protop = "http";
}
HTTPS protocol for URL with missing scheme part (and thus also bypass protocol guessing mentioned in (obsolete) answer by #FatalError) can be set with option
--proto-default https
since version 7.45.0 from October 2015. See also https://github.com/curl/curl/pull/351.
It can be put into ~/.curlrc.
Example:
$ curl -v example.org
* Trying XXXXIPv6redacted:80...
* Connected to example.org (XXXXIPv6redacted) port 80 (#0)
> GET / HTTP/1.1
...
$ curl --proto-default https -v example.org
* Trying XXXXIPv6redacted:443...
* Connected to example.org (XXXXIPv6redacted) port 443 (#0)
* ALPN: offers h2
...
(Note that it's not a magic option to assure security. It e.g. won't affect http proxy, if set, according to the manual.)

Phantom messages in Exchange Calendar folder

I'm developing an IMAP client, and testing it against a Microsoft Exchange server. When I select the Calendar mailbox, UID SEARCH suggests that there are messages in the mailbox, but UID FETCH cannot fetch them:
S: * OK The Microsoft Exchange IMAP4 service is ready.
...
C: 38 SELECT "Calendar"
S: * 13 EXISTS
S: * 0 RECENT
S: * FLAGS (\Seen \Answered \Flagged \Deleted \Draft $MDNSent)
S: * OK [PERMANENTFLAGS (\Seen \Answered \Flagged \Deleted \Draft $MDNSent)] Permanent flags
S: * OK [UIDVALIDITY 37122] UIDVALIDITY value
S: * OK [UIDNEXT 190] The next unique identifier value
S: 38 OK [READ-WRITE] SELECT completed.
C: 39 UID SEARCH OR UNSEEN FLAGGED
C: 40 UID SEARCH SEEN UNFLAGGED SINCE 19-Oct-2014
S: * SEARCH
S: 39 OK SEARCH completed.
S: * SEARCH 90 183
S: 40 OK SEARCH completed.
C: 41 UID FETCH 90,183 (ENVELOPE INTERNALDATE FLAGS BODY.PEEK[])
S: 41 OK FETCH completed.
I presume that Calendar is some kind of virtual mailbox, but this behaviour doesn't make sense to me — why would the server return the message uids in UID SEARCH if it's not possible to FETCH them? Is there any way to fetch these messages, or somehow manipulate them using IMAP?
Update: After being upgraded to 2010 SP3, it seems like the Exchange server had a change of heart. Now it at least returns something in response to the FETCH command:
C: 120 UID FETCH 773,948 (ENVELOPE INTERNALDATE FLAGS BODY.PEEK[])
S: * 16 FETCH (ENVELOPE (NIL "Retrieval using the IMAP4 protocol failed for the following message: 773" (("Microsoft Exchange Server 2010" NIL NIL ".MISSING-HOST-NAME.")) NIL NIL (("██████████" NIL "██████████" "██████████")) NIL NIL NIL NIL) INTERNALDATE "05-Mar-2015 17:22:02 +0000" FLAGS (\Seen) BODY[] {783}
S: <783 bytes omitted>
S: UID 773)
S: * 17 FETCH (ENVELOPE (NIL "Retrieval using the IMAP4 protocol failed for the following message: 948" (("Microsoft Exchange Server 2010" NIL NIL ".MISSING-HOST-NAME.")) NIL NIL (("██████████" NIL "██████████" "██████████")) NIL NIL NIL NIL) INTERNALDATE "11-Mar-2015 17:56:04 +0000" FLAGS (\Seen) BODY[] {747}
S: <747 bytes omitted>
S: UID 948)
S: 120 OK FETCH completed.
The messages look like this:
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
From: Microsoft Exchange Server 2010
To: ███████████ <████████#██████████.com>
Subject: Retrieval using the IMAP4 protocol failed for the following
message: 773
Content-Transfer-Encoding: quoted-printable
The server couldn't retrieve the following message:
Subject: "████ █████ ███████ ██ ████ ███ █████ ███████.. ███ ███ ██████ ███=
██!"
From: "███████████ ██████████" ("/O=3D████████/OU=3D████████ █████████████=
█ █████ (███████████████)/CN=3D██████████/CN=3D███████████ █████████████")
Sent date: 3/5/2015 5:22:02 PM
The message hasn't been deleted. You might be able to view it using either =
Outlook or Outlook Web App. You can also contact the sender to find out wha=
t the message says.=
Sounds like a server bug. It is not a secret that Exchange "implaments IMAP" in order to be able to tick a required checkbox when evaluating tenders. You will see other bugs in there as well.
It might be interesting to get a list of all UIDs which that mailbox allegedly contains, and to try fetching them one-by-one to see if at least some of them return some usable value. However, considering that this is a mailbox called "Calendar", chances are that it's just some random Exchange export of data which only make sense via MAPI. Random googling results in https://technet.microsoft.com/en-us/library/aa998606%28v=exchg.150%29.aspx which suggests that Exchange can be configured to offer these calendar data in various formats. Maybe your server is "misconfigured".
TL;DR: it's Exchange. Don't waste time on it.

Resources