FAQs regarding RazorSafe fetchers?

Issue

What are some common questions regarding RazorSafe fetchers?

Solution

The following are some frequently asked questions (FAQ) regarding RazorSafe fetcher functionality:

What is the polling interval when IMAP is used?

The IMAP fetcher only polls for new messages when it has fetched everything it could originally see. So IMAP polling works as follows: Once it fetches a chunk of messages it sleeps for an amount of time that is inverse-exponentially proportional to the number of messages just downloaded (this time varies from 0 to 60 seconds)

When a fetcher sees N  messages, will all N  messages be fetched?

If the fetcher sees N  messages on the IMAP server, all N  will be fetched minus the emails that have already been downloaded; whether from this account or from another account.

If the previous fetch takes longer than the next scheduled poll, that poll will be skipped, right?

Polling does not occur in a thread and is not sequential. It occurs after all messages discovered from the previous poll are fetched.

At what unit is expunging executed?

Expunging varies based on whether you are using quick-delete IMAP or wait-until-archive-to-delete IMAP.

If you are using quick-delete, the fetcher attempts to download up to 1000 messages. After doing so the mailbox is expunged. Notice that this is up to 1000. So, if 1000 messages do not exist, the fetcher will download what does exist and then expunge.

If you are using wait-until-archive-to-delete, then it is assumed that most users only archive messages once per day so it is not necessary to check for newly archived emails very often. This IMAP fetcher downloads up to 20,000 messages at a time and, at the end of every chunk, checks to see if enough time has expired to check the database records for emails to delete from the mail server. At least an hour must pass between checks.

Is expunge scheduled based?

No, expunges are not scheduled.  

What are the reason(s) to implement unit based expunge? Performance for IMAP protocol, internal database, or others?

Expunge is generally considered an expensive operation. It would slow down fetching significantly to call expunge after every download, or even delete. Marking messages as deleted from the mail server is also done in batch because it is also a very slow operation (e.g., about 10-15 single deletes can be done per second on various mail servers). In the case of wait-until-archive-to-delete fetchers, RazorSafe also waits to delete/expunge in order to put less stress on the internal database.