IMAP Extension for Incremental Threading (INCTHREAD)

Online IMAP clients which want to conserve the required bandwidth and also show messages in threads to their users have an option to delegate the message threading to the IMAP server through mechanisms outlines in . Using the UID SEARCH command, clients do not have to download the message headers (like the Message-Id, References and In-Reply-To), and can instead fetch the resulting thread mapping of all messages in a mailbox. Unfortunately, the savings in transferred data are significantly reduced when clients have to fetch the thread mapping over and over again, for example when a new message arrives. Even if the client downloads the relevant headers of new arrivals, these data alone are not sufficient to determine a proper place where to insert the newly arriving message. Furthermore, a single newly arriving message could potentially affect placement of many messages (or even all of them in a pathological case due to joining of adjacent threads). This issue prevents using approach similar to the CONTEXT=SEARCH and CONTEXT=SORT extensions where only a position of the new arrival is communicated in an incremental manner. This extension builds on Arnt Gulbrandsen's work and reuses the INTHREAD search key defined in said draft. This search key is used to inform the server that the search conditions are to refer to all threads containing any messages which match the original search criteria. However, as the untagged THREAD response does not contain any data about the position of the affected thread among other threads in the mailbox, support for INTHREAD alone does not relieve the clients from performing additional operations due to missing information. A naive workaround where the affected threads are always placed at the logical end of the mailbox would yield results different from the complete THREAD command when copying older messages. Similarly, attempting to reuse the original thread position would significantly limit the usefulness of the REFS algorithm which sorts threads with "fresh messages" at the end of the view. Because the THREAD response cannot transmit the position of the resulting thread relative to other threads in the mailbox, the ESEARCH response is used and the UID THREAD command is extended to allow for specifying the return options in manner consistent with how SEARCH and UID SEARCH were modified by ESEARCH. Finally, two new ESEARCH return options, the THREAD and the INCTHREAD, are defined. These modifications together allow clients to delegate the threading operation completely to the server side without significantly increasing network traffic even on busy mailboxes.

This extension can still transfer excessive amounts of data because the commands return complete threads instead of incremental difference updates. However, this approach allows for reusing the clients' and servers' existing facilities, both for parsing and response processing. In addition, unless the protocol mandated history tracking of the threading tree, a much intrusive and resource-demanding feature, the incremental updates would be only possible in case where the mailbox is currently selected. This extension affirms the decision about threading requests to remain on the client side, letting it use its policies about when to request full threading information and when to use the incremental updates. No support for automated updates of the threading data in the sense of the CONTEXT extension are defined at this point. This might change based on feedback from other server and client vendors.

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119.

This document reuses the INTHREAD search key from with no changes in its meaning or semantics. Please note that the "INTHREAD" search key and the "INCTHREAD" ESEARCH return option are two distinct features.

The UID THREAD command is extended with the following two return options: Return the threading information through the ESEARCH response using syntax similar to the untagged THREAD response. Return a dedicated INCTHREAD section for each thread found in the result set. The THREAD and INCTHREAD return options are mutually exclusive. Servers MUST return a tagged BAD response if a client specifies both return options in a single UID THREAD command. Servers MUST use the ESEARCH response instead of the untagged THREAD response when responding to the extended form of the UID THREAD command. The THREAD and UID THREAD are two distinct commands. Because clients are not expected to rely on transient identifiers like the message sequence numbers for threading retrieval and storage and because of the requirement of using UIDs in the INCTHREAD response, no modification to the THREAD command is defined. This might change in future iterations of this draft if client authors express sufficient interest.

Servers announcing the INCTHREAD capability support two new search return options: Method for conveying the threading information in a form similar to the THREAD untagged response. Response contains the threading information along with the specification of a UID of the previous thread root.

The THREAD return value uses the same format as the threading data in the untagged THREAD responses . This return value is defined to allow clients to easily match data received over network with the tag of the command which caused them, as per the usual ESEARCH rules.

The INCTHREAD return value consists of "previous thread root UID" followed by "threading response containing a single thread". The UID of the previous thread root MUST be zero if the thread sorts first in the resulting list of threads. A dedicated INCTHREAD record MUST be present for each thread contained in the result set.

At first, the client removes any messages referenced in the received INCTHREAD response from its thread mapping. This step is crucial to allow new arrivals joining previously independent threads together. In the second step, the client extracts the threading information from the received INCTHREAD response. The threading data is parsed as in and the newly formed thread is inserted just behind a thread whose root message has UID specified in the "uid" argument of the INCTHREAD response. If the client supports incremental threading, the INCTHREAD responses MUST be processed in order. In particular, inserting the newly formed threads to a proper location shall happen immediately as the new thread is created as the subsequent responses COULD refer to the root message of the just-inserted thread. If the UID is said to be zero (0), it is interpreted specially to mean that the newly formed thread is the very first one among the list of threads in the mailbox. If valid, the UID of the previous thread root message MUST refer to the previous thread in a mapping which contains all messages from a mailbox. In particular, no matter what additional searching criteria the client has used, the previous thread MUST always be identified without any search criteria being applied. In future, this might be changed to allow for a subsequent search specification to accommodate clients working on a subset of messages. Servers MUST NOT send a number which does not refer to any thread root UID unless the number is 0 to indicate the very first thread. Clients MUST deal with servers sending a UID which does not refer to any thread root or any message in the mailbox. Is is implementation-defined at which position such thread shall be inserted, but the thread MUST appear in the list of threads.

This document adds two new IMAP capabilities, the ETHREAD and INCTHREAD. Servers announcing the ETHREAD capability support the extended UID THREAD command syntax and the THREAD return option. Servers supporting the INCTHREAD capability MUST support and announce the ETHREAD capability as well.

This section contains a few examples to illustrate how the INCTHREAD extension operates.

Using the proposed extension, a typical communication between two compliant IMAP protocol speakers might look like the following:

The actual resulting message threading looks like the following:

Consider the following threading for a mailbox:

Such a response corresponds to the following threading:

A new message arrives and the client asks for the reading information:

The updated threading information should look like the following:

In this case, the thread with thread root with UID 4 is still considered "fresher" by the selected thread algorithm.

The following example shows a more complicated scenario where independent threads are joined together. This illustrates the need for clients to remove the referenced messages from their thread mapping:

Such a response corresponds to the following threading:

A new response arrives, joining the first two threads in the mailbox together:

The newly formed thread remains at the beginning of a mailbox:

This extension builds upon the SEARCH=INTHREAD extension and the THREAD extension .

IMAP4 capabilities are registered by publishing a standards track or IESG approved experimental RFC. The registry is currently located at: http://www.iana.org/assignments/imap4-capabilities This document defines the ETHREAD and INCTHREAD IMAP capabilities. IANA will be asked to add these capability to the registry.

The following syntax specification uses the Augmented Backus-Naur Form (ABNF) notation as specified in . Non-terminals referenced but not defined below are as defined by , , , or .

from [RFC3501] modifier-ethread = "THREAD" modifier-incthread = "INCTHREAD" thread-return-opt = modifier-thread / modifier-incthread ;; Similar to from [RFC4466] ret-data-thread = "THREAD" [SP 1*thread-list] ;; from [RFC5256] ret-data-incthread = "INCTHREAD" SP uid SP thread-list ;; from [RFC3501] ;; from [RFC5256] search-return-data =/ ret-data-thread / ret-data-incthread ;; from [RFC4466] thread =/ "UID" SP "THREAD" [thread-return-opts] SP thread-alg SP search-criteria ;; and from [RFC5256] ;; from [RFC3501] as amended by ;; [I-D.ietf-morg-inthread] ;; ;; The thread-return-opts MUST contain exactly one of ;; modifier-thread or modifier-incthread thread-return-opts = SP "RETURN" SP "(" [thread-return-opt *(SP thread-return-opt)] ")" ;; similar to the from [RFC4466] ]]>

This document is believed to not have any security implications besides those already implied by and .