Enschede NL stephan@rename-it.nl

General sieve duplicate deliveries This document defines a new test command "duplicate" for the "Sieve" email filtering language. This test adds the ability to detect duplicate message deliveries. The main application for this new test is handling duplicate deliveries commonly caused by mailing list subscriptions or redirected mail addresses. The detection is normally performed by matching the message ID to an internal list of message IDs from previously delivered messages. For more complex applications, the "duplicate" test can also use the content of a specific header or other parts of the message.

This is an extension to the Sieve filtering language defined by RFC 5228. It adds a test to determine whether a certain message was seen before by the delivery agent in an earlier execution of the Sieve script. This can be used to detect and handle duplicate message deliveries. Duplicate deliveries are a common side-effect of being subscribed to a mailing list. For example, if a member of the list decides to reply to both the user and the mailing list itself, the user will get one copy of the message directly and another through mailing list. Also, if someone cross-posts over several mailing lists to which the user is subscribed, the user will receive a copy from each of those lists. In another scenario, the user has several redirected mail addresses all pointing to his main mail account. If one of the user's contacts sends the message to more than one of those addresses, the user will likely receive more than a single copy. Using the "duplicate" extension, users have the means to detect and handle such duplicates, e.g. by discarding them, marking them as "seen", or putting them in a special folder. Duplicate messages are normally detected using the Message-ID header field, which is required to be unique for each message. However, the "duplicate" test is flexible enough to use different (weaker) criteria for defining what makes a message a duplicate, for example based on the subject line or parts of the message body. Other applications of this new test command are also possible, as long as the tracked unique value is a string.

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in . Conventions for notations are as in Section 1.1, including use of the "Usage:" label for the definition of action and tagged arguments syntax.

] [":header" / ":uniqueid" ] [":seconds" ] ]]>

In its basic form, the "duplicate" test keeps track of which messages were seen before by this test during an earlier Sieve execution. Messages are identified by their message ID as contained in the Message-ID header. The "duplicate" test evaluates to "true" when the message was seen before and it evaluates to "false" when it was not. As a side-effect, the "duplicate" test adds the message ID to an internal duplicate tracking list once the Sieve execution finishes successfully. This way, the same test will evaluate to "true" during the next Sieve execution. Implementations MUST prevent making any definitive modifications to the internal duplicate tracking list until the Sieve script execution finishes successfully. If failing script executions would add the message ID to the duplicate tracking list, all "duplicate" tests in the Sieve script would erroneously yield "true" for the next delivery attempt of the same message, which can -- depending on the action taken for a duplicate -- easily lead to discarding the message without further notice. However, deferring the definitive modification of the tracking list to the end of a successful Sieve script execution is not without problems. It can cause a race condition when a duplicate message is delivered in parallel before the tracking list is updated. This way, a duplicate message could be missed by the "duplicate" test. More complex implementations could use a locking mechanism to prevent this problem. But, irrespective of what implementation is chosen, situations in which the "duplicate" test erroneously yields "true" MUST be prevented at all costs. The "duplicate" test MUST only check for duplicates amongst message ID values encountered in previous executions of the Sieve script; it MUST NOT consider ID values encountered earlier in the current Sieve script execution as potential duplicates. This means that all "duplicate" tests in a Sieve script execution, including those located in scripts included using the "include" extension, MUST always yield the same result if the arguments are identical. Implementations SHOULD limit the number of entries in the duplicate tracking list. When limiting the number of entries, implementations SHOULD discard the oldest ones first. Also, implementations SHOULD let entries in the tracking list expire after a short period of time. The user can explicitly control the length of this expiration time by means of the ":seconds" argument, which is always specified in seconds. If the ":seconds" argument is omitted, an appropriate default value MUST be used. A default expiration time of around 7 days is deemed to be appropriate. Sites SHOULD impose a maximum limit on the expiration time. If that limit is exceeded, the maximum value MUST silently be substituted; exceeding the limit MUST NOT produce an error. If the ":seconds" argument is zero, the "duplicate" test MUST yield "false" unconditionally. By default, the content of the message's Message-ID header field is used as the unique ID for duplicate tracking. For more complex applications, the "duplicate" test can also be used to detect duplicate deliveries based on other message text. Then, the tracked unique ID can be an arbitrary string value extracted from the message. By adding the ":header" argument with a message header field name, the content of the specified header field can be used as the tracked unique ID instead of the default Message-ID header. Alternatively, the tracked unique ID can be specified explicitly using the ":uniqueid" argument. The ":header" and ":uniqueid" arguments are mutually exclusive and specifying both for a single "duplicate" test command MUST trigger an error. If the tracked unique ID value is extracted directly from a message header field, i.e. when the ":uniqueid" argument is not used, leading and trailing whitespace (see Section 2.2 of RFC 5228) MUST first be trimmed from the value before performing the actual duplicate verification. When the ":uniqueid" argument is used, such normalization concerns are the responsibility of the user. If the header field specified using the ":header" argument exists multiple times in the message, only the first occurrence MUST be used for duplicate tracking. If the specified header field is not present in the message, the "duplicate" test MUST yield "false" unconditionally. In that case the duplicate tracking list is left unmodified by this test, since no unique ID value is available. The same rules apply with respect to the Message-ID header field for the basic "duplicate" test without a ":header" or ":uniqueid" argument, since that header field could also be missing or occurring multiple times. The string parameter of the ":uniqueid" argument can be composed from arbitrary text extracted from the message using the "variables" extension. To extract text from the message body, the "foreverypart" and "extracttext" extensions need to be used as well. This provides the user with detailed control over what identifies a message as a duplicate. Note that the "duplicate" test does not support either the "index" , or "mime" extensions directly, meaning that none of the ":index", ":mime:" or associated arguments are added to the "duplicate" test when these extensions are active. The ":uniqueid" argument can be used in combination with the "variables" extension to achieve the same result indirectly. The tracked unique ID value MUST be matched case-sensitively, irrespective of whether it originates from a header or is specified explicitly using the ":uniqueid" argument. To achieve case-insensitive behavior, the "set" command added by the "variables" extension can be used in combination with the ":uniqueid" argument to normalize the tracked unique ID value to upper or lower case. Using the ":handle" argument, the duplicate test can be employed for multiple independent purposes. The message is recognized as a duplicate only when the tracked unique ID was seen before in an earlier script execution by a "duplicate" test with the same ":handle" argument. NOTE: The necessary mechanism to track duplicate messages is very similar to the mechanism that is needed for tracking duplicate responses for the "vacation" action. One way to implement the necessary mechanism for the "duplicate" test is therefore to store a hash of the tracked unique ID and, if provided, the ":handle" argument.

A Sieve implementation that defines the "duplicate" test command will advertise the capability string "duplicate".

In this basic example message duplicates are detected by tracking the Message-ID header. Duplicate deliveries are stored in a special folder contained in the user's Trash folder. If the folder does not exist, it is created automatically using the "mailbox" extension. This way, the user has a chance to recover messages when necessary. Messages that are not recognized as duplicates are stored in the user's inbox as normal.

This example shows a more complex use of the "duplicate" test. The user gets network alerts from a set of remote automated monitoring systems. Multiple notifications can be received about the same event from different monitoring systems. The Message-ID of these messages is different, because these are all distinct messages from different senders. To avoid being notified multiple times about the same event the user writes the following script:

The subjects of the notification message are structured with a predictable pattern which includes a description of the event. In the script above the "duplicate" test is used to detect duplicate alert events. The message subject is matched against a pattern and the event description is extracted using the "variables" extension. If a message with that event in the subject was received before, but more than a minute ago, it is not detected as a duplicate due to the specified ":seconds" argument. In the the event of a duplicate, the message is marked as "seen" using the "imap4flags" extension. All alert messages are put into the "Alerts" mailbox irrespective of whether those messages are duplicates or not.

This example shows how the "duplicate" test can be used to limit the frequency of notifications sent using the "enotify" extension. Consider the following scenario: a mail user receives XMPP notifications about new mail through Sieve, but sometimes a single contact sends many messages in a short period of time. Now the user wants to prevent being notified of all of those messages. The user wants to be notified about messages from each person at most once per 30 minutes and writes the following script:

The example shown above uses the message envelope sender rather than the Message-ID header as the unique ID for duplicate tracking. The example can be extended to allow multiple messages from the same sender in close succession as long as the discussed subject is different. This can be achieved as follows:

This uses a combination of the message envelope sender and the subject of the message as the unique ID for duplicate tracking.

A flood of unique messages could cause the list of tracked message ID values to grow indefinitely. Implementations therefore SHOULD implement limits on the number and lifespan of entries in that list.

The following template specifies the IANA registration of the Sieve extension specified in this document:

]]>

This information should be added to the list of sieve extensions given on http://www.iana.org/assignments/sieve-extensions.

This document describes the "date" and "index" extensions to the Sieve email filtering language. The "date" extension gives Sieve the ability to test date and time values in various ways. The "index" extension provides a means to limit header and address tests to specific instances of header fields when header fields are repeated. [STANDARDS-TRACK] Harvard University

1350 Mass. Ave. Cambridge MA 02138 - +1 617 495 3864 sob@harvard.edu

General keyword In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. Authors who follow these guidelines should incorporate this phrase near the beginning of their document: The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119. Note that the force of these words is modified by the requirement level of the document in which they are used. The Sieve Email Filtering "include" extension permits users to include one Sieve script inside another. This can make managing large scripts or multiple sets of scripts much easier, and allows a site and its users to build up libraries of scripts. Users are able to include their own personal scripts or site-wide scripts. [STANDARDS-TRACK] This document describes a language for filtering email messages at time of final delivery. It is designed to be implementable on either a mail client or mail server. It is meant to be extensible, simple, and independent of access protocol, mail architecture, and operating system. It is suitable for running on a mail server where users may not be allowed to execute arbitrary programs, such as on black box Internet Message Access Protocol (IMAP) servers, as the base language has no variables, loops, or ability to shell out to external programs. [STANDARDS-TRACK] This document defines extensions to the Sieve email filtering language to permit analysis and manipulation of the MIME body parts of an email message. [STANDARDS-TRACK] In advanced mail filtering rule sets, it is useful to keep state or configuration details across rules. This document updates the Sieve filtering language (RFC 5228) with an extension to support variables. The extension changes the interpretation of strings, adds an action to store data in variables, and supplies a new test so that the value of a string can be examined. [STANDARDS-TRACK] Recent discussions have shown that it is desirable to set different IMAP (RFC 3501) flags on message delivery. This can be done, for example, by a Sieve interpreter that works as a part of a Mail Delivery Agent.</t><t> This document describes an extension to the Sieve mail filtering language for setting IMAP flags. The extension allows setting of both IMAP system flags and IMAP keywords. [STANDARDS-TRACK] This memo defines an extension to the Sieve mail filtering language (RFC 5228) for accessing mailbox and server annotations, checking for mailbox existence, and controlling mailbox creation on "fileinto" action. [STANDARDS-TRACK] Users go to great lengths to be notified as quickly as possible that they have received new mail. Most of these methods involve polling to check for new messages periodically. A push method handled by the final delivery agent gives users quicker notifications and saves server resources. This document does not specify the notification method, but it is expected that using existing instant messaging infrastructure such as Extensible Messaging and Presence Protocol (XMPP), or Global System for Mobile Communications (GSM) Short Message Service (SMS) messages will be popular. This document describes an extension to the Sieve mail filtering language that allows users to give specific rules for how and when notifications should be sent. [STANDARDS-TRACK] This document describes a profile of the Sieve extension for notifications, to allow notifications to be sent over the Extensible Messaging and Presence Protocol (XMPP), also known as Jabber. [STANDARDS-TRACK] This document describes an extension to the Sieve email filtering language for an autoresponder similar to that of the Unix "vacation" command for replying to messages. Various safety features are included to prevent problems such as message loops. [STANDARDS-TRACK]