Semantic Delta Encoding with HTTP

The Web's traditional fetch interaction has been to transfer a page and related resources from origin server to browser, to allow the user to interact with the page, and then to to submit a new browser request that begins the process of loading a new page. As the Web has become more interactive interactions between a page and the origin server between page loads are more common. Many of these interactions take the form of synchronising the latest information to display to the user. Synchronisation of very small quantities of state that have undemanding timing constraints can either fetch the updated data when needed or poll the data periodically. Larger quantities of state and tighter timing constraints can introduce additional complexity. State synchronisation problems such as this one can lead to solutions that do not scale as well as the traditional Web interaction model and can lead to reliability problems. Some solutions require a TCP connection to be held open between the client and the origin server for the duration of the page's lifetime. Some require other state information to be stored by the origin server on behalf of each client. The types of solution can violate the REST stateless constraint with the result that scalability and reliability of applications can be reduced, and it can be more difficult for intermediaries such as caches to involve themselves effectively to improve performance and deal with other considerations. The Semantic Delta Encoding protocol is a state synchronisation mechanism that complies with the REST stateless constraint. An origin server maintains both the current state to be synchronised and a buffer of recent changes whose size can be bounded by the server. Each client synchronises an initial replica of the state, and then subsequently keeps track of where it is up to in the buffer of recent changes (the "delta buffer"). At appropriate times the client will make a request that identifies the position it is up to in order to request to be brought up to date with the current state. This "delta fetch" interaction transfers only the changes that have occurred, so allows a large resource to be synchronised with minimal ongoing data transfer overhead and potentially reduced processing overhead. This approach also increases the fidelity of information transfer. By transferring the changes rather than a new state snapshot the client does not need to compute what has changed in the resource state but can read this information directly from the message. The mechanism is intended for two main use cases: A large set of state that changes in relatively small way needs to be synchronised, and transferring the entire state each time a change occurs would be inefficient The changes to a set of state are more important to the client than the state itself, meaning that the client would prefer to obtain changes than to obtain new snapshots of the state Outside of the Web this mechanism is able to be used to synchronise data sets in a standard manner. For example: It can be used to synchronise lists of alarms or realtime alerts between systems with minimal overhead It can be used to synchronise the state of a large number of individual variables or properties between systems in a low overhead manner that can be explicit about the nature of changes that have occurred. The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in . Comments are solicited and should be addressed to the author(s).

Semantic Delta Encoding is intended to ensure the following properties: The mechanism shall be usable in conjunction with any media type. It should not define or require any specific media type for initial state or for deltas The mechanism shall use standard HTTP/1.1 () requests to transfer initial state and to transfer deltas The mechanism shall allow the origin server to place a constant bound on the size of the delta buffer The mechanism shall support efficient delivery of delta responses by leveraging existing cache infrastructure The mechanism should minimise the number of requests required to bring a given client up to date The mechanism is explicitly intended not to try and update the bytes of the representation of the main resource, but instead to provide a semantic delta describing the changes.

The Semantic Delta Encoding model consists of one or more clients, a main resource, a delta buffer, and a set of delta resources. The main resource captures all of the state that needs to be synchronised to a client, and the delta resources identify specific locations within the delta buffer. Each client will begin by issuing a HTTP GET request with headers of its choosing to the main resource. This request is intended to bring the client up to date with the current state of the resource. As part of the response this main resource will identify the next delta resource, and the client stores this resource identifier away. When the client next wishes to be brought up to date with the main resource state it queries the delta resource using the identifier that was returned to it. This query will return all of the deltas that have been added to the delta buffer since the client issued its request to the main resource. This single query brings the client up to date, and also informs the client of the identifier of the next delta resource that the client should use. If a client queries its delta resource when no new deltas have been added, the server responds 204 No Content to indicate that nothing has changed and the client should use the same delta resource identifier in its next request. If a client queries its delta resource when the location that this resource identifies within the buffer has expired or when the delta buffer itself is no longer valid, the server responds 410 Gone to indicate that the client will need to query the main resource in order to reestablish synchronisation. For example:

GET /activity-feed HTTP/1.1 Host: example.com Accept: application/activity-feed+xml HTTP/1.1 200 OK Link: <http://example.com/activity-feed/delta/233>; rel="delta" Content-Type: application/activity-feed+xml Cache-Control: max-age=5 Content-Length: ... ... current activity feed state ... GET /activity-feed/delta/233 HTTP/1.1 Host: example.com Accept: application/activity-feed-delta+xml HTTP/1.1 204 No Content Cache-Control: max-age=5 (nothing has changed, don't retry for 5 seconds) GET /activity-feed/delta/233 HTTP/1.1 Host: example.com Accept: application/activity-feed-delta+xml HTTP/1.1 200 OK Link: <http://example.com/activity-feed/delta/238>; rel="next" Cache-Control: max-age=5 Content-Type: application/activity-feed-delta+xml ... all 5 changes since record 233 ... GET /activity-feed/delta/238 HTTP/1.1 Host: example.com Accept: application/activity-feed-delta+xml HTTP/1.1 410 Gone (Too many changes have occurred since record 238. The client will need to fetch the main resource again) This technique is stateless when a bounded number of buffers are present in the origin server (the delta buffers are shared by multiple clients) and the buffer sizes are determined and bounded by the origin server. The origin server has control over the size of the delta buffer and can forget about individual clients between requests. Both main and delta resources can be cached to be returned to multiple clients. Note: In the example above two different media types are used to convey main resource state and delta resource state, respectively. Because of the semantic nature of the deltas it is likely that in many circumstances a media type will be able to be constructed that serves the needs of both the main resource and the delta resources, so a single media type will often be used instead.

This specification can apply to a wide range of main resources. Any resource that wishes to publish it's ability to support delta encoding MUST include a "delta" link in response to GET and HEAD requests. The link is supplied in the HTTP link header specified in with relationship type "delta". At the end of processing the response to its main resource GET request the client will have a synchronised representation of the resource state, plus a link to fetch changes that have occurred or will occur to the resource state after the time of the included representation.

Every delta resource identifies a particular location within the delta buffer. These locations correspond to particular states that the main resource either has or has had at some time in the past. When a fetch to the main resource returns a specific location in the delta buffer it becomes possible for the consumer to fetch changes since that representation to construct a new replica of the resource state, or to process changes to the resource state since that time for other purposes. If a delta resource identifies a location in the delta buffer that has been expired from the buffer it MUST return a 410 Gone status code in response to GET and HEAD requests. If a delta resource identifies a position in the delta buffer after which no further deltas have been inserted it MUST return a 204 No Content status code in response to GET and HEAD requests. If a delta resource identifies a position in the delta buffer after which further deltas have been inserted it MUST return a 200 OK status code in response to GET and HEAD requests. Its response MUST include all deltas up to the end of the delta buffer in a single response, and it SHOULD include a link to the next delta. A delta resource SHOULD include equivalent cache control directives to the main resource in response to GET and HEAD requests. This ensures that multiple clients seeking to access the same delta are able to effectively utilise caches to bring themselves up to date.

The exact nature of delta resource identifiers is origin server-specific and is not defined by this specification. However, delta resource identifiers should have a number of properties: As with most resources a delta resource identifier SHOULD be unique across space and time, and MUST NOT be reused while clients may still hold references to them. Servers MAY embed a form of globally unique identifier within the resource identifier for this purpose. A delta resource identifier MUST identify both the delta buffer and the point within the delta buffer that the consumer is up to in its synchronisation process.

It will often be the case that different clients have different requirements for what must be synchronised. As this is a stateless protocol, origin servers SHOULD NOT allocate a different delta buffer for each active consumer. This will compromise scalability and reliability outcomes. Rather, the number of delta buffers allocated by an origin server SHOULD be a bounded set. To deal with the needs of different clients the recommended solution is to share a common delta buffer between many clients, but to allow each consumer to select a distinct subset of the delta buffer entries by identifying a set of filter criteria within the delta resource identifier. For example:

GET /feed?keywords=ietf,web HTTP/1.1 Host: example.com Accept: application/activity-feed+xml HTTP/1.1 200 OK Link: <http://example.com/feed/delta/233;ietf,web>; rel="delta" Content-Type: application/activity-feed+xml Cache-Control: max-age=5 Content-Length: ... ... current activity feed state ... GET /feed/delta/233;ietf,web HTTP/1.1 Host: example.com Accept: application/activity-feed-delta+xml HTTP/1.1 200 OK Link: <http://example.com/feed/delta/238;ietf,web>; rel="next" Cache-Control: max-age=5 Content-Type: application/activity-feed-delta+xml ... all 3 changes relating to ietf or web since record 233 ... This interaction will allow the client that wants to see deltas for these specific topics to use the same delta buffer as a client that wants to see a different subset of the available deltas. As the origin server is able to forget each consumer's place in the delta buffer and their selected filter criteria at the end of each request this filtering model complies with the REST stateless constraint. Since the client has no control over the delta resource identifier, any filter information present in the delta resource identifier must be derived by the origin server from the semantics of the request to the main resource. Header information from the delta request MAY be taken into account by the origin server for filtering purposes, such as information about the identity of the user requesting the delta. Note: This specification does not describe any particular syntax to use in specifying filter criteria, or any other part of the delta resource identifier. All resource identifiers for main and delta resources are part of the specification of individual origin servers and are outside the scope of this specification.

The delta buffer held by a server may contain redundant information. For example, multiple changes may have occurred to the same property of the main resource and it may not be necessary for the client to see each of these changes. In this case the server MAY summarise the delta buffer by removing redundant information. For example, the server may replace two changes in the buffer with a single change that communicates the final state to the client. If summarization occurs the server MUST ensure that the consumer sees a consistent resource state at the end of each response message.

This section describes additional semantics for status codes that can be returned from delta resources.

The 200 OK status code from a delta resource indicates a set of deltas that are included in the message body. The client SHOULD process these deltas to identify changes or to update its current model of the main resource's state. The client SHOULD examine the response for a "next" link header and process it as indicated below. If no "next" link is present in the message the client SHOULD treat the message as having brought it up to date with the current state of the resource, but for which the delta buffer is now no longer available. The client SHOULD NOT make any further requests to the delta resource, and SHOULD instead query the main resource in its next request if it requires further synchronisation.

The 204 No Content status code from a delta resource indicates that no further deltas have been added, and this delta resource remains the correct resource to query for additional deltas. The client SHOULD retain the current resource identifier for its next query. The origin server SHOULD include a max-age Cache-Control directive in the 204 No Content response, and the client SHOULD NOT issue a new request to this resource until the max-age timeout has expired. This allows the origin server to control the rate at which clients poll its delta buffer for new deltas.

The 410 Gone status code from a delta resource indicates that the position it identifies in the delta buffer has been expired out of the buffer, or that the delta buffer itself is no longer valid. The delta resource can no longer be used to bring the consumer up to date with the current state of the main resource. The client SHOULD query the main resource in its next request if it requires further synchronisation, and MAY do so immediately on recieving the 410 Gone response.

Other status codes should be processed according to their respective specifications. If the client would have to treat the response from a delta resource as a failure, it SHOULD instead handle the code as if it were 410 Gone and therefore resynchronise the main resource instead of triggering failure logic.

This section describes the new "delta" link relation type, and additional semantics for the "next" link relation type for a delta resource.

The "delta" link relation is used by a main resource to indicate the identifier of the next delta resource. This delta resource will always indicate the current end of the delta buffer, so will return a 204 No Content response if queried before any new deltas are added to the buffer. The client SHOULD retain this resource identifier to use to query for deltas when it next needs to bring its state up to date with that of the main resource. The client SHOULD NOT immediately issue a request to the delta resource unless it is using a long poll (described below), but instead SHOULD wait until it next needs to access the main resource state.

The "next" link relation is used by a delta resource to indicate the identifier of the next delta resource. This delta resource will always indicate the current end of the delta buffer, so will return a 204 No Content response if queried before any new deltas are added to the buffer. The client SHOULD retain this resource identifier to use to query for deltas when it next needs to bring its state up to date with that of the main resource. The client SHOULD NOT immediately issue a request to the delta resource unless it is using a long poll (described below), but instead SHOULD wait until it next needs to access the main resource state.

The default mechanisms for a consumer that wishes to stay synchronised with the state of the main resource are either to fetch the current delta whenever they need to access the state of the main resource, or to poll the delta resource periodically if they need to trigger processing based on changes to the resource. If the polling mechanism is selected then the polling period must be carefully controlled to balance the need for timely processing when the state changes against the network and processing overhead associated with issuing delta requests. If the processing must be triggered within a short period of the change then either a rapid polling rate is required or an alternative technique is warranted. This specification defines a long poll option that allows the server to hold onto the most recent delta request from a given client until either a delta is generated by the origin server or a timeout occurs. When either of these events occur the server generates a response according to requirements of a normal request that would have arrived at that time. The client is able to indicate its preference for a long poll with the Request-Timeout header. The "Request-Timeout" header is a end-to-end request header that indicates the maximum time that a client is prepared to await a response.

Request-Timeout = "Request-Timeout" ":" timeout-value timeout-value = 1*DIGIT A client adds a Request-Timeout header to any request for which they are prepared to await a response. The client sets the header to the maximum time that they are prepared to wait. The value of the Request-Timeout header is a single integer value in seconds. An origin server interprets this header as the time between receipt of a complete request and the time that it generates and begins sending the response. A client will observe a longer time interval between request and response, as network transit and processing by intermediaries add delays. If this time is critical, a client SHOULD allow for delays in setting a value for the header. An origin server MAY apply a lower value to the timeout based on local policy. An origin server MAY choose to take longer to produce a response, at the risk that the client is no longer able to use the response. An HTTP intermediary MAY reduce the value of a Request-Timeout header based on local policy. An intermediary MAY add a Request-Timeout header if none is present. The value in the Request-Timeout header MUST NOT be increased or removed. If no new delta occurs before the value specified in the Request-Timeout header expires, the origin server SHOULD return 204 No Content. If the server is unwilling or unable to keep the long poll open for the requested Request-Timeout header duration it MAY return 204 No Content at any time before the next delta is added to the delta buffer. The Semantic Delta Encoding model combined with long poll is able to emulate real-time publish/subscribe semantics for a given main resource without requiring an explicit subscription model, allowing the server to shed long poll clients whenever this may be required without breaching the contract between itself and its clients. If a client that specifies a Request-Timeout header sees a 204 No Content response, it MUST ensure that the time between the previous request and the next request is consistent with a short polling period. As with normal 204 No Content handling () the server SHOULD include a max-age Cache-Control directive in the 204 No Content response, and the client SHOULD NOT issue a new request to this resource until the max-age timeout has expired. These provisions are intended to avoid entering a degenerate rapid polling mode when a server refuses to participate in the long poll and immediately returns 204 No Content.

If the delta buffer is consistently too short for a given client to use then the client will see a 410 Gone response each time it attempts to query its delta resource. This will usually cause the client to immediately issue a new GET request to the main resource. In this degenerate case the semantic delta encoding mechanism will cause the client to alternate between issuing a new GET request to the main resource and a GET request to a delta resource returning 410 Gone. This case exhibits similar behaviour to the case where the client polls only the main resource, although latency is effectively increased. Because this degenerate case can occur clients SHOULD be able to fall back to computing the set of changes by comparing the previous synchronised state to the new state of the main resource whenever it queries the main resource, or otherwise bring itself up to date by purging the old synchronised state and replacing it with the new state from the main resource. If changes are occurring rapidly to the main resource the number of bytes added to the delta buffer may exceed the number of bytes that the client is able to fetch in a given period of time. If this occurs, each request will fetch a larger set of deltas than the previous request until either a maximum message size limit is encountered or an expired delta resource (returning 410 Gone) is encountered. The 410 Gone response will result in the resynchronisation case already described, but it is also worth considering the case when this has not already occurred. As each message increases in size it will take longer for the message to be transferred and processed. This will result in increasing latency between deltas being added to the buffer and those deltas being processed by a given client. Clients that are sensitive to latency SHOULD place an upper bound on the size of the biggest delta they are prepared to process. They SHOULD treat an over-sized delta as if it were a 410 Gone response. A good rule of thumb for determining how large a set of deltas both for the client to process and for the server to generate is that the set of deltas should not be significantly larger than the size of the main resource when clients are sensitive to latency. If the Request-Timeout header is specified by the client but is not honoured by the server the client will immediately see a 204 No Content response to each of its requests. As noted in , the client MUST maintain a minimim gap between request messages when it requests a long poll but sees a 204 No Content response. This will cause the interaction to degenerate into a standard poll.

This specification assumes that the state of all resources (including main resources and all delta resources) will be able to be transferred in a single message from server to client. If this is not the case then a slightly more complex mechanism is required, which is outside the scope of this specification. The simplest approach to building such a mechanism would be to blur the distinction between main resources and delta resources to allow current state and deltas to be transferred side by side in a single message. Each new request that a client makes should return as many deltas as possible to the client, plus (if any space is left over) as much additional state as possible to the client. For example, a client could fetch an initial subset of the overall resource state. The next request would fetch any deltas that have been added since the last request, plus an additional quantity of state. When all state has been successfully transferred to the client the state can be said to be synchronised, and it is completely up to date with the server's resource at that time. This mechanism can be further optimised by only transferring deltas relating to the synchronised portion of the main resource state in responses. It would be possible to define a general mechanism that facilitates this kind of transfer, but it would be more complex than the mechanism defined in this specification. Moreover, opportunities to make use of this increased genericity would be limited on the Web where resources are typically not large enough to require it.

One of the implications of using this mechanism is that caches may find themselves caching both main resources and delta resources related to that main resource. There is a risk that the cache will become polluted with redundant information that reduces its efficiency. If the max-age caching model is used then this impact can be reduced. All clients that request the resource within the max-age expiry period will retrieve the same cached delta, and will therefore obtain the same next delta link as part of the fetch. This group can continue to move forwards through the set of deltas as a block that only needs to interact with their local cache to obtain new deltas. Moreover, as they will only fetch a new delta when the old one expires there will only be one cached delta active for the group For example, if the max-age on both the main resource and the delta resources is 10 seconds then the first client to GET to the main resource will cause the cache to become populated with the main resource state. Any other client that requests that state within the 10 second period will retrieve the local cached representation along with a link to the same delta resource. The first client to request the delta will cause the cache to become populated with the delta. If clients from the same group each fetch within the 10 second window they will continue to be part of the group. The group as a whole will continue to progress through the same cached deltas and through the same "next" links. The group can become fragmented if they take longer than 10 seconds to fetch their deltas. In this case a new delta will be cached with a new next link that may have no relationship to the earlier "next" link. In this case the group will split into the set that fetched the delta within the first 10 second period and the group that fetched the delta within a later 10 second period. If all clients fell into a unique 10 second period then the group would split up into a set of single member groups. The mechanism that can allow these groups to become merged again is when the rate of change to the main resources slows such that independent groups are directed to the same "next" link. This allows multiple groups to be joined back into a single group. In general the mechanism will work most effectively with the least group fragmentation and highest cache efficiency when the period between client GET requests on average is less than the max-age period, and less than the average period between main resource changes. With the long poll mechanism in place a cache must be able to hold off on issuing the redundant GET requests in order to operate with high efficiency. If the cache lets a single GET request through to the resource while blocking any redundant requests then it will be able to wait for a response and return that response to all blocked clients for whom the response is appropriate, ie the blocked clients whose requests are equivalent to the first request for caching purposes. The cache can then allow further blocked requests through that may have request headers that prevent the first response from being used, for example because they specified a different Accept header. If the max-age caching model is in appropriate and Last-Modified or ETags are used instead the groups of clients accessing a particular cache can readily become fragmented leading to inefficiency.

When the number of delta bytes being generated approaches that of the available network bandwidth the most efficient transfer mechanism would be a continuous stream, without any breaks in transmission. The semantic delta encoding mechanism enforces a break in the transmission that begins when the last byte of the previous delta is transmitted. The normal processing of a response might wait for the response to be generated and processed before issuing a new request. Under this model the next delta cannot begin to be transmitted until the previous delta has been transmitted and processed, and a new client GET request to the new delta has been transmitted to the server. In theory an efficient implementation could reduce this transmission delay by reading the link header from the previous response message before it is completely transmitted. This link could be used to issue a new long poll GET request to the next delta before the previous response has been completely transmitted or processed. However, a client SHOULD NOT request the next delta until the previous response has been completely received. This is due to the problem of multiple deltas potentially ending up on multiple TCP connections and competing for the available bandwidth during their concurrent transfers. Even if a client uses pipelining to ensure that the next delta transmission begins only when the previous response message has been completely transmitted an intermediary such as a caching proxy may employ its own connection management strategy that causes the different deltas to compete for available bandwidth at that layer.

This specification MAY be combined with the LOCK method defined in to synchronously synchronise state, however this is not recommended unless it is specifically required. In general methods such as LOCK that would affect the state of the main resource SHOULD be applied to the main resource rather than a delta resource.

The Request-Timeout header definition in this specification is based on the Timeout header in draft-loreto-http-timeout-00. "Request-Timeout" is used instead to avoid conflict with . The term summarization comes from . The mechanism described in this specification treats the state of the main resource as an estimated variable (consensus-free, master/slave), however only the null estimation function is supported at intermediaries such as caches. Clients MAY apply their own estimation functions if they see fit to do so. This protocol allows the client to select between an efficient version of the REST style (fetching deltas when required), the REST with Polling style (fetching deltas periodically), or an approximation of the Asynchronous REST style (deltas are returned immediately if a long poll is used). defines a delta encoding mechanism for HTTP, but one that is quite different from that which is included in this specification. RFC3229 is intended for use by caches to update the bytes of a given representation rather than to transfer a semantic delta to client programs. The RFC3229 mechanism is more difficult, restrictive, and inefficient to use for semantic delta encoding purposes.

This specification updates the Link Relation Type Registry with the following new entry: Relation Name: delta Description: Refers to a resource that contains semantic differences between the current version of this resource and a future version of this resource. Reference: [this document]

This specification updates the Message Header registry with the following new entry: Header field: Request-Timeout Applicable protocol: http Status: experimental Author/Change controller: IETF (iesg@ietf.org) Internet Engineering Task Force Specification document(s): [this document]

The use of a single delta buffer between different clients requires care to ensure that clients only see information they are entitled to see. The delta buffer must have the same access control mechanisms in place as the main resource, otherwise it will become a backdoor mechanism for accessing the data.