RPKI Repository Delta Protocol

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in .

The current RPKI Resource Certificate Repository Structure (RFC6480 & RFC6481) uses rsync as both a delta and transfer protocol and recommends that repositories be set up in a hierarchical way such that relying parties (validation tools) can fetch all updates in a single repository efficiently while performing top-down validation. This structure has its benefits. In particular it has allowed for early deployment of RPKI without the need to re-invent a delta protocol, and this has allowed early adopters of RPKI to build up operational experience more quickly. The delta protocol also has benefits for relying party tools, allowing them to quickly retrieve what’s new in a repository limiting fetch time and bandwidth usage. Operational experience shows that the current RPKI infrastructure works fairly well under most circumstances. However, operational experience, as well as lab testing, have also shown that there are emergent concerns with the current infrastructure that justify that the WG thinks about improvements in this space.

Rsync is a very efficient tool when used 1:1 between a client and server. The problem is that in a globally deployed RPKI we can expect in the order of 40k clients, roughly corresponding to the number of ASNs, to connect regularly to a repository server. When the rsync built-in delta protocol is used (recursive fetching), the server is computationally involved in calculating the delta for each connected client. Performance measurements in a lab have shown that the maximum clients that can be processed per second (throughput), and the maximum number of concurrent clients are both linearly dependent on the repository size. The throughput is limited by server CPU. Concurrency is limited by server memory. As a result a sufficiently large repository has to invest heavily in running multiple rsyncd instances to cope with the load of a large number of clients, and to counter the risks of DDoS attacks.

The retrieval of signed object is described in an RFC6480 (section 6). There are no formal limits imposed by this informational RFC on the update frequency, but to prevent the overloading of repository servers as described above, the typical update interval of current tools is between 1-24 hours. In previous discussions in the WG it was suggested that human scale propagation times (i.e. up to 24 hours) are good enough for the problem that we are trying to solve. There are however good reasons why much faster retrieval of new signed objects desirable.

A ROA exists for a prefix and ASN, but the prefix needs to be announced from another ASN. In many cases the PP can know this in advance and create an appropriate ROA well in advance, but there are also failure cases possible where this was not foreseeable.

A wrong ROA is mistakenly issued and picked up by some RPs, the situation needs to be corrected asap.

The BGPSec protocol is still being discussed. However there are indications that it would be desirable if propagation times for new router certificates and CRLs could be reduced.

There is only one known implementation of rsync and the standard is not described by any RFC. The implementation is non-modular, making it impossible to use the code as a library even when coding in the same language. As a result all current implementations of relying party tooling have had no option but to use rsync as a pre-installed external process. This has several major draw backs for the quality of implementations: RP tools require that rsync is installed on a system, and they can not assume which version is installed. Because there is only one implementation all RPKI repositories can be affected by a single bug or exploit in rsync. Calling an external process is expensive limiting the benefits that can be gained from parallel processing. Parsing downloaded objects is inefficient -- objects have to be downloaded to disk first before they can be read and parsed. Dealing with errors is complicated -- exit codes are not always clear, stderr may have to be parsed. Exit codes and messages are not guaranteed to be the same across rsync versions.

Rsync is non-transactional and as a result RPs may get partially updated CA repository objects if it happens to fetch while the objects on disk are being updated. This is confusing to RPs who can not tell the difference between this and an attack involving partial replay or withholding. Current wisdom for the RP is to try again and hope for the best.

A future delta protocol should be transport agnostic, allowing agility in future transport protocols between RPs and repositories, and sharing of deltas between RPs. In this document I will use http as an example protocol.

A future delta protocol should allow for transactional updates with regard to *all* objects for CA certificate publication point.

In a full deployment scenario a large number of RPs is expected to approach a single repository server regularly. Estimates of how large this number of RPs is, and how regularly they will fetch updates vary. However as a starting point one might expect one RP tool for each ASN, currently 40k, to fetch updates every five minutes. Whatever the real numbers may be it should be noted that the repository server has very little control over these numbers. Because of this it makes sense to look into a delta protocol where the number of clients and frequency of fetching, has the least possible effect on the central repository server. Pre-computing updates as described later on in this document allows for such minimal involvement. Deltas can be offloaded to caches or CDNs. Although this results in a protocol that is less efficient 1:1 for the Relying Party, the trade-off chosen here is that offloading CPU cycles to, say, 40k RPs every 5 minutes scales better than spending them on the server.

Higher update frequencies and shorter propagation times are desired. For this reason a delta protocol would do well to support update notifications to RPs. Two basic strategies exist: poll and push. Because the poll model is simpler to implement I will focus on this here. This does not exclude work on a push model though.

RPs are vulnerable to attacks where information is withheld or replayed. RPs can notice such attacks if they rely on manifests to inform them about: which objects should be expected when the next update is expected at the latest If deltas were signed it would be possible for RPs to detect attacks or transport errors sooner. However, signing deltas comes at a cost of complexity. In particular it will be difficult to communicate keys used for signing in a secure and dynamic (allow rolls) way. The benefit seems too limited to warrant this.

Relying Parties regularly poll a small update notification file with pointers to deltas that can be used for getting up to date quickly, and full dumps if there are no deltas available for this. The idea is that the notification file is small enough and can be pre-calculated so that there should be no problem with 10.000s of RPs polling for this file every couple of minutes. Caching of this file by CDNs could be allowed to reduce dependency on the back-end server, as long as this file is not cached for too long, eg 5 minutes. Note that full dumps and deltas are immutable. Even if one considers the practice of replacing objects sharing the same rsync URI, the actual objects themselves are unchanged, they are just replaced. And this replacement does not change the fact that the URI used to be associated with another object. Following this logic full dumps and deltas are made be available over truly unique URLs, making it possible to let CDNs cache them indefinitely (space permitting). This can help speed (more local delivery) and reduces the dependency on the back-end servers being available.

For now I will assume that xml can be used, but at this stage it’s really an implementation detail. The important thing to discuss first is the content of the files and the usage semantics. One thing to note though is that all files include headers with all relevant context information. In other words this is not just implied by the URI where the files were found, so they can be retrieved from different locations, or shared between RPs.

This file can be used by RPs to discover whether new objects exist, and where the files containing full and/or incremental dumps can be retrieved.

The header contains the following information: Element Description Session_Id UUID with time encoding identifying the session id of the publication server. File_Type UPDATE NOTIFICATION Version Integer representing the current version of the repository. The current version number is intended to be continuous. If a repository server gets confused it MUST reset the session id and start with version 0. If the number reaches max integer... reset session id and start with version 0

A list of all full repository dumps that are made available with the following information: Element Description Version The version number of the repository for this full dump. URI A unique URI pointing to this full dump. It is RECOMMENDED to construct this URI based on the session id and version. E.g. http://host/dumps/SESSION_ID/1.xml Hash The hash of that file (no added security, just helps detect transport errors)

A list of all available deltas: Element Description Version_From The first version contained in the delta. Version_To The last version contained in the delta. URI A unique URI pointing to this full dump. It is RECOMMENDED to construct this URI based on the session id and versions for this delta. E.g. http://host/deltas/SESSION_ID/1/2.xml Hash The hash of that file (no added security, just helps detect transport errors)

It is RECOMMENDED that publication servers aggregate deltas after some time to reduce the number of fetches that RPs need to perform. There are several strategies possible to optimise this, so this is not limited strictly here. The publication server MUST provide at least a minimal set of deltas that allows RPs to synchronise to the current version number, starting with any of the full dumps listed. The RP MAY need to process more than one delta file to achieve this. A synchronisation algorithm for RPs is desribed later in this document.

A full dump MUST contain all objects currently published by the publication server.

Similar to the update notification file header, but contains slightly different informtation: Element Description Session_Id Same as above File_Type FULL DUMP Version Integer representing the current version for this dump.

The following information is listed for each published RPKI object: Element Description URI A reference to how the current version of this object might be fetched directly Object Base64 Encoded Object The URI is useful for current RP implementations that use URIs to work out relations in the PKI. It should be noted though that RPs can also work this out based on manifests and object properties such as key identifiers, signatures and the hash of the binary object. See: http://www.ietf.org/internet-drafts/draft-tbruijnzeels-sidr-validation-local-cache-00.txt The extension of the filename can be used by the RP to determine the type of object.

An incremental dump file contains all changes between the start and end versions.

Similar to the update notification file header, but contains slightly different informtation: Element Description Session_Id Same as above File_Type DELTA Version_From Integer representing the start version for this delta. Version_To Integer representing the last version for this delta.

All changes are listed. Note that this follows the publication protocol closely. The following information is listed for each version covered by this detla and each object that changed in that version: Element Description Version The version when the change occurred. publish/withdraw Informs the RP whether the object was published or withdrawn (see also publication protocol).. URI Same as full dump Object Same as full dump

The location of the update notification files for a CA certificate publication point can be communicated by including an SIA pointer for this. This should probably be recognisable by protocol. E.g. fnpn*://rpki.ripe.net/updates.xml *: fancy new protocol name (yes, this is obviously a working title..) The protocol implies the actual transport protocol, eg. http, and semantics described in this document.

It is recommended that the RP uses a local cache of unvalidated objects for validation and does not rely on URIs to build up its internal representation of the PKI tree. For one way describing how to do this see: http://www.ietf.org/internet-drafts/draft-tbruijnzeels-sidr-validation-local-cache-00.txt This appraoch has the advantage that it's irrelevant where objects were retrieved from, and whether they were found once or many times. The RPKI objects are *defined* by their content: the bits. The name is a useful indicator, but not the definition of what the object *is*. So building the PKI tree based on this, makes it easier to deal with multiple possible fetch mechanisms such as rscync or the protocol proposed here, as well as the possible occurrence of more than one URI for each protocol. It is no longer relevant to know objects by name. In particular an RP can just add any new objects it finds to its local cache and ignore that objects have been withdrawn. This makes it much easier to deal with updates as described below because it allows that only one local unvalidated cache is used, and it largely eliminates the need to keep a detailed record of the current state compared to repositories; except some minimal information to keep track of what the latest version seen is. Because the local cache is ultimately space limited RPs MAY delete objects it no longer needs. For example old superseeded manifests and CRLs may be purged. If the manifest is used as the authoritative list of objects to validate then any object no longer referenced by any still relevant manifest can be deleted as well.

If a Relying Party notices a 'fnpn' SIA pointer on CA certificates it SHOULD prefer to use this method to get all objects from this publication server. If a RP finds the same pointer on multiple CA certificates it may assume that the publication server includes the products for all those CAs. If a RP finds a valid CA certificate that points to a different publication server, or one that only has the current rsync directory and manifests pointers, then it should use the retrieval mechanisms appropriate to that publication point.

The RP should retrieve the update file and store the session id and version locally. Since the RP will have no prior session for this update server it will give itself version 0. Now the RP can find the full dump with the highest number in the notification file and fetch it. Adding any objects not previously known to its own local cache. The latest full dump MAY not have all the latest changes. The RP can update further as described below.

It is RECOMMENDED that the RP regularly retrieves the update files and checks the session id and current versions for any 'fnpn' Update Locations it has learnt.

If the notification file lists a higher current version number than what we currently have then the RP can try to fetch updates using the following recursive approach: As long as the RPs current version is smaller than the published current version Find deltas with version-from smaller than or equal to the RPs version If no such deltas can be found, fall back to the latest full dump listed and update from there Otherwise select the delta with the highest version-to Replay it, adding all unknown objects to the local cache Update the current RP version to 'version-to' of the processed delta Repeat until in sync with current version on update notification file

If the session id changed the RP MUST reset its own current version, and retrieve a full dump and deltas as described in the 'New Update Location' exchange.

It is RECOMMENDED that the RP uses ALL publication servers it knows about to protect itself against outages or other problems (eg cause by attacks) with any one repository. If one local unvalidated cache is used as described above the overhead of doing this should be minimal.

It is no coincidence that the proposed delta format closely resembles the messages used in the current draft for the publication protocol. One thing to note is that it MUST be possible to combine multiple publish and withdraw elements in a single message from a client CA to a publiction server. This allows the CA to provide updates in units of 'consistent sets' that it is responsible for, and it minimises the the work needed to compute deltas for the publication server. It can just take such a message as a single increment and easily translate the publish/withdraw elements to the format described above.

Publication Servers SHOULD use unique URLs, eg bases on a UUID and the versions involved, for full dumps and deltas. The reason is that as long as these resources can be uniquely identified, it is much easier to offload serving them from http caches or CDNs.

It is recommended that the update notification file is not cached, or at least not cached longer than say 5 minutes, except when the back-end publication server is unavailable for some reason.

It is recommended that all other (unique) resources are cached. It should be safe to drop full dumps and deltas no longer referenced any recent update notification file, e.g. last referenced 72 hours ago...

TBD