NFS Protocol Extension: Retrospect and Prospect

This document examines the subject of protocol extension within the NFS family of protocols. In order to better understand the issues that exist going forward with NFSv4, we examine the history of protocol extension throughout the development of NFS including both the pre-IETF period and the development of successive NFSv4 minor versions. We then use this history as a basis upon which to explore the issues involved in providing a modified extension paradigm that builds on the work already done, but is more flexible.

Often, protocols require means by which they can be extended. Such extension may be needed to meet new requirements, to correct protocol weaknesses exposed by experience, or even to correct protocol bugs (as can happen when protocols are published as RFC's without fully fleshed-out implementations). We need to distinguish here between protocol "extension" and "versioning". Versioning is a form of protocol extension but not every form of protocol extension can be accommodated within a versioning paradigm. When a versioning paradigm is in place, groups of extensions are conceived of as ordered, allowing extensions in subsequent versions to build upon those in previous versions. When multiple extensions are combined into a single version, each of the extensions may be built assuming that the others will be present as well. In such cases, there can be the opportunity to make design changes in the protocol, allowing elements of the protocol to be restructured, sometimes in major ways. When a versioning paradigm is in effect and extensions are optional, extensions cannot build upon one another, since the presence of any particular extension cannot be assumed. In such cases, the ability to restructure the protocol is reduced, but smaller changes may be introduced more easily. In this latter case, it is not clear that the word "versioning" is appropriate. Nevertheless, in this document, we will, as in the phrase "NFSv4 minor versioning" use the existing terminology without necessarily subscribing to the view that "versioning" is the appropriate description.

Some factors that are often relevant in deciding on the means by which a protocol will be extended. Whether extensions are to be individually selectable (i.e. optional) or assumed to be always present, allowing one to build upon another earlier one. The size and scope of extensions that will be made. Compatibility issues with existing implementations. Issues that relate to ensuring that when individual extensions, separately arrived at, are each optionally allowed, that the ones used are compatible and can be used effectively together. The overall implementation framework. For example, RPC-based protocols may do extension by means of the RPC version mechanism. While it is possible to use different sorts of extension mechanisms for different sorts of extensions, protocols typically do not take advantage of that flexibility. On the other hand, protocols do, as NFS has done, change their preferred extension mechanisms in response to long-term changes in requirements. However, once having done so, they rarely switch back. Changing extension mechanisms is a big step, both conceptually and in implementation terms, and is not frequently repeated.

Often, protocols will be designed with specific mechanisms, designed to allow protocol extension. An example is the provision for TCP options (see and .) Most often, such mechanisms are designed to allow individual extensions to be designed and implemented independently, with any dependency relations between extensions specified separately and not enforced by the extension mechanism itself.

RPC-based protocols may, and often do, provide for protocol extension by replacing the XDR for one version with that for another and using the RPC versioning mechanism to manage selection of the proper protocol variant. The use of the RPC versioning mechanism enforces a versioning paradigm of this sort on protocols using this extension mechanism. This extension mechanism allows very extensive protocol changes, up to and including the replacement of one protocol by an entirely different one. For some kinds of protocol extensions, this seems the only way to effect the change.

It is possible to replace an XDR definition by one which is an extension in the sense that The set of messages described by the second definition is a superset of that described by the first. Each message within the set of messages described by the first definition is recognized as having exact the same structure/ interpretation by the second definition. Each message within the set of messages described by the second definition but not the first definition must be recognized as part of an unsupported extension. Within an XDR/RPC framework, extensions can be arrived at by: Addition of previously unspecified RPC operation codes. Addition of new, previously unused, values to existing enums. Addition of previously unassigned bit values to a flag word. Addition of new cases to existing switches, provided that the existing switch did not contain a default case. Such an extension relation between XDR descriptions is reflexive and transitive and thus defines a partial order. In practice, provisions have to be made to make sure that two extensions of the same description are compatible (i.e. either one is an extension of the other, or there is a another description that is a valid extension of both). To put things in concrete terms, such compatibility can be assured if measures are taken to ensure: That the same operation code is not used for two different RPC operations. That the same enum value is not assigned two different meanings. That the same bit flag value is not assigned two different meanings. That whenever the same case value is added to the same switch in two different extensions, the content assigned to the two matching added cases is the same. That default cases are never added to existing switches.

It is possible to use multiple of these means of protocol extension simultaneously. When this is done, the result is a composite extension mechanism. For example, if the XDR replacement or XDR extension mechanism is adopted, a protocol-specific mechanism can be added to it, if the protocol-specific mechanism is built on objects whose XDR definition is sufficiently generic. (e.g. opaque arrays or feature bitmasks). It can be argued that the NFSv4 attribute model provides such an embedded protocol-specific extension mechanism, since sets of attribute values are specified as XDR opaque arrays and attribute sets are specified as variable-length arrays of 32-bit integers allowing new attribute bits to be defined outside of the XDR definition framework. Note that there exists specification text that suggests that attributes are part of the XDR specification, making it hard to reach a firm conclusion on the matter. However, the resolution of this question does not affect the other matters discussed below, since, in either case, we have an extension mechanism that allows optional extensions.

NFSv2 and NFSv3 were described by the informational RFC's and . These documents each described existing interoperating client and server implementations. Thus they started with running code. If there was a rough consensus in effect, it was that these were useful protocols to use and thus that someone building a client or server had to interoperate with the existing implementations. The following characteristics of protocol development during that period are noteworthy. Most client implementations were implemented on very similar systems, in terms of the API's supported and many specifics of the local filesystems exported by servers. Often, the important client and server implementations were done by the same organization. As a result of these commonalities, specifications tended to avoid a lot of detail that would have been required in a more diverse environment. New features were thought of in terms of generally understood client and server implementation frameworks and it was generally clear which of those could be implemented without markedly changing that framework.

There were a number of significant changes involved, but only the first two were of major importance. Converting file sizes and offsets from 32-bit to 64-bit unsigned integers. Support for uncommitted WRITEs and the COMMIT request. Provision for WRITE to return atomic pre- and post-write file attributes, in order to allow a client to determine whether another client was writing the file. Interestingly enough, this feature was not carried over into NFSv4. The READDIRPLUS request. The addition of NFS3ERR_JUKEBOX (the precursor of NFS4ERR_DELAY). Of these only the first actually needed something as drastic as the XDR replacement model. The others could have been handled simply by adding new RPC requests and an enum value to an existing NFSv2 XDR. Since, NFS's extension mechanism was then XDR replacement, such choices were not available.

NFSv4 was the first NFS version published as a Standards track document. Although an initial , entitled "NFS version 4 protocol" was published as a Proposed Standard, it was never implemented due to issues with the design of locking. Subsequently, , entitled "Network File System (NFS) version 4 Protocol" was published as a Proposed Standard, obsoleting . Currently, there are bis documents, and , nearing publication. For more information about these documents and general discussion of issues regarding documents obsoleting or updating minor version protocol specifications, see . The set of changes made to create NFSv4 was larger by far than that for NFSv3. A partial list follows. Conversion to a stateful protocol, including support for locking. Locking included support for OPENs (with share reservations), byte-range locking (optionally including mandatory byte-range locks) and delegation. The COMPOUND operation. Conversion to a bi-directional protocol, by the addition of (optional) callbacks. Internationalization. Support for filesystems doing case-insensitive name matching. A new, extensible file attribute model, including support for acls, and conversion of user and group to a string model, opening the way for multi-domain support. Support for named attributes. Merger of ancillary protocols (e.g. MOUNT, NSM, NLM) into the NFS Protocol proper. Support for crossing of filesystem boundaries within a server's file name space (originally done for the incorporation of MOUNT functionality). Support for such multi-server operations as migration, replication, and referral. Referrals were not explicitly mentioned in and are explained in . Creation of a minor versioning model (to be discussed in ) to allow further protocol extension. These features/extensions were implemented via the XDR replacement model. This was the only realistic alternative, not only because of the size of the list, but also because some of the changes undercut some central design elements of the pre-IETF NFS protocol.

The minor versioning model for NFSv4 is an XDR extension model. It was presented within a versioning paradigm but the fact that all the added features were to be (at least initially) optional indicated that features were intended to be built independently, and that clients were expected to deal with their presence or absence. Note that the term "features" is not explicitly defined. We assume that the definition includes operations within COMPOUND or CB_COMPOUND, attributes, aded flag bits and enum values, and new cases of XDR switch definitions. Now let's look at some specifics of the minor version rules established for NFSv4 in . Note that some of these were significantly modified by and , as discussed in . No RPC requests may be added. Thus COMPOUND (and NULL) are to be the only requests within all NFSv4 minor versions. Similarly for callbacks, CB_COMPOUND and CB_NULL are the only requests within callback program. The arguments for COMPOUND and CB_COMPOUND contain a 32-bit minorversion field. Although this field is part of the minor versioning paradigm, it is not clear how useful it is, as long as all extensions are optional. For a more detailed discussion of this issue, see . Features may be defined as optional, recommended, or mandatory. These designations apply to implementation by the server. For clients, no operations are mandatory, although it is hard to imagine an NFSv4.0 client that does not use PUTFH or SETCLIENTID, for example. Features may be upgraded or downgraded along the optional/recommended/mandatory scale. Features may be declared "mandatory to not implement". This allows the deletion of a feature while retaining as reserved the value previously assigned. Clients and servers that support a particular minor version must support all previous minor versions as well. New features may not be designated as mandatory in the minor version in which they are introduced. Clients are not allowed to use stateids, filehandles, or similar returned objects from the COMPOUND procedure with one minor within another COMPOUND procedure with a different value of the minorversion field. This model was subsequently modified in and in . See and for details. Many of the events anticipated in the model presented above have never been realized and it may be that they never will be realized. See for some details. Examples are: There have never been recommended operations. There have never been optional attributes. Features have never been upgraded or downgraded in the transition between minor versions.

NFSv4.1 made a major change to NFSv4.0. It was able to do so using an XDR extension model although it did not follow the rules laid out in . Specifically, some features were declared "infrastructural" and thus mandatory upon introduction. Note that at the same time, the requirement that clients and servers support previous minor versions changed from a "must" to a "SHOULD". Presumably, this change reflects the fact that a minor version with substantial infrastructural changes is essentially a new protocol, making the "must" seem dubious. Whether the "SHOULD" here meets the requirements of needs to be explored. NFSv4.1 was described in and , each of which was published as a Proposed Standard. The following features were added as infrastructural features. Support for a sessions model including support for EOS (exactly-once semantics). Note that COMPOUND was taken advantage of to avoid adding slot and sequence information to the request header. Instead this information is packaged in a SEQUENCE or CB_SEQUENCE operation at the start of the COMPOUND or CB_COMPOUND. A new set of operations were added which enable the client and server to identify themselves to one another. Although these are often thought of as part of the sessions model, in fact they are logically distinct. RECLAIM_COMPLETE to allow better server sequencing of lock reclaim operations. There also a number of optional features. Parallel NFS WANT_DELEGATION to allow delegations to be obtained apart from opens. Directory delegations and notifications. The FS_LOCATIONS_INFO and FS_STATUS attributes. Note that there has been little implementation work on the last two of these. Parallel NFS created an alternate protocol extension mechanism for NFS. New pNFS mapping types could be added. Existing mapping types might have their own extension mechanisms. There also exists the possibility that features might be added within the NFSv4 protocol proper, designed to, or capable of, interacting with particular mapping types. This document will not address these issues but eventually, the NFSv4 Protocol will have to deal with them.

While NFSv4.2 has not been defined in an RFC, its development has been going on for some time and it is unlikely to experience additions to or deletions from its feature set before completion. The descriptions in and can serve as useful references for this analysis. This is despite the fact that some of the features may later experience significant change. The following features (all optional) are provided for in NFSv4.2: Support for labeled NFS. Server-side copy. An operation fence option on EXCHANGE_ID. Application data holes (formerly application data blocks). Disk-space reservation (nominally "recommended" since it is implemented by an attribute and attributes have never been declared "optional"). Hole-punching operations. READ_PLUS. Note that there are two pieces of infrastructure that are used by multiple features above. These are not "infrastructural" in the sense mentioned in (i.e. they are not mandatory), but they do serve an infrastructural role in that are required to be present if one of the optional features that use them are supported. WRITE_PLUS used to implement (ordinary) hole punching and application data holes. OFFLOAD operations/callbacks used to support WRITE_PLUS and server-side copy.

As noted above, there have been changes made by and in the NFSV4 minor versioning model. NFSv4.1 (in introduced the concept of "infrastructural" features (i.e. those defined as mandatory at initial introduction). NFSv4.2 (in added the concept of feature obsolescence allowing implementers to get early notice of the expectation that some features are on the path to becoming mandatory-to-not-implement. With these changes, we can classify potential minor versions, starting with those that currently exist. Minor version zero which introduced a new (major) version of the NFS protocol. All of the operations within it are new and a subset are effectively mandatory. Minor versions which introduce a new operation and make it mandatory (based on its being infrastructural). Currently, the only such version is minor version one, although there may be others in the future. All other minor versions. These add only optional/recommended features, each present or absent on the server with clients needing to be able to deal individually with their presence or absence. Currently, the only such version is minor version two. It is likely there will be others in the future and it may be that all future minor versions will be of this character. We term versions in the first two categories "infrastructure-level versions" Such versions form an ascending sequence in which the difference between, for example, NFSv4.0 and NFSv4.1, is very similar to the difference between NFSv2 and NFSv3. Clients may be designed for one or the other, or for both but a client capable of interacting with both is really choosing between two different protocols. We term versions in the last category "optional-feature-only versions". Note that although the concept of optional features being upgraded to mandatory status remains, it is likely that it will not be used very much, if at all, in the future. The situation is similar for the case of features being downgraded to mandatory-to-not-implement. Given the diversity of NFS clients and servers, it is highly unlikely that a new non-infrastructural feature will be so broadly necessary/desirable that a consensus to make it mandatory would be likely to arise. Such a decision would prevent servers not implementing such a feature from incorporating other later-developed features. It is only when a feature is judged so useful by users that people will not use servers without it, that adoption will become universal. At this point, a decision to make it mandatory would merely ratify what had already happened on its own. Except in the case of a universally recognized mistake, any downgrading to mandatory-to-not-implement, would only happen when a replacement becomes mandatory so the considerations above make that situation equally unlikely to occur. Still, it is possible that versions making such feature status changes will be created in the future. We will call any such "mandatory-feature-change" versions.

Minor versions which are infrastructure-level or which are, mandatory-feature-change versions form an ascending sequence in which we also have a versioning paradigm, implemented using XDR extension. Optional-feature-only versions are fundamentally different. Each NFSv4.2 server implements the same protocol as NFSv4.1 with a particular set of optional features beyond those that are mandatory. This set may range from the null set all the way to all of the optional features. Here, it appears that the versioning paradigm is not appropriate to the reality of the extension mechanism. As a way of illustrating the basic point here, let us consider two servers each of which only supports operations within NFSv4.1: The first server "supports" NFSv4.2 but none of the optional features added in . In this case, any attempt by a client to use one of those features will result in an NFS4ERR_OPNOTSUPP being returned. Let us say that the second server does not support NFSv4.2 and supports precisely the same set of features. In this case, a request will be rejected (with error NFS4RR_....) if its COMPOUND minorversion field is two and if the field is one, any unsupported NFSv4.2 operation will be rejected with NFS4ERR_OP_INVAL. Although this obeys the rules as they stand, there is no real value for the client, the server, or the protocol in making these artificial distinctions. Optional-feature-only minor versions such as NFSv4.2 are not minor versions in the same sense that NFSv4.0 and NFSv4.1 are. In this case the minorversion field is not providing any information, while the set of operations supported is the important thing that the server implementer chooses and the client needs to know. In later sections we will discuss how this mismatch might be best addressed as NFSv4 development proceeds.

To summarize protocol extension as it applies to the NFSV4 protocols: NFSV4.0 was implemented using the XDR replacement approach inherited from NFSv2 and NFSv3. As was to be expected given the nature and scope of the changes, its development took considerable time. It defined a protocol extension approach based on the XDR extension mechanism which was designed to enable future development of minor versions. However, this mechanism was not used as part of the implementation of NFSv4.0. NFSV4.1 was implemented using the XDR extension mechanism. To implement sessions, it was forced to modify the extension approach in the only way that was viable in the circumstances. As a result, the specification process took a long time, since it made significant structural changes to the protocol and also because it had to specify the entre protocol, and not just a set of extensions. NFSV4.2 returned to the original XDR extension mechanism and was intended to be a small incremental update with a one-hundred page (or less) specification. The fact that this turned out to be a multi-year effort has occasioned concern and we will attempt to see how the process can be streamlined and otherwise made more effective. The history of NFSv4.2 development serves to illustrate some of the inherent problems in the current approach to minor versioning. Since similar problems can be expected to recur unless that approach is changed, attention needs to be paid to understanding why such difficulties are experienced.

The following pattern was followed for NFSv4.2, and, unless changes are made, seems likely to persist. Various features are sketched out in individual drafts The working group reaches a decision (i.e. by rough consensus) as to the extensions/features to be included in the minor version. At the time this decision is made, the features may vary as to their maturity. Some have individual drafts which sketch them out while others do not. Any existing individual drafts are combined into a draft of a working group document intended to eventually evolve into an RFC describing the new minor version. These are then supplemented with descriptions of the other features chosen for inclusion. This document goes though further refinement and cycles of working group document review. At some point a companion -dot-x document is prepared and reviewed by the working group as well. The two documents go through working group last call, IESG review, and RFC publication. This pattern of development is not a good fit for the kind of minor version that NFSv4.2 is and many future such minor versions will be. Such versions consist of a set of mostly unrelated features, each individually selectable or not by implementers, artificially yoked together. In essence, we have a "feature batch" rather than a minor version.

A number of issues have been noted with the current process for NFSv4.2, leading to the conclusion that the process needs to be revised in some way for future minor versions, of the same sort. It takes too long to get a minor version drafted and through working group and IESG review. Despite the fact that NFSv4.2 was intended to be a fairly minimal minor version, describable in a one-hundred-page spec, it looks like the pace of development is such that there will be over a three-year delay from the time that the first NFSv4.2 draft was submitted to the point at which it is ready to be submitted to the IESG. The timeline for some of the features within NFSv4.2 is even longer. It will take about five years for the inter-server copy feature to go from initial draft to IESG submission of the minor version of which it is a part. We still do not have significant active implementations in which proposed last-minute protocol changes can be tested for validity. As an example of the problem, consider the decision to pass source stateids to the COPY op. If there were an implementation of inter-server server-side copy, the problems that this created (since stateids are tied to clientids in NFSv4.1 and beyond) would have quickly become manifest. Many features within NFSv4.2 have not received the kind of searching review appropriate to this stage of specification development. Some examples are discussed below. Some instances of problems/issues ascribable to a lack of searching document review are described below. Rather than requiring the necessary review prior to feature acceptance, a common pattern has been that important issues are only discovered on those occasions in which it appears that work on the minor version is coming to a close and that there are issues that have to be addressed before they create difficulties for prospective implementers. The state of the IO hints feature is most unsatisfactory. It is not clear how, or even if, it is possible to specify this in a way that interoperable clients and servers can be written which respond appropriately to such hints so that useful performance improvements can be demonstrated. It was the general understanding within the group that labeled NFS required use of RPCSEC_GSSv3, when in fact, only one case of labelled NFS, the server-enforced one, required it. When it became clear that RPCSEC_GSSV3 had not been worked on for a long time, the working group had to address that issue seriously. The security for inter-server copy was specified to be dependent on RPCSEC_GSSv3, yet, when it was found that RPCSEC_GSSv3 seemed not to be on the horizon, it turned out that a simple alternative was available. After a great deal of uncertainty about whether the functionality needed for inter-server copy would really be doable in RPCSEC_GSSv3, the working group had a range of choices for inter-server copy security. Later, after much work was done on RPCSEC_GSSv3, the working group has the option of going back to an approach dependent on RPCSEC_GSSv3. Application data blocks and related features have had a complicated history within NFSv4.2. Application data blocks were superseded by application data holes. There was little interest in implementing the feature since, as specified, a server implementation would require extensive filesystem extensions. Also, client-side API's were lacking, meaning that the only possible client-side implementations would be in special-purpose clients tightly bound to a particular implementation. Nevertheless, the feature continued in this unsatisfactory state for a long while. Eventually, it became clear that, as the feature was defined, it imposed significant implementation constraints even on NFSv4.2 clients not implementing the feature. At this point, it became clear that significant changes had to be made. How the issue will be resolved is not yet clear as this is written. If we look at the problems above, we can understand better how such problems can arise. In short, the decision as to what features to include within a minor version, is not a good use of the rough consensus model and in proceeding on that basis, the group created a set of perverse incentives that undercut the process. Also, as the process goes on for a long time, as is likely, these perverse incentives are intensified. Consider the following points: It is not clear exactly what the consensus as to proposed minor version contents actually means. Working group members might interpret it as meaning "These features are worth pursuing and they should be pursued". However, if they thought the definition was more like, "each of these features is so important that, if it is not ready, any other feature, including the one I'm interested in, should be delayed also", then it is hard to imagine any such rough consensus existing. Note that, given the minor versioning implementation laid out above (in ), the latter definition is, for functional purposes, the effective meaning, of the minor version content consensus. Given that many features are linked together, any delay in one feature, once it is accepted as part of the feature batch, delays all of the features, making it hard for people to comment forthrightly on any significant specification inadequacies. Not only will it delay your preferred feature, but if the problems are not fixed, the only recourse is an extreme penalty. As a result, it often seems not worth pursuing these sorts of issues. As the version turnaround cycle is so long, it is very difficult to remove a feature from a minor version feature batch. Given that these are all features that have enough interest to be in the minor version, it is hard to transfer the feature into the next minor version, given that doing so will certainly result in a multi-year delay, at a minimum. As a result, features, once accepted, have an implicit guarantee of inclusion in the minor version, undercutting the motivation of the proposer and others to work to move the feature forward. On the other hand, uncertainty about the time of specification completion, often makes it is hard to plan for and allocate resources to development of client and server implementations. Given that responsibility for a minor version is transferred to the editor of the minor version definition document at an early stage, we have a process in which it is not clear who has responsibility to follow up on the work necessary other than the minor version editor who may not have the required time and expertise in all cases. There is not a designated feature owner with responsibility to make the feature happen.

As we work to correct the issues noted above, and fill out the details of a modified extension paradigm, we will have to take note of the design considerations put forth in .

In some cases it is necessary to update the specification of an NFSv4 minor version, to deal with cases where the existing specification is unclear or incorrect and the changes are too large to address via the errata mechanism. In the case of NFSv4.0, the following documents will update or obsolete , if approved. There is a set of documents being prepared to replace the existing specification of the NFSv4.0 protocol. These are mostly to clarify documents but in some cases there are changes to the protocol (e.g. explicit discussion of referrals, rework of internationalization), although the new specification matches existing implementations. is marked as obsoleting . The companion xdr specification (), while not marked as obsoleting or updating , effectively replaces the XDR specification within that document. There is a document being worked on () to correct the handling of transparent state movement in the event of fs migration. That document is currently marked as updating . Although these documents change the specification of the protocol, they do not make any substantive changes in the XDR definition. It is generally held to be the case, that a document updating a minor version RFC, is not allowed to extend the XDR. Sometimes typedefs are added to make it clearer how particular fields are used. Also comments have been added and minimal reformatting done, but even addition of new error codes, as opposed to adding existing error codes to the list of those allowed to be returned by a given operation, has not been allowed.

Given that we have an XDR extension paradigm in place, it may not make sense to prohibit XDR extensions to be made in these update documents, as it may be necessary to fix protocol (as opposed to specification) bugs. Although the working group has been able to avoid XDR extension in fixing protocol bugs, that sort of luck might not continue. While the working group was able to compatibly correct the state migration issues without an XDR change, doing so made the process more complicated than it would be otherwise, since client implementations needed to infer trunking patterns from server behavior, rather than having this information directly provided to them. It should be noted that the prohibition of such XDR changes is not explicitly mentioned anywhere, to my knowledge. Rather, it seems to be a piece of NFSv4 versioning folklore which needs to be either justified or discarded. Acknowledging this change would enable the working group to do the following sorts of things (in addition to the correction of the kinds of specification bug fixes now recognized as allowed) in bis documents and other documents which update minor version definition RFC's. While it would be theoretically possible to add entirely new features, working group and IESG review should keep additions limited to the following two classes of items. Adding new operations to correct protocol bugs, subject to the proviso that a server implementing a replacement operation must also support the operation replaced. In this case, uniqueness for values such as operation and attribute numbers would assured by defining them in a later minor version's XDR definition document, or some update thereof. If the bug fix does not apply to that later minor version, it can be treated there as mandatory-to-not-implement, to match what it replaces. Backporting of features whose usefulness has been proven in a subsequent minor version and can be easily made available in an earlier minor version. In this case, values such as operation and attribute numbers are already assured of uniqueness, due to their assignment as unique in the later minor version. Doing things this way would address the issues that have given rise to the perceived need for "micro-versioning". Note that the sorts of changes that would be made would not require any change in the minorversion field at all.

The following requirements will govern construction of a possible new protocol extension approach for NFSv4. That individual extensions not be documented together (i.e. in the same document) unless there are good reasons to do so (e.g. the extensions use common facilities not documented anywhere else, there is a dependency relationship among the extensions allowing one to be used only when others are available). This would allow for shorter documents and a less trying review process. The process should allow for protocol bugs to be fixed, even if the problem is found after RFC publication. Just as we have the errata process to fix spec issues, we should be able to fix bugs/oversights in the XDR, as long as compatibility issues with existing implementations are addressed. That the process gives the working group an appropriate opportunity to review extensions and reject those that are architecturally inappropriate. That the process gives the working group and IESG an appropriate opportunity to review extension specifications and get them fixed, without adversely affecting other unrelated features. That the process appropriately assigns the responsibility for making proposed extensions real on those proposing them. This should include, at an appropriate time, work to get "running code" (i.e. client and server implementations) to demonstrate implementation feasibility.

The following principles seem the best way to meet these requirements without a disruptive major-version-scale shift in the NFSv4 definition (i.e. something as big as the shift from NFSv3 to NFSv4 or from NFSv4.0 to NFSv4.1) That the extension mechanism be usable to correct protocol bugs in bis documents and other RFC's updating existing RFC's. See for details. That a new, more flexible workflow be designed for ongoing protocol extension development. This should include some recognition of the role of the development of implementations in the maturing of a feature. That appropriate feature discovery mechanisms be added to allow clients to discover what facilities a server supports, without having to attempt to use them all.

The following set of steps are necessary in many possible ways of proceeding along the way to a new extension approach for NFSv4. Details and actual sequencing will reflect choices that the working group makes. Creation of a new standards-track document defining how protocol extension and versioning are to work in NFSv4. Since it would supersede existing treatments of the issues, it should be recorded as updating specifications for NFSv4.0 ( or the RFC arising from , when approved), for NFSv4.1 (), and for NFSv4.2 (the RFC arising from when approved). Establishment of a framework for extension discovery (and negotiation if that is judged necessary) to replace testing of operations, switch arms, and flag bits for errors indicating lack of support. The operations and attributes for this framework might be documented in the document defining the protocol extension framework, in a free-standing standards-track document, or as an infrastructural feature in a new minor version. In any of those cases, they could be incorporated as optional in earlier minor versions by documents updating the minor version specification RFC's. Design of processes to review proposed extensions for architectural suitability, to reserve necessary operation codes, attribute numbers, and enum and flag values, and to decide when and if minor versions should be created to upgrade or downgrade features. Such processes might well be documented in a working group informational document, but any such documentation should probably be done only after the working group is satisfied that the processes are working well. As far as the value reservation issue, we will have to decide whether an IANA-based approach, as recommended by , is required or whether simpler procedures, implemented within the working group, are adequate. As far as the document specifying the NFSv4 extension and versioning framework, the following are important elements: Clearly separate the concepts of protocol extension and versioning to allow the basic XDR extension approach to be used in contexts other than creation of a minor version. Clearly specify rules and expectations for updates to already defined minor versions. Discourage any future incorporation of the definition of protocol extensions in minor version definition documents, except where the extension is infrastructural. Thus the basic function of minor version definition documents would be to specify what features already defined are included and their status (experimental, optional, recommended or mandatory) in that minor version. Resolve and clarify cases where the original minor versioning rules don't match the extension/versioning model as it has evolved (e.g. how far the obligation of a client to support earlier versions extends, across what sorts of version changes does it make sense to require non-use of filehandles, stateids, etc.)

Since no substantive protocol changes are proposed here, no security considerations apply. As extensions are designed and specified, their security issues will be addressed and each extension will receive the appropriate security review from the NFSv4 working group and IESG.

The current document does not require any actions by IANA. Depending on decisions that the working group makes about how to address the issues raised here, future documents may require actions by IANA.

The author wishes to thank Chuck Lever of Oracle for his helpful document review and many important suggestions.