Vary presence and requirement of properties with CRUD operation

[pplr]

Properties of a schema could be marked as readOnly or writeOnly if they should not be used respectively in a request or response. Additionally, required properties are present in the required field. Request vs response is the only criteria that can alter a resource schema.

This was not sufficient to describe our legacy REST APIs where presence and requirement of resource's properties is highly dependent of CRUD operation. The was due, in part, to wrong design choices. However, we faced generic situations that cannot be expressed for example:

• Write once / immutable property: for properties like slug or identifier. Those properties should not be part of update request.
• Server side default value: this kind of property is required only in response.

All this can be achieved using composition. I don't think that's a reasonable solution because:

• contract is not more human readable as resources are split in many schema
• for polymorphic resources (using oneOf), there is too many different schema to create and duplication is inevitable

We used following vendor extensions to be able to produce the documentation of our API:

Field Name	Type	Description
x-createOnly	boolean	Relevant only for Schema "properties" definitions. Declares the property as "create only". This means that it MAY be sent as part of a POST request but SHOULD NOT be sent as part of the response or any other request type. If the property is marked as x-createOnly being true and is in the required list, the required will take effect on the POST request only. A property MUST NOT be marked as both x-createOnly and readOnly, writeOnly, x-updateOnly, x-createForbidden or x-updateForbidden being true. Default value is false.
x-updateOnly	boolean	Relevant only for Schema "properties" definitions. Declares the property as "update only". This means that it MAY be sent as part of a PUT request but SHOULD NOT be sent as part of the response or any other request type. If the property is marked as x-updateOnly being true and is in the required list, the required will take effect on the PUT request only. A property MUST NOT be marked as both x-updateOnly and readOnly, writeOnly, x-createOnly, x-createForbidden or x-updateForbidden being true. Default value is false.
x-createForbidden	boolean	Relevant only for Schema "properties" definitions. Declares the property as "create forbidden". This means that it MAY be sent as part of a PUT request or be sent as part of the response but SHOULD NOT be sent as part of any other request type. If the property is marked as x-createForbidden being true and is in the required list, the required will take effect on the PUT request and the response only. A property MUST NOT be marked as both x-createForbidden and readOnly, writeOnly, x-createOnly, x-updateOnly or x-updateForbidden being true. Default value is false.
x-updateForbidden	boolean	Relevant only for Schema "properties" definitions. Declares the property as "create and read only". This means that it MAY be sent as part of a POST request or be sent as part of the response but SHOULD NOT be sent as part of any other request type. If the property is marked as x-updateForbidden being true and is in the required list, the required will take effect on the POST request and the response only. A property MUST NOT be marked as both x-updateForbidden and readOnly, writeOnly, x-createOnly, x-updateOnly or x-createForbidden being true. Default value is false.
x-requiredCreate	[string]	List properties required in the POST request.
x-requiredUpdate	[string]	List properties requires in the PUT request.
x-requiredRead	[string]	List properties required in the response.

Being able to describe requirement and presence of resource's properties based on usage context could be a great enhancement.

[handrews]

@pplr I want to explore your use case a bit with existing JSON Schema features. This also comes up in hyper-schema a lot so if possible I'd like to align Hyper-Schema (the spec for which I edit) and OpenAPI on this.

All this can be achieved using composition. I don't think that's a reasonable solution because:
contract is not more human readable as resources are split in many schema

I've seen this get unmaintainable when people treat the JSON Schema like an additive definition system, but it works well when JSON Schema is used for what it is: a constraint layering system.

The "base" schema should define all possible fields, and only mark as "required", "readOnly", etc. the ones that should meet that constraint in all situations.

Then for each usage, just layer on the "required", "readOnly", etc., and filter out parameters by setting their property schemas to {"not": {}} (the simplest impossible schema - in new JSON Schema you can make this more clear and encourage tools to optimize by setting a boolean schema of false, but that's no available in OpenAPI yet).

Example:

#/components/schemas/base:

{
    "required": ["alwaysHere"],
    "properties": {
        "id": {"type": "integer", "minimum": 0, "readOnly": true},
        "alwaysHere": {"type": "string"},
        "onlyOnCreate": {"type": "boolean"},
        "onlyOnUpdate": {"type": "string"},
        "sensitiveInformation": {"type": "string", "writeOnly": true}
    }
}

For POST to create:

{
    "allOf": [{"$ref": "#/components/schemas/base"}],
    "required": ["onlyOnCreate"],
    "properties": {
        "id": {"not": {}},
        "onlyOnUpdate": {"not": {}}
    }
}

For PUT requests:

{
    "allOf": [{"$ref": "#/components/schemas/base"}],
    "required": ["id"],
    "properties": {
        "onlyOnCreate": {"not": {}}
    }
}

For GET responses:

{
    "allOf": [{"$ref": "#/components/schemas/base"}],
    "required": ["id", "onlyOnUpdate"],
    "properties": {
        "onlyOnCreate": {"not": {}},
        "sensitiveInformation": {"not": {}}
    }
}

I think this covers most of your examples. It is more verbose, but it works and clearly shows the difference between the base and each usage. It also produces the behavior out of simple constraints rather than being a complex function of the extension keywords plus which other constraints are present with which value. In general, JSON Schema keywords that involve adjacent keywords are much harder to support and work with.

for polymorphic resources (using oneOf), there is too many different schema to create and duplication is inevitable

@pplr Is the duplication due to also using "additionalProperties": false or do you need to redefine properties across the oneOfs for other reasons? The next JSON Schema draft (which admittedly won't help OpenAPI 3.x) will have a keyword that fixes the problems of using "additionalProperties": false with the *Of keywords.

[mewalig]

I think PUT should be for creating a resource where the resource id is specified by the client/user (for example, a product model that has a user-specified model ID), PATCH for updating a resource, POST for creating something where the id will be specified by the server (for example, a purchase transaction, where multiple POSTs of the same data should create multiple transactions, with a server-generated unique transaction id) (in case helpful to reference: https://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html and https://stackoverflow.com/questions/630453/put-vs-post-in-rest. I understand that often POST is used for broader purposes but in my view that approach is highly flawed and since this thread is not meant to cover that discussion I won't go into that here).

[MikeRalphson]

As Mark Nottingham says:

Don’t use RFC2616. Delete it from your hard drives, bookmarks, and burn (or responsibly recycle) any copies that are printed out.

https://tools.ietf.org/html/rfc7231#section-4.3.4 covers PUT and s.4.3.3 POST

[handrews]

@mewalig this issue is really about describing existing APIs rather than best practices for designing new ones.

[pplr]

@handrews thank you for your help.

The way you use the not field is unusual.
Understanding "property": {"not": {}} meaning is not straightforward!

We use OpenAPI and JSON schema to help developers using our APIs.
We want to be closer as possible to the code they need to write.
When reading not: {}, it must be "translated" and coded as "forbidden field".

I agree with you that it totally covers my examples.

in new JSON Schema you can make this more clear and encourage tools to optimize by setting a boolean schema of false, but that's no available in OpenAPI yet

It will be possible to write this ?

{
    "allOf": [
        {"$ref": "#/components/schemas/base"},
        {
            "required": ["onlyOnCreate"],
            "properties": {
                "id": false,
                "onlyOnUpdate": false
            }
        }
    ]
}

@pplr Is the duplication due to also using "additionalProperties": false or do you need to redefine properties across the oneOfs for other reasons ?

I don't see any way to avoid duplication when the base resource is polymorphic, even with the "not": {} trick. For example:

{
    "required": ["alwaysHere"],
    "properties": {
        "id": {"type": "integer", "minimum": 0, "readOnly": true}
    },
    "oneOf": [
      {
        "properties": {
          "onlyOnUpdate": {"type": "string"},
          "alwaysHereA": {"type": "string"}
        }
      },
      {
        "properties": {
          "onlyOnCreate": {"type": "boolean"},
          "alwaysHereB": {"type": "string"}
        }
      }
    ]
}

[handrews]

@pplr

The way you use the not field is unusual.
Understanding "property": {"not": {}} meaning is not straightforward!

This is why in draft-06 we defined boolean schemas: true is equivalent to {}, and false is equivalent to {"not": {}}. This is exactly how true and false behave for additionalProperties and additionalItems, we just made that syntax legal anywhere a schema can appear.

So yes, in response to your "it will be possible to write this?", yes, that's exactly what it will look like.
It's a lot more clear, and is specifically intended to allow tooling to optimize false and true instead of having to guess about not. I'm bringing this up because it's important to not introduce more complex solutions for things that are solved quite simply by updating the JSON Schema support.

In the meantime, you can put {"not": {}} in a definition or as it's own schema file and do something like:

{
    "allOf": [{"$ref": "#/components/schemas/base"}],
    "required": ["id", "onlyOnUpdate"],
    "properties": {
        "onlyOnCreate": {"$ref": "#/components/schemas/forbidden"},
        "sensitiveInformation": {"$ref": "#/components/schemas/forbidden"}
    }
}

which gets the job done.

I don't see any way to avoid duplication when the base resource is polymorphic, even with the "not": {} trick. For example:

The not trick isn't the main technique for dealing with "polymorphism", but I need to understand more about your use case and what, exactly, you mean by "polymorphism" to tell you how to solve it (if it's solvable). In general, mapping OOP concepts directly into JSON Schema validation does not work well (hence awkward constructs like OpenAPI's discriminator). Are you trying to map OOP base/child classes into JSON Schema? If so can you give an example that produces too much duplication? It's hard to see these problems with trivial examples, they're too easy to refactor.

And, of critical importance: Are you trying to use "additionalProperties": false anywhere? That has a huge impact on how difficult this is.

[mewalig]

OK so use https://tools.ietf.org/html/rfc7231#section-4.3.4 instead. The same reasoning still applies.

@handrews re "this issue is really about describing existing APIs rather than best practices for designing new ones": I assume you are not suggesting that supporting existing practices means not supporting best practices where it is easy to support both. I can't imagine that OpenAPI would not want to support best practices. My only intention here is to suggest that any documented solution is presented in a manner that makes it clear how to apply the solution to a best-practices case (unless there are cases where the solution can't apply to both, which I can't think of).

[stueynz]

Now let's make a suitably realistic non-trivial base with structured objects rather than just a simple list of properties and figure out how we're going to make some nested properties required for certain operations.

#/components/schemas/Person:

{
       "properties": {
        "id": {"type": "integer", "minimum": 0, "readOnly": true},
        "name": {
            "type": "object",
            "properties" : {
                "firstName" : "string",
                "middleName" : {  "type" : "array", "items": "string" },
                "lastName" : "string"
            },
        },
       "demographics": { 
            "type" : "object",
            "properties" : {
                 "gender" : "string",
                 "ethnicity" : { "type" : "array", "items" : "string"}
            }
      }
}

...and then we have POST operation with body

{
   "schema" : {
       "allOf" : [ {"$ref: "#/components/schemas/Person" } ],
      
       // I bet this isn't allowed .... but it's what  I want to say
       "required" : [ "name.firstName", "name.lastName", "demographics.gender" ]
       ] 
   }
}

Is there a way of doing this with the current edition of OpenAPI / JSON Schema?

[handrews]

@stueynz There is a deepRequired proposal for JSON Schema, although it probably won't be considered until draft-09 (which will quite possibly be out before OpenAPI updates JSON Schema support anyway, as draft-08 should be out in two months or so). https://github.com/json-schema-org/json-schema-spec/issues/203 if you want to see details. I've been skeptical of it recently but I keep going back and forth- it's definitely still in the running.

@mewalig I am trying to stay focused on describing APIs as they are, which includes both the legacy APIs such as @pplr specifically mentioned in the original comment, and APIs designed with best practices in mind. It's not necessary to debate best practices such as the proper use of PUT, POST, etc. in this issue- there are many forums for that.

[fantavlik]

And, of critical importance: Are you trying to use "additionalProperties": false anywhere? That has a huge impact on how difficult this is.

I'm only a year+ late this discussion 😃 but my team does have schemas that really should have "additionalProperties": false in practice but because of the limitations and lack of deepRequired in OA3 we are forced to omit this.

Thank you @handrews for the elegant "property": {"not": {}} pattern. One limitation of breaking out the schemas like this is that I think it sends a signal to code generation that you should generate three separate models for PUT requests v. POST requests v. responses using composition where some properties are removed and some are marked required.

The use-case we would like to support is to signal to code generation to use a single model for these three+ scenarios where marshaling/unmarshaling handles all of the property omission depending on the http method/read/write context. I think the x-createOnly, etc vendor extensions mentioned above would help us unify into a single schema to achieve this although I worry about that language becoming a bit verbose and also the PUT debate above re: definition of create vs update.

Here is a proposal for extending the readOnly and writeOnly properties with x-readMethods and x-writeMethods to be more specific when needed -- I doubt it has a place in JSON schema since it's tied to http methods but it seems like it could solve our use-case in OpenAPI schemas:

Legend:
put? means the field is optional for PUT
'*' means required for all methods
('*?' could mean optional for all methods?)

SingleModel:
  type: object
  properties:
    immutableProp:
      description: optional on create, invalid for update, required for responses
      type: integer
      x-readMethods: ['*'] # equivalent to [get,put,post,delete,options,head,patch,trace]
      x-writeMethods: [post?,put?]
    alwaysHere:
      description: rendered in all responses and requests
      type: string
      required: true
    requiredForCreate:
      type: string
      x-readMethods: ['*'] # equivalent to [get,put,post,delete,options,head,patch,trace]
      x-writeMethods: [post,put,patch?]
    onlyOnUpdate:
      type: string
      x-readMethods: ['*'] # equivalent to [get,put,post,delete,options,head,patch,trace]
      x-writeMethods: [patch?]
    sensitiveInformation:
      description: optional for create/update, never returned
      type: string
      writeOnly: true # equivalent to x-readMethods: []
    createdOn:
      description: only returned in responses
      type: string
      format: date-time
      readOnly: true # equivalent to x-writeMethods: []
      required: true

[handrews]

@fantavlik see #1622 for why method-specific validation behavior is a problem for JSON Schema compatibility, which is a goal of OAS 3.1.

While this approach is better in that it uses new keywords instead of changing the behavior of existing keywords (readOnly and writeOnly have been part of JSON Schema for several drafts now), it's still essentially defining a schema template and asking for multiple schemas to be generated behind-the-scenes. There needs to be a clear separation between what is and is not a JSON Schema or else people will try to run the not-quite-json-schema through a validator and it's not going to work because validators don't understand HTTP context. Nor should they.

[codyaray]

Definitely would like to see something like this built in to OpenAPI... sort of like how many tools know to ignore readOnly: true on POST/PUT/PATCH operations, we need something that says "ignore this on update (PUT/PATCH) but allowed it to be set". As suggested by the op, immutable is a good choice for this:

Write once / immutable property: for properties like slug or identifier. Those properties should not be part of update request.

Otherwise, we need separate requestBody schemas for "create" and "update" operations. :(

Similarly, I have a use case for "onlyOnCreateResponse" such as secret attributes that are only returned once, on create. In our use case, we have an API Key management but we only want to return the secret part of the API Key in the POST response.