Validate Complex JSON Schema With References in Postman
Once I found out about writing tests in Postman, I went down the rabbit-hole on JSON Schema and how to deploy a complex data structure to validate another complex data structure.
The task was to migrate a grown and complex API to a new backend as a drop-in replacement. This means, the current API needed to be rewritten without changing any of the specs.
We've got a bunch of the endpoints in Postman to test some cases. The API was greatly documented, but not strictly validated. In order to make sure the rewritten API matches the old one, I decided to write some tests.
I basically "zäumte" the horse up from the back.
After two days I ended up on the docs of Ajv JSON schema validator and JSON Schema and oh boy did this make my head spin.1
Now I don't want to go into details about JSON Schema or AJV too much. This post is more about how to utilize this in Postman. And for that, you are better comfortable with how to use variables in Postman and how to set up mock servers, to test the JSON Schema against the responses.
And you should know a few things about JSON Schema.
The Easy Way
Postman includes Tiny Validator (tv4) and Another JSON-Schema Validator (AJV) and it's quite easy to validate a response against a JSON Schema:
const schema = {
"properties": {
"alpha": {
"type": "boolean"
}
}
};
pm.test('Schema is valid', function() {
pm.response.to.have.jsonSchema(schema);
});This video explains everything: (JSON Schema validation in Postman - YouTube
This is relatively straightforward, but unfortunately not really documented by Postman (as far as I can tell).
Postman relies on the implementation of the Chai.js testing suite and this seems to be a good choice if the schemas are simple. I wasn't able to make it work with complex schemas, that uses $ref's and because I'm lazy and rather spend 10 hours to learn the JSON Schema specs, I went and made it work in AJV.2
This is the JSON object I was reverse engineering:
{
"status": {
"code": 200,
"description": "User retrieved successfully."
},
"payload": {
"user": {
"firstName": "Joe",
"lastName": "Doe",
"role": 3,
"email": "doe@example.com",
"customerID": "",
"projects": [
"6IXG5mEg6QLl9rhVSE6m",
"Hs1bHiOIqKclwwis3CNf",
"8C2OUGVZXU35FA7iwRn4"
],
"status": "Status",
"id": "c1BSZnKLdHSRYqrU5hqiQo733j13"
}
}
}With this jsonSchema tool, I can turn my data into a JSON Schema.
It's naive to think that this is all I need to do to thoroughly validate the JSON, but with this test script, I can get a first glimpse of how helpful it is, to define all validation rules in one JSON object.
var schema = {
"$schema": "http://json-schema.org/draft-07/schema",
"$id": "http://example.com/example.json",
"type": "object",
"required": [
"status",
"payload"
],
"properties": {
"status": {
"type": "object",
"required": [
"code",
"description"
],
"properties": {
"code": {
"type": "integer",
"$id": "#/properties/status/properties/code"
},
"description": {
"type": "string",
"$id": "#/properties/status/properties/description"
}
},
"$id": "#/properties/status"
},
"payload": {
"type": "object",
"required": [
"contactPerson"
],
"properties": {
"contactPerson": {
"type": "object",
"required": [
"firstName",
"lastName",
"email",
"phone",
"mobile",
"street",
"streetNr",
"city",
"postalcode",
"position",
"responsibility",
"company",
"projects",
"updated",
"id"
],
"properties": {
"firstName": {
"type": "string",
"$id": "#/properties/payload/properties/contactPerson/properties/firstName"
},
"lastName": {
"type": "string",
"$id": "#/properties/payload/properties/contactPerson/properties/lastName"
},
"email": {
"type": "string",
"$id": "#/properties/payload/properties/contactPerson/properties/email"
},
"phone": {
"type": "string",
"$id": "#/properties/payload/properties/contactPerson/properties/phone"
},
"mobile": {
"type": "string",
"$id": "#/properties/payload/properties/contactPerson/properties/mobile"
},
"street": {
"type": "string",
"$id": "#/properties/payload/properties/contactPerson/properties/street"
},
"streetNr": {
"type": "integer",
"$id": "#/properties/payload/properties/contactPerson/properties/streetNr"
},
"city": {
"type": "string",
"$id": "#/properties/payload/properties/contactPerson/properties/city"
},
"postalcode": {
"type": "string",
"$id": "#/properties/payload/properties/contactPerson/properties/postalcode"
},
"position": {
"type": "string",
"$id": "#/properties/payload/properties/contactPerson/properties/position"
},
"responsibility": {
"type": "string",
"$id": "#/properties/payload/properties/contactPerson/properties/responsibility"
},
"company": {
"type": "string",
"$id": "#/properties/payload/properties/contactPerson/properties/company"
},
"projects": {
"type": "array",
"items": {
"type": "string",
"$id": "#/properties/payload/properties/contactPerson/properties/projects/items"
},
"$id": "#/properties/payload/properties/contactPerson/properties/projects"
},
"updated": {
"type": "integer",
"$id": "#/properties/payload/properties/contactPerson/properties/updated"
},
"id": {
"type": "string",
"$id": "#/properties/payload/properties/contactPerson/properties/id"
}
},
"$id": "#/properties/payload/properties/contactPerson"
}
},
"$id": "#/properties/payload"
}
}
};
pm.test('Schema is valid', function() {
pm.response.to.have.jsonSchema(schema);
});The problem is, I had many of those structures, with a lot of duplication. So I rather create JSON Schema snippets with $references and reuse them as much as possible, than creating 20 schemas that are based on the same response pattern.
The Complicated and Head-Spinning Way
The API I had to describe in JSON Schema was not perfect in form but followed some standards.
The most helpful is the use of plural and a singular form for the keys.
payload.users describes a collection of a single user object. This is consistent throughout the whole API and makes things easier.
With this setting, it was easy to describe a single user object.
// initialize a global variable in the collection root
schema = {};
schema.userProperties = {
"$schema": "http://json-schema.org/draft-07/schema",
"$id": "http://example.com/userProperties.json",
"type": "object",
"required": [
"firstName",
"lastName",
"role",
"email",
"customerID",
"status",
"id"
],
"properties": {
"firstName": {
"$id": "#/user/properties/firstName",
"type": "string"
},
"lastName": {
"$id": "#/user/properties/lastName",
"type": "string"
},
"role": {
"$id": "#/user/properties/role",
"type": "integer"
},
"email": {
"$id": "#/user/properties/email",
"type": "string"
},
"customerID": {
"$id": "#/user/properties/customerID",
"type": "string"
},
"projects": {
"$id": "#/user/properties/projects",
"type": "array",
"items": {
"$id": "#/user/properties/projects/items",
"type": "string"
}
},
"status": {
"$id": "#/user/properties/status",
"type": "string"
},
"id": {
"$id": "#/user/properties/id",
"type": "string"
}
}
};
It is important to use the $id identifier in order to stitch together schema fragments. The $id is a URI, but it doesn't have to be an accessible web address. It's more to describe external and internal references (/externalSchema.json vs. #/path/to/key or #/path_to_key) and the developer gets an idea of how the referenced schema is structured.
Even though schemas are identified by URIs, those identifiers are not necessarily network-addressable. They are just identifiers. Generally, implementations don’t make HTTP requests (https://) or read from the file system (file://) to fetch schemas. Instead, they provide a way to load schemas into an internal schema database. When a schema is referenced by its URI identifier, the schema is retrieved from the internal schema database. -- Docs
In my original example, the properties describe the user object, which is nestled into the payload object.
So almost every call of the API responses with something similar to this:
schema.base = {
"$schema": "http://json-schema.org/draft-07/schema",
"$id": "http://example.com/baseSchema.json",
"type": "object",
"required": [
"status",
"payload"
],
"properties": {
"status": {
"$ref": "simpleBaseSchema.json#/properties/status"
},
"payload": {
"$id": "#/properties/payload",
"type": "object",
// all possible properties
"properties": {
// single user object
"user": { "$ref": "userProperties.json" },
// a collection of user objects
"users": {
"type": "array",
"items": { "$ref": "userProperties.json" }
}
}
}
}
}
schema.simpleBase = {
"$schema": "http://json-schema.org/draft-07/schema",
"$id": "http://example.com/simpleBaseSchema.json",
"type": "object",
"required": [
"status"
],
"properties": {
"status": {
"$id": "#/properties/status",
"type": "object",
"required": [
//"code",
"description"
],
"properties": {
"code": {
"$id": "#/properties/status/properties/code",
"type": "integer"
},
"description": {
"$id": "#/properties/status/properties/description",
"type": "string"
}
}
}
}
}
There is a status and a payload key in the root. The status holds messages and status codes. The payload hold the different data objects.
{
"status": {},
"payload": {},
}In payload.properties I describe the possible data objects, like a single user, or a collection of users or other data objects that I leave out here for sake of simplification.
In both cases, I $reference { "$ref": "userProperties.json" } because that's the object I expect.
I also reference a certain part in the simpleBaseSchema.json that holds the structure of a simple response without a payload and its status object is everywhere the same.
Now I have three different schemas that share certain data objects.
Since I haven't tested any other JSON Validator I can't say yet how reusable this code is. But I believe if I save each JSON Schema as a file I should be able to read and use them in the JSON Schema implementation of another programming language.
For the tests in Postman, AJV it is.
There are several ways how to initialize AJV but what I did was the following.
All schemas and part of the AJV initialization code go into the Postman Collection Pre-request Script.
// @see https://ajv.js.org/guide/combining-schemas.html
// initialize AJV
var Ajv = require('ajv');
ajv = new Ajv({ allErrors: true, verbose: true });
// add all schemas
ajv.addSchema(schema);
The following part goes into the Postman Test section of each response.
// create a validation method that is referencing the baseSchema.json
const validate = ajv.getSchema('http://example.com/baseSchema.json');
// use the validation method in the Postman script
pm.test('Schema is valid', function() {
pm.expect(validate(pm.response.json()), JSON.stringify(validate.errors)).to.be.true;
});It creates the validate() method based on the schema that gets referenced in the getSchema() method. It's the schema $id of the schema.base variable/schema.
If I don't expect a payload, like after a DEL request, I can build a validate() method by doing this: ajv.getSchema('http://example.com/simpleBaseSchema.json')
This way I can use the building blocks of JSON objects very easily.
Bonus
In order to find out where the JSON might not be valid, the AJV errors can be displayed in the console or within the test by stringifying the error object JSON.stringify(validate.errors). Depending on the complexity, this can be really overwhelming so a better way is to just display what is necessary.
For that, I declared a global helper function in the collection script, called utils.
utils = {
/**
* Formatted AJV Error Messages
* @see https://ajv.js.org/api.html<small class="hashtag text-monospace text-muted">#validation-errors</small>
*/
errorMsg: function(err) {
console.log(err);
if(!err){
return true;
}
let errors = [];
err.forEach(e => {
let text = `\`data${e.dataPath}\` ${e.message}`;
errors.push(text);
})
return errors.join(', ');
}
};
and then in the test script use this:
pm.expect(validate(pm.response.json()), utils.errorMsg(validate.errors)).to.be.true;
Conclusion
In order to validate JSON, learn about JSON-Schema. It pays off, it's widely used and there are many implementations.
Postman is not the only tool to test APIs, but it's relatively easy to use and not as restricted for small teams (3 people).
The ability to design an API and to use Mock Servers and scripting is pretty nice. It's very helpful.
Todo
- Evaluating remote schemas served from Github or so (https://ajv.js.org/guide/managing-schemas.html#asynchronous-schema-loading)
- Check PHP implementation
Some Links
- Schema validation — Postman Quick Reference Guide Version 1.7.0 - July 2021 documentation
- Test Ajv in the brower with RunKit + npm: ajv
