Node.js and OpenAPI

Introduction

What is the need?

As introduced in the Introduction to Node.js codelab, Node.js has a unique advantage because developers that write JavaScript for the browser can write the server-side code in addition to the client-side code without the need to learn a completely different language. Node.js is thus a very good candidate for implementing an OpenAPI specification on the server side.

Since we use the OpenAPI standard for documenting our RESTful API, it is important to note that there also exist code generation tools that allow us to generate an Express Node.js application based on an OpenAPI specification document. We won’t however use these tools in this codelab.

What you’ll build

You will first realize the Planet API Node.js application, using the OpenAPI specification. After a stepwise implementation of the RESTful API, you will be able to serve the Planet API and to test it using a basic client.

You will then realize the first version of your API as specified in the Introduction to OpenAPI codelab. This step will naturally require some additional processing for delivering the data to the client based on the specifications.

What you’ll learn

The fundamentals of Node.js Express applications.
How to implement an OpenAPI specification step by step as an Express application.
The architecture of the Node.js Express application used for delivering the OpenAPI specifications.

What you’ll need

WebStorm.
The Node.js framework.
The Introduction to Node.js and Introduction to OpenAPI codelabs are prerequisites for this codelab.

Introduction to Express

Express - Node.js web application framework is a minimalist web framework widely used for creating Node.js web applications. A very good tutorial (see Installing Express and following steps) is given on the official Express website and one should read this introduction .carefully for a good understanding. This will allow everyone to well understand the architecture of the Node.js Express application that we will build in this codelab.

It is important to note that Express is not a replacement of the http Node.js module, but that it is rather built on top of it. Rewriting the “Hello World!” Node.js application of the Introduction to Node.js codelab is thus very easy:

index_with_express.js

const http = require('http');
const express = require('express');

const hostname = '127.0.0.1';
const port = 3000;

let app = express();

app.use((req, res) => {
  res.statusCode = 200;
  res.setHeader('Content-Type', 'text/plain');
  res.end('Hello, World!\n');
});

let server = http.createServer(app);
server.listen(port, hostname, () => {
  console.log(`Server running at http://${hostname}:${port}/`);
});

As you can see from this code, Express does not replace the http Node.js module, that still listen for incoming requests. Express is in fact a very effective way of helping in designing callbacks for serving different routes.

Note also that very often, Express applications are using shorthands that make the http module invisible, as shown below:

index_express_shorthand.js

const express = require('express');
const app = express();

const hostname = '127.0.0.1';
const port = 3000;

app.use((req, res) => {
  res.statusCode = 200;
  res.setHeader('Content-Type', 'text/plain');
  res.send('Hello World!');
});

app.listen(port, hostname, () => {
  console.log(`Server running at http://${hostname}:${port}/`);
});

In the code above, the app.listen() method does exactly the same as what was done in the previous example. This shows what Express is done for: it helps to build the request handler of your RESTful API Node.js application.

If you add this file into your project, install the Express module, and run the application, you should be able to open a new tab with the ” http://localhost:3000” in your browser and see the “Hello World!” displayed on the web page.

After installing the Express module, your “package.json” file should contain the dependency to Express as shown below:

package.json

{
  "name": "nodejs-openapi",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "dependencies": {
    "express": "^4.18.2"
  }
}

The syntax for specifying the version of the Express module to be used is well described in semver. Version numbers are specified using "major.minor.patch" tuples and the notation with ^ indicates that it allows patch and minor updates but not major updates.

Using Routes with Express

Eventhough your RESTful API provides only one or two paths, often a RESTful API provides hundreds of paths, each supporting multiple methods. Such an API thus provides hundreds of path/method combinations, each combination being defined as a route. Each route must be processed by your Node.js application and it is therefore necessary, for the sake of modularity, to provide a specific request handler function for each route.

As already explained, with Express, routes determine how an application responds to a client request to a particular endpoint, which is a URI (or path) and a specific HTTP request method (e.g. GET or POST). So routing refers to setting a specific handler for each route. This is accomplished by setting one or more handler function for each route defined by the API, using the syntax:

app.METHOD(PATH, HANDLER);

where:

app is an instance of express,
METHOD is the HTTP request method,
PATH is the path or URI, relative to the server entry point,
and HANDLER is the callback function to be executed when the route is matched.

If we apply routing to our Express application, it can be rewritten as:

index_express_with_routing.js

const express = require('express');
const app = express();

const hostname = '127.0.0.1';
const port = 3000;

let noRouteFound = (req, res) => {
  let route = req.method + ' ' + req.url;
  res.end('You asked for ' + route);
};

let helloWorldRoute = (req, res) => {
  res.statusCode = 200;
  res.setHeader('Content-Type', 'text/plain');
  res.send('Hello World!');
};

app.get('/', helloWorldRoute);
app.get('*', noRouteFound);

app.listen(port, hostname, () => {
  console.log(`Server running at http://${hostname}:${port}/`);
});

In the example, we set two routes, one for the method ‘GET’ on the ‘/’ path and one for the method ‘GET’ on all other paths.

Note

In Express, routing/middleware functions are executed sequentially, therefore the order of routing/middleware inclusion is important. If you invert the order of both app.get() methods in the example above, all requests will be handled by the ‘*’ route and no request will ever reach the ‘/’ route.

Note

A special routing method that may be used for setting one handler function for all HTTP request methods exists. This method is named ‘all’.

A detailed description of the routing principle is given on Express routing. Understanding the way endpoints are served with Express applications is important, since endpoints definitions are an important part of any OpenAPI specification.

Using Middlewares with Express

The concept of middleware is also essential for Express applications. The concept is well explained on Using Express middleware. Essentially, you may think of middlewares as functions that are added in the pipe handling the request, where each request goes through each function in the configuration order before the response is ultimately sent to the client.

In versions of Express anterior to 4.0, many middlewares were built-in with Express. Today, these middlewares are delivered as separate modules through a middleware layer for Node.js.

Among these many middleware functions, it is useful to point out a few ones. The Express cors middleware is a middleware that can be used to enable Cross-Origin Resource Sharing (CORS). Note that understanding CORS is important when developing Web applications and this topic will be covered in a separate codelab. Another middleware, the Express body-parser middleware, is a middleware that parses incoming request bodies before the request is handled by the application. It allows for instance to parse a JSON request body. The basic use of these middleware functions is illustrated below:

index_express_middleware.js

const express = require('express');
const app = express();
const cors = require('cors');
const bodyParser = require('body-parser');

const hostname = '127.0.0.1';
const port = 3000;

// enable CORS
app.use(cors());

// parse application/json body content
app.use(bodyParser.json());

let noRouteFound = (req, res) => {
  let route = req.method + ' ' + req.url;
  res.end('You asked for ' + route);
};

let helloWorldRoute = (req, res) => {
  res.statusCode = 200;
  res.setHeader('Content-Type', 'text/plain');
  res.send('Hello World!');
};

app.get('/', helloWorldRoute);
app.get('*', noRouteFound);

app.listen(port, hostname, () => {
  console.log(`Server running at http://${hostname}:${port}/`);
});

In this example above, middleware functions are used with no mount path. This means that the cors() and the bodyParser.json() functions will be executed every time the application receives a request. It is also possible to mount middleware functions on specific mount paths, as follows:

middleware_mount.js

app.use('/users', (req, res, next) => {
  console.log('Request Type:', req.method);
  next();
});

app.get('/users', (req, res, next) => {
  console.log('Request Type:', req.method);
  next();
});

In this example, the first function is executed for any type of HTTP request on the ‘/users’ path and the second function handles GET requests on the same path. Both functions pass control to the next middleware function by calling next(). A middleware function can also end the request-response cycle by calling the res.send() function for sending the response to the client.

Defining Router-level Middlewares

For improving the modularity in handling routes, one can use router-level middlewares which work in the same way as application-level middlewares.

A router object is an isolated instance of middleware and routes. It is somehow a separate “mini-application,” capable only of performing middleware and routing functions. It allows organizing routes into separate files, thus promoting modularity.

The definition of a route using the Router() concept looks like:

routes/helloWorld.js

const express = require('express');
const router = express.Router();

// will handle any request on the root path
// that is relative to where the router is used (path in use())
router.route('/').get((req, res, next ) => {
  // do something
  res.statusCode = 200;
  res.setHeader('Content-Type', 'text/plain');
  res.send('Hello World!');

  // at this point, the response is sent to the client,
  // so do not call the next middleware
  // next();
});


// Export the router to make it accessible for "requirers" of this file
module.exports = router;

The use of this router in another module is implemented as follows:

index_express_routes.js

const express = require('express');
const app = express();
const helloWorldRoute = require('./routes/helloWorld');

const hostname = '127.0.0.1';
const port = 3000;

let noRouteFound = (req, res) => {
  let route = req.method + ' ' + req.url;
  res.end('You asked for ' + route);
};

app.use('/', helloWorldRoute);
app.get('*', noRouteFound);

app.listen(port, hostname, () => {
  console.log(`Server running at http://${hostname}:${port}/`);
});

As you can see in the example above, a router behaves like a middleware, so you can use it as an argument to app.use().

Using Controllers for Serving Routes

By using Router-level Middlewares, it is possible to organize different routes into separate files/modules. One step further is to separate the code that routes requests from the code that actually processes the requests. This can be accomplished by using a popular Node.js architecture that implements controllers. With this architecture:

Routes forward the supported requests and any information encoded in request URLs to the appropriate controller functions.
Controller functions get the requested data from the models and based on this date, return the response to the client.

A diagram of the architecture is shown below. Applying such an architecture allows us to implement a modular structure for the code handling the routes and to clearly separate the routing code from the request handler functions. This architecture will be applied for the implementation of our OpenAPI applications.

**Write the Express Application for the Planet API**

In this section, we implement an Express application for the Planet API, that we specified in the Introduction to OpenAPI codelab. For this implementation, we will use the concept of routes, middleware functions and controllers as presented in the previous step.

We will also use the express-openapi-validator module. This module allows us to automatically validate API requests and responses using an OpenAPI specification. It also allows us to automatically generate routes based on the specification document.

The use of a validator module is quite simple and is as follows :

Write your Express application skeleton as documented earlier in this codelab. Add the CORS and body-parser middlewares to your application.
Define the path of the OpenAPI specification document that will be used in the next steps. Note that the OpenAPI specification document is expected to be stored at the root of your application.
index.js
```
...
const path = require('path');
...
const spec = path.join(__dirname, 'planetsAPI.json');
...
```

Define the controllers for delivering the requested data from the models

controllers/planets_controller.js

// return the all planets responses
exports.allPlanets = (req, res) => {
  const planets = [
    {"name": "Mercury", "position": 1},
    {"name": "Venus", "position": 2},
    {"name": "Earth", "position": 3},
    {"name": "Mars", "position": 4},
    {"name": "Jupiter", "position": 5},
    {"name": "Saturn", "position": 6},
    {"name": "Uranus", "position": 7},
    {"name": "Neptune", "position": 8},
  ];
  res.send(planets);
};

exports.onePlanet = (req, res) => {
  const detailedPlanetList = [
    {"name": "Mercury", "position": 1, "confirmed_moons":0, "provisional_moons":0},
    {"name": "Venus", "position": 2, "confirmed_moons":0, "provisional_moons":0},
    {"name": "Earth", "position": 3, "confirmed_moons":1, "provisional_moons":0},
    {"name": "Mars", "position": 4, "confirmed_moons":2, "provisional_moons":0},
    {"name": "Jupiter", "position": 5, "confirmed_moons":53, "provisional_moons":26},
    {"name": "Saturn", "position": 6, "confirmed_moons":53, "provisional_moons":29},
    {"name": "Uranus", "position": 7, "confirmed_moons":27, "provisional_moons":0},
    {"name": "Neptune", "position": 8, "confirmed_moons":14, "provisional_moons":0},
  ];
  res.send(detailedPlanetList[req.params.planetId - 1]);
};

Note that in our case, for the sake of simplicity, the models are simply represented as JSON arrays that are accessed in the exported controller functions. In a more complex case, the model would certainly be extracted from a database. Note that at this stage, the planetId parameter validity is not checked. This will be done in a later step.

Define the routes for serving the /planets and /planets/:planetId paths. The router makes use of the controller defined in the previous step.

index_express_routes.js

const express = require('express');
const router = express.Router();

// Require controller modules
const planetsController = require('../controllers/planets_controller');

router.get('/:planetId', planetsController.onePlanet);
// GET all planets at root
router.get('/', planetsController.allPlanets);

module.exports = router;

In the application file, add some code for serving the OpenAPI specification document as a static page.
index.js
```
...
// (optionally) Serve the OpenAPI spec
app.use('/spec', express.static(spec));
...
```
Install the OpenApiValidator module for your Express application. When installing the validator, choose to validate both requests and responses, which allows to validate that both requests are formulated correctly by the clients and that your Express application formulates the responses correctly as well. The validation is based on your OpenAPI specification and in particular on the way the schemas are specified.
index.js
```
...
const openApiValidator = require('express-openapi-validator');
...
app.use(
  OpenApiValidator.middleware({
        apiSpec: spec,
        validateRequests: true, // (default)
        validateResponses: true, // false by default
  }),
);
...
```

Define the routes to be used by your application

index.js

...
const planetsRoute = require('./routes/planets');
...
app.use('/planets', planetsRoute);
...

Add an error handler for the Express application. The way errors are constructed by the application is not yet specified in the OpenAPI specification document, which will be done next.

index.js

...
app.use((err, req, res, next) => {
  // dump error to console for debug
  console.error(err);
  // format the error (if no status is specified, make it 500)
  res.status(err.status || 500).json({
        message: err.message,
        status: err.status || 500,
  });
});
...

At this point, if you start your Express application, you should be able to test the application as follows by opening the links below in a browser window:

Open “localhost:3000/spec”: the OpenAPI specification should be displayed in JSON format, as expected.
Open “localhost:3000/planets”: you should see the list of all planets in JSON format, as expected.
Open “localhost:3000/planets/1”: you should see the detailed description of “Mercury” in JSON format, as expected.
Open “localhost:3000/planet”: you should see a “not found” message with a “404” status. In addition, you should see an error message generated by OpenApiValidator in the console of your Express application. This is as expected.
Open “localhost:3000/planets/0”: you should see an empty response with a “200” status. We expect an error message in this case and the behavior is not as expected.

At this stage, we need to fix the way errors are specified and handled.

Add Error Specification and Handling

In the OpenAPI specification document, errors and status codes must also be specified. This can be done by

Adding a specific error status code for a specific path and method. In the specification below, we add a status code for the “GET” method on the /planets/{planetId} path and we define that the response for this status code must follow the “NotFound” response format:

planetsAPI.json

...
"/planets/{planetId}": {
   "get": {
     ...
     "responses": {
        "200": {
         ...
        },
        "404": {
         "$ref": "#/components/responses/NotFound"
        }
      }
    }
...

We then need to define the “NotFound” response format:

planetsAPI.json

...

"components": {
  "responses": {
    "NotFound":{
      "description": "The specified resource was not found",
      "content": {
        "application/json": {
        "schema": {
          "$ref": "#/components/schemas/Error"
        }
      }
    }
  }
}
...

And finally we must define the “Error” schema:

planetsAPI.json

...

"components": {
  "schemas": {
    "Error": {
      "type": "object",
      "properties": {
       "status": {
         "type": "number",
         "description": "error code",
         "example": 404
       },
      "message": {
        "type": "string",
        "description": "detailed error description",
        "example": "The specified resource was not found"
      }
    },
    "required" : [
      "status",
      "message"
    ]
  },

...

Based on this description, we can now add the error handling code in the controller:

controllers/planets_controller.js

// return the all planets responses
exports.allPlanets = (req, res) => {
  const planets = [
    {"name": "Mercury", "position": 1},
    {"name": "Venus", "position": 2},
    {"name": "Earth", "position": 3},
    {"name": "Mars", "position": 4},
    {"name": "Jupiter", "position": 5},
    {"name": "Saturn", "position": 6},
    {"name": "Uranus", "position": 7},
    {"name": "Neptune", "position": 8},
  ];
  res.send(planets);
};

exports.onePlanet = (req, res) => {
  const detailedPlanetList = [
    {"name": "Mercury", "position": 1, "confirmed_moons":0, "provisional_moons":0},
    {"name": "Venus", "position": 2, "confirmed_moons":0, "provisional_moons":0},
    {"name": "Earth", "position": 3, "confirmed_moons":1, "provisional_moons":0},
    {"name": "Mars", "position": 4, "confirmed_moons":2, "provisional_moons":0},
    {"name": "Jupiter", "position": 5, "confirmed_moons":53, "provisional_moons":26},
    {"name": "Saturn", "position": 6, "confirmed_moons":53, "provisional_moons":29},
    {"name": "Uranus", "position": 7, "confirmed_moons":27, "provisional_moons":0},
    {"name": "Neptune", "position": 8, "confirmed_moons":14, "provisional_moons":0},
  ];
  if (req.params.planetId < 1 || req.params.planetId > detailedPlanetList.length) {
    let error = new Error();
    error.statusCode = 404;
    error.statusMessage = "Invalid planet id";
    throw error;
  }
  else {
    res.send(detailedPlanetList[req.params.planetId - 1]);
  }
};

As you can observe, the error is generated as specified with a message and a status property. The status property gets the 404 value (as a number). This allows proper error validation and proper error handling by the Express error handler defined in “index.js”.

You can now test again the “localhost:3000/planets/0” path and you now should see a “not found” message with a “404” status. In addition, you should see an error message generated by OpenApiValidator in the console of your Express application. This shows that this error is now generated and handled correctly.

Understanding Request and Response Validation

One very interesting feature of the OpenApiValidator module is that it automatically validates both requests and responses. For testing the validation process, you may:

Modify the planet_list function by generating an invalid response (e.g. an empty response).
Generate an invalid request by opening for instance “localhost:3000/planets?q=2” with a query parameter that is not defined in the API.

In both cases, you should observe responses with corresponding status codes in the browser window and detailed error messages in the Node.js console. This demonstrates how effectively and easily the OpenAPI implementation can be made robust to invalid requests and responses.

Deliverables

You first need to deliver the code of your Express application serving the Planet API to your gitlab repository. The code must be delivered in a way that makes it runnable after pulling the code and running npm install.
In your report, you must add a test report section that shows that you tested all cases described in this codelab. This means a test for the 5 cases described above, for the case where error handling is implemented correctly and for the two cases where invalid response and requests are detected.
Finally, you need to modify your OpenAPI specification document to make sure that the response sent for both /planets and /planets/{planetId} paths follows exactly the specified schema. It means that based on the specification document, if the response does not contain the required properties, then OpenApiValidator should generate an error.

Write the Express Application for your RESTful API

After implementing the Planet API In the previous step, you are now ready to implement the Express application for your RESTful API and to make the server application ready for the client application. Your RESTful API was specified in the Introduction to OpenAPI codelab. In this codelab, you specified the RESTful API and wrote the OpenAPI specification document that you may use now in this implementation.

At first, you need to reproduce all steps described in the previous section, while adapting the implementation to your API. By doing so, you will get the Express application working and delivering the required endpoints. What is missing at this point is the implementation of the model delivering the API data, as specified in the RESTful API document. You thus need to implement it.

The Data Source

The data specific to your API must be provided from a CSV file. This CSV file may be available online or stored on your own server.

The COVID-19 Data Source as an example

For example, the data delivered by the COVID-19 API is delivered from a CSV file available online (Covid cases Switzerland). You can download this file and open it using a CSV editor (like Excel or OpenOffice). The structure of the file is quite simple:

The first column contains the date in the format DD.MM.YYYY.
Further columns contain data for each Canton as cumulative number of cases (e.g. AG) or as daily number of cases (e.g. AG_diff). It also stores the change in percentage for the number of cases (e.g. AG_pc or AG_diff_pc).

Note that empty cells mean that the value of the cell is 0. The csv file is updated every day and a row is thus appended to the file every day. Note also that some Cantons (like Fribourg) update the statistics only on a weekly basis and for those Cantons, the last known value is added on a daily basis and updated when the statistics is updated by the Canton’s office.

Read Data from the Data Source

For implementing the model delivering data, you must first implement a function that allows you to read this data from the CSV file. For this purpose, you may use Node.js Streams as presented in the Node.js codelab.

The COVID-19 Data Source as an example

For the COVID-19 API, the signature of the function to develop would be “models/cases.js”

models/cases.js

...

async function readData() {
...
}

...

In this function, one would first open a Stream from an URL. For this purpose, the needle Node.js module would be used for performing the HTTP request in {{ node }} and downloading the CSV file as a stream of data, as shown below:

models/cases.js

...
const needle = require('needle');
...

async function readData() {
  const url =
   'https://raw.githubusercontent.com/daenuprobst/covid19-cases-switzerland/master/covid19_cases_switzerland_openzh.csv';

  const stream = needle.get(url, { compressed: false });
...
}
...

Once the stream of data is open, it may be piped through the following modules: * autodetect-decoder-stream that allows you to automatically detect the encoding of the data delivered by the needle stream * csv-reader that allows you to parse the CSV data delivered by the autodetect-decoder-stream component.

Callback functions on the csv-reader pipe may be added using the on() function: * .on('data', ...) allows a function to be called each time a row is parsed by the csv-reader instance. This allows you to push the parsed rows to an internal structure used for delivering the required data to the function caller. * .on('end', ...) allows a function to be called once all data has been parsed. This is not mandatory in our case.

Finally, the function may return a promise as shown below: “models/cases.js”

models/cases.js

...

async function readData() {
...
  return new Promise(function(resolve, reject) {
    stream.on('done', (err) => {
      ...
    });
  });
}
...

In this example, the readData() function immediately returns a Promise object to the caller. Once the stream reading is done (.on('done', ...)), the promise must be resolved in case of success or rejected in case of failure (meaning that err is not null). In case of success, the accumulated array of rows is returned.

Parse the Data

The readData() function allows you to read the data from the CSV file stored online. This data must then be parsed for extracting the required information

Parsing the COVID-19 Data as an example

In this exmaple, the readData() function allows you to read the data from the CSV file stored online. This data must then be parsed for extracting the required information from the raw data. For this purpose, one may develop a function called getCases(). This function receives as parameter the start and end dates, as specified in the query:

models/cases.js

...
async function getCases(startDate, endDate) {
  // get the entire data from the csv file
  let entire_data = await readData();
  ...
}
...

As you can read, this function first calls the readData() function using the async/await syntax mechanism. It must then parse the entire_data instance, for returning the data as specified in the COVID API to the caller. In this case, the caller is the controller serving the ‘/positives’ endpoint:

controllers/positivesController.js

const model = require('../models/cases');

// return the positive cases responses
exports.positiveCases = async (req, res) => {
  let cases = await model.getCases(req.query.start, req.query.end);
  res.send(cases);
};

Deliverables

You must deliver the code of your Express application serving your RESTful API. The code must be delivered in a way that makes it runnable after pulling the code and running npm install.
Your implementation must use the OpenApiValidator module for validating both requests and responses. Your RESTful API specification document must be completed such that both requests and responses can be validated entirely. -