Bootiful CMS part 3 - Microservice with Test-Driven Documentation

In this blog post, we will build a microservice that will replace the H2 database as the post store, and we will document its API with Spring Rest Docs.

Build with Gradle

As usual, we’ll build this project with Gradle. The build-file is very similar to the previous ones. This time we’re going to use JPA, so we will need the Spring Data JPA starter and H2 database dependencies.

services/post-service/build.gradle
def vJavaLang = '1.8'

buildscript {
    ext.springRepo = 'http://repo.spring.io/libs-release'

    repositories {
        maven { url springRepo }
    }

    dependencies {
        classpath "org.springframework.boot:spring-boot-gradle-plugin:1.2.5.RELEASE"
    }
}

apply plugin: 'war'
apply plugin: 'spring-boot'

targetCompatibility = vJavaLang
sourceCompatibility = vJavaLang

repositories {
    maven { url springRepo }
}

dependencies {
    compile("org.springframework.boot:spring-boot-starter-web")
    compile("org.springframework.boot:spring-boot-starter-actuator")
    compile("org.springframework.boot:spring-boot-starter-data-jpa")
    compile("com.h2database:h2")
    compile("org.springframework.cloud:spring-cloud-starter-eureka:1.0.3.RELEASE")
}

task wrapper(type: Wrapper) {
	gradleVersion = '2.7'
}

Application

Skeleton

The post service is a Spring Boot application again, so we need an Application class and a configuration file.

services/post-service/src/main/java/be/beeworks/service/post/Application.java
package be.beeworks.service.post;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.builder.SpringApplicationBuilder;
import org.springframework.boot.context.web.SpringBootServletInitializer;
import org.springframework.cloud.netflix.eureka.EnableEurekaClient;

import javax.servlet.ServletContext;
import javax.servlet.ServletException;

@SpringBootApplication
@EnableEurekaClient
public class Application extends SpringBootServletInitializer {
    public static void main(String[] args) {
        SpringApplication.run(Application.class, args);
    }

    @Override
    protected SpringApplicationBuilder configure(SpringApplicationBuilder application) {
        return application.sources(Application.class);
    }
}
services/post-service/src/main/resources/application.yml
server:
  port: 9080
spring:
  application:
    name: post-service

Since we already set up our discovery server in the previous blog post, we can add the Eureka client to the post service immediately. The name of the service in Eureka will be post-service and it will run on port 9080.

Model

We’re building on the previous blog posts, so the post model will be the same. Let’s just add some validation to the title property. This will come in handy later on.

services/post-service/src/main/java/be/beeworks/service/post/model/Post.java
package be.beeworks.service.post.model;

import javax.persistence.Entity;
import javax.persistence.GeneratedValue;
import javax.persistence.Id;
import javax.validation.constraints.NotNull;
import javax.validation.constraints.Size;

@Entity
public class Post {
    @Id
    @GeneratedValue
    private Long id;
    @NotNull
    @Size(min = 1, max = 200)
    private String title;
    private String content;

    public Post() {}

    public Post(String title, String content) {
        this.title = title;
        this.content = content;
    }

    public Long getId() {
        return id;
    }

    public void setId(Long id) {
        this.id = id;
    }

    public String getTitle() {
        return title;
    }

    public void setTitle(String title) {
        this.title = title;
    }

    public String getContent() {
        return content;
    }

    public void setContent(String content) {
        this.content = content;
    }
}

The repository is a simple Spring Data CrudRepository.

services/post-service/src/main/java/be/beeworks/service/post/repository/PostRepository.java
package be.beeworks.service.post.repository;

import be.beeworks.service.post.model.Post;
import org.springframework.data.repository.CrudRepository;

public interface PostRepository extends CrudRepository<Post,Long> {
}

Controller

Now that we have our model and repository, let’s expose the post service API, through a Spring MVC RestController. We’ll allow the usual CRUD operations, except for the Delete. We need 4 calls:

  1. a GET request /posts, that returns a list of posts as JSON

  2. a GET request /posts/{id}, that returns a single post as JSON

  3. a POST request /posts, with a post as JSON body, that returns HTTP status 201 (Created)

  4. a PATCH request /posts/{id}, with a post as JSON body.

services/post-service/src/main/java/be/beeworks/service/post/controller/PostController.java
package be.beeworks.service.post.controller;

import be.beeworks.service.post.model.Post;
import be.beeworks.service.post.repository.PostRepository;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.HttpStatus;
import org.springframework.web.bind.annotation.*;

@RestController
@RequestMapping("/posts")
public class PostController {
    @Autowired private PostRepository postRepository;

    @RequestMapping(value = "/{id}", method = RequestMethod.GET)
    public Post getPost(@PathVariable("id") Long id) {
        return postRepository.findOne(id);
    }

    @RequestMapping(value = "", method = RequestMethod.POST)
    @ResponseStatus(HttpStatus.CREATED)
    public void createPost(@RequestBody Post post) {
        postRepository.save(new Post(post.getTitle(), post.getContent()));
    }

    @RequestMapping(value = "/{id}", method = RequestMethod.PATCH)
    @ResponseStatus(HttpStatus.NO_CONTENT)
    public void updatePost(@PathVariable("id") Long id, @RequestBody Post post) {
        Post existingPost = postRepository.findOne(id);
        existingPost.setTitle(post.getTitle());
        existingPost.setContent(post.getContent());
        postRepository.save(existingPost);
    }

    @RequestMapping(value = "", method = RequestMethod.GET)
    public Iterable<Post> listPosts() {
        return postRepository.findAll();
    }
}

This is all very basic Spring MVC, perfect for our little demo microservice. You should be able to start it already, but make sure you start the discovery service first, or you’ll see a lot of stacktraces in the logs.

cd discovery
gradle bootRun
cd services/post-service
gradle bootRun

Now you can create a post with Curl, and retrieve a list of posts:

curl 'http://localhost:9080/posts' -i -X POST \
 -H 'Content-Type: application/json' \
 -d '{"title" : "sample post 1 from REST service","content" : "sample content 1"}'

HTTP/1.1 201 Created
Server: Apache-Coyote/1.1
X-Application-Context: post-service:9080
Content-Length: 0
Date: Sun, 13 Nov 2016 13:09:05 GMT
curl 'http://localhost:9080/posts'

[{"id":1,"title":"sample post 1 from REST service","content":"sample content 1"}]

Great! We have a working microservice. Let’s get to the interesting part of this story: documenting the API.

Documentation

There are quite a few tools available for documenting REST APIs. At the company I’m working for right now, we’ve been using RAML, but a very popular alternative is Swagger. RAML has the obvious drawback that you need to maintain it manually, which means it’s almost certainly not in sync with reality. Swagger generates documentation at runtime, so it’s much closer to reality, but it has other drawbacks. Adding Swagger to your project adds lots of transitive dependencies, which is something you might not want. Also, it leaks internal implementation details, and in some cases it doesn’t take all the (hidden) complexity of a Spring MVC app into account. Also the only way of adding further documentation to a REST call is by using annotations in your code, which is a really bad way to write documentation. A drawback of both RAML and Swagger is their URL-centric approach. The URLs of your calls are front and center, and a good API documentation starts from what you might want to do with the API, rather than from the URLs you might use.

That’s why I propose to use Spring REST Doc. Spring REST Doc combines hand-written documentation in AsciiDoc format with code snippets, generated through MockMVC tests. So you have the advantage that you can write nice, human-readable documentation, focussing on use cases rather that URLs, and most of all, all your code examples are guaranteed to work, because they are the result of tests against the actual code. And your production code contains no traces of documentation libraries.

Adding Spring REST Doc

Let’s start by adding Spring REST Doc to the Gradle build file.

services/post-service/build.gradle
def vJavaLang = '1.8'

buildscript {
    ext.springRepo = 'http://repo.spring.io/libs-release'

    repositories {
        maven { url springRepo }
    }

    dependencies {
        classpath "org.springframework.boot:spring-boot-gradle-plugin:1.2.5.RELEASE"
        classpath "org.asciidoctor:asciidoctor-gradle-plugin:1.5.2"
    }
}

apply plugin: 'war'
apply plugin: 'spring-boot'
apply plugin: 'org.asciidoctor.gradle.asciidoctor'

targetCompatibility = vJavaLang
sourceCompatibility = vJavaLang

repositories {
    maven { url springRepo }
}

dependencies {
    compile("org.springframework.boot:spring-boot-starter-web")
    compile("org.springframework.boot:spring-boot-starter-actuator")
    compile("org.springframework.boot:spring-boot-starter-data-jpa")
    compile("com.h2database:h2")
    compile("org.springframework.cloud:spring-cloud-starter-eureka:1.0.3.RELEASE")
    testCompile('org.springframework.restdocs:spring-restdocs-mockmvc:1.0.0.RELEASE')
}

test {
    outputs.dir file("build/generated-snippets")
}

asciidoctor {
    sourceDir 'src/main/asciidoc'
    attributes 'snippets': file("build/generated-snippets"),
            'doctype': 'book',
            'icons': 'font',
            'source-highlighter': 'highlightjs',
            'toc': 'left',
            'toclevels': 4,
            'sectlinks': true

    inputs.dir file("build/generated-snippets")
    dependsOn 'test'
}

war {
    dependsOn 'asciidoctor'

    Task asciidoc = getTasks().getByName("asciidoctor")

    from ("${asciidoc.outputDir}/html5") {
        into 'static/docs'
    }
}

task wrapper(type: Wrapper) {
	gradleVersion = '2.7'
}

Here’s what the updated build.gradle does, compared to the original one:

  • add the asciidoctor plugin to the project

  • add the testCompile dependency to org.springframework.restdocs:spring-restdocs-mockmvc:1.0.0.RC1

  • tell Spring REST Doc to put generated snippets in the directory build/generated-snippets

  • tell AsciiDoctor to look for asciidoc source file in src/main/asciidoc and for code snippets in build/generated-snippets

  • make the asciidoctor task depend on the test task

  • make the war task depend on the asciidoctor task

  • put the AsciiDoctor output in the war-file, in directory static/docs

Main documentation

First, we setup the basic structure of our documentation.

services/post-service/src/main/asciidoc/index.adoc
= {project-name} Getting Started Guide

:revnumber: {project-version}

include::introduction.adoc[]

[[conventions]]
= Conventions

include::conventions/verbs.adoc[]

include::conventions/status-codes.adoc[]

[[resources]]
= Resources

// \include::resources/some-resource.adoc[]
services/post-service/src/main/asciidoc/introduction.adoc
{project-name} is a RESTful web service for dealing with blog posts in the BeeWorks CMS.
services/post-service/src/main/asciidoc/conventions/verbs.adoc
[[overview-http-verbs]]
== HTTP verbs

{project-name} tries to adhere as closely as possible to standard HTTP and REST conventions in its
use of HTTP verbs.

|===
| Verb | Usage

| `GET`
| Used to retrieve a resource

| `POST`
| Used to create a new resource

| `PUT`
| Used to update a resource. Updating a resource with PUT means the resource will be completely replaced by the data in the request, so fields that are missing in the request will be set to NULL.

| `PATCH`
| Used to partially update a resource. Only the fields that are actually included in the request will be updated. Other fields on the resource will not be touched.

| `DELETE`
| Used to delete an existing resource
|===
services/post-service/src/main/asciidoc/conventions/status-codes.adoc
[[overview-http-status-codes]]
== HTTP status codes

{project-name} tries to adhere as closely as possible to standard HTTP and REST conventions in its
use of HTTP status codes.

|===
| Status code | Usage

| `200 OK`
| The request completed successfully

| `201 Created`
| A new resource has been created successfully. The resource's URI is available from the response's
`Location` header

| `204 No Content`
| An update to an existing resource has been applied successfully

| `400 Bad Request`
| The request was malformed. The response body will include an error providing further information

| `404 Not Found`
| The requested resource did not exist
|===

Although you are absolutely free to structure your API documentation any way you want, the authors of Spring REST Docs encourage this layout. Let’s go through the contents of the files:

  • The index.adoc file contains the structure of the documentation. All other adoc files are included here.

  • In the introduction.adoc file, you describe what the microservice does (and doesn’t do).

  • Next, the documentation will contain a description of our REST conventions: what HTTP verbs are used throughout the service and what status codes are returned when. A microservice should be consistent in its use of these, so it doesn’t make much sense to repeat this information with every resource in the service, like most tools do. The files provided here, verbs.adoc and status-codes.adoc reflect the conventions that should be used throughout De Persgroep.

  • The last section, the resources, is where we will describe the individual resources, and where we will include the code snippets the MockMVC tests generate.

Documenting a resource

With Spring REST Docs, documenting a resource means testing a resource. How’s that for a win-win?

Writing the test

Let’s start by documenting the resource for retrieving a list of all posts. The first thing we need to do is create a test for the controller:

services/post-service/src/test/java/be/beeworks/service/post/controller/PostControllerTest.java
package be.beeworks.service.post.controller;

... (imports omitted)

@RunWith(SpringJUnit4ClassRunner.class)
@SpringApplicationConfiguration(classes = Application.class)
@WebAppConfiguration
public class PostControllerTest {
    @Rule
    public final RestDocumentation restDocumentation = new RestDocumentation("build/generated-snippets");

    @Autowired
    private WebApplicationContext context;

    @Autowired
    private PostRepository postRepository;

    private MockMvc mockMvc;

    private RestDocumentationResultHandler document;

    @Before
    public void setUp() {
        this.document = document("{method-name}",
                preprocessRequest(prettyPrint()),
                preprocessResponse(prettyPrint()));

        this.mockMvc = MockMvcBuilders.webAppContextSetup(this.context)
                .apply(documentationConfiguration(this.restDocumentation))
                .alwaysDo(this.document)
                .build();
    }

    @Test
    public void listPosts() throws Exception {
        createSamplePost("sample 1", "content 1");
        createSamplePost("sample 2", "content 2");

        this.document.snippets(
                responseFields(
                        fieldWithPath("[].id").description("The post ID"),
                        fieldWithPath("[].title").description("The post title"),
                        fieldWithPath("[].content").optional().description("The content of the post"))
        );

        this.mockMvc.perform(get("/posts").accept(MediaType.APPLICATION_JSON))
                .andExpect(status().isOk());
    }

    private Post createSamplePost(String title, String content) {
        return postRepository.save(new Post(title, content));
    }
}

In addition to the standard RunWith annotation, the test class is annotated with the Spring Boot SpringApplicationConfiguration annotation, so the Spring Boot features will be added to the context configuration. The WebAppConfiguration tells the context loader that we want to test a web application (in this case with MockMVC).

Inside the class, we first add a JUnit test rule to bootstrap the Spring REST docs snippet generation. Note the path build/generated-snippets we pass here. This should be the same path that gets configured in the Gradle build.

Next we inject the WebApplicationContext and the PostRepository, so we can create sample posts in our tests.

In the test setup, we first create a Spring REST Docs RestDocumentationResultHandler, which is a Spring MVC Test ResultHandler for generating the snippets. The filename of the snippet will be the name of the test method, and the JSON output of the request and response will be pretty printed.

Finally, we build an instance of MockMVC, applying the documentation configuration and ensuring the documentation ResultHandler is always called.

After the setup, we are ready to test and document the listPosts method of our controller. First, we create 2 sample posts, using the injected repository. Then, we tell the documentation ResultHandler to document 3 fields in the response JSON: the id, the title and the content of each post in the array (note the syntax: [].id means the id property of each object in an array). Then we perform a GET request on the /posts resource, and we expect the status to be OK.

Running the test

Let’s run this test, either in your IDE or using Gradle:

gradle test

The test should succeed and as a result, 4 AsciiDoc snippets should have appeared in build/generated-snippets/list-posts:

  • curl-request.adoc, which is an example of how to call the /posts resource using CURL

  • http-request.adoc, containing the full GET request the test performed

  • http-response.adoc, containing the full response the test received

  • response-fields.adoc, containing a table with all the fields the response should contain

Note that the field descriptors we add to the documentation ResultHandler will also be validated by the test, and if the response does not contain the expeced fields, the test will fail. If the response contains fields that were not described, the test will also fail. This ensures that the documentation is always correct.

Including the snippets in the documentation

We can now add the documentation for the /posts resource to the index.adoc file:

services/post-service/src/main/asciidoc/resources/posts.adoc
[[resources-posts]]
== Posts

The Posts resource is used to create and list blog posts

[[resources-posts-list]]
=== Listing blog posts

A `GET` request will list all of the service's blog posts.

==== Response structure

Unresolved directive in <stdin> - include::{snippets}/list-posts/response-fields.adoc[]

==== Example request

Unresolved directive in <stdin> - include::{snippets}/list-posts/curl-request.adoc[]

==== Example response

Unresolved directive in <stdin> - include::{snippets}/list-posts/http-response.adoc[]

You must also include this file in index.adoc of course:

services/post-service/src/main/asciidoc/index.adoc
[[resources]]
= Resources

include::resources/posts.adoc[]

The description of the resource is just plain AsciiDoc. What’s interesting is the includes of the 3 snippet files. The {snippets} fragment is configured by Spring REST Docs and points to build/generated-snippets.

We can compile the documentation with the asciidoctor Gradle task:

gradle asciidoctor

The result is a file index.html in build/asciidoc/html5 that looks like this:

Posts resource documentation

Performing a gradle build will include this documentation in the war file, under the /static/docs path:

gradle build
java -jar build/libs/post-service-0.1.0-SNAPSHOT.war
  1. and open this URL in your browser: http://localhost:9080/static/docs/index.html. You should see the same page appearing.

Documenting Bean Validation constraints

One of the more interesting abilities of Spring REST Docs is documenting the bean validation constraints you add to your model using javax.validation annotations, although at this point (Spring REST Docs is at the RC1 of version 1.0.0 at the time of writing) it’s still a bit too much work. But let’s try it anyway.

As noted above, the title of a blog post has 2 constraints: it should not be null and it should be between 1 and 200 characters long.

@NotNull
@Size(min = 1, max = 200)

This information should be reflected in the documentation, specifically for creating and updating blog posts. Let’s add a test for the createPost method of the PostController:

services/post-service/src/main/java/be/beeworks/service/post/controller/PostController.java
...

    @Autowired
    private ObjectMapper objectMapper;

...

    @Test
    public void createPost() throws Exception {
        Map<String, String> newPost = new HashMap<String,String>();
        newPost.put("title", "created from REST");
        newPost.put("content", "content from REST");

        ConstrainedFields fields = new ConstrainedFields(Post.class);

        this.document.snippets(
                requestFields(
                        fields.withPath("title").description("The post title"),
                        fields.withPath("content").optional().description("The content of the post")
                )
        );
        this.mockMvc.perform(
                post("/posts").contentType(MediaType.APPLICATION_JSON).content(
                        this.objectMapper.writeValueAsString(newPost)))
                .andExpect(status().isCreated());

    }

    private static class ConstrainedFields {

        private final ConstraintDescriptions constraintDescriptions;

        ConstrainedFields(Class<?> input) {
            this.constraintDescriptions = new ConstraintDescriptions(input);
        }

        private FieldDescriptor withPath(String path) {
            return fieldWithPath(path).attributes(key("constraints").value(StringUtils
                    .collectionToDelimitedString(this.constraintDescriptions
                            .descriptionsForProperty(path), ". ")));
        }
    }

There are a number of things going on here, let’s walk through them:

  • First, we need the Jackson ObjectMapper, to create the JSON we will send with the POST request

  • Inside the test method, we first create a new blog post, as a Map. This will then be converted to JSON using the Jackson ObjectMapper

  • Next we create an instance of ConstrainedFields for the blog post. ConstrainedFields is a helper class for building FieldDescriptors containing the bean validation constraints as a separate attribute.

  • The rest of the test should be familiar, not much different from the previous one

To get this new FieldDescriptor attribute in the documentation, we need to do one more, slightly annoying, thing: we need to override the template for the request-fields code snippet.

services/post-service/src/test/resources/org/springframework/restdocs/templates/request-fields.snippet
|===
|Path|Type|Description|Constraints

{{#fields}}
|{{path}}
|{{type}}
|{{description}}
|{{constraints}}

{{/fields}}
|===

Note the extra column constraints. As before, we should now add the create blog post section the posts resource documentation file:

services/post-service/src/main/asciidoc/resources/posts.adoc
[[resources-posts-post]]
=== Creating a blog post

A `POST` request with a post as JSON body will create a new blog post.


==== Request structure

Unresolved directive in <stdin> - include::{snippets}/create-post/request-fields.adoc[]

==== Example request

Unresolved directive in <stdin> - include::{snippets}/create-post/curl-request.adoc[]

==== Example response

Unresolved directive in <stdin> - include::{snippets}/create-post/http-response.adoc[]

With the new resource test and documentation set up, we can run the test and the asciidoctor tasks again:

gradle test asciidoctor
  1. and the constraints will be documented in the new create blog post section of the documentation.

Create post documentation

Conclusion

Spring REST Docs is a convenient way to write API documentation that is guaranteed to be correct, thanks to its test-driven nature. It can do more than what we’ve covered here, so check the official reference documentation.

In the next blog post, we will replace the post repository in the blog post editor backend with the service we created today, and we will use NetFlix Feign, Ribbon and Hystrix to make the consumer resilient!