Frontend integration test framework with CodeceptJS and Playwright

On my current project we use the widely known test pyramid. That means we have a small set of slow end-to-end tests that runs against a fully deployed application: frontend, backend and a database with meaningful data. One level below that we have our so called frontend integration tests. Also called UI integration tests, frontend only tests etc. These are tests where we do want to fully render the app in a real browser, but don’t want to test the whole stack and the tests don’t have to run in multiple browsers.

We were using a framework using Protractor for driving the browser and ng-apimock for mocking the http calls. Since Protractor is considered end of life, we needed to migrate to newer solution.

CodeceptJS

We eventually settled on a framework consisting of CodeceptJS and Playwright. If you don’t know CodeceptJS, it is more of a wrapper than a standalone testing tool. It supports frameworks like Testcafé, Protractor, WebdriverIO, Playwright and more. The advantage is that you can write your tests and page objects separate from the underlying framework that controls the browser.

Since we are considering using Testcafé for our new end-to-end framework and Playwright for the frontend integration tests, using CodeceptJS means we could reuse our page objects.

Playwright

Playwright is a relative new framework to automate browser testing that’s fast, reliable and evergreen. It is from the same team of developers that originally developed Puppeteer.

Getting started

In this blog we will go step by step creating the framework. You will need at least Node.js > 12.0. I will include all code examples as gists.
I also included my github repository where you can find the whole framework:

richardhendricksen/codeceptjs-playwright-with-mocking

Application under test

For this guide we will use a simple Todo app for our application under test. We will use the Todo-Backend app for this. It comes with a separate frontend and multiple backends to choose from. I have chosen for the Java 8 with Spring Boot backend. The resulting url for this app is shown below:

Todo-Backend client

The exact choice for the backend part isn’t that relevant since we will mock it anyway. But it is easier to debug tests when you have a fully functional app.

Setting up CodeceptJS

Create an empty folder, then initialize npm and/or git as usual. Install the CodeceptJS, Playwright and Typescript dependencies:
npm install --save-dev codeceptjs playwright typescript ts-node

Now setup CodeceptJS using npx codeceptjs init . We will be using Typescript as our main language where possible. The final step will be to add an initial feature to test. Use settings shown below:

Typescript

Since we will be using Typescript for our test code, we need to add a tsconfig.json file to our project:

Next we need to add ts-node to our generated config file:

Remove the unneeded jsconfig.json file.

We can now run the framework using npx codeceptjs run . If everything went correctly you will now see 1 passing test that does exactly nothing.

Adding first test

When initializing the CodeceptJS framework we already created an empty test: list_todos.spec.ts . Now we can add our code to test our first scenario.

Tests in the CodeceptJS framework consists of a file that first describe the feature that is tested. In this feature there are one or more scenarios. In theses scenarios we use the I object to interact with the browser. As described by CodeceptJS:

Each test is described inside a Scenario function with the I object passed into it. The I object is an actor, an abstraction for a testing user. The I is a proxy object for currently enabled Helpers.

Our first scenario will be to open the app with an empty todo list. We will then check if the title is shown, that we can make new todo items and no todo list is shown:

Run the test with npx codeceptjs run .

Success!

If you want to see more detail about the steps in the test, add --steps to the runner:

Page objects

To make the code more readable and reusable, we will migrate to using page objects. CodeceptJS has support for page objects. But since I want to use Typescript and don’t want to use their dependency injection I will use a different setup.

In the sub folder pages we will create our first page object: todo.page.ts. It will will contain the needed selectors and methods for getting or modifying the state of the app. The page object itself will not do any assertions, this will be done in the test itself. As stated by Martin Fowler on page objects:

Page objects are commonly used for testing, but should not make assertions themselves. Their responsibility is to provide access to the state of the underlying page. It’s up to test clients to carry out the assertion logic.

Therefore we won’t be using any of the see* methods anymore, since they do their own assertions, and I don’t want those in my page objects. We will use the grab* methods instead, which will return the state, instead of asserting them. We then use chai to do the assertion in the test instead of the page object.

async/await

The CodeceptJS code looks synchronous, but it actually asynchronous, as mentioned in the How it Works section:

Behind the scenes all actions are wrapped in promises, inside of the I object. Global promise chain is initialized before each test and all I.* calls will be appended to it, as well as setup and teardown.

However when using the grab* methods CodeceptJS states:

If you need to get element’s value inside a test you can use grab* methods. They should be used with await operator inside async function

In other words, test execution doesn’t pause when using the grab* methods, since they aren’t part of the global promise chain. Therefore we will need to use async/await setup when using those methods. To keep our test code consistent we will use it for all our test code.

Custom Steps

In our initial test we used seeElement and dontSeeElement methods. These methods asserts if the element is visible or not visible, respectively. CodeceptJS currently doesn’t have a method to grab the visible state of an element. The one method that comes close is grabNumberOfVisibleElements . This method however returns the number of elements that is visible given a selector. So we need to add a custom method canSee that returns the boolean we can assert.

When we initialized the project, CodeceptJS created a steps_file.js where we can add custom steps that extend the I object. But since we are using Typescript, rename the file to steps_file.ts. Then add our custom method:

Fix our config by replacing steps_file.js with steps_file.ts in the include section:

include: {
    I: './steps_file.ts'
},

Finally to provide autocompletion for our new method in the IDE, we need to update the Typescript definition file to include this new method.
Run: npx codeceptjs def

Combining all information above, the todo.page.ts will look as follows:

We have to do some extra plumbing for the getTitle method, because the grabTextFrom method returns either a string or string[] depending on the selector. Since we only expect a single string, when an array is returned we only take the first value. Another solution would be to throw a warning or error since we aren’t expecting an array of strings.

We can now edit our existing test to use the page object. For our assertions we will use the chai framework. Install it with npm install --save-dev chai .

We now have one scenario: an empty todo list. For the next scenarios we want the app to contain todos. To able to test this we would need to set the required state in the database before each test. Since we only are testing the frontend this is not some we want to do. The solution is to mock the backend request that returns the list of todos.

Mocking

Before adding more scenarios we need to add support for mocking the backend. CodeceptJS includes limited mocking support, but this is only implemented for Puppeteer and Webdriver. This means we will have to implement this ourselves.

Extending the Playwright helper in CodeceptJS is made easy by creating a custom helper. This helper will extend the I object, adding framework specific methods to handle the mocking.

Playwright has a method called page.route(url, handler) in their API. This method provides the capability to modify network requests that are made by a page.

To create a custom helper run npx codeceptjs gh . Use settings shown below:

As you can see above the helpers are written in Javascript instead of Typescript. This is because CodeceptJS expects custom helpers to be ES6 classes that extend the abstract Helper class. I have not been able to get this to work in Typescript yet.

The next thing we need to do is to extend our config to include our new custom helper. Replace the helpers section of the config with:

helpers: {
    Playwright: {
        url: 'https://www.todobackend.com/client/index.html?https://todo-backend-spring4-java8.herokuapp.com/todos',
        show: true,
        browser: 'chromium'
    },
    PlaywrightHelper: {
        require: './helpers/playwright.helper.js',
    }
},

Writing the helper

The method we will add will be called mockEndpoint. It expects the following parameters:

• url : A regex string that matches the http request
• method: the HTTP method type, e.g. GET, POST etc.
• baseDir: the directory where the mocked response can be found
• scenario: the filename of the mocked JSON response that contains the data to be returned

After writing this I figured out that the Playwright page.route doesn’t support HTTP method matching, it only matches the url. I still think it is a good idea to store this information though, so I have left it in.

Since the first 3 params are fixed for each endpoint we will combine them together in one object.

Create a directory mocking and add a endpoints.ts file containing:

It does not contain any endpoints yet, but we will add those later.

Now edit the playwright.helper.js file and add our method mockEndpoint. It uses the page object from the Playwright Helper provided by CodeceptJS. The page object contains the route method we need. It will match a regex http request and returns the JSON content of a file instead of the real request.

This provides us with an I.mockEndpoint(endpoint, scenario) method to use in our tests.

Typescript

To provide autocompletion for our new method in the IDE, we need to update the Typescript definition file again to include this new method.
Run: npx codeceptjs def

Mocking the GET todos endpoint

To mock the GET todos endpoint we need go back to the app:

Todo-Backend client

When checking the network tab you can see it does the following network request to get the list of todos when opening the page:

GET https://todo-backend-spring4-java8.herokuapp.com/todos/

It initially returns an empty JSON list:

[]

Now add a todo item and refresh. It will now return a JSON list with one item:

[
  {
    "id": 1,
    "title": "My first todo",
    "completed": false,
    "order": 1,
    "url": "https://todo-backend-spring4-java8.herokuapp.com/todos/1"
  }
]

So now we have all the info we need to mock this endpoint. Let’s add this info to our Endpoints class in the endpoints.ts file:

export class Endpoints {
    static TODOS: Endpoint = {url: '^https://todo-backend-spring4-java8.herokuapp.com/todos/$', method: 'GET', baseDir: 'todos'};
}

Let’s create a mock data files for this endpoint. In the directory mocking/data/todos create empty.json containing [] .

Next create singleTodo.json with:

[
  {
    "id": 1,
    "title": "My first todo",
    "completed": false,
    "order": 1,
    "url": "https://todo-backend-spring4-java8.herokuapp.com/todos/1"
  }
]

Updating first scenario

Our first scenario worked because the state of the backend was aligning with our test: the list of todos was empty. But we don’t want to call the backend, so edit the first scenario to mock the backend call so it returns an empty todo list:

Scenario('empty todo list', async (I) => {
    await I.mockEndpoint(Endpoints.TODOS, 'empty');
...

Adding second scenario

Let’s add a new scenario to our existing test file. We will mock the endpoint to return a single todo item and then check if that todo item is shown correctly.

Expanding page object

Since we now have a todo item in the frontend, we want to verify that:

• The title of the todo item is My first todo
• The todo item is not completed
• The number of todo items is 1

We need to expand our page object to be able to retrieve this information from the page:

Then we can update our test again:

Run the test with npx codeceptjs run --steps

Conclusion

While I was writing this guide I got more and more excited about the possibilities of CodeceptJS and Playwright. It was easy to setup, incredibly fast and so far really stable.

While the I.<method> BDD style reads really well if your tests only consists of these methods, it gets a bit messy if you use them in conjunction with page objects. What doesn’t help is that the example code on the CodeceptJS website does exactly this.
Solution for this would be to put everything in page objects, but then you shouldn’t use the I.see* methods anymore if you don’t want to do the assertions in page objects. However, not all see methods have an equivalent grab method, resulting in having to create more custom code yourself.

What I’m really curious about is how easy it is to reuse this framework and its page objects for e2e testing using TestCafé, since this is one of my reasons for choosing CodeceptJS in conjunction with Playwright instead of just Playwright alone. Perhaps I’ll add a future blog with my experiences on this matter.