Running TestCafe tests in Docker, locally and in Jenkins

While you can run TestCafe tests on any machine that has Node.js installed, it is more convenient to run them in a Docker container. This has the advantage when running the tests you don’t need to think about any dependencies, Node.js version etc. It also makes tests less flaky, because the environment the tests run in is always the same, either locally or in the CI environment. This is even more important when using TestCafe for visual regression testing, because then the screenshots will look the same wherever you run the tests.

Setting up

Make sure you have the following software installed:

• Recent version of Docker.
• Node.js on your local development machine so we can test our code before creating the Docker image.
• Checkout my example git repository. It contains a TestCafe framework and tests. You can find it here:

richardhendricksen/testcafe-mocking-typescript-framework

Creating the Docker image

The goal is to have a Docker image that contains all the executables needed to run our TestCafe tests. This means software like Node.js, the required browsers and other configuration that ensures functionality like screenshots and reporting works. It will not include the test and testframework code itself in the Docker container. The testcode changes a lot when creating new tests, and I don’t want to recreate the Docker image every time we add a new test.

We will use volume mounts when running the tests locally in Docker. In CI environment we will use git to pull our testcode in our Docker container.
This keeps our image lean and evergreen.

Most other guides use the testcafe/testcafe Docker image as a base image. This image is created by the TestCafe team. While I was experimenting with Dockerizing my tests I found this Docker image to be limiting. It has only the testcafe npm package pre-installed. My test frameworks usually depend on more packages than only the testcafe package, so the image is too barebones for me. Also it expects you run your test with the testcafe command, so if you like me use custom runners written in Typescript it won’t work out of the box.

After some searching I found the circleci/node:browsers image. This image contains Node.js, Java, all the browsers we need and is regularly updated. The only thing that isn’t available out of the box is TestCafe itself. But we can fix this easily by installing it and all other npm dependencies using the yarn install command:

What this Dockerfile does:

• It copies the package.json from our project into the Docker image in the runner directory. There it installs the dependencies using yarn. This results that all our needed dependencies (including testcafe) are installed inside the Docker image.
• Finally it sets the WORKDIR to a subdirectory called framework. When we want to run our tests we will mount our testcode inside that directory. Why we do this is that we don’t want to mount our local node_modules directory inside the Docker container, because of OS differences this will lead to issues. And to prevent overriding the already installed node_modules, we mount the test code in a subdirectory. When Node.js cannot find the required dependencies in the node_modules folder in the current directory it will look into the node_modules folder in the parent directory etc.

Build the Docker image usingyarn docker:build
Or run docker build . -t testcafe-runner .

Running the tests locally

To run our tests inside the Docker container from our local machine we need to make our tests available inside the Docker container. We will use volume mounts for this. We need to mount the following directories:

• The testframework and tests.
• The report directory so when the tests are finished the reports are saved on our local machine and not in the Docker container that gets deleted.
• We don’t want to mount our local node_modules folder inside the Docker container, so we create an empty volume for this directory.

We can achieve this by running:

docker run -it --rm \
-v ${PWD}:/runner/framework \
-v ${PWD}/reports/allure:/runner/allure \
-v /runner/framework/node_modules \
testcafe-runner yarn test:ci

This command will spin up a container of type testcafe-runner, remove it after it is done running, mounts the needed volumes inside the container and will execute the yarn test:ci command inside the container.

This can also be executed using the yarn docker:run:test command.

After the tests have run you can check the Allure test report using the
yarn test:report:open command.

Running the tests in Jenkins (or any other CI env)

When running the tests in CI we don’t need to volume mount our testcode inside the container. We will use git to retrieve the latest code from the repository. For this demo we will use Jenkins, but any CI environment like BitBucket Pipelines, GitHub Actions will do.

To try this out locally you need a Jenkins instance with Docker support. To set this up easily I created a Docker compose file that sets up a Jenkins container with Docker in Docker support:

Run with docker compose up -d .

The Jenkins container it creates has no security features whatsoever, so only use for local testing!

Now navigate to http://localhost:8080/ and click on Create a job .

Create a Pipeline job:

And use the following pipeline script:

Run the pipeline. It will now:

• Retrieve the Docker image and start the container
• Checkout the code using git inside the container
• Install the npm dependencies using Yarn
• Run the TestCafe tests
• Create and archive the Allure report

Afterwards you can view the report using the Test results link in the menu:

This will open the Allure HTML report, with screenshots for failed tests:

Conclusion

This blog should give a quick guide on how to quickly get started with running TestCafe tests in Docker. The CI setup for Jenkins should as easily also work for Bitbucket Pipelines and Github Actions. Examples of these can be found on my example Github repo.