Skip to main content

2.5 Unit, integration, and end-to-end testing

Intermediate
Tutorial

Overview​

Testing code is an important stage of any development workflow. Without proper testing before the code is put into production, bugs and errors that may have been caught during testing can be detrimental to production environments. At this point of the developer ladder, it's time to explore tools that can be used to test our code.

There are three primary types of testing:

  • Unit testing: Tests individual functions and calculations to assure that they generate data or return the expected result; tests a single unit at a time.

  • Integration testing: Tests several functions, calculations, and portions of the code together; tests how different parts integrate with one another. A common form of integration testing is known as continuous integration testing, or CI testing.

  • End-to-end (e2e) testing: Tests the app's complete workflow, including buttons, forms, and frontend assets; tests the app from end to end.

Motoko unit testing​

The Motoko base library contains a series of test files that can be used for unit testing. There are tests for individual types, such as Text, Array, Func, etc. Check out the full library of test files.

Many of these tests use a library called Motoko Matchers. Motoko Matchers is a test library that contains several packages that can be imported and used to provide testing functionalities.

Let's take a look at a simple unit test. This will test the program's error messages:

import M "mo:matchers/Matchers";
import T "mo:matchers/Testable";
import Suite "mo:matchers/Suite";

let suite = Suite.suite("My test suite", [
    Suite.suite("Nat tests", [
        Suite.test("10 is 10", 10, M.equals(T.nat(10))),
        Suite.test("5 is greater than three", 5, M.greaterThan<Nat>(3)),
    ])
]);
Suite.run(suite);

This unit test does the following:

  • First, it imports the Suite, Testable and Matchers packages from the Motoko Matchers project.

    • Suite is a package that defines a simple test runner.

    • Testable is a package that provides functionality for units to be compared in a test.

    • Matchers is a package that provides functionality for composable assertions that can be used in tests.

  • Then, it defines a Suite, which is used to build a tree of tests to be run.

  • In the suite, 2 tests are defined. One tests if the Nat value 10 is equal to 10. The second tests if the Nat value of 5 is greater than 3.

  • Lastly, the suite is run with the line Suite.run(suite).

Canister unit testing​

In addition to the Suite, Testable and Matchers packages, Motoko Matchers also contains a Canister package. The Canister package can be used to execute unit tests for canisters. Below is an example that uses this Canister package:

import Canister "canister:CanisterName";
import C "mo:matchers/Canister";
import M "mo:matchers/Matchers";
import T "mo:matchers/Testable";

actor {
    let it = C.Tester({ batchSize = 8 });
    public shared func test() : async Text {

        it.should("greet me", func () : async C.TestResult = async {
          let greeting = await Canister.greet("Bob");
          M.attempt(greeting, M.equals(T.text("Hello, Bob!")))
        });

        it.shouldFailTo("greet my friend", func () : async () = async {
          let greeting = await Canister.greet("George");
          ignore greeting
        });

        await it.runAll()
        // await it.run()
    }
}

This unit test does the following:

  • First, it imports a canister called CanisterName. This allows the result of functions within the canister to be tested.

  • Then, it imports the Canister, Testable and Matchers packages from the Motoko Matchers project.

  • A batchSize is defined; this determines how many times the test will be run.

  • Two tests are defined. Both test the result of the canister's greet function:

    • The first tests if the canister's greeting function is passed the text 'Bob', does the result text equal 'Hello, Bob!'?

    • The second tests if the greeting doesn't greet 'Bob' and instead greets 'George', the greeting should be ignored.

  • Then, it runs all tests in the batch, indicated by the runAll() call.

Want to look at a more complex, running example? You can take a look at the unit tests that are used in the invoice canister example.

Want to learn more about running large, long-running batches of tests? Take a look at the Motoko BigTest library.

Rust unit testing​

In this developer ladder series, you'll continue to focus on exploring and using Motoko. However, it's important to make note of the resources available for Rust testing.

Learn more about Rust unit testing.

Tests using PocketIC​

PocketIC provides local canister testing using a Python library. Using PocketIC, developers can write tests for their canister in Python, then deploy the scripts to interact with their local ICP replica. An example test with PocketIC might look like:

from pocket_ic import PocketIC

pic = PocketIC()
canister_id = pic.create_canister()

# use PocketIC to interact with your canisters
pic.add_cycles(canister_id, 1_000_000)
response = pic.update_call(canister_id, method="greeting", ...)
assert(response == 'Hello, PocketIC!')

Learn more about PocketIC.

Tests using Light Replica​

Light Replica is a community-developed project designed to provide a local testing and development environment similar to Ethereum's Truffle or Hardhat tools. Light Replica creates a local node designed to replicate the behavior of a real ICP node that includes additional logging and functions to assist with testing. Using this local node, any Wasm file can be tested and interacted with as if it were deployed on a live production node. The Light Replica's codebase includes tests primarily written in Rust and Typescript, and can be beneficial as a tool for Rust development testing, but can be used with any canister that has been compiled into a Wasm file.

Learn more about Light Replica.

End-to-end (e2e) testing​

Now, let's take a look at creating a project and setting up end-to-end (e2e) testing for the project.

Prerequisites​

Before you start, verify that you have set up your developer environment according to the instructions in 0.3 Developer environment setup.

Creating a new project​

To get started, create a new project in your working directory. Open a terminal window, navigate into your working directory (developer_journey), then use the following commands to start dfx and create a new project:

dfx start --clean --background
dfx new e2e_tests

You will be prompted to select the language that your backend canister will use. Select 'Motoko':

? Select a backend language: ›
❯ Motoko
  Rust
  TypeScript (Azle)
  Python (Kybra)

dfx versions v0.17.0 and newer support this dfx new interactive prompt. Learn more about dfx v0.17.0.

Then, select a frontend framework for your frontend canister. Select 'No frontend canister':

  ? Select a frontend framework: ›
  SvelteKit
  React
  Vue
  Vanilla JS
  No JS template
❯ No frontend canister

Lastly, you can include extra features to be added to your project:

  ? Add extra features (space to select, enter to confirm) ›
⬚ Internet Identity
⬚ Bitcoin (Regtest)
⬚ Frontend tests

Then, navigate into the new project directory:

cd e2e_tests

Setting up the project​

Next, install vitest and isomorphic-fetch with the command:

npm install --save-dev vitest isomorphic-fetch

Then, open your package.json file and insert "test": "vitest" in the scripts category, such as:

package.json
...
  "scripts": {
    "build": "webpack",
    "prebuild": "dfx generate",
    "start": "webpack serve --mode development --env development",
    "deploy:local": "dfx deploy --network=local",
    "deploy:ic": "dfx deploy --network ic",
    "generate": "dfx generate e2e_tests_backend",
    "test": "vitest"
  },
...

Need the full package.json file? Check out the repo containing the full project for this tutorial.

Now let's create a folder to contain our test files. It is a common practice to store them in the ./src/tests/ folder of your project, so let's create that folder with the command:

mkdir src/tests/

Creating an agent​

Now, you'll create a mixture of JavaScript and TypeScript files to create a utility that'll be used to create an agent using generated declarations. This file will be called actor.js.

You'll be diving a bit into using an agent in this portion of the tutorial. Note that the developer ladder has briefly mentioned agents thus far, but hasn't fully explored agents yet. The developer ladder will cover agents in a future tutorial, but for this tutorial it is important to know that an agent is a library used to make calls to the Internet Computer public interface. You'll be using it to communicate the canister's public methods, defined in the declarations/e2e_tests_backend/e2e_tests_backend.did.js, to the file containing our the code for our tests.

Create a new file in src/tests/ called actor.js, then insert the following content:

src/tests/actor.js
import { Actor, HttpAgent } from "@dfinity/agent";
import fetch from "isomorphic-fetch";
import canisterIds from ".dfx/local/canister_ids.json";
import { idlFactory } from "../declarations/e2e_tests_backend/e2e_tests_backend.did.js";

export const createActor = async (canisterId, options) => {
  const agent = new HttpAgent({ ...options?.agentOptions });
  await agent.fetchRootKey();

  // Creates an actor with using the candid interface and the HttpAgent
  return Actor.createActor(idlFactory, {
    agent,
    canisterId,
    ...options?.actorOptions,
  });
};

export const e2e_tests_backendCanister = canisterIds.e2e_tests_backend.local;

export const e2e_tests_backend = await createActor(e2e_tests_backendCanister, {
  agentOptions: { host: "http://127.0.0.1:4943", fetch },
});

This agent file does the following:

  • Sets up file handlers by reading the canister IDs from their associated JSON file.

  • Fetches the root key since you are testing locally.

  • Imports the interface description language (IDL) from the declarations file (../declarations/e2e_tests_backend/e2e_tests_backend.did.js).

  • Creates a default actor.

This example uses fetchRootKey. It is not recommended that dapps deployed on the mainnet call this function from the ICP JavaScript agent, since using fetchRootKey on the mainnet poses severe security concerns for the dapp that's making the call. It is recommended to put it behind a condition so that it only runs locally.

This API call will fetch a root key for verification of update calls from a single replica, so it’s possible for that replica to respond with a malicious key. A verified mainnet root key is already embedded into the ICP JavaScript agent, so this only needs to be called on your local replica, which will have a different key from mainnet that the ICP JavaScript agent does not know ahead of time.

Creating a test file​

Now it's time to create our test file. Create a new file in src/tests/ called e2e_tests_backend.test.ts, then insert the following content:

src/tests/e2e_tests_backend.test.ts
import { expect, test } from "vitest";
import { Actor, CanisterStatus, HttpAgent } from "@dfinity/agent";
import { Principal } from "@dfinity/principal";
import { e2e_tests_backendCanister, e2e_tests_backend } from "./actor";

test("should handle a basic greeting", async () => {
  const result1 = await e2e_tests_backend.greet("test");
  expect(result1).toBe("Hello, test!");
});

In this test file, the following things happen:

  • The necessary packages are imported.

  • The necessary agent functionalities are imported from the actor.js file.

  • A test method is defined which accepts two arguments - a test name and a function.

  • Inside the test method, expect is used to check the result of the greet function against the expected result.

Running a basic test​

Next let's deploy the canister and run our test with the commands:

dfx deploy
dfx generate
npm test

The test should be successful and return output such as:

 ✓ src/tests/e2e_tests_backend.test.ts (1)
   ✓ should handle a basic greeting

 Test Files  1 passed (1)
      Tests  1 passed (1)
   Start at  16:24:03
   Duration  205ms (transform 32ms, setup 0ms, collect 68ms, tests 11ms, environment 0ms, prepare 41ms)


 PASS  Waiting for file changes...
       press h to show help, press q to quit

Running a complex test​

Now let's create a more complex test that checks the canister's metadata using the CanisterStatus API. Insert the following code at the end of the src/tests/e2e_tests_backend.test.ts file:

src/tests/e2e_tests_backend.test.ts
test("Should contain a candid interface", async () => {
  const agent = Actor.agentOf(e2e_tests_backend) as HttpAgent;
  const id = Principal.from(e2e_tests_backendCanister);

  const canisterStatus = await CanisterStatus.request({
    canisterId: id,
    agent,
    paths: ["time", "controllers", "candid"],
  });

  expect(canisterStatus.get("time")).toBeTruthy();
  expect(Array.isArray(canisterStatus.get("controllers"))).toBeTruthy();
  expect(canisterStatus.get("candid")).toMatchInlineSnapshot(`
    "service : {
      greet: (text) -> (text) query;
    }
    "
  `);
});

Need the full src/tests/e2e_tests_backend.test.ts file? Check out the repo containing the full project for this tutorial.

This test simply checks that our canister's metadata contains a Candid interface.

Then, run the test again with the npm test command. This time, the output should reflect 2 successful tests:

 ✓ src/tests/e2e_tests_backend.test.ts (2)
   ✓ should handle a basic greeting
   ✓ Should contain a candid interface

 Test Files  1 passed (1)
      Tests  2 passed (2)
   Start at  16:27:26
   Duration  247ms (transform 32ms, setup 0ms, collect 66ms, tests 62ms, environment 0ms, prepare 39ms)


 PASS  Waiting for file changes...
       press h to show help, press q to quit

Integration testing​

Now that you have a mini tutorial project to use, you'll use it to demonstrate continuous integration testing. CI testing is a common automated testing workflow. One example of CI testing is done through Github, where a repository can be configured to run a test every time a change is pushed to the repository's code.

Before you upload your code to Github, first edit the package.json file. In the scripts section of this file, add the following lines:

package.json
"ci": "vitest run",
"preci": "dfx stop; dfx start --clean --background; dfx deploy; dfx generate"

This tells the code to automatically start dfx and deploy the project's canister.

Then, add a Github workflow configuration. To do this, create a .github folder, with a workflows subfolder in your project with the command:

mkdir .github
mkdir .github/workflows

Create a new file .github/workflows/e2e.yml, then insert the following content into this file:

.github/workflows/e2e.yml
name: End to End

on:
  pull_request:
    types:
      - opened
      - reopened
      - edited
      - synchronize

jobs:
  test:
    runs-on: ${{ matrix.os }}
    strategy:
      fail-fast: false
      matrix:
        os: [ubuntu-20.04]
        ghc: ['8.8.4']
        spec:
          - '0.16.1'
        node:
          - 16

    steps:
      - uses: actions/checkout@v2
      - name: Use Node.js ${{ matrix.node }}
        uses: actions/setup-node@v1
        with:
          node-version: ${{ matrix.node }}

      - run: npm install

      - run: echo y | sh -ci "$(curl -fsSL https://sdk.dfinity.org/install.sh)"

      - run: npm run ci
        env:
          CI: true
          REPLICA_PORT: 8000

To learn more about Github workflows, check out the Github documentation.

Need help?​

Did you get stuck somewhere in this tutorial, or feel like you need additional help understanding some of the concepts? The ICP community has several resources available for developers, like working groups and bootcamps, along with our Discord community, forum, and events such as hackathons. Here are a few to check out:

Next steps​

To wrap up level 2 of the developer ladder, you'll dive into Motoko level 2 and learn more about Motoko's fundamentals.