Filtering & Controlling Test Runs

Japa gives us a number of options to dictate which tests run. We've already covered some of the CLI-based options we have to filter our tests, within our Test Runner lesson, but we can also take it a step further with per-test designations.

Let's start again by creating a new spec

node ace make:test controlling_runs --suite=unit

  

    

      Copied!
      

        

      
    
  
  
    
    
    
    
  

Okay, first let's set up a simple test that uses our test service to simulate a ping to an external service to see whether we're able to connect.

test.group("Controlling runs", () => {
  const testService = new TestService();

  test("successfully ping external service", async ({ assert }) => {
    const result = await testService.pingExternalServiceExample("success");
    assert.equal(result, "success");
  });
});

  

    

      Copied!
      

        

      
    
  
  
    
    
    
    
  

Great, we can see this works a-okay... but let's say maybe this service is being sunset and we're migrating to a new one. We no longer need this test to run, but want to keep it until the migration is complete. We can use the skip method to do exactly that.

test.group("Controlling runs", () => {
  const testService = new TestService();

  test("successfully ping external service", async ({ assert }) => {
    const result = await testService.pingExternalServiceExample("success");
    assert.equal(result, "success");
  }).skip();
});

  

    

      Copied!
      

        

      
    
  
  
    
    
    
    
  

Now, when this runs, the test runner will skip actually running this test. Instead, it'll print it out in yellow within the terminal with a dot icon to note that it was skipped. Furthermore, we can designate a specific reason as to why we've skipped the test.

test.group("Controlling runs", () => {
  const testService = new TestService();

  test("successfully ping external service", async ({ assert }) => {
    const result = await testService.pingExternalServiceExample("success");
    assert.equal(result, "success");
  }).skip(true, "pending migration to new service");
});

  

    

      Copied!
      

        

      
    
  
  
    
    
    
    
  

Now, when the skip flag is truthy, the skip reason will print below the test's name, giving more context about it being skipped. The first argument can also be a variable or even an async function that returns a boolean. For example, maybe you need to ping a service to ensure it's available first.

test("ping external service and skip on failure", async ({ assert }) => {
  const result = await testService.pingExternalServiceExample("success");
  assert.equal(result, "success");
}).skip(async () => {
  const result = await testService.pingExternalServiceExample("failure");
  return result === "failure";
}, "External service is down");

  

    

      Copied!
      

        

      
    
  
  
    
    
    
    
  

Next, let's say we have a few tests we're debugging. Rather than run a whole suite or group, we can instead pin the individual tests we want to focus on using the pin method.

test("only run this pinned test", async ({ assert }) => {
  const result = await testService.pingExternalServiceExample("success");
  assert.equal(result, "success");
}).pin();

  

    

      Copied!
      

        

      
    
  
  
    
    
    
    
  

When any tests are pinned, the test runner will only run those tests. Additionally, it'll note that the test is pinned with a [PINNED] prefix.

Now, if you have tests running as part of your Continuous Integration (CI) pipeline, you will probably want to ensure no tests are pinned. That brings us to our first Japa plugin.

Japa plugins are defined within the bootstrap file via this plugins array right here. Japa comes with a disallowPinnedTests plugin we can use to conditionally disallow pinned tests.

The first two plugins already in here add Japa's assertion library and AdonisJS integration. We want to add disallowPinnedTests as a third plugin, and for now, let's hard-code this to true just to see what we get.

++import { disallowPinnedTests } from "@japa/runner/plugins";

export const plugins: Config["plugins"] = [
  assert(),
  pluginAdonisJS(app),
  disallowPinnedTests({
    dissallow: true,
  }),
];

  

    

      Copied!
      

        

      
    
  
  
    
    
    
    
  

You notice that when we attempt to run our tests with a pinned test, we now get an error noting pinned tests are not allowed. We can also run

node ace test --list-pinned

  

    

      Copied!
      

        

      
    
  
  
    
    
    
    
  

to see which tests are pinned.

Great! So, we can go ahead and switch this to a boolean dependent on the existence of a CI variable within our environment variables. Though, depending on your CI, you may need to alter this.

import { disallowPinnedTests } from "@japa/runner/plugins";

export const plugins: Config["plugins"] = [
  assert(),
  pluginAdonisJS(app),
  disallowPinnedTests({
    dissallow: !!process.env.CI,
  }),
];

  

    

      Copied!
      

        

      
    
  
  
    
    
    
    
  

Okay, so that we can continue working, let's go ahead and comment out this pinned test so we still have it for reference.

Lastly, we can assign tags to tests to give us a way to filter tests by their assigned tags. So let's take our original test here and add an actual test that will run for the success and failure flags of our ping.

test("successfully ping external service", async ({ assert }) => {
  const result = await testService.pingExternalServiceExample("success");
  assert.equal(result, "success");
}).tags(["external", "external-success"]);

test("fail to ping external service", async ({ assert }) => {
  const result = await testService.pingExternalServiceExample("failure");
  assert.equal(result, "failure");
}).tags(["external", "external-failure"]);

  

    

      Copied!
      

        

      
    
  
  
    
    
    
    
  

We'll use the tags method to give both a tag of external and then we'll add a variant for their expected status. If we now run and filter to the external tags

node ace test --tags="external"

  

    

      Copied!
      

        

      
    
  
  
    
    
    
    
  

Japa will only run the tests tagged with "external". That applies across all our suites and files as well. If we run again, focusing on "external-success"

node ace test --tags="external-success"

  

    

      Copied!
      

        

      
    
  
  
    
    
    
    
  

We see that only the one test with that tag is now run. We can specify multiple tags as well by just adding another tags argument

node ace test --tags="external-success" --tags="external-failure"

  

    

      Copied!