Test Automation Framework

Work

Introduction

The goal of this test automation framework is to help the developers and QE engineers to simply/reduce the efforts on automation test cases. In most cases, the engineers should focus only on the test cases itself instead of worrying about these routine jobs:

1. Making a restful API call with headers/request body and validate the response.
2. Inserting/updating data to databases like MySQL/Cassandra.
3. Validating data inside the database.
4. Sending a Kafka message or validate a message in the Kafka broker.

This test automation framework is going to take care of the trivial things above without writing any java code in most cases.

How it works

1. Automation Framework Work Flow

howItWorks.png?raw=true未知大小

2. Sample Excel with test case

A sample Excel file with the test cases looks like the one below(click for large picture):

excel_sample.png?raw=true未知大小

Sample Http Request

A sample data in httpRequest excel column including API endpoint, http method, headers and body as below

{
  "endpoint": "${api.base.uri.policyengine}/policy-engine-app/rs/v1/decision",
  "method": "POST",
  "headers": {
    "accept": "application/json",
    "content-type": "application/json",
    "wm_consumer.tenant_id": "0",
    "wm_consumer.vertical_id": "2"
  },
  "body": {
    "tenant": 0,
    "vertical": 2,
    "context": {
      "flowType": "CASPR",
      "flowSubType": "BOOK_SLOT",
      "deviceType": "IOS",
      "customer": {
        "cid": "3ebws2dz-t6cc-9b0s-e6ad-fc3h03aiwqta",
        "isAssociate": false,
        "isGuestSignUp": false,
      },
      "deviceInfo": {
        "vtc": "PORqWxdC8AQPYYMKlInF",
        "deviceId": "123232323as",
        "ipV4Address": "10.12.34.56"
      },
      "slotDetail": {
        "slotId": "yjwv",
        "reservationId": "R109",
        "slotDate": 1589742000000,
        "zoneOffset": null,
        "accesspoint": "DELIVERY"
      }
    }
  }
}

Sample Http Response

A sample data in httpResponse excel column as the expected Http response including status code and response body:

{
  "status": 200,
  "body": {
    "decision": "ALLOW"
  }
}

Each row in the excel file is a single test case, it may or may not have dependencies with the other test cases based on what test cases you have in the excel.

The test automation framework will read this excel file line by line to run each test case including calling API and validating response, manipulating database, or sending/validating a message to/in Kafka: as the excel column names indicate the features.

For example: Once the framework gets the listed Http sample above, it will start to call the API with the specified endpoint, method, headers, and body, internally, after getting a response, it will auto compare actual data with expected Http response, raise the error if they are not same or pass the test if they match with each other.

We will go through the details on this excel file in the following sections.

Since the automation framework takes care of the most work, thus, the engineers can focus only on writing test cases in excel.

Getting Started

1. Create Automation Git Repo

Create your automation git repository with the structure like below:

├── pom.xml
├── src
│   ├── main
│   │   ├── java
│   │   └── resources
│   └── test
│       ├── java
│       │   └── com
│       │       └── walmart
│       │           └── policy
│       │               └── engine
│       │                   └── api
│       │                       ├── PolicyAdminAppTests.java
│       │                       └── PolicyEngineDecisionTests.java
│       └── resources
│           ├── PolicyAdminAppTests.csv
│           ├── PolicyAdminAppTests.xls
│           ├── PolicyEngineDecisionTests.csv
│           ├── PolicyEngineDecisionTests.xls
│           ├── application-local.yml
│           ├── application-qaint.yml
│           ├── log4j2.xml
│           └── testng.xml
└── tools
    ├── pre-commit
    └── xls2txt

The following sections explain the structure.

2. Required dependency in your pom

pom.xml

Add test automation framework dependency to your pom.xml

<dependencies>
    <dependency>
        <groupId>com.walmart.test.automation</groupId>
        <artifactId>test-engine-automation</artifactId>
        <version>1.0.1</version>
    </dependency>
</dependencies>

3. Test cases inside excel files

PolicyEngineDecisionTests.xls

The Excel file with test cases in it as we mentioned above.

You can view the raw excel file here, with this raw file, follow the template to create your own Excel test file, remember to put it into src/test/resources

4. An empty Java file

PolicyEngineDecisionTests.java

Each excel based test case file should have a Java class to trigger, so the file PolicyEngineDecisionTests.xls above should have a corresponding Java file with the naming convention like: PolicyEngineDecisionTests.java:

package com.walmart.policy.engine.api;
import com.walmart.test.automation.AutomationBaseTests;
public class PolicyEngineDecisionTests extends AutomationBaseTests {
    // no java code is required, only excel test data in PolicyEngineDecisionTests.xls
}

This Java file could be empty like above(focus on test in Excel) if you don't have any special case to handle, but it should extend AutomationBaseTests and be put into src/test/java

You can view the raw Java file here.

5. Auto-generated CSV text file

PolicyEngineDecisionTests.csv

Since Excel file is a binary file which is unable to diff as a text file, this CSV file as text file is auto-converted from the corresponding Xsl and has equivalent content while the users can keep using Excel with more feature supported like color/format.

Copy tools folder into the root of your git repo and copy tools/pre-commit into your GIT_REPO/.git/hooks/, after that, if only there are changes in the excel files, the corresponding CSV files will be auto-generated when committing your changes.

6. Yaml application configuration files

application-qaint.yml

This YAML file is where you have all the application configurations like API base Uri, Cassandra/Kafka/Mysql connection info, including the Jira integration(explain later). Just replace the values of Cassandra/Kafka/Mysql below with yours. You can also define your own configurations here. Not only these configurations will be used internally in the framework, but also you can use them in your java test case or even in the Excel file.

For example, ${api.base.uri.policyengine} below is referred by the Http sample above, Line 2.

api.base.uri:
  policyengine: https://ws.qaint.policyengine.walmart.com
  policyadmin: https://ws.qaint.policyadmin.walmart.com
  electrode: https://policy.governance.qaint.walmart.com
cassandra:
  contact-points: cass-491076148-2-515233908.dev.policy-engine-dev.ms-df-cassandra.stg-az-westus-2.prod.us.walmart.net
  local-data-center: westus
  username: app
  password: xxxxxx
  keyspace: policy_engine
  port: 9042
  schema-action: NONE
kafka:
  bootstrap.servers: kafka-358735030-1-886374088.prod-southcentralus-az.kafka-shared-non-prod.ms-df-messaging.prod-az-southcentralus-1.prod.us.walmart.net:9092
  key.serializer: org.apache.kafka.common.serialization.StringSerializer
  value.serializer: org.apache.kafka.common.serialization.StringSerializer
  key.deserializer: org.apache.kafka.common.serialization.StringDeserializer
  value.deserializer: org.apache.kafka.common.serialization.StringDeserializer
mysql.datasource:
  jdbc-url: jdbc:mysql://db.stg.policy-engine-qaint-stg.ms-df-cloudrdbms.glb.us.walmart.net:3306/policy_engine
  username: policy_engine
  password: xxxxxx
  driver-class-name: com.mysql.cj.jdbc.Driver
  max-pool-size: 20
  min-idel: 10
  pool-name: HikariReadWritePool
jira:
  enable: false
  createForEach: false
  groupPriority: P2
  defaultPriority: P3
  project: CEPPE
  issuetype: Task
  reporter: j0z05z1
  assignee: j0z05z1
  summary: Test case(s) failed in automation suite
  api:
    url: https://jira.walmart.com/rest/api/2/issue
    authentication: GuRwNsXaTdygO0pG4Nw1JVFUMkyr73t5rZbSL+Zzq9rdij6UjgwAkI+FV8bian0Lsk0ekhQagswvdGGOaO

It's also possible to use these configurations in a test Java file like below:

package com.walmart.policy.engine.api;
public class PolicyEngineDecisionTests extends AutomationBaseTests {
    @Autowired
    private Environment environment;
    @Test
    public void testSample(ITestContext iTestContext) {
        // Resolve the placeholders by the values from application-qaint.yml or excel file, even support list
        System.out.println(this.environment.resolvePlaceholders("hello: ${api.base.uri.policyengine} world"));
    }
}

The file naming convention indicates the environment this configuration is working for:

application-qaint.yml is for qaint environment
application-local.yml is for the local box.

By default, it will use application-qaint.yml as the default Yaml file to run test cases unless you specify system property like below.

mvn test -Dspring.profiles.active=local

this system property also works when you run test cases in IDE like Intellij.

application-local.yml

api.base.uri:
  policyengine: http://localhost:8080
  policyadmin: http://localhost:8080
  electrode: http://localhost:8080

Inside application-local.yml, define only the key/values you want to overwrite like above, in this case, the framework will switch the endpoint to your local but it will still use the rest of info like Kafka/Mysql/Cassandra defined in default application-qaint.yml.

7. Jira Integration

With the Jira configuration in Yaml configuration below, the automation framework will automatically create a Jira ticket for any test failures with a specified summary and detailed error information in the Jira description.

jira:
  enable: false
  createForEach: false
  groupPriority: P2
  defaultPriority: P3
  project: CEPPE
  issuetype: Task
  reporter: j0z05z1
  assignee: j0z05z1
  summary: Test case(s) failed in automation suite
  api:
    url: https://jira.walmart.com/rest/api/2/issue
    authentication: GuRwNsXaTdygO0pG4Nw1JVFUMkyr73t5rZbSL+Zzq9rdij6UjgwAkI+FV8bian0Lsk0ekhQagswvdGGOaO

The values above defined in the YAML file are default values, they can be reset or update when starting a test like below:

mvn test -Djira.enable=true -Djira.createForEach=true

You can also set individual Jira priority for each test case when it fails and jira.enable=true and jira.createForEach=true. Check "priority" field in testConfig.

The sample Jira tickets created by automation:
One Jira per test failure:
https://jira.walmart.com/browse/CEPPE-357
A single Jira contains multiple test failures:
https://jira.walmart.com/browse/CEPPE-352

9. Your own Java test case

If you think the automation doesn't cover all the scenarios you want, you can also add more Java test cases like the below:

public class PolicyEngineDecisionTests extends AutomationBaseTests {
    @Autowired
    private Environment environment;
    @Test
    public void testTestCaseId(ITestContext iTestContext) {
        TestData testData = (TestData) iTestContext.getAttribute("Test_1_1");
        System.out.println(testData.getActualHttpResponse());
        // Resolve the placeholders by the values from application-qaint.yml or excel file, even support list
        System.out.println(this.environment.resolvePlaceholders("hello: ${Test_1_1.httpResponse.body.decision}"));
        System.out.println(this.environment.resolvePlaceholders("hello: ${Test_2_1.vars.cid}"));
        TestData testData_2_1 = (TestData) iTestContext.getAttribute("Test_2_1");
        System.out.println(testData_2_1.getHttpRequest());
    }
}

Not only the framework will run the test cases inside an excel/CSV file, but also it will run your java test case above. Line 8 returns all the data you want and line 11 explains how to resolve a string with a placeholder. Check details in PolicyEngineDecisionTests.java.

10. log4j2.xml and testng.xml

Nothing special for them, just use them as usual, don't forget to put test Java files into testng.xml to run even if you don't have any additional test case in it since these Java files will read corresponding excel test files to run the test cases.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite guice-stage="DEVELOPMENT" name="Policy Tests">
  <test name="Policy Tests">
    <classes>
      <class name="com.walmart.policy.engine.api.PolicyEngineDecisionTests"/>
      <class name="com.walmart.policy.engine.api.PolicyAdminAppTests"/>
    </classes>
  </test>
</suite>

Run with

$ mvn test

11. Sample Git Repository

This section explains everything on the structure, you can always copy/paste samples from Policy Automation to create your own.

Usage: Excel test case file

Before explaining the meaning of each column above, you can also view those two sample excel files as the sample to understand while reading the following explanation.

PolicyEngineDecisionTests.xls
PolicyAdminAppTests.xls

1. Column testCaseId (required)

You can put any unique string here as a test case id. The sample Test_1_1 is suggested since the first number 1 indicates groupId, the second 1 means the first case in group 1. This case id should be unique since java codes or the other excel cells may refer to this test case id for data, explain this later.

2. Column testCaseDescription (Optional)

Just put any description here for the purpose of the test case. It will be in the log if the assertion fails.

3. Column vars (Optional)

A column you can define a JSON here as the global variable which will be used in java codes or the other excel cells.

For example, you can have this JSON below in the vars cell of the row of Test_2_1

{
  "cid": "7m5kemzz-xb7u-4skk-8jbu-bzb6uttrjori",
  "vtc": "K4mNGN445oE4bxELseLT"
}

And then, in any cell of the excel, you can refer to this variable. For example, in the httpRequest cell you can define a request body like below with ${Test_2_1.vars.cid} as the placeholder.

{
"body": {
  "context": {
    "customer": {
      "cid": "${Test_2_1.vars.cid}",
    },
    "deviceInfo": {
      "vtc": "${Test_2_1.vars.vtc}",
    },
    "slotDetail": {
      "slotId": "tybe",
    }
  }
}
}

Also ${Test_2_1.vars.cid} here indicates how we use testCaseId.

4. Column kafkaRequest (Optional)

Any JSON data in this column will be sent to the specified Kafka topic by the automation framework.

{
  "topic": "policy-caspr-feed",
  "key": "[slotId-1, slotId2]",
  "data": {
    "header": {
      "action": "RESERVATION_CANCEL"
    },
    "payload": {
      "reservationId": [
        "slotId-1",
        "slotId-2"
      ]
    }
  }
}

The JSON should follow the schema like below:

Kafka connection info should be pre-configured in Yaml configuration.

5. Column sqlUpsert (Optional)

Any JSON data in this column will be inserted or updated to the specified database by the automation framework.

{
  "cassandra": [
    "INSERT INTO velocity_snapshot (tenant_id, vel_var, vel_key, create_date, ref_id)
    VALUES (0, 'COUNT_ORDERS_BY_CID', 'WZV73LJL-0V0Y-GNY1-LRJ3-1E1A6SJK90GY', toTimeStamp(now()), 'order_509')",
    "INSERT INTO velocity_snapshot (tenant_id, vel_var, vel_key, create_date, ref_id)
    VALUES (0, 'COUNT_ORDERS_BY_VTC', 'K8LGU0TN1IA9GULHNISH', toTimeStamp(now()), 'order_509')"
  ],
  "mysql": 
    "update policies set description = 'Customer auth failure more than limit(s)'
    where id = '1458fe0b-28c7-4831-8ea4-e249a138d0bc'"
}

The JSON should follow the schema like below:

DB connection info should be pre-configured in Yaml configuration.

6. Column httpRequest (Optional)

The framework will call the API with the specified endpoint/method/headers/body in the httpRequest column.

{
  "endpoint": "${api.base.uri.policyengine}/policy-engine-app/rs/v1/decision",
  "method": "POST",
  "headers": {
    "accept": "application/json",
    "content-type": "application/json",
    "wm_consumer.tenant_id": "0",
    "wm_consumer.vertical_id": "2"
  },
  "body": {
    "tenant": 0,
    "vertical": 2,
    "context": {
      "flowType": "CASPR",
      "flowSubType": "BOOK_SLOT",
      "deviceType": "IOS",
      "customer": {
        "cid": "3ebws2dz-t6cc-9b0s-e6ad-fc3h03aiwqta",
        "isAssociate": false,
        "isGuestSignUp": false,
      },
      "deviceInfo": {
        "vtc": "PORqWxdC8AQPYYMKlInF",
        "deviceId": "123232323as",
        "ipV4Address": "10.12.34.56"
      },
      "slotDetail": {
        "slotId": "yjwv",
        "reservationId": "R109",
        "slotDate": 1589742000000,
        "zoneOffset": null,
        "accesspoint": "DELIVERY"
      }
    }
  }
}

The cell data should follow the schema like below:

7. Column httpResponse (Optional)

Put any expected response data in this column, after calling API in httpRequest column, the framework will compare the actual response with the expected ones recursively. If the response is a complex JSON, you don't have to list all the fields of the response, you can just pick up the ones you are interested in to validate.

{
  "status": 200,
  "body": {
    "decision": "ALLOW"
  }
}

If your actual response is not 200 or the decision is "DECLINE", your test case will then fail.

The cell data should follow the schema like below:

8. Column sqlValidation (Optional)

Put SQL query and expected result set in this column, so that the framework will query DB and compare the actual result set with the expected ones.

{
  "cassandra": [
    {
      "query": "SELECT count(1) FROM velocity_snapshot where vel_var = 'COUNT_ORDERS_BY_CID' AND vel_key = 'WZV73LJL-0V0Y-GNY1-LRJ3-1E1A6SJK90GY' ALLOW FILTERING",
      "result": [{"count": 1}]
    },
    {
      "query": "SELECT count(1) FROM velocity_snapshot where vel_var = 'COUNT_ORDERS_BY_VTC' AND vel_key = 'K8LGU0TN1IA9GULHNISH' ALLOW FILTERING",
      "result": [{"count": 1}]
    }
  ],
  "mysql":
    {
      "query": "select state, approval_level from policies where id = '${Test_1_1.httpResponse.body.id}'",
      "result": {"state": "PREVIEW", "approval_level": null}
    }
}

The JSON should follow the schema like below:

DB connection info should be pre-configured in Yaml configuration.

9. Column kafkaValidation (Optional)

The automation framework will start a Kafka listener internally if this column is populated, and once you trigger a Kafka message from somewhere like kafkaRequest or httpRequest, this listener will expect to receive a kafka message you specified as below, if the listener fails to receive any messages in 45 seconds or the actual message received is not expected after comparing with the expected data below, an assertion error will be raised.

{
  "topic": "policy-caspr-feed",
  "key": "[slotId-1, slotId2]",
  "data": {
    "header": {
      "action": "RESERVATION_CANCEL"
    },
    "payload": {
      "reservationId": [
        "slotId-1",
        "slotId-2"
      ]
    }
  }
}

The JSON should follow the schema like below:

Kafka connection info should be pre-configured in Yaml configuration.

10. Column testConfig (Required)

This testConfig column is introduced to avoid duplicate test data everywhere in the excel. With the proper overwrite settings on all(default) columns or specified columns, you can focus only on the data which is changing but keep the others as before without even mentioning them.

a) new strategy on default columns

For example, you have the first row of excel as below:

testCaseId

Test_1_1

httpRequest

{
  "endpoint": "${api.base.uri.policyengine}/policy-engine-app/rs/v1/decision",
  "method": "POST",
  "headers": {
    "content-type": "application/json"
  },
  "body": {
    "tenant": 0,
    "vertical": 2,
    "context": {
      "flowType": "CASPR",
      "customer": {
        "cid": "3ebws2dz-t6cc-9b0s-e6ad-fc3h03aiwqta",
        "isAssociate": false,
        "isGuestSignUp": false
      },
      "deviceInfo": {
        "vtc": "PORqWxdC8AQPYYMKlInF"
      },
      "slotDetail": {
        "slotId": "yjwv",
        "slotDate": 1589742000000,
        "accesspoint": "DELIVERY"
      }
    }
  }
}

httpResponse

{
  "status": 200,
  "body": {
    "decision": "ALLOW"
  }
}

testConfig

{
  "overwrite": {
    "default": "new"
  }
}

The "default": "new" above indicates whatever existing data in all(default) columns are new/base/template/original data. The framework will use them directly without any changes for the current case Test_1_1, but all the following rows/cases will generate new case based on the current data in this case until a new rows/cases have the same "default": "new" showing up in its testConfig column

b) merge strategy on default columns

For the second row/test case in the excel what if the changes are on only some fields of httpRequest but not all and we are expecting the same httpResponse. Since we don't want to repeat duplicate test data in excel you can have the second row as below:

testCaseId

Test_1_2

httpRequest

{
"body": {
  "context": {
    "customer": {
      "cid": "${Test_2_1.vars.cid}",
    },
    "deviceInfo": {
      "vtc": "${Test_2_1.vars.vtc}",
    },
    "slotDetail": {
      "slotId": "tybe",
    }
  }
}
}

httpResponse

leave blank

testConfig

{
  "overwrite": {
    "default": "merge"
  }
}

If we specify the merge strategy above, the framework will update the changes to the base data we have in the previous row Test_1_1 to get a new request, but keep the other fields as before, also the default key above indicates merge will happen on all columns including httpRequest and httpResponse.

So in this way you don't have to repeat duplicate fields in request/response if they don't have changes.

In this case, you will have a new context.customer.cid, context.deviceInfo.vtc and context.slotDetail.slotId before sending reqest for Test_1_2, but the other info like endpoint, method, headers body.context.flowType remains same as Test_1_1, and httpResonse is leaving blank since we will reuse the base/original data in Test_1_1 which is

{
  "status": 200,
  "body": {
    "decision": "ALLOW"
  }
}

This mechanism will save efforts on duplicate data and update changes only in the base/original/template case to reflect them to all the 'merge' strategy cases.

Check this sample file PolicyEngineDecisionTests.xls to get more senses.

c) remove insert strategy on specific columns

Besides the new, merge strategy, you can also have the other strategy like remove or insert for more scenarios, and also you can specify the strategy applies only on some column instead of all the columns.

For example, we have the third row for the 3rd case which is missing the context.customer.cid in httpRequest and we are expecting a bad request 400 with detailed error messages in httpResponse.

testCaseId

Test_1_3

httpRequest

{
"body": {
  "context": {
    "customer": {
      "cid": "3ebws2dz-t6cc-9b0s-e6ad-fc3h03aiwqta",
    }
  }
}
}

httpResponse

{
  "status": 400,
  "body": {
    "message": "API Field validation failed",
    "fieldErrors": [
      {
        "field": "context.customer.cid",
        "message": "must not be blank"
      }
    ]
  }
}

testConfig

{
  "overwrite": {
    "default": "merge",
    "httpRequest": "remove",
    "httpResponse": "insert"
  }
}

"httpRequest": "remove" means we need to remove/delete context.customer.cid from the base request in Test_1_1 which has the new strategy as the base/template data.

httpResponse": "insert" means the httpResponse is totally different from the one in Test_1_1 so I need to ignore the base response without any merge anymore but use the entire current httpResponse instead. The difference between new and insert strategy is insert works only for the current case and it will be ignored by the following test cases, while new strategy will reset the current data as base data so that the following test cases will respect the new base data.

d) "priority" field in testCofnig

{
  "overwrite": {
    "default": "new"
  },
  "priority": "P2"
}

"priority" field can be set with the values like "P1/P2/P3/P4/None", this field value will be used to set individual Jira priority for each test case when test failure happens on the test case itself. If you don't set a value on this field, the Jira.defaultPriority defined inside the YAML file will be set. See details in Jira Integration.

11. The ReExp and placeholder in Excel

RegExp

In some cases, you want to validate a random ID in httpResponse after calling create policy API for example. Although you can't assume the actual ID in your test case, if you know the format of the ID you can do something like below:

testCaseId

Test_1_1

httpResponse column

{
  "status": 200,
  "body": {
    "id": "#{[a-z0-9]{8}-[a-z0-9]{4}-[a-z0-9]{4}-[a-z0-9]{4}-[a-z0-9]{12}}",
    ...
  }
}

This form should be respected to use RegExp:

"#{regular expression}"

So that if the actual value matches this regular expression you defined, it will pass otherwise fail.

Placeholder

1. Refer to any previous cell data in excel

After creating a policy, the next case is going to reuse this new policy id to call update policy API, so you should know the policy id coming from the previous case, to do this, check this sample

testCaseId

Test_1_2

httpRequest column

{
  "endpoint": "${api.base.uri.policyadmin}/policy-admin-app/rs/v1/policy/${Test_1_1.httpResponse.body.id}",
  "method": "PUT",
  ...
}

${Test_1_1.httpResponse.body.id} means in Test_1_2, we will use the policy id returning from the previous create policy API.

This placeholder works for any columns and json data inside recursively:

${testCaseId.columnName.JsonField.subJsonField}

2. Refer the key/value from Yaml configuration as placeholder in excel

"endpoint": "${api.base.uri.policyadmin}/policy-admin-app/rs/v1/policy"

${api.base.uri.policyadmin} is coming from src/test/resources/application-qaint.yml or application-local.yml based on your predefined env.