Skip to content

Latest commit

 

History

History
281 lines (249 loc) · 12 KB

validation_files.md

File metadata and controls

281 lines (249 loc) · 12 KB
Raw

Validation Files

Quilla operates through validation files, which are JSON files that define the environment and process for performing a validation in a declarative way. Each validation file requires the following:

  • One or more target browsers (Firefox, Chrome, Edge)
  • A starting URL path
  • A list of steps

Each step can be either a setup step, or a validation step. Setup steps do not ordinarily provide any reports if they are successful, and Quilla will abort the test if any steps cause an error, as it will assume that the error is not recoverable. In this case, a StepFailureReport will be produced, and the ReportSummary will indicate that there is a critical failure.

Validation steps will produce a ValidationReport, which can result in either a success or a failure. Although the hope is that every test you write will result in a success, Quilla will not abort if a validation fails, and will continue the test until it is finished or otherwise aborted. However, if in attempting to perform the validation some unrecoverable error occurs, Quilla will instead produce a StepFailureReport with the exception that occurred and abort the execution.

All Quilla integration tests are written as quilla tests and therefore can be referenced as examples when writing new Quilla tests.

The Quilla docs also include a cookbook to give examples for each of the available actions.

Supported actions

The table below summarizes all the supported actions, what they do, and what they require. The Validate and OutputValue actions are omitted from this table and are shown in later sections.

Action Description Target Parameters
Refresh Refreshes the current page None None
NavigateBack Navigates back to the last page None None
NavigateForward Navigates forward to the next page None None
NavigateTo Navigates to the target URL URL None
Click Clicks an element on the page XPath None
Clear Clears the text from an element on the page XPath None
Hover Hovers the cursor over the target element on the page XPath None
SendKeys Types the specified keys onto the target element on the page XPath data
WaitForExistence Waits until a target element exists on the page XPath timeoutInSeconds
WaitForVisibility Waits until a target element is visible on the page XPath timeoutInSeconds
SetBrowserSize Sets the browser size to a specific width & height None width, height

Output Values

Quilla is able to create 'outputs', which allows users to use values from the page within their own validations. This can come in handy when you need to react to the data on the page. A classic example is an application that displays some text that it requires the user to then type into a text field: instead of hardcoding the value for the UI tests (which may be impossible if the values are dynamically generated), a Quilla test could instead read the value from the XPath and use it in the same validation.

Below is a table displaying the supported output sources, the targets they support, and the parameters they require. Every single output requires an outputName parameter.

Source Description Target type Parameters
Literal Creates an output from the literal value of the target string None
XPathText Creates an output from the inner text value of the target XPath None
XPathProperty Creates an output from the value of the specified property name of the target XPath parameterName

Validations

Each validation is performed when the "action" is set to Validate. The kind of validation is provided by the "type" key, and by default Quilla currently supports the URL and XPath types.

Each validation type supports a series of states provided by the "state" key, and requires some form of target to specify what is being validated.

XPath Validation

Each XPath validation requires a "target" key with an XPath that describes an element on the page. The state of this XPath will then be validated, according to the valid states.

Some XPath validations also require parameters, such as the Attribute and Parameter-based validations of web elements.

The table below describes what the supported states are, and what they are validating

State Description Parameters
Exists The specified target exists on the page None
NotExists The specified target does not exist on the page None
Visible The specified target is visible on the page None
NotVisible The specified target is not visible on the page None
TextMatches Validates that the element text matches a regular expression pattern pattern
NotTextMatches Validates that the element text does not match a regular expression pattern pattern
HasProperty Ensures that the element has a property with a matching name name
NotHasProperty Ensures that the element does not have a property with a matching name name
HasAttribute Ensures that the element has an attribute with a matching name name
NotHasAttribute Ensures that the element does not have an attribute with a matching name name
PropertyHasValue Ensures that the property has a value matching the one specified name, value
NotPropertyHasValue Ensures that the property does not have a value matching the one specified name, value
AttributeHasValue Ensures that the attribute has a value matching the one specified name, value
NotAttributeHasValue Ensures that the attribute does not have a value matching the one specified name, value
VisualParity Checks previous baseline images pixel-by-pixel to ensure that sections have not changed baselineID

Note: The VisualParity state is discussed more at length in the visual parity section. For information on how to write storage plugins for VisualParity to use, check out the "Storage Plugins" section of the plugins docs.

URL Validation

Each URL validation requires a "target" key with a string. The precise target will be defined by the "state" property.

State Description
Equals The specified target is exactly equal to the URL of the page at the time the validation runs
NotEquals The specified target is not equal to the URL of the page at the time the validation runs
Contains The specified target is a substring of the URL of the page at the time the validation runs
NotContains The specified target is not a substring of the URL of the page at the time the validation runs

Examples

Searching Bing for puppies

Example written in 2021-06-16

{
    "definitions": {
        "HomePage": {
            "SearchTextBox": "//input[@id='sb_form_q']",
            "SearchButton": "//label[@for='sb_form_go']",
        },
        "ResultsPage": {
            "MainInfoCard": "//div[@class='lite-entcard-main']"
        }
    },
    "targetBrowsers": ["Firefox"],
    "path": "https://www.bing.com",
    "steps": [
        {
            "action": "Validate",
            "type": "URL",
            "state": "Contains",
            "target": "bing"
        },
        {
            "action": "Validate",
            "type": "XPath",
            "state": "Exists",
            "target": "${{ Definitions.HomePage.SearchTextBox }}"
        },
        {
            "action": "Validate",
            "type": "XPath",
            "state": "Exists",
            "target": "${{ Definitions.HomePage.SearchButton }}"
        },
        {
            "action": "SendKeys",
            "target": "${{ Definitions.HomePage.SearchTextBox }}",
            "parameters": {
                "data": "Puppies"
            }
        },
        {
            "action": "Click",
            "target": "${{ Definitions.HomePage.SearchButton }}"
        },
        {
            "action": "WaitForExistence",
            "target": "${{ Definitions.ResultsPage.MainInfoCard }}",
            "parameters": {
                "timeoutInSeconds": 10
            }
        },
        {
            "action": "Validate",
            "type": "URL",
            "state": "Contains",
            "target": "search?q=Puppies"
        }
    ]
}

Signing In to Github

Example written on 2021-06-24

{
    "definitions": {
        "Username": "${{ Environment.GITHUB_EXAMPLE_USER_USERNAME }}",
        "Password": "${{ Environment.GITHUB_EXAMPLE_USER_PASSWORD }}",
        "WelcomePage": {
            "SignInButton": "//div[@class='position-relative mr-3']/a"
        },
        "SignInPage": {
            "UsernameInputField": "//input[@id='login_field']",
            "PasswordInputField": "//input[@id='password']",
            "SubmitButton": "//input[@class='btn btn-primary btn-block']"
        },
        "HomePage": {
            "UserMenuIcon": "//div[@class='Header-item position-relative mr-0 d-none d-md-flex']",
            "YourProfileDropdown": "//details-menu/a[@href='/${{ Definitions.Username }}']"
        }
    },
    "targetBrowsers": ["Firefox"],
    "path": "https://github.com",
    "steps": [
        {
            "action": "Validate",
            "type": "URL",
            "state": "Contains",
            "target": "github"
        },
        {
            "action": "Validate",
            "type": "XPath",
            "state": "Exists",
            "target": "${{ Definitions.WelcomePage.SignInButton }}"
        },
        {
            "action": "Validate",
            "type": "XPath",
            "target": "${{ Definitions.WelcomePage.SignInButton }}",
            "state": "TextMatches",
            "parameters": {
                "pattern": "[Ss]ign[ -]?[Ii]n"
            }
        },
        {
            "action": "Click",
            "target": "${{ Definitions.WelcomePage.SignInButton }}"
        },
        {
            "action": "Validate",
            "type": "URL",
            "state": "Contains",
            "target": "/login"
        },
        {
            "action": "Clear",
            "target": "${{ Definitions.SignInPage.UsernameInputField }}"
        },
        {
            "action": "Clear",
            "target": "${{ Definitions.SignInPage.PasswordInputField }}"
        },
        {
            "action": "SendKeys",
            "target":"${{ Definitions.SignInPage.UsernameInputField }}",
            "parameters": {
                "data": "${{ Definitions.Username }}"
            }
        },
        {
            "action": "SendKeys",
            "target": "${{ Definitions.SignInPage.PasswordInputField }}",
            "parameters": {
                "data": "${{ Definitions.Password }}"
            }
        },
        {
            "action": "Validate",
            "type": "XPath",
            "target": "${{ Definitions.SignInPage.PasswordInputField }}",
            "state": "HasAttribute",
            "parameters": {
                "name": "id"
            }
        },
        {
            "action": "Validate",
            "type": "XPath",
            "target": "${{ Definitions.SignInPage.PasswordInputField }}",
            "state": "AttributeHasValue",
            "parameters": {
                "name": "id",
                "value": "password"
            }
        },
        {
            "action": "Click",
            "target": "${{ Definitions.SignInPage.SubmitButton }}"
        },
        {
            "action": "Validate",
            "type": "URL",
            "state": "Equals",
            "target": "https://github.com/"
        },
        {
            "action": "Click",
            "target": "${{ Definitions.HomePage.UserMenuIcon }}"
        },
        {
            "action": "Click",
            "target": "${{ Definitions.HomePage.YourProfileDropdown }}"
        },
        {
            "action": "Validate",
            "type": "URL",
            "state": "Contains",
            "target": "${{ Definitions.Username }}"
        }
    ]
}