Gherkin scenarios
The acceptance scenarios of features/, one section per product. Run runs a whole feature or one scenario against the stack mise run up started; the run also lands in Test results.
The Driven by filter keeps the features of one tool. Run feature runs every scenario of a feature, Run one scenario. The Gherkin runner starts pytest-bdd against the running stack, the verdict shows above with the pytest output, and the run is filed in Test results like a CI job. The stack down, the scenarios skip and say how to start it.
live products write for real when their scenarios run (Redis visits, orders, accounts, emails); static ones only read.
Each feature says what drives its scenarios, a badge beside its name from its @tool- tag: pytest (pytest-bdd steps calling the stack over HTTP or the bucket), playwright (steps clicking the storefront page in a real Chromium, written from a recorded journey), schemathesis (requests generated from a contract), k6 (a load played by k6, its summary read by the steps). Every one still runs through pytest-bdd, so the console runs and files them alike.
Customer welcome live
Greets a returning customer by first name, counts the visits and shows the bank balance.
🛍️ Try it by hand in Live experimentation: the same product the scenarios below prove, opened alone on the storefront.
A customer typing their username is greeted by first name, told which visit this is, and shown the balance the partner bank holds for them. The greeting must never invent a balance nor greet someone the company does not serve.
Goal the happy path through the three neighbours at once: the database gives the first name, the partner bank the balance, formatted in euros from integer cents.
- When "ada" asks for a greeting
- Then the greeting starts with "Hello Ada!"
- And the greeting shows a balance in EUR
Goal the visit counter in Redis really grows by one per greeting, so a returning customer is recognised.
- When "grace" asks for a greeting
- And "grace" asks for a greeting again
- Then the second visit number is one more than the first
Goal a 404 of the partner bank means no account, not an error: the greeting goes on and invents no amount.
- When "zoe" asks for a greeting
- Then the greeting starts with "Hello Zoe!"
- And the greeting shows no balance
Goal a customer the company no longer serves gets a 404, before any call to the bank or any visit counted.
- When "dennis" asks for a greeting
- Then the storefront answers 404
Ten customers greet themselves for half a minute, each call going through the storefront, the API, the database, the partner bank and Redis. k6 plays the load, the scenario reads its summary: the same numbers a load test job files in Test results.
Goal the greeting stays fast and reliable with several customers at once, not only for one tester clicking: under 1% of failed calls, 95% of the greetings in less than 500 ms.
- When k6 plays "customer-welcome-baseline.js" against the storefront
- Then less than 1 percent of the calls fail
- And 95 percent of the greetings answer within 500 ms
The journey a customer clicks, recorded and written in the template of a recorded scenario: who plays, from which data, what is seen. Playwright drives a real Chromium on the storefront of the running stack.
Goal what the customer really sees, not only what the API answers: the page greets them by first name and shows the balance the bank holds.
- Given "ada", a customer of the company who can see their bank balance
- And the partner bank holds an account for "ada"
- When "ada" greets themselves on the storefront page
- Then the page greets "Ada" with a balance in EUR
Checkout live
Places an order and follows it while stock is reserved, the invoice issued and the payment captured.
🛍️ Try it by hand in Live experimentation: the same product the scenarios below prove, opened alone on the storefront.
The checkout journey a customer clicks, recorded and written in the template of a recorded scenario. Playwright fills the order form of the storefront and watches the workflow steps the page draws as the worker runs them.
Goal the page follows the asynchronous workflow to the end on its own, the customer never reloads: three steps turn done, the order reads paid.
- Given "ada", a customer of the company who can order
- And the stock of "BK-0004" holds at least 1 item
- When "ada" orders 1 "BK-0004" on the storefront page
- Then the page shows every step done and the order paid
An order is accepted at once, then a worker reserves the stock, issues the invoice and captures the payment, one event per step. An order the stock cannot serve is cancelled before any payment. Each run of the first scenario takes one item of the stock of the running stack.
Goal the asynchronous workflow runs to the end: accepted at once with a 202, then stock, invoice and payment each done by the worker, in order.
- When "ada" orders 1 "BK-0004"
- Then the order is accepted at once
- And the workflow ends "completed" with the order "paid"
- And the step "capture_payment" is "done"
Goal a failing step stops the workflow: the order is cancelled and the customer is never charged.
- When "grace" orders 1 "EL-0001"
- Then the order is accepted at once
- And the workflow ends "failed" with the order "cancelled"
- And the step "capture_payment" is "skipped"
Goal the API refuses a closed customer itself, before any event reaches the queue.
- When "dennis" orders 1 "BK-0004"
- Then the storefront answers 404
Account access live
Creates an account, signs in, resets a password, with every email landing in the QA mailbox.
🛍️ Try it by hand in Live experimentation: the same product the scenarios below prove, opened alone on the storefront.
Every email of the tenant lands in the QA mailbox, each test account being a plus tag of one address. A scenario reads the email of its own address, opens the link it carries and goes on, as a customer would.
Background
- Given a new address of the QA mailbox
Goal creating an account really sends an email, readable by a test like a customer reads an inbox.
- When the customer creates an account with it
- Then the mailbox receives "choose your password" for that address
- And the email holds a password link
Goal the whole onboarding, from the email link to the first sign-in.
- When the customer creates an account with it
- And the customer sets "Foundation-2026" from the link of "choose your password"
- Then the customer signs in with "Foundation-2026"
Goal a leaked or replayed link cannot change the password again.
- When the customer creates an account with it
- And the customer sets "Foundation-2026" from the link of "choose your password"
- Then setting "Another-2026" from the same link is refused
Goal a customer who forgot the password can get back in through the mailbox.
- When the customer creates an account with it
- And the customer sets "Foundation-2026" from the link of "choose your password"
- And the customer asks to reset the password
- Then the mailbox receives "Reset your" for that address
Goal brute force is stopped at ten tries, and the real owner is told by email.
- When the customer creates an account with it
- And the customer sets "Foundation-2026" from the link of "choose your password"
- And the customer signs in 10 times with "Wrong-password-1"
- Then the mailbox receives "blocked" for that address
- And the customer cannot sign in with "Foundation-2026"
The sign-in a customer clicks, recorded and written in the template of a recorded scenario. The account is test data like any other: the scenario creates its own, with a password, so it never depends on the account of whoever recorded it.
Goal a customer with a password reaches their account from the page, the form sending what the tenant expects and showing who is signed in.
- Given "a new customer", a customer account who can sign in with a password
- And a new account of the QA mailbox whose password is "Foundation-2026"
- When the customer signs in on the storefront page with "Foundation-2026"
- Then the page says the customer is signed in
The simulated Auth0 tenant signs a customer in with their email and password. It never says whether an account exists, tells a blocked customer to check their email, asks a new customer to set a password first, and asks a customer with a second factor for the code of their authenticator app.
Goal the plain sign-in works for a customer with an email and a password.
- When "qa-mailbox+ada@mycompany.test" signs in with "Foundation-2026"
- Then the sign-in is accepted
Goal the tenant never tells an attacker which emails have an account.
- When "qa-mailbox+nobody@mycompany.test" signs in with "Foundation-2026"
- Then the tenant answers 401 "Wrong email or password"
Goal a blocked account says why and where to go next, with its own status code.
- When "qa-mailbox+grace@mycompany.test" signs in with "Ultrabook-2026"
- Then the tenant answers 423 "Account blocked, check your email"
Goal an account created without a password cannot be entered until its link is used.
- When "qa-mailbox+zoe@mycompany.test" signs in with "Anything-2026"
- Then the tenant answers 403 "No password yet, open the link of your email"
Goal the password alone is not enough with a second factor: the tenant asks for the code, then accepts it.
- When "qa-mailbox+lin@mycompany.test" signs in with "Authenticator-2026"
- Then the tenant answers 403 "mfa_required"
- When the customer sends the code of their authenticator app
- Then the sign-in is accepted
Partner contracts static
Checks that a third party still answers what the contract we copied promises, with requests generated from that contract.
The bank is not ours. We copied its contract, Microcks replays it to our tests and the greeting trusts it: if the bank changes what it answers, every test stays green on the replay while the real greeting breaks. Schemathesis reads the copy, generates requests from it and checks each answer of the real partner against it. Red here means our copy is no longer true.
Goal usernames of every shape the contract allows (empty, accented, very long) never crash the bank, and every answer is one the contract describes, the "no account" 404 included.
- Given the contract of the Bank Partner
- When schemathesis sends 50 requests generated from it
- Then no answer is a server error
- And every status code is documented by the contract
- And every body matches the schema of its status code
Goal the answers the greeting really uses, a known customer's balance, keep the fields and types of the contract, the amount in integer cents.
- Given the contract of the Bank Partner
- When schemathesis asks the balance of "<username>"
- Then the partner answers 200
- And every body matches the schema of its status code
Examples, one run per row
| username |
|---|
| ada |
| grace |
Goal the bank only answers its clients: without the long-lived key it issued us (kept in Vault), the answer is a 401, documented by the contract.
- Given the contract of the Bank Partner
- When schemathesis asks the balance of "ada" without the key
- Then the partner answers 401
- And every body matches the schema of its status code
Bucket checks static
The same generic checks on every team bucket, each bucket bringing its own contract (names, format, freshness, manifest).
Teams exchange files through buckets: finance exports its invoices every day, the catalog publishes its feeds. The same generic checks run on every bucket, each bucket bringing its own expectations as a row of examples: where the files are, how they are named, which format they hold, how fresh they must be.
Goal the files another team drops are where, named, shaped and as fresh as agreed, before our jobs read them.
- Given the files of bucket "<bucket>" under "<prefix>"
- Then there is at least one file
- And every file name matches "<pattern>"
- And no file is empty
- And every file reads as <format>
- And the newest file is less than <hours> hours old
Examples, one run per row
| bucket | prefix | pattern | format | hours |
|---|---|---|---|---|
| finance-exports | daily/ | ^daily/[0-9]{4}-[0-9]{2}-[0-9]{2}[.]csv$ | csv | 720 |
| catalog-feeds | feeds/ | ^feeds/[a-z]+[.]json$ | json | 720 |
Goal the manifest and the files agree both ways: nothing listed is missing, nothing present is unlisted.
- Given the files of bucket "<bucket>" under "<prefix>"
- When the manifest "manifest.json" of the bucket is read
- Then every file it lists exists
- And every file under the prefix is listed
Examples, one run per row
| bucket | prefix |
|---|---|
| finance-exports | daily/ |
| catalog-feeds | feeds/ |
Quality Console static
The back office of the quality platform, its tools and the company data, read-only.
The back office lists what the company runs and lets anyone browse its data without the right to change it. A name nobody listed never reaches a query.
Goal the catalog the tests' metadata relies on is the one the console shows.
- When I open the console page "/catalog"
- Then the page shows "Storefront"
- And the page shows "Quality platform"
Goal anyone can read the company data without a database client.
- When I open the console page "/databases/mycompany"
- Then the page shows "30 tables"
Goal a name from the URL never reaches a SQL statement unless MySQL listed it.
- When I open the console page "/databases/mycompany/tables/salaries"
- Then the console answers 404