Agentic AI Test Engineer — Zero-to-Hero ...

Agentic AI Test Engineer — Zero-to-Hero Project Runbook

Nov 11, 2026
44

Organized complete Agentic AI Test Engineer — Zero-to-Hero guide around the actual workflow we built on your MacBook Air M2: local Ollama installation, Python application, AI-generated test cases, JSON validation, Robot Framework, Flask mock API, failure analysis, GitLab Runner, CI pipeline, and the errors we encountered.

The guide will include copy-and-paste commands, complete scripts for the core components, troubleshooting steps, and verification commands. One important distinction throughout: the AI proposes tests, but the API requirements determine the correct expected results.

Environment: MacBook Air M2, macOS, Python, Ollama, Flask, Robot Framework, GitLab CI and a self-hosted GitLab Runner.

Project location: ~/agentic-test-engineer

Local AI model: llama3.2:3b

Purpose: Build an AI-assisted test engineering system that converts requirements into test cases, validates them, generates Robot Framework automation, executes tests, analyzes failures, and publishes results through GitLab CI.


1. What we built

1.1 Architecture

Requirements
input/subscriber_requirements.md
          |
          v
Python application
app/run_local_ai.py
          |
          v
Ollama local LLM
127.0.0.1:11434
llama3.2:3b
          |
          v
AI-generated JSON
generated/test_cases.json
          |
          v
JSON Schema validation
app/validate_test_cases.py
          |
          v
Deterministic test generator
app/generate_robot_tests.py
          |
          v
Robot Framework suite
generated/subscriber_api.robot
          |
          v
Flask mock subscriber API
127.0.0.1:5001
          |
          v
Robot Framework execution
          |
          +--> results/output.xml
          +--> results/log.html
          +--> results/report.html
          |
          v
Failure analysis
app/analyze_robot_results.py
          |
          v
generated/failure_analysis.json
          |
          v
GitLab CI artifacts

1.2 What makes this agentic?

The project uses AI to propose test cases and connects that output to validation, code generation, test execution, and result analysis.

Strictly speaking, the initial implementation is an AI-assisted test engineering pipeline. To call it a fully autonomous agent, it would need controlled tool use, decision-making, result evaluation, and a bounded retry or investigation loop.

The key safety principle is:

  • Requirements define expected behavior.

  • The AI proposes test cases.

  • Validators reject invalid or unsupported proposals.

  • Deterministic templates generate executable tests.

  • Robot Framework provides execution evidence.

  • AI may suggest explanations, but must not invent confirmed root causes.


2. Install the required tools

Open Terminal on your MacBook.

2.1 Check your system

uname -m
sw_vers
python3 --version
git --version
brew --version

On your Apple Silicon Mac, uname -m should normally return:

arm64

Install missing dependencies with Homebrew:

brew install python git

2.2 Install Ollama

Install the Ollama macOS application from its official website:

https://ollama.com/

Open the application and verify the installation:

ollama --version
ollama list
curl -sS http://127.0.0.1:11434/api/tags

The API request should return JSON describing the locally available models.

If the connection is refused, open/start the Ollama application and try again.

2.3 Download the model

ollama pull llama3.2:3b
ollama show llama3.2:3b
ollama list

Run an interactive test:

ollama run llama3.2:3b

Enter a simple prompt. Exit the interactive session when finished.

How your MacBook was used

The model runs locally through Ollama. Your Python code communicates with the Ollama service at:

http://127.0.0.1:11434

This means the model inference uses your MacBook's CPU/GPU resources and local memory, rather than requiring a hosted LLM API for each prompt.


3. Create the project from scratch

mkdir -p ~/agentic-test-engineer
cd ~/agentic-test-engineer

mkdir -p app input generated results
touch app/__init__.py

3.1 Create a Python virtual environment

python3 -m venv .venv
source .venv/bin/activate

python --version
which python

The Python path should point inside:

~/agentic-test-engineer/.venv/

Whenever you open a new Terminal for local work, activate the environment again:

cd ~/agentic-test-engineer
source .venv/bin/activate

3.2 Create requirements.txt

ollama
robotframework
robotframework-requests
requests
jsonschema
flask

Install the dependencies:

python -m pip install --upgrade pip
python -m pip install -r requirements.txt

Verify:

python -c "import ollama, robot, requests, jsonschema, flask; print('All dependencies imported successfully')"

3.3 Recommended directory structure

agentic-test-engineer/
├── .gitignore
├── .gitlab-ci.yml
├── requirements.txt
├── app/
│   ├── __init__.py
│   ├── test_case_schema.py
│   ├── ollama_generator.py
│   ├── run_local_ai.py
│   ├── validate_test_cases.py
│   ├── mock_subscriber_api.py
│   ├── generate_robot_tests.py
│   ├── classify_failures.py
│   └── analyze_robot_results.py
├── input/
│   └── subscriber_requirements.md
├── generated/
│   ├── test_cases.json
│   ├── subscriber_api.robot
│   └── failure_analysis.json
└── results/
    ├── output.xml
    ├── log.html
    └── report.html

3.4 Create .gitignore

.venv/
__pycache__/
*.py[cod]
.DS_Store
.env
results/*.xml
results/*.html
results/*.log
generated/test_cases.json
generated/subscriber_api.robot
generated/failure_analysis.json

Generated outputs can be saved as CI artifacts instead of being committed. If you want to review and version-control generated tests, remove their corresponding ignore rules.


4. Define the API requirements

Create input/subscriber_requirements.md:

# Subscriber API requirements

## Endpoint
- Method: POST
- Path: /subscribers

## Business rules
- A subscriber must have a unique subscriberId.
- A valid create request returns HTTP 201.
- A duplicate subscriberId returns HTTP 409.
- phoneNumber is mandatory.
- A request missing phoneNumber returns HTTP 422.
- Invalid requests must not create a subscriber.
- Error responses contain a machine-readable error code.

## Not specified
- Authentication requirements.
- Complete request and response schemas.
- Validation precedence when multiple fields are invalid.
- Whether null or empty subscriberId is valid.
- HTTP status for malformed JSON.
- phoneNumber format and length constraints.

Why document the unspecified behavior? Because an LLM can generate plausible tests for behavior the API contract never defined.

For example, the requirements explicitly define HTTP 201, 409 and 422 for particular scenarios. They do not define the malformed-JSON status. Do not treat HTTP 400 as contractually required unless the API specification confirms it.


5. Build the JSON Schema

Create app/test_case_schema.py:

TEST_CASE_SCHEMA = {
    "type": "object",
    "required": [
        "requirement_summary",
        "assumptions",
        "test_cases"
    ],
    "properties": {
        "requirement_summary": {
            "type": "string"
        },
        "assumptions": {
            "type": "array",
            "items": {
                "type": "string"
            }
        },
        "test_cases": {
            "type": "array",
            "items": {
                "type": "object",
                "required": [
                    "name",
                    "method",
                    "url",
                    "expected_status_code"
                ],
                "properties": {
                    "name": {
                        "type": "string",
                        "minLength": 1
                    },
                    "method": {
                        "type": "string",
                        "enum": [
                            "GET",
                            "POST",
                            "PUT",
                            "PATCH",
                            "DELETE"
                        ]
                    },
                    "url": {
                        "type": "string",
                        "minLength": 1
                    },
                    "expected_status_code": {
                        "type": "integer",
                        "minimum": 100,
                        "maximum": 599
                    }
                },
                "additionalProperties": True
            }
        }
    },
    "additionalProperties": True
}

This schema ensures that the model returns the required structure.

Important: A schema-valid test is not automatically a correct test. A case expecting HTTP 422 for the wrong reason can still pass schema validation.


6. Generate test cases using local Ollama

6.1 Create app/ollama_generator.py

import json

from jsonschema import ValidationError, validate
from ollama import chat

from app.test_case_schema import TEST_CASE_SCHEMA


def generate_test_cases(requirements: str) -> dict:
    response = chat(
        model="llama3.2:3b",
        format=TEST_CASE_SCHEMA,
        messages=[
            {
                "role": "system",
                "content": (
                    "You are a senior API QA test engineer. "
                    "Generate test cases strictly from the supplied requirements. "
                    "Follow the required JSON schema. "
                    "Every test case must include name, method, url, "
                    "and expected_status_code. "
                    "Do not invent undocumented expected results. "
                    "Record uncertainty in assumptions. "
                    "Return only structured JSON."
                ),
            },
            {
                "role": "user",
                "content": (
                    "Generate structured API test cases from these requirements:\n\n"
                    + requirements
                ),
            },
        ],
    )

    try:
        result = json.loads(response.message.content)
        validate(
            instance=result,
            schema=TEST_CASE_SCHEMA
        )

    except json.JSONDecodeError as exc:
        raise ValueError(
            f"Ollama returned invalid JSON: {exc}"
        ) from exc

    except ValidationError as exc:
        location = ".".join(
            str(part) for part in exc.absolute_path
        )

        raise ValueError(
            f"Generated test cases failed schema validation "
            f"at '{location}': {exc.message}"
        ) from exc

    if not result["test_cases"]:
        raise ValueError(
            "The model generated zero test cases."
        )

    return result

6.2 Create app/run_local_ai.py

import json
from pathlib import Path

from app.ollama_generator import generate_test_cases

ROOT = Path(__file__).resolve().parent.parent

REQUIREMENTS_FILE = (
    ROOT / "input" / "subscriber_requirements.md"
)

OUTPUT_FILE = (
    ROOT / "generated" / "test_cases.json"
)


def main() -> None:
    requirements = REQUIREMENTS_FILE.read_text(
        encoding="utf-8"
    )

    result = generate_test_cases(requirements)

    OUTPUT_FILE.parent.mkdir(
        parents=True,
        exist_ok=True
    )

    OUTPUT_FILE.write_text(
        json.dumps(
            result,
            indent=2,
            ensure_ascii=False
        ) + "\n",
        encoding="utf-8"
    )

    print(
        f"Generated {len(result['test_cases'])} test case(s)."
    )
    print(f"Output: {OUTPUT_FILE}")


if __name__ == "__main__":
    main()

6.3 Run the generator

From the project root:

cd ~/agentic-test-engineer
source .venv/bin/activate

python -m app.run_local_ai
cat generated/test_cases.json

The workflow is:

  1. Python reads the requirements.

  2. Python sends the prompt to Ollama.

  3. Ollama runs llama3.2:3b.

  4. The model returns structured JSON.

  5. Python validates the response.

  6. Python writes generated/test_cases.json.

We used the Python Ollama library rather than sending prompts to an external hosted LLM API.


7. Validate the generated JSON

Create app/validate_test_cases.py:

import json
import sys
from pathlib import Path

from jsonschema import ValidationError, validate

from app.test_case_schema import TEST_CASE_SCHEMA

ROOT = Path(__file__).resolve().parent.parent

INPUT_FILE = (
    ROOT / "generated" / "test_cases.json"
)


def main() -> int:
    try:
        data = json.loads(
            INPUT_FILE.read_text(encoding="utf-8")
        )

        validate(
            instance=data,
            schema=TEST_CASE_SCHEMA
        )

    except FileNotFoundError:
        print(f"FAIL: file not found: {INPUT_FILE}")
        return 1

    except json.JSONDecodeError as exc:
        print(f"FAIL: invalid JSON: {exc}")
        return 1

    except ValidationError as exc:
        location = ".".join(
            str(part) for part in exc.absolute_path
        )

        print(
            f"FAIL: schema validation at "
            f"'{location}': {exc.message}"
        )
        return 1

    cases = data.get("test_cases", [])

    if not cases:
        print("FAIL: no test cases generated")
        return 1

    print(
        f"PASS: schema valid; {len(cases)} test case(s)"
    )

    for case in cases:
        print(
            f"- {case['name']} | "
            f"{case['method']} {case['url']} | "
            f"expected HTTP {case['expected_status_code']}"
        )

    return 0


if __name__ == "__main__":
    sys.exit(main())

Run:

python -m app.validate_test_cases
python -m json.tool generated/test_cases.json

Inspect each test's expected status, endpoint, assumptions and relationship to the source requirements before executing it.


8. Generate Robot Framework tests

The deterministic generator is an important safety layer. It should not execute arbitrary code produced by the LLM.

Our generator was designed to:

  1. Read generated/test_cases.json.

  2. Validate it using the shared JSON Schema.

  3. Normalize scenario names.

  4. Map recognized names to reviewed Robot templates.

  5. Verify the HTTP method, endpoint and expected status.

  6. Reject unsupported or ambiguous scenarios.

  7. Deduplicate equivalent cases.

  8. Write generated/subscriber_api.robot.

8.1 Scenario rules

ScenarioExpected statusContract statusValid subscriber creation with unique ID201Explicitly specifiedDuplicate subscriber ID409Explicitly specifiedMissing phone number422Explicitly specifiedMissing subscriber ID422Requires contract confirmation or an approved assumptionNull subscriber ID422Not specifiedMalformed request400Not specified

Do not turn an assumption into a contractual requirement just to get a green pipeline.

8.2 Name-normalization issue we encountered

Different model runs generated names such as:

SubscriberIdUnique
ExistingSubscriberIdConflict
Create Subscriber with Unique Id
Create Subscriber with Mandatory PhoneNumber
Create Subscriber with Missing Mandatory PhoneNumber
ExistingSubscriberIdConflictNoPhoneNumber

The original classifier did not recognize every name. The pipeline failed even though the model had generated a potentially relevant scenario.

The correct solution is to add reviewed aliases and validate the semantics. Do not simply accept any name containing the word subscriber.

For example, ExistingSubscriberIdConflictNoPhoneNumber is ambiguous because the requirements do not specify validation precedence between a duplicate ID and a missing phone number.

Run the generator and inspect its output:

python -m app.generate_robot_tests
sed -n '1,240p' generated/subscriber_api.robot

If the generator rejects a case, inspect the exact case in the JSON and either add a reviewed alias, clarify the requirements, or reject the unsupported scenario.


9. Create the Flask mock API

The mock API gives the test pipeline a controlled target. It is not the real subscriber service.

Create app/mock_subscriber_api.py:

from flask import Flask, jsonify, request

app = Flask(__name__)

SUBSCRIBERS = {}


@app.get("/health")
def health():
    return jsonify({"status": "ok"}), 200


@app.post("/subscribers")
def create_subscriber():
    payload = request.get_json(silent=True)

    if not isinstance(payload, dict):
        return jsonify({
            "error": {
                "code": "invalid_request",
                "message": "Request body must be a JSON object",
            }
        }), 400

    subscriber_id = payload.get("subscriberId")
    phone_number = payload.get("phoneNumber")

    if not subscriber_id:
        return jsonify({
            "error": {
                "code": "missing_subscriber_id",
                "message": "subscriberId is required",
            }
        }), 422

    if subscriber_id in SUBSCRIBERS:
        return jsonify({
            "error": {
                "code": "existing_subscriber_id",
                "message": "subscriberId already exists",
            }
        }), 409

    if not phone_number:
        return jsonify({
            "error": {
                "code": "missing_phone_number",
                "message": "phoneNumber is required",
            }
        }), 422

    SUBSCRIBERS[subscriber_id] = {
        "subscriberId": subscriber_id,
        "phoneNumber": phone_number,
    }

    return jsonify(
        SUBSCRIBERS[subscriber_id]
    ), 201


@app.delete("/test-support/subscribers")
def reset_subscribers():
    SUBSCRIBERS.clear()
    return "", 204


if __name__ == "__main__":
    app.run(
        host="127.0.0.1",
        port=5001,
        debug=False
    )

9.1 Start the mock server

In Terminal 1:

cd ~/agentic-test-engineer
source .venv/bin/activate

python -m app.mock_subscriber_api

Leave this process running.

9.2 Test the API manually

In Terminal 2:

curl -i http://127.0.0.1:5001/health

Expected: HTTP 200.

Test a successful create:

curl -i -X POST \
  http://127.0.0.1:5001/subscribers \
  -H 'Content-Type: application/json' \
  -d '{"subscriberId":"sub-1001","phoneNumber":"+15550001001"}'

Expected for this mock: HTTP 201.

Important: The mock's handling of malformed JSON, missing IDs and other edge cases must be confirmed against the actual API contract before being treated as authoritative.


10. Execute Robot Framework tests

The generated Robot suite uses RequestsLibrary to call the Flask API.

Generate the suite:

python -m app.generate_robot_tests

Run the suite:

mkdir -p results

.venv/bin/robot \
  --output results/output.xml \
  --log results/log.html \
  --report results/report.html \
  generated/subscriber_api.robot

A previous local run completed:

5 tests, 5 passed, 0 failed

This is a historical result from the project; a future run can produce different cases or results.

10.1 Open the reports

open results/log.html
open results/report.html

Files:

  • results/output.xml: machine-readable execution results.

  • results/log.html: detailed test and keyword execution.

  • results/report.html: summary of the suite.

Robot Framework returns a nonzero exit code when tests fail. That is expected for a genuine failed assertion.


11. Failure analysis

We added an analysis step that reads Robot Framework XML and creates structured failure analysis.

A failure message is evidence of an observed mismatch, not proof of the application's root cause.

For example:

actual-value != expected-value

confirms that the assertion values differ. It does not establish whether the product, test input, requirement or assertion is wrong.

11.1 Create a controlled failure

Create results/controlled_failure.robot:

*** Test Cases ***
Controlled Failure For AI Analysis
    Should Be Equal    actual-value    expected-value

Run it:

.venv/bin/robot \
  --output results/controlled_failure.xml \
  --log results/controlled_failure_log.html \
  --report results/controlled_failure_report.html \
  results/controlled_failure.robot

This test is supposed to fail. A nonzero exit code is expected.

11.2 Deterministic classifier

Create or verify app/classify_failures.py:

def classify_failure(failure: dict) -> dict:
    keyword = failure.get("keyword", "")
    message = failure.get("message", "")

    if (
        keyword == "Should Be Equal"
        and " != " in message
    ):
        actual, expected = message.split(" != ", 1)

        return {
            "category": "ASSERTION_VALUE_MISMATCH",
            "observation": (
                "Actual and expected values differ."
            ),
            "evidence": {
                "keyword": keyword,
                "actual": actual,
                "expected": expected,
            },
            "confirmed": True,
            "application_root_cause": "UNKNOWN",
            "next_checks": [
                "Confirm that the expected value matches the requirement.",
                "Trace the actual value to the API response or test input.",
                "Determine whether the mismatch is intentional or a product defect.",
            ],
        }

    return {
        "category": "UNCLASSIFIED_FAILURE",
        "observation": (
            message or "No failure message was captured."
        ),
        "evidence": {
            "keyword": keyword,
            "message": message,
        },
        "confirmed": False,
        "application_root_cause": "UNKNOWN",
        "next_checks": [
            "Inspect the complete Robot log and related application evidence."
        ],
    }

This is a classifier function, not a complete analyzer by itself. The analyzer must parse Robot XML, extract each failure, invoke the classifier and write generated/failure_analysis.json.

The project also has app/analyze_robot_results.py. Check its current CLI before running it:

python -m app.analyze_robot_results --help

If the script does not implement --help, inspect the argument parser:

sed -n '1,260p' app/analyze_robot_results.py

A previous analysis run produced plausible but unsupported AI root-cause suggestions. The improvement is to classify known failure patterns deterministically first, then label any LLM explanation as a hypothesis until independent evidence confirms it.


12. Configure GitLab CI

12.1 Why we used a self-hosted runner

Your Ollama service runs on the Mac at:

http://127.0.0.1:11434

GitLab's shared runners cannot reach that local service through your Mac's loopback address.

We used a self-hosted GitLab Shell Runner on the Mac, tagged:

mac-ollama

The runner executes CI job commands on the Mac. Ollama then processes the prompt locally.

Check the environment on the runner host:

which git
git --version
echo "$PATH"

ollama list
curl -sS http://127.0.0.1:11434/api/tags

The runner must be online and configured with the mac-ollama tag. Use the current GitLab project settings and official GitLab Runner instructions to register or reconfigure it. Do not commit runner authentication tokens.

12.2 GitLab CI pipeline

Create .gitlab-ci.yml:

stages:
  - generate
  - validate
  - test
  - analyze

default:
  tags:
    - mac-ollama
  before_script:
    - python3 -m venv .venv
    - .venv/bin/python -m pip install --upgrade pip
    - .venv/bin/pip install -r requirements.txt

variables:
  OLLAMA_HOST: "http://127.0.0.1:11434"
  OLLAMA_MODEL: "llama3.2:3b"
  PIP_DISABLE_PIP_VERSION_CHECK: "1"

generate_test_cases:
  stage: generate
  script:
    - curl --fail --silent --show-error "$OLLAMA_HOST/api/tags"
    - ollama show "$OLLAMA_MODEL"
    - .venv/bin/python -m app.run_local_ai
    - .venv/bin/python -m app.validate_test_cases
  artifacts:
    when: always
    paths:
      - generated/test_cases.json
    expire_in: 7 days

validate_and_generate_robot:
  stage: validate
  script:
    - .venv/bin/python -m app.validate_test_cases
    - .venv/bin/python -m app.generate_robot_tests
    - test -s generated/subscriber_api.robot
  artifacts:
    when: always
    paths:
      - generated/test_cases.json
      - generated/subscriber_api.robot
    expire_in: 7 days

robot_api_tests:
  stage: test
  script:
    - mkdir -p results
    - .venv/bin/python -m app.mock_subscriber_api > results/mock_api.log 2>&1 &
    - echo $! > results/mock_api.pid
    - |
      for i in $(seq 1 30); do
        if curl --fail --silent http://127.0.0.1:5001/health; then
          break
        fi
        sleep 1
      done
    - curl --fail --silent http://127.0.0.1:5001/health
    - .venv/bin/robot --output results/output.xml --log results/log.html --report results/report.html generated/subscriber_api.robot
  after_script:
    - if [ -f results/mock_api.pid ]; then kill "$(cat results/mock_api.pid)" 2>/dev/null || true; fi
  artifacts:
    when: always
    paths:
      - results/output.xml
      - results/log.html
      - results/report.html
      - results/mock_api.log
    expire_in: 7 days

analyze_results:
  stage: analyze
  when: always
  script:
    - |
      if [ -s results/output.xml ]; then
        .venv/bin/python -m app.analyze_robot_results results/output.xml generated/failure_analysis.json
      else
        echo "No Robot output.xml was produced; skipping analysis."
      fi
  artifacts:
    when: always
    paths:
      - generated/failure_analysis.json
    expire_in: 7 days

Important before using this pipeline

  • Verify the analyzer accepts positional arguments input.xml output.json. If its parser expects flags, update the command accordingly.

  • Later jobs must receive artifacts from earlier jobs.

  • The runner needs Python, Git, Ollama, the model and access to the local ports.

  • 127.0.0.1 only reaches services on the machine running the job.

  • Run the pipeline in a branch before merging changes.

  • A model-generated test suite should not automatically become a trusted regression suite without semantic checks and review.

12.3 Execute the pipeline

  1. Commit and push the project to GitLab.

  2. Open your GitLab project.

  3. Navigate to Build → Pipelines.

  4. Open the latest pipeline.

  5. Inspect each job's log.

  6. Open the robot_api_tests job.

  7. Find Job artifacts, Browse, or Download artifacts.

  8. Download the archive and open results/log.html or results/report.html.

Artifacts are configured to expire after seven days.


13. Errors encountered and their solutions

Error 1: git: command not found

Observed:

/usr/bin/bash: line 182: git: command not found
ERROR: Job failed: exit code 127

Cause: Git was missing from the runner environment or not on its PATH.

Check:

which git
git --version
echo "$PATH"

Solution on a Homebrew-managed Mac:

brew install git

Ensure the runner service inherits the correct PATH. A service can have a different environment from your interactive Terminal.

Error 2: Ollama connection refused

Check:

curl -v http://127.0.0.1:11434/api/tags
ollama list
ollama show llama3.2:3b

Solution: start Ollama, verify the model exists, and ensure the job is executing on the Mac's mac-ollama Shell Runner.

Error 3: ModuleNotFoundError: No module named 'app'

Cause: the command was run from the wrong directory.

Solution:

cd ~/agentic-test-engineer
source .venv/bin/activate
python -m app.run_local_ai

Use package-module execution from the project root.

Error 4: Invalid JSON or schema validation failure

Check:

python -m app.run_local_ai
python -m app.validate_test_cases
python -m json.tool generated/test_cases.json

Solution: inspect the exact JSON or schema error, then fix the prompt, schema or output handling. Do not bypass validation to force a green pipeline.

Error 5: Generated scenario not recognized

Examples:

SubscriberIdUnique
Create Subscriber with Mandatory PhoneNumber

Cause: model-generated names vary, while the original classifier supported a limited set of names.

Solution: normalize names and add reviewed aliases for unambiguous scenarios. Verify the expected status and preconditions. Reject ambiguous or unsupported scenarios.

Error 6: Duplicate ID and missing phone number in one test

Example:

ExistingSubscriberIdConflictNoPhoneNumber

Cause: the requirements do not define which validation takes precedence.

Solution: clarify the contract or test the conditions independently. Do not guess whether HTTP 409 or 422 should win.

Error 7: HTTP 400 or 422 generated without a requirement

Cause: the model inferred an expected response not specified in the requirements.

Solution: confirm the API contract, or explicitly label and approve the behavior as an assumption. A mock API response does not establish the real API contract.

Error 8: Robot cannot connect to Flask

Check:

curl -i http://127.0.0.1:5001/health
lsof -nP -iTCP:5001 -sTCP:LISTEN

Solution: start the mock API, inspect results/mock_api.log, and ensure the API and Robot process run on the same host.

Error 9: Where is log.html?

Local execution:

open results/log.html
open results/report.html

GitLab execution:

Build → Pipelines → pipeline → robot_api_tests → Job artifacts.

Download or browse the artifact archive and open the report files.

Error 10: AI analysis invents a root cause

Cause: the LLM produced a plausible explanation without sufficient evidence.

Solution: parse Robot XML, capture the actual failure message and keyword, classify known patterns deterministically, and keep the application root cause UNKNOWN until evidence confirms it.

Error 11: Controlled failure exits with a nonzero status

Cause: the test was deliberately written to fail.

Solution: this is expected for the controlled-failure experiment. Keep it separate from the regular passing suite.

Error 12: Artifacts missing in a later job

Check:

ls -lh generated results

Solution: ensure the producing job creates the files and declares them under artifacts:paths. Check artifact downloading configuration and expiry.

Error 13: Kubernetes localhost:8080 refused

This occurred in the separate Kubernetes practice environment, not in the core Ollama/Flask workflow.

Check:

kubectl config get-contexts
kubectl config current-context
echo "$KUBECONFIG"
ls -l ~/.kube/config

If the intended context exists, select it:

kubectl config use-context docker-desktop
kubectl cluster-info
kubectl get nodes

Only use docker-desktop if it appears in kubectl config get-contexts.


14. Git workflow

From the project root:

cd ~/agentic-test-engineer

git status
git remote -v
git branch --show-current

For a new repository:

git init
git add .
git status
git commit -m "Build AI-assisted Robot Framework test pipeline"

Add the actual GitLab repository URL:

git remote add origin <YOUR_GITLAB_REPOSITORY_URL>
git branch -M main
git push -u origin main

If origin already exists, inspect it with git remote -v instead of adding it again.

Do not commit credentials, runner tokens or sensitive test data.


15. Full local execution checklist

Start Ollama and verify that the model is installed. Start the mock API in Terminal 1 when ready to execute Robot tests.

In Terminal 2:

cd ~/agentic-test-engineer
source .venv/bin/activate

# 1. Check Ollama
curl --fail --silent --show-error \
  http://127.0.0.1:11434/api/tags

# 2. Generate test cases
python -m app.run_local_ai

# 3. Validate generated JSON
python -m app.validate_test_cases

# 4. Generate Robot tests
python -m app.generate_robot_tests

# 5. Check mock API health
curl --fail --silent \
  http://127.0.0.1:5001/health

# 6. Execute Robot tests
mkdir -p results

.venv/bin/robot \
  --output results/output.xml \
  --log results/log.html \
  --report results/report.html \
  generated/subscriber_api.robot

# 7. Open reports
open results/log.html
open results/report.html

# 8. Inspect generated files
ls -lh generated results

The mock API must be running before step 6.


16. Final verification checklist

  • Ollama installed and running.

  • llama3.2:3b downloaded.

  • Python virtual environment created.

  • Required packages installed.

  • Requirements documented.

  • AI generates generated/test_cases.json.

  • JSON Schema validation passes.

  • Scenario generator rejects unknown and ambiguous cases.

  • Expected status codes are backed by requirements or approved assumptions.

  • Flask /health returns HTTP 200.

  • Robot Framework executes the generated suite.

  • XML and HTML reports are produced.

  • Controlled failure analysis is tested separately.

  • Failure analysis distinguishes facts from hypotheses.

  • GitLab Shell Runner is online and tagged mac-ollama.

  • The runner can reach local Ollama.

  • CI artifacts include test JSON, Robot suite, XML, HTML reports and mock API logs.

  • No credentials are committed.


17. Next improvements

  1. Add requirement IDs to every test case for traceability.

  2. Add semantic validation beyond JSON Schema.

  3. Require human approval before generated tests enter the trusted regression suite.

  4. Use OpenAPI or another formal API contract when available.

  5. Pin dependency versions for reproducible CI.

  6. Isolate test data before running tests in parallel.

  7. Add unit tests for every classifier alias and status mismatch.

  8. Run deterministic failure classification before LLM-based analysis.

  9. Require independent evidence before declaring a root cause.

  10. Extend the same architecture to telecom scenarios such as SIP registration, call setup, MO/MT flows, and HTTP/2 SBA APIs after defining protocol-specific requirements.

Final principle

Requirements define expected behavior. AI proposes test cases. Validators reject unsupported proposals. Deterministic templates generate automation. Robot Framework provides execution evidence. Failure analysis separates observations from hypotheses. GitLab CI preserves the artifacts for review and debugging.

That is the foundation of a reliable AI-assisted test engineering system.

Important note

This runbook consolidates the workflow and errors we discussed. Some scripts—especially app/generate_robot_tests.py and app/analyze_robot_results.py—were developed iteratively, and their exact latest contents are not all present in this conversation. The runbook documents their responsibilities and the verified patterns, but it should not be treated as a byte-for-byte backup of every version you currently have on your Mac.

For a fully reproducible backup of your actual project, run this from Terminal:

cd ~/agentic-test-engineer
find app input generated results -maxdepth 2 -type f -print

Then create an archive of your project, excluding the virtual environment and Git metadata:

cd ~
tar --exclude='agentic-test-engineer/.venv' \
    --exclude='agentic-test-engineer/.git' \
    -czf agentic-test-engineer-backup.tar.gz \
    agentic-test-engineer

That archive will preserve the scripts and configuration files that actually exist in your current working copy, which is the safest way to retain the precise implementation you built.

Подобається цей допис?

Купити для DigeratiDaily каву

Більше від DigeratiDaily

КонфіденційністьУмови