Organized complete Agentic AI Test Engineer — Zero-to-Hero guide around the actual workflow we built on your MacBook Air M2: local Ollama installation, Python application, AI-generated test cases, JSON validation, Robot Framework, Flask mock API, failure analysis, GitLab Runner, CI pipeline, and the errors we encountered.
The guide will include copy-and-paste commands, complete scripts for the core components, troubleshooting steps, and verification commands. One important distinction throughout: the AI proposes tests, but the API requirements determine the correct expected results.
Environment: MacBook Air M2, macOS, Python, Ollama, Flask, Robot Framework, GitLab CI and a self-hosted GitLab Runner.
Project location: ~/agentic-test-engineer
Local AI model: llama3.2:3b
Purpose: Build an AI-assisted test engineering system that converts requirements into test cases, validates them, generates Robot Framework automation, executes tests, analyzes failures, and publishes results through GitLab CI.
1. What we built
1.1 Architecture
Requirements
input/subscriber_requirements.md
|
v
Python application
app/run_local_ai.py
|
v
Ollama local LLM
127.0.0.1:11434
llama3.2:3b
|
v
AI-generated JSON
generated/test_cases.json
|
v
JSON Schema validation
app/validate_test_cases.py
|
v
Deterministic test generator
app/generate_robot_tests.py
|
v
Robot Framework suite
generated/subscriber_api.robot
|
v
Flask mock subscriber API
127.0.0.1:5001
|
v
Robot Framework execution
|
+--> results/output.xml
+--> results/log.html
+--> results/report.html
|
v
Failure analysis
app/analyze_robot_results.py
|
v
generated/failure_analysis.json
|
v
GitLab CI artifacts1.2 What makes this agentic?
The project uses AI to propose test cases and connects that output to validation, code generation, test execution, and result analysis.
Strictly speaking, the initial implementation is an AI-assisted test engineering pipeline. To call it a fully autonomous agent, it would need controlled tool use, decision-making, result evaluation, and a bounded retry or investigation loop.
The key safety principle is:
Requirements define expected behavior.
The AI proposes test cases.
Validators reject invalid or unsupported proposals.
Deterministic templates generate executable tests.
Robot Framework provides execution evidence.
AI may suggest explanations, but must not invent confirmed root causes.
2. Install the required tools
Open Terminal on your MacBook.
2.1 Check your system
uname -m
sw_vers
python3 --version
git --version
brew --versionOn your Apple Silicon Mac, uname -m should normally return:
arm64Install missing dependencies with Homebrew:
brew install python git2.2 Install Ollama
Install the Ollama macOS application from its official website:
Open the application and verify the installation:
ollama --version
ollama list
curl -sS http://127.0.0.1:11434/api/tagsThe API request should return JSON describing the locally available models.
If the connection is refused, open/start the Ollama application and try again.
2.3 Download the model
ollama pull llama3.2:3b
ollama show llama3.2:3b
ollama listRun an interactive test:
ollama run llama3.2:3bEnter a simple prompt. Exit the interactive session when finished.
How your MacBook was used
The model runs locally through Ollama. Your Python code communicates with the Ollama service at:
http://127.0.0.1:11434This means the model inference uses your MacBook's CPU/GPU resources and local memory, rather than requiring a hosted LLM API for each prompt.
3. Create the project from scratch
mkdir -p ~/agentic-test-engineer
cd ~/agentic-test-engineer
mkdir -p app input generated results
touch app/__init__.py3.1 Create a Python virtual environment
python3 -m venv .venv
source .venv/bin/activate
python --version
which pythonThe Python path should point inside:
~/agentic-test-engineer/.venv/Whenever you open a new Terminal for local work, activate the environment again:
cd ~/agentic-test-engineer
source .venv/bin/activate3.2 Create requirements.txt
ollama
robotframework
robotframework-requests
requests
jsonschema
flaskInstall the dependencies:
python -m pip install --upgrade pip
python -m pip install -r requirements.txtVerify:
python -c "import ollama, robot, requests, jsonschema, flask; print('All dependencies imported successfully')"3.3 Recommended directory structure
agentic-test-engineer/
├── .gitignore
├── .gitlab-ci.yml
├── requirements.txt
├── app/
│ ├── __init__.py
│ ├── test_case_schema.py
│ ├── ollama_generator.py
│ ├── run_local_ai.py
│ ├── validate_test_cases.py
│ ├── mock_subscriber_api.py
│ ├── generate_robot_tests.py
│ ├── classify_failures.py
│ └── analyze_robot_results.py
├── input/
│ └── subscriber_requirements.md
├── generated/
│ ├── test_cases.json
│ ├── subscriber_api.robot
│ └── failure_analysis.json
└── results/
├── output.xml
├── log.html
└── report.html3.4 Create .gitignore
.venv/
__pycache__/
*.py[cod]
.DS_Store
.env
results/*.xml
results/*.html
results/*.log
generated/test_cases.json
generated/subscriber_api.robot
generated/failure_analysis.jsonGenerated outputs can be saved as CI artifacts instead of being committed. If you want to review and version-control generated tests, remove their corresponding ignore rules.
4. Define the API requirements
Create input/subscriber_requirements.md:
# Subscriber API requirements
## Endpoint
- Method: POST
- Path: /subscribers
## Business rules
- A subscriber must have a unique subscriberId.
- A valid create request returns HTTP 201.
- A duplicate subscriberId returns HTTP 409.
- phoneNumber is mandatory.
- A request missing phoneNumber returns HTTP 422.
- Invalid requests must not create a subscriber.
- Error responses contain a machine-readable error code.
## Not specified
- Authentication requirements.
- Complete request and response schemas.
- Validation precedence when multiple fields are invalid.
- Whether null or empty subscriberId is valid.
- HTTP status for malformed JSON.
- phoneNumber format and length constraints.Why document the unspecified behavior? Because an LLM can generate plausible tests for behavior the API contract never defined.
For example, the requirements explicitly define HTTP 201, 409 and 422 for particular scenarios. They do not define the malformed-JSON status. Do not treat HTTP 400 as contractually required unless the API specification confirms it.
5. Build the JSON Schema
Create app/test_case_schema.py:
TEST_CASE_SCHEMA = {
"type": "object",
"required": [
"requirement_summary",
"assumptions",
"test_cases"
],
"properties": {
"requirement_summary": {
"type": "string"
},
"assumptions": {
"type": "array",
"items": {
"type": "string"
}
},
"test_cases": {
"type": "array",
"items": {
"type": "object",
"required": [
"name",
"method",
"url",
"expected_status_code"
],
"properties": {
"name": {
"type": "string",
"minLength": 1
},
"method": {
"type": "string",
"enum": [
"GET",
"POST",
"PUT",
"PATCH",
"DELETE"
]
},
"url": {
"type": "string",
"minLength": 1
},
"expected_status_code": {
"type": "integer",
"minimum": 100,
"maximum": 599
}
},
"additionalProperties": True
}
}
},
"additionalProperties": True
}This schema ensures that the model returns the required structure.
Important: A schema-valid test is not automatically a correct test. A case expecting HTTP 422 for the wrong reason can still pass schema validation.
6. Generate test cases using local Ollama
6.1 Create app/ollama_generator.py
import json
from jsonschema import ValidationError, validate
from ollama import chat
from app.test_case_schema import TEST_CASE_SCHEMA
def generate_test_cases(requirements: str) -> dict:
response = chat(
model="llama3.2:3b",
format=TEST_CASE_SCHEMA,
messages=[
{
"role": "system",
"content": (
"You are a senior API QA test engineer. "
"Generate test cases strictly from the supplied requirements. "
"Follow the required JSON schema. "
"Every test case must include name, method, url, "
"and expected_status_code. "
"Do not invent undocumented expected results. "
"Record uncertainty in assumptions. "
"Return only structured JSON."
),
},
{
"role": "user",
"content": (
"Generate structured API test cases from these requirements:\n\n"
+ requirements
),
},
],
)
try:
result = json.loads(response.message.content)
validate(
instance=result,
schema=TEST_CASE_SCHEMA
)
except json.JSONDecodeError as exc:
raise ValueError(
f"Ollama returned invalid JSON: {exc}"
) from exc
except ValidationError as exc:
location = ".".join(
str(part) for part in exc.absolute_path
)
raise ValueError(
f"Generated test cases failed schema validation "
f"at '{location}': {exc.message}"
) from exc
if not result["test_cases"]:
raise ValueError(
"The model generated zero test cases."
)
return result6.2 Create app/run_local_ai.py
import json
from pathlib import Path
from app.ollama_generator import generate_test_cases
ROOT = Path(__file__).resolve().parent.parent
REQUIREMENTS_FILE = (
ROOT / "input" / "subscriber_requirements.md"
)
OUTPUT_FILE = (
ROOT / "generated" / "test_cases.json"
)
def main() -> None:
requirements = REQUIREMENTS_FILE.read_text(
encoding="utf-8"
)
result = generate_test_cases(requirements)
OUTPUT_FILE.parent.mkdir(
parents=True,
exist_ok=True
)
OUTPUT_FILE.write_text(
json.dumps(
result,
indent=2,
ensure_ascii=False
) + "\n",
encoding="utf-8"
)
print(
f"Generated {len(result['test_cases'])} test case(s)."
)
print(f"Output: {OUTPUT_FILE}")
if __name__ == "__main__":
main()6.3 Run the generator
From the project root:
cd ~/agentic-test-engineer
source .venv/bin/activate
python -m app.run_local_ai
cat generated/test_cases.jsonThe workflow is:
Python reads the requirements.
Python sends the prompt to Ollama.
Ollama runs
llama3.2:3b.The model returns structured JSON.
Python validates the response.
Python writes
generated/test_cases.json.
We used the Python Ollama library rather than sending prompts to an external hosted LLM API.
7. Validate the generated JSON
Create app/validate_test_cases.py:
import json
import sys
from pathlib import Path
from jsonschema import ValidationError, validate
from app.test_case_schema import TEST_CASE_SCHEMA
ROOT = Path(__file__).resolve().parent.parent
INPUT_FILE = (
ROOT / "generated" / "test_cases.json"
)
def main() -> int:
try:
data = json.loads(
INPUT_FILE.read_text(encoding="utf-8")
)
validate(
instance=data,
schema=TEST_CASE_SCHEMA
)
except FileNotFoundError:
print(f"FAIL: file not found: {INPUT_FILE}")
return 1
except json.JSONDecodeError as exc:
print(f"FAIL: invalid JSON: {exc}")
return 1
except ValidationError as exc:
location = ".".join(
str(part) for part in exc.absolute_path
)
print(
f"FAIL: schema validation at "
f"'{location}': {exc.message}"
)
return 1
cases = data.get("test_cases", [])
if not cases:
print("FAIL: no test cases generated")
return 1
print(
f"PASS: schema valid; {len(cases)} test case(s)"
)
for case in cases:
print(
f"- {case['name']} | "
f"{case['method']} {case['url']} | "
f"expected HTTP {case['expected_status_code']}"
)
return 0
if __name__ == "__main__":
sys.exit(main())Run:
python -m app.validate_test_cases
python -m json.tool generated/test_cases.jsonInspect each test's expected status, endpoint, assumptions and relationship to the source requirements before executing it.
8. Generate Robot Framework tests
The deterministic generator is an important safety layer. It should not execute arbitrary code produced by the LLM.
Our generator was designed to:
Read
generated/test_cases.json.Validate it using the shared JSON Schema.
Normalize scenario names.
Map recognized names to reviewed Robot templates.
Verify the HTTP method, endpoint and expected status.
Reject unsupported or ambiguous scenarios.
Deduplicate equivalent cases.
Write
generated/subscriber_api.robot.
8.1 Scenario rules
ScenarioExpected statusContract statusValid subscriber creation with unique ID201Explicitly specifiedDuplicate subscriber ID409Explicitly specifiedMissing phone number422Explicitly specifiedMissing subscriber ID422Requires contract confirmation or an approved assumptionNull subscriber ID422Not specifiedMalformed request400Not specified
Do not turn an assumption into a contractual requirement just to get a green pipeline.
8.2 Name-normalization issue we encountered
Different model runs generated names such as:
SubscriberIdUnique
ExistingSubscriberIdConflict
Create Subscriber with Unique Id
Create Subscriber with Mandatory PhoneNumber
Create Subscriber with Missing Mandatory PhoneNumber
ExistingSubscriberIdConflictNoPhoneNumberThe original classifier did not recognize every name. The pipeline failed even though the model had generated a potentially relevant scenario.
The correct solution is to add reviewed aliases and validate the semantics. Do not simply accept any name containing the word subscriber.
For example, ExistingSubscriberIdConflictNoPhoneNumber is ambiguous because the requirements do not specify validation precedence between a duplicate ID and a missing phone number.
Run the generator and inspect its output:
python -m app.generate_robot_tests
sed -n '1,240p' generated/subscriber_api.robotIf the generator rejects a case, inspect the exact case in the JSON and either add a reviewed alias, clarify the requirements, or reject the unsupported scenario.
9. Create the Flask mock API
The mock API gives the test pipeline a controlled target. It is not the real subscriber service.
Create app/mock_subscriber_api.py:
from flask import Flask, jsonify, request
app = Flask(__name__)
SUBSCRIBERS = {}
@app.get("/health")
def health():
return jsonify({"status": "ok"}), 200
@app.post("/subscribers")
def create_subscriber():
payload = request.get_json(silent=True)
if not isinstance(payload, dict):
return jsonify({
"error": {
"code": "invalid_request",
"message": "Request body must be a JSON object",
}
}), 400
subscriber_id = payload.get("subscriberId")
phone_number = payload.get("phoneNumber")
if not subscriber_id:
return jsonify({
"error": {
"code": "missing_subscriber_id",
"message": "subscriberId is required",
}
}), 422
if subscriber_id in SUBSCRIBERS:
return jsonify({
"error": {
"code": "existing_subscriber_id",
"message": "subscriberId already exists",
}
}), 409
if not phone_number:
return jsonify({
"error": {
"code": "missing_phone_number",
"message": "phoneNumber is required",
}
}), 422
SUBSCRIBERS[subscriber_id] = {
"subscriberId": subscriber_id,
"phoneNumber": phone_number,
}
return jsonify(
SUBSCRIBERS[subscriber_id]
), 201
@app.delete("/test-support/subscribers")
def reset_subscribers():
SUBSCRIBERS.clear()
return "", 204
if __name__ == "__main__":
app.run(
host="127.0.0.1",
port=5001,
debug=False
)9.1 Start the mock server
In Terminal 1:
cd ~/agentic-test-engineer
source .venv/bin/activate
python -m app.mock_subscriber_apiLeave this process running.
9.2 Test the API manually
In Terminal 2:
curl -i http://127.0.0.1:5001/healthExpected: HTTP 200.
Test a successful create:
curl -i -X POST \
http://127.0.0.1:5001/subscribers \
-H 'Content-Type: application/json' \
-d '{"subscriberId":"sub-1001","phoneNumber":"+15550001001"}'Expected for this mock: HTTP 201.
Important: The mock's handling of malformed JSON, missing IDs and other edge cases must be confirmed against the actual API contract before being treated as authoritative.
10. Execute Robot Framework tests
The generated Robot suite uses RequestsLibrary to call the Flask API.
Generate the suite:
python -m app.generate_robot_testsRun the suite:
mkdir -p results
.venv/bin/robot \
--output results/output.xml \
--log results/log.html \
--report results/report.html \
generated/subscriber_api.robotA previous local run completed:
5 tests, 5 passed, 0 failedThis is a historical result from the project; a future run can produce different cases or results.
10.1 Open the reports
open results/log.html
open results/report.htmlFiles:
results/output.xml: machine-readable execution results.results/log.html: detailed test and keyword execution.results/report.html: summary of the suite.
Robot Framework returns a nonzero exit code when tests fail. That is expected for a genuine failed assertion.
11. Failure analysis
We added an analysis step that reads Robot Framework XML and creates structured failure analysis.
A failure message is evidence of an observed mismatch, not proof of the application's root cause.
For example:
actual-value != expected-valueconfirms that the assertion values differ. It does not establish whether the product, test input, requirement or assertion is wrong.
11.1 Create a controlled failure
Create results/controlled_failure.robot:
*** Test Cases ***
Controlled Failure For AI Analysis
Should Be Equal actual-value expected-valueRun it:
.venv/bin/robot \
--output results/controlled_failure.xml \
--log results/controlled_failure_log.html \
--report results/controlled_failure_report.html \
results/controlled_failure.robotThis test is supposed to fail. A nonzero exit code is expected.
11.2 Deterministic classifier
Create or verify app/classify_failures.py:
def classify_failure(failure: dict) -> dict:
keyword = failure.get("keyword", "")
message = failure.get("message", "")
if (
keyword == "Should Be Equal"
and " != " in message
):
actual, expected = message.split(" != ", 1)
return {
"category": "ASSERTION_VALUE_MISMATCH",
"observation": (
"Actual and expected values differ."
),
"evidence": {
"keyword": keyword,
"actual": actual,
"expected": expected,
},
"confirmed": True,
"application_root_cause": "UNKNOWN",
"next_checks": [
"Confirm that the expected value matches the requirement.",
"Trace the actual value to the API response or test input.",
"Determine whether the mismatch is intentional or a product defect.",
],
}
return {
"category": "UNCLASSIFIED_FAILURE",
"observation": (
message or "No failure message was captured."
),
"evidence": {
"keyword": keyword,
"message": message,
},
"confirmed": False,
"application_root_cause": "UNKNOWN",
"next_checks": [
"Inspect the complete Robot log and related application evidence."
],
}This is a classifier function, not a complete analyzer by itself. The analyzer must parse Robot XML, extract each failure, invoke the classifier and write generated/failure_analysis.json.
The project also has app/analyze_robot_results.py. Check its current CLI before running it:
python -m app.analyze_robot_results --helpIf the script does not implement --help, inspect the argument parser:
sed -n '1,260p' app/analyze_robot_results.pyA previous analysis run produced plausible but unsupported AI root-cause suggestions. The improvement is to classify known failure patterns deterministically first, then label any LLM explanation as a hypothesis until independent evidence confirms it.
12. Configure GitLab CI
12.1 Why we used a self-hosted runner
Your Ollama service runs on the Mac at:
http://127.0.0.1:11434GitLab's shared runners cannot reach that local service through your Mac's loopback address.
We used a self-hosted GitLab Shell Runner on the Mac, tagged:
mac-ollamaThe runner executes CI job commands on the Mac. Ollama then processes the prompt locally.
Check the environment on the runner host:
which git
git --version
echo "$PATH"
ollama list
curl -sS http://127.0.0.1:11434/api/tagsThe runner must be online and configured with the mac-ollama tag. Use the current GitLab project settings and official GitLab Runner instructions to register or reconfigure it. Do not commit runner authentication tokens.
12.2 GitLab CI pipeline
Create .gitlab-ci.yml:
stages:
- generate
- validate
- test
- analyze
default:
tags:
- mac-ollama
before_script:
- python3 -m venv .venv
- .venv/bin/python -m pip install --upgrade pip
- .venv/bin/pip install -r requirements.txt
variables:
OLLAMA_HOST: "http://127.0.0.1:11434"
OLLAMA_MODEL: "llama3.2:3b"
PIP_DISABLE_PIP_VERSION_CHECK: "1"
generate_test_cases:
stage: generate
script:
- curl --fail --silent --show-error "$OLLAMA_HOST/api/tags"
- ollama show "$OLLAMA_MODEL"
- .venv/bin/python -m app.run_local_ai
- .venv/bin/python -m app.validate_test_cases
artifacts:
when: always
paths:
- generated/test_cases.json
expire_in: 7 days
validate_and_generate_robot:
stage: validate
script:
- .venv/bin/python -m app.validate_test_cases
- .venv/bin/python -m app.generate_robot_tests
- test -s generated/subscriber_api.robot
artifacts:
when: always
paths:
- generated/test_cases.json
- generated/subscriber_api.robot
expire_in: 7 days
robot_api_tests:
stage: test
script:
- mkdir -p results
- .venv/bin/python -m app.mock_subscriber_api > results/mock_api.log 2>&1 &
- echo $! > results/mock_api.pid
- |
for i in $(seq 1 30); do
if curl --fail --silent http://127.0.0.1:5001/health; then
break
fi
sleep 1
done
- curl --fail --silent http://127.0.0.1:5001/health
- .venv/bin/robot --output results/output.xml --log results/log.html --report results/report.html generated/subscriber_api.robot
after_script:
- if [ -f results/mock_api.pid ]; then kill "$(cat results/mock_api.pid)" 2>/dev/null || true; fi
artifacts:
when: always
paths:
- results/output.xml
- results/log.html
- results/report.html
- results/mock_api.log
expire_in: 7 days
analyze_results:
stage: analyze
when: always
script:
- |
if [ -s results/output.xml ]; then
.venv/bin/python -m app.analyze_robot_results results/output.xml generated/failure_analysis.json
else
echo "No Robot output.xml was produced; skipping analysis."
fi
artifacts:
when: always
paths:
- generated/failure_analysis.json
expire_in: 7 daysImportant before using this pipeline
Verify the analyzer accepts positional arguments
input.xml output.json. If its parser expects flags, update the command accordingly.Later jobs must receive artifacts from earlier jobs.
The runner needs Python, Git, Ollama, the model and access to the local ports.
127.0.0.1only reaches services on the machine running the job.Run the pipeline in a branch before merging changes.
A model-generated test suite should not automatically become a trusted regression suite without semantic checks and review.
12.3 Execute the pipeline
Commit and push the project to GitLab.
Open your GitLab project.
Navigate to Build → Pipelines.
Open the latest pipeline.
Inspect each job's log.
Open the
robot_api_testsjob.Find Job artifacts, Browse, or Download artifacts.
Download the archive and open
results/log.htmlorresults/report.html.
Artifacts are configured to expire after seven days.
13. Errors encountered and their solutions
Error 1: git: command not found
Observed:
/usr/bin/bash: line 182: git: command not found
ERROR: Job failed: exit code 127Cause: Git was missing from the runner environment or not on its PATH.
Check:
which git
git --version
echo "$PATH"Solution on a Homebrew-managed Mac:
brew install gitEnsure the runner service inherits the correct PATH. A service can have a different environment from your interactive Terminal.
Error 2: Ollama connection refused
Check:
curl -v http://127.0.0.1:11434/api/tags
ollama list
ollama show llama3.2:3bSolution: start Ollama, verify the model exists, and ensure the job is executing on the Mac's mac-ollama Shell Runner.
Error 3: ModuleNotFoundError: No module named 'app'
Cause: the command was run from the wrong directory.
Solution:
cd ~/agentic-test-engineer
source .venv/bin/activate
python -m app.run_local_aiUse package-module execution from the project root.
Error 4: Invalid JSON or schema validation failure
Check:
python -m app.run_local_ai
python -m app.validate_test_cases
python -m json.tool generated/test_cases.jsonSolution: inspect the exact JSON or schema error, then fix the prompt, schema or output handling. Do not bypass validation to force a green pipeline.
Error 5: Generated scenario not recognized
Examples:
SubscriberIdUnique
Create Subscriber with Mandatory PhoneNumberCause: model-generated names vary, while the original classifier supported a limited set of names.
Solution: normalize names and add reviewed aliases for unambiguous scenarios. Verify the expected status and preconditions. Reject ambiguous or unsupported scenarios.
Error 6: Duplicate ID and missing phone number in one test
Example:
ExistingSubscriberIdConflictNoPhoneNumberCause: the requirements do not define which validation takes precedence.
Solution: clarify the contract or test the conditions independently. Do not guess whether HTTP 409 or 422 should win.
Error 7: HTTP 400 or 422 generated without a requirement
Cause: the model inferred an expected response not specified in the requirements.
Solution: confirm the API contract, or explicitly label and approve the behavior as an assumption. A mock API response does not establish the real API contract.
Error 8: Robot cannot connect to Flask
Check:
curl -i http://127.0.0.1:5001/health
lsof -nP -iTCP:5001 -sTCP:LISTENSolution: start the mock API, inspect results/mock_api.log, and ensure the API and Robot process run on the same host.
Error 9: Where is log.html?
Local execution:
open results/log.html
open results/report.htmlGitLab execution:
Build → Pipelines → pipeline → robot_api_tests → Job artifacts.
Download or browse the artifact archive and open the report files.
Error 10: AI analysis invents a root cause
Cause: the LLM produced a plausible explanation without sufficient evidence.
Solution: parse Robot XML, capture the actual failure message and keyword, classify known patterns deterministically, and keep the application root cause UNKNOWN until evidence confirms it.
Error 11: Controlled failure exits with a nonzero status
Cause: the test was deliberately written to fail.
Solution: this is expected for the controlled-failure experiment. Keep it separate from the regular passing suite.
Error 12: Artifacts missing in a later job
Check:
ls -lh generated resultsSolution: ensure the producing job creates the files and declares them under artifacts:paths. Check artifact downloading configuration and expiry.
Error 13: Kubernetes localhost:8080 refused
This occurred in the separate Kubernetes practice environment, not in the core Ollama/Flask workflow.
Check:
kubectl config get-contexts
kubectl config current-context
echo "$KUBECONFIG"
ls -l ~/.kube/configIf the intended context exists, select it:
kubectl config use-context docker-desktop
kubectl cluster-info
kubectl get nodesOnly use docker-desktop if it appears in kubectl config get-contexts.
14. Git workflow
From the project root:
cd ~/agentic-test-engineer
git status
git remote -v
git branch --show-currentFor a new repository:
git init
git add .
git status
git commit -m "Build AI-assisted Robot Framework test pipeline"Add the actual GitLab repository URL:
git remote add origin <YOUR_GITLAB_REPOSITORY_URL>
git branch -M main
git push -u origin mainIf origin already exists, inspect it with git remote -v instead of adding it again.
Do not commit credentials, runner tokens or sensitive test data.
15. Full local execution checklist
Start Ollama and verify that the model is installed. Start the mock API in Terminal 1 when ready to execute Robot tests.
In Terminal 2:
cd ~/agentic-test-engineer
source .venv/bin/activate
# 1. Check Ollama
curl --fail --silent --show-error \
http://127.0.0.1:11434/api/tags
# 2. Generate test cases
python -m app.run_local_ai
# 3. Validate generated JSON
python -m app.validate_test_cases
# 4. Generate Robot tests
python -m app.generate_robot_tests
# 5. Check mock API health
curl --fail --silent \
http://127.0.0.1:5001/health
# 6. Execute Robot tests
mkdir -p results
.venv/bin/robot \
--output results/output.xml \
--log results/log.html \
--report results/report.html \
generated/subscriber_api.robot
# 7. Open reports
open results/log.html
open results/report.html
# 8. Inspect generated files
ls -lh generated resultsThe mock API must be running before step 6.
16. Final verification checklist
Ollama installed and running.
llama3.2:3bdownloaded.Python virtual environment created.
Required packages installed.
Requirements documented.
AI generates
generated/test_cases.json.JSON Schema validation passes.
Scenario generator rejects unknown and ambiguous cases.
Expected status codes are backed by requirements or approved assumptions.
Flask
/healthreturns HTTP 200.Robot Framework executes the generated suite.
XML and HTML reports are produced.
Controlled failure analysis is tested separately.
Failure analysis distinguishes facts from hypotheses.
GitLab Shell Runner is online and tagged
mac-ollama.The runner can reach local Ollama.
CI artifacts include test JSON, Robot suite, XML, HTML reports and mock API logs.
No credentials are committed.
17. Next improvements
Add requirement IDs to every test case for traceability.
Add semantic validation beyond JSON Schema.
Require human approval before generated tests enter the trusted regression suite.
Use OpenAPI or another formal API contract when available.
Pin dependency versions for reproducible CI.
Isolate test data before running tests in parallel.
Add unit tests for every classifier alias and status mismatch.
Run deterministic failure classification before LLM-based analysis.
Require independent evidence before declaring a root cause.
Extend the same architecture to telecom scenarios such as SIP registration, call setup, MO/MT flows, and HTTP/2 SBA APIs after defining protocol-specific requirements.
Final principle
Requirements define expected behavior. AI proposes test cases. Validators reject unsupported proposals. Deterministic templates generate automation. Robot Framework provides execution evidence. Failure analysis separates observations from hypotheses. GitLab CI preserves the artifacts for review and debugging.
That is the foundation of a reliable AI-assisted test engineering system.
Important note
This runbook consolidates the workflow and errors we discussed. Some scripts—especially app/generate_robot_tests.py and app/analyze_robot_results.py—were developed iteratively, and their exact latest contents are not all present in this conversation. The runbook documents their responsibilities and the verified patterns, but it should not be treated as a byte-for-byte backup of every version you currently have on your Mac.
For a fully reproducible backup of your actual project, run this from Terminal:
cd ~/agentic-test-engineer
find app input generated results -maxdepth 2 -type f -printThen create an archive of your project, excluding the virtual environment and Git metadata:
cd ~
tar --exclude='agentic-test-engineer/.venv' \
--exclude='agentic-test-engineer/.git' \
-czf agentic-test-engineer-backup.tar.gz \
agentic-test-engineerThat archive will preserve the scripts and configuration files that actually exist in your current working copy, which is the safest way to retain the precise implementation you built.
