Introduction: Why I Started Experimenting with AI in Testing
Over the past year, there’s been a lot of noise around AI in software development especially in testing. Tools promise autonomous QA, self-healing tests, and intelligent pipelines.
But in reality, most teams (including mine) are still juggling:
- Jira tickets
- Test cases in separate tools
- CI/CD pipelines
- And a lot of manual glue in between
So I started experimenting.
This blog post is not a “final solution” or polished framework. It’s an MVP (minimum viable product) a practical, working experiment to explore:
What happens if we treat Jira as the single source of truth and let AI handle the rest of the testing and development pipeline?
The result is an early-stage AI-native test-driven development (TDD) pipeline where:
- Humans define intent in Jira
- AI agents generate and run tests
- Results flow back automatically
It’s not perfect, but it’s already surprisingly powerful.
The Core Idea: AI-Driven Testing Starts with One Source of Truth
One of the biggest problems in testing today is fragmentation.
Test intent lives everywhere:
- Jira tickets
- Confluence pages
- Test management tools
- Developer notes
This MVP flips that model completely:
Jira becomes the single place where both humans and AI agents understand what needs to be tested.
Instead of syncing tools, copying requirements, or maintaining separate test suites:
- The Jira ticket is the test definition
- The AI agent reads it directly
- The pipeline executes based on it
This is the foundation of AI-driven testing at scale.
The MVP Architecture (Simple but Powerful)
To keep things practical, I structured this as a lean 3-layer MVP.
1. Task Discovery (Jira → AI Agent)
Everything starts in Jira.
A typical ticket might include:
- A clear test scenario
- Acceptance criteria
- Environment details
- Success conditions
The key difference?
The ticket is written in a way that AI can understand it, not just humans.
Why Jira works well here:
- It already exists in most teams
- It has structured fields + APIs
- It provides history and audit logs
- Everyone already uses it daily
2. Test Generation & Execution (AI Agent → Test Framework)
Once a ticket is picked up, the AI agent:
- Reads the Jira ticket via API
- Extracts the testing intent
- Generates test code (e.g. pytest, Playwright)
- Runs the test against the target environment
- Captures results (logs, XML reports, screenshots)
This is where AI in automated testing really starts to shine:
- No manual test writing
- No duplication of effort
- Faster feedback loops
3. Result Sync (Test Framework → Jira)
After execution, everything flows back into Jira:
- Test results (pass/fail)
- Execution logs
- Metrics
- Attachments (screenshots, reports)
This closes the loop:
Jira is no longer just planning work, it’s actively tracking test execution in real time.
What Makes This an MVP (and Not a Final Solution)
It’s important to be clear:
This setup is experimental.
Right now, it’s:
- A collection of scripts and transformers
- A simple agent loop (not fully autonomous yet)
- A proof-of-concept for AI-driven TDD
It’s missing things like:
- Robust orchestration
- Advanced error handling
- Scalable agent infrastructure
- Smart prompt optimization
But that’s the point.
This MVP is about learning what works before over-engineering the system.
Key Components (The “Glue” That Makes It Work)
Instead of building a massive platform, I focused on small, composable pieces.
1. Jira → Test Case Transformer
This script converts a Jira ticket into something an LLM can use:
def jira_to_ai_prompt(ticket_key: str) -> dict:
"""Convert a Jira ticket into an AI-ready prompt."""
It:
- Parses the ticket
- Extracts requirements
- Formats them into a structured prompt
This is critical because it decouples Jira from the AI model.
2. Test Execution Layer
The AI doesn’t reinvent testing, it uses existing frameworks, for example in my current usecase:
- pytest
- Playwright
This keeps things stable and familiar to be reused and verifed by test engineers if needed.
3. JUnit → Xray Normalization
Since Jira (via Xray) expects specific formats, I added a preprocessing step:
- Inject test keys
- Normalize descriptions
- Fix formatting issues
This avoids duplicate tests and broken imports.
4. Format Conversion Utilities
Small scripts handle:
- XML cleanup
- Metadata alignment
- Schema conversion
Nothing fancy but essential for reliability.
The Workflow: From Jira Ticket to Executed Test
Here’s what a full cycle looks like in practice:
- Product owner or system specialist generates the initial jira ticket (with help of AI tools such as: https://github.com/mattpocock/skills/blob/main/grill-me/SKILL.md)
- The QA agent detects a “ready” development task
- The ticket is parsed into a structured prompt that generates TDD criteria for the ticket and moves the ticket to Ready for development
- QA Agent generates the test code
- Designer&Development agent takes the ticket from Ready for development and generates code to pass the tests
- The QA agent detects a “ready” testing task
- Tests run automatically
- Results are processed and normalized
- Jira is updated with outcomes
- Feedback is logged for iteration
The key idea:
The Jira ticket becomes the conversation between humans and AI. All development and test code live in the same mono repositorty
How This Changes Test-Driven Development (TDD)
Traditional TDD:
- Write test → fail → fix → refactor
AI-driven TDD (with this MVP):
- Describe intent in Jira
- AI generates code and runs the test
- Feedback loops automatically
- Humans focus on improving logic, not writing boilerplate
It becomes:
Human intent + AI execution = faster, scalable TDD
Why This Approach Matters (Especially for AI in Testing)
Even in MVP form, a few things became obvious.
1. Less Manual Test Maintenance
No separate test management system needed.
2. Faster Feedback Loops
Tests can be generated and executed in minutes.
3. Built-in Traceability
Everything links back to the Jira ticket.
4. Easy to Scale
More agents = more throughput.
Challenges I Ran Into (And What to Watch Out For)
This approach isn’t without issues.
AI Hallucination in Tests
Sometimes generated tests don’t match intent.
Mitigation:
- Structured prompts
- Validation layers
- Human review when needed
Duplicate Test Cases
Missing metadata can cause duplication in Xray.
Fix:
- Inject test keys during preprocessing
Race Conditions
Multiple agents picking the same ticket.
Fix:
- Use Jira workflows and locking strategies
Where This Could Go Next
If this MVP proves useful, the next steps are clear:
Short Term
- Add a proper agent loop
- Improve prompt quality
- Automate result syncing
Mid Term
- Add analytics (test coverage, performance)
- Detect flaky tests
- Introduce smarter retries
Long Term
- Self-improving agents
- Domain-specific test generation
- Full CI/CD integration
Final Thoughts: Is AI-Driven Testing Actually Practical?
I went into this expecting a cool experiment.
What I found instead is something more interesting:
Even a simple MVP can meaningfully reduce friction in testing workflows.
It’s not replacing QA engineers.
It’s not fully autonomous.
It’s not production-ready.
But it does show that:
- AI can understand test intent
- AI can generate useful tests
- Jira can act as a real execution hub—not just a backlog
If You’re Exploring AI in Testing
My recommendation:
Don’t start with a big platform.
Start small:
- Pick one workflow
- Use Jira as your source of truth
- Add a simple AI layer
- Iterate
That’s exactly what this MVP is.
TL;DR
- This is an MVP experiment in AI-driven testing
- Jira is used as the single source of truth
- AI agents:
- Read tickets
- Generate tests
- Execute them
- Report results back
- It’s not production-ready but it’s a strong foundation