B.S. Computer Science
Carnegie Mellon University, Pittsburgh, PA
AI Evaluation Specialist & LLM Red-Teamer
Evaluating & Red-Teaming Large Language Models
Computer Science graduate from Carnegie Mellon University specializing in designing adversarial evaluation prompts, auditing LLM responses for factual & logical errors, and documenting findings with precise, structured written reports.
1,200+ Assignments Completed (>95% Approval)
About Me
AI Evaluation Specialist with a CS degree from Carnegie Mellon University focused on evaluating, red-teaming, and fine-tuning Large Language Models.
Professional Summary
Hi, I’m Richard Benson — LLM Red-Teamer & AI Quality Evaluation Expert
I hold a B.S. in Computer Science from Carnegie Mellon University (2011–2015). My career centers on evaluating and red-teaming large language models (LLMs) to ensure factual accuracy, logical consistency, safety, and domain relevance.
Having authored 180+ technical evaluation reports (where over 40% led to measurable model improvements in subsequent releases), I establish robust inter-annotator agreement protocols (achieving κ = 0.82) and develop comprehensive evaluation rubrics across code, medical reasoning, and complex technical tasks.
LLM Red-Teaming
Adversarial prompting & safety boundary audits.
Response Auditing
Hallucination detection & factual verification.
Rubric Design
Structured grading & multi-tier rubric creation.
Kappa Reliability
Inter-annotator agreement protocols (κ = 0.82).
Career & Education Journey
Software Engineer
Duolingo | Pittsburgh, PA
ML & AI Eval Specialist
Abridge & OmbuLabs
Freelance AI Evaluation
Remote | 1,200+ Assignments
“Elevating AI reliability through rigorous adversarial testing, structured grading rubrics, and empirical error analysis.”
1,200+ Assignments
>95% Approval Rating
180+ Technical Reports
κ = 0.82 Team Reliability
Technical Skills & Core Competencies
Specialized methodologies, tools, and frameworks for evaluating, auditing, and red-teaming LLMs.
Adversarial Prompting & Red-Teaming
ExpertDesigning multi-turn jailbreak prompts, evaluating safety boundaries, and testing model resilience to malicious inputs.
Response Auditing & Hallucinations
ExpertDetecting factual inconsistencies, logical fallacies, and subtle model hallucinations against ground truth data.
Rubric Design & Structured Grading
ExpertCreating standardized evaluation rubrics for code, medical reasoning, technical writing, and domain-specific knowledge.
Fact-Checking & Primary Sources
AdvancedRigorous verification of model-generated text against authoritative academic, clinical, and primary source literature.
Python (Transformers, OpenAI, Pandas)
AdvancedProgrammatic evaluation scripting, batch API processing, dataset visualization, and metric computation.
Inter-Annotator Reliability (κ = 0.82)
AdvancedEstablishing consensus metrics, calculating Cohen's Kappa, and training annotation teams on quality protocols.
Technical Stack & Toolbox
- Python, Pandas, NumPy, Statistics
- OpenAI API, Hugging Face Transformers
- Model Evaluation Frameworks & Benchmarks
- Node.js, REST APIs, Microservices, Git
- Prompt Engineering & Few-Shot Optimization
Evaluation Philosophy
"Effective LLM evaluation requires both deep technical intuition and analytical precision. By combining computer science fundamentals with structured grading rubrics and empirical error taxonomies, I turn subjective output observations into actionable fine-tuning signals for AI engineering teams."
Resume & Experience
Detailed breakdown of my professional background in LLM evaluation, machine learning auditing, and software engineering.
Work History
Over 9 years of combined expertise across software engineering, machine learning evaluation, and adversarial red-teaming.
Freelance AI Trainer / Evaluation Specialist
Remote
2024 – Present
- Evaluate large language model outputs through structured feedback on code, reasoning, and domain-specific tasks.
- Completed 1,200+ assignments with >95% approval rating.
- Authored 180+ evaluation reports identifying model failure patterns; approximately 40% led to measurable improvements in subsequent model releases.
- Established inter-annotator agreement protocols achieving κ = 0.82 across a five-person team.
AI Model Evaluation Specialist
OmbuLabs | Philadelphia, PA
2023 – 2024
- Evaluated custom AI and machine learning models in production for client organizations across multiple domains.
- Conducted rigorous testing of model outputs for accuracy, bias detection, and real-world performance against baseline metrics.
- Authored technical evaluation reports with actionable recommendations for model fine-tuning and optimization.
- Assisted in developing evaluation frameworks for LLM-based applications including prompt optimization and response quality assessment.
Machine Learning Evaluation Specialist
Abridge | Pittsburgh, PA
2021 – 2022
- Evaluated large language model outputs for factual accuracy, logical consistency, and clinical relevance in healthcare documentation.
- Developed structured evaluation rubrics for code, medical reasoning, and clinical knowledge domains.
- Maintained 96% annotation accuracy and trained eight team members on evaluation protocols.
- Collaborated with ML engineers to document failure modes impacting model improvement cycles.
Software Engineer
Duolingo | Pittsburgh, PA
2015 – 2022
- Designed and deployed full-stack features for language-learning platform serving millions of users.
- Built microservices architecture using Python and Node.js, supporting 2x user growth.
- Implemented automated test suites achieving 92% code coverage, reducing production bugs by 34%.
- Optimized API response times by 40%.
- Led team of six engineers on core infrastructure projects.
- Mentored four junior engineers on software design and system architecture.
Educational Background
Rigorous computer science foundation from Carnegie Mellon University.
2011 – 2015
Pittsburgh, PA
B.S. Computer Science
Carnegie Mellon University
Focused on computer science core fundamentals, algorithms, software design, systems architecture, and machine learning principles. Built deep technical foundations that power rigorous adversarial testing, model evaluation, and software quality auditing.
Core Technical Skills
Overview of technical competencies and domains of expertise.
LLM Evaluation & Red-Teaming
Adversarial Prompting
LLM Red-Teaming
Hallucination Detection
Response Auditing
Fact-Checking Against Primary Sources
Bias Detection
Evaluation Frameworks & Quality Protocols
Rubric Design & Structured Grading
Inter-Annotator Agreement (κ = 0.82)
Technical Documentation & Reporting
Prompt Optimization
Failure Mode Taxonomy
Programming, Data & Engineering
Python (Transformers, OpenAI API, Pandas)
Statistics & Data Analysis
Node.js
Microservices Architecture
Automated Test Suites (92% Coverage)
Git & GitHub
Evaluation Case Studies & Projects
Representative examples of LLM red-teaming, clinical factual auditing, rubric design, and evaluation protocols.
- All Projects
- Red-Teaming
- Factual Auditing
- Evaluation Frameworks
- Software Engineering
Red-Teaming
LLM Red-Teaming & Jailbreak Audit
Designed multi-turn adversarial prompts to test safety boundaries, prompt injection resistance, and PII leakage prevention.
Factual Auditing
Clinical Healthcare LLM Accuracy Audit
Evaluated clinical documentation outputs at Abridge for medical reasoning, factual consistency, and 96% annotation accuracy.
Frameworks
Inter-Annotator Agreement Protocol
Established annotation consistency guidelines across a 5-person team, achieving high inter-rater agreement (κ = 0.82).
Factual Auditing
Failure Pattern Taxonomy Reports
Authored 180+ structured technical reports; 40% directly contributed to model quality gains in subsequent releases.
Frameworks
Production Client ML Model Audits
Conducted bias detection, accuracy benchmarking, and fine-tuning recommendations for enterprise client models.
Software Engineering
Microservices & Test Automation
Engineered scalable Python/Node.js microservices and automated test suites at Duolingo achieving 92% code coverage.
Professional Endorsements
Feedback from machine learning leads, engineering managers, and annotation team members.
Services & Offerings
Specialized expert contributor services in LLM evaluation, red-teaming, and model training.
LLM Red-Teaming
Designing adversarial prompts to stress-test safety guardrails, detect prompt injections, and eliminate unwanted output behaviors.
Response Auditing & Fact-Checking
Auditing complex LLM responses for factual accuracy, logical consistency, and source integrity across technical and domain-specific fields.
Rubric & Grading Framework Design
Crafting tailored evaluation rubrics, multi-dimensional scoring matrices, and quality standards for model training pipelines.
Inter-Annotator Agreement Protocols
Setting up guidelines, consensus procedures, and Cohen's Kappa measurement to ensure high multi-annotator reliability.
Technical Evaluation Reporting
Producing exhaustive written reports detailing failure pattern taxonomies, root causes, and actionable fine-tuning recommendations.
Prompt Optimization & Alignment
Refining system prompts, zero/few-shot templates, and chain-of-thought instructions to maximize downstream output quality.
Frequently Asked Questions
Common questions regarding my LLM evaluation expertise, methodology, and engagement types.
01
What sets your LLM evaluation methodology apart?
My work combines formal computer science training (B.S. from CMU) with extensive software engineering and evaluation experience. Rather than superficial grading, I analyze output logic, perform primary source fact-checking, categorize failure modes, and provide structured technical reports that enable engineering teams to fine-tune models effectively.
02
How do you achieve high inter-annotator agreement (κ = 0.82)?
I establish unambiguous rubric definitions, edge-case disambiguation guidelines, and calibration sessions. By measuring inter-annotator agreement via Cohen's Kappa, I ensure consensus and data quality across annotation teams.
03
What domains do you specialize in evaluating?
I specialize in code evaluation, mathematical & logical reasoning, technical documentation, clinical healthcare summaries (from my work at Abridge), and adversarial safety red-teaming.
04
Are you available for contract or expert contributor roles?
Yes, I am actively seeking expert contributor roles in LLM evaluation, red-teaming, and AI model training on both contract and full-time bases.
Contact Me
Get in touch for expert contributor work, LLM evaluation consulting, or red-teaming projects.
Send a Message
Feel free to reach out regarding model evaluation opportunities or technical queries.