From a13caa3a8f09a0daf9a2b67b77af3314e9638674 Mon Sep 17 00:00:00 2001 From: Miao Liu Date: Tue, 1 Sep 2026 16:52:30 +0200 Subject: [PATCH 1/2] Add feedback proposal for FlakyGuard (PR #2935) --- .../feedback/miaoli-jingze/README.md | 22 +++++++++++++++++++ 1 file changed, 22 insertions(+) create mode 100644 contributions/feedback/miaoli-jingze/README.md diff --git a/contributions/feedback/miaoli-jingze/README.md b/contributions/feedback/miaoli-jingze/README.md new file mode 100644 index 000000000..77a4b1b9e --- /dev/null +++ b/contributions/feedback/miaoli-jingze/README.md @@ -0,0 +1,22 @@ +# Assignment Proposal + +## Title + +Feedback on Scientific paper - FlakyGuard: Automatically Fixing Flaky Tests at Industry Scale + +## Names and KTH ID + + - Miao Liu (miaoli@kth.se) + - Jingze Guo (jingze@kth.se) + +## Deadline + +- Task 3 + +## Category + +- Feedback + +## Description + +We have contacted the responsible group (Jiaxun Wei (jiaxun@kth.se) and Matyas Kozar (kozar@kth.se)) for the paper 'FlakyGuard: Automatically Fixing Flaky Tests at Industry Scale' and have gotten the approval from them to provide feedback. https://github.com/KTH/devops-course/pull/2935 From 5b3d6d6ede95023aee315e8d86c17827ed4638e0 Mon Sep 17 00:00:00 2001 From: Miao Liu Date: Wed, 9 Sep 2026 21:50:22 +0200 Subject: [PATCH 2/2] Add executable tutorial proposal: RAG Quality Gates with RAGAS --- .../miaoli-shunkang/README.md | 27 +++++++++++++++++++ 1 file changed, 27 insertions(+) create mode 100644 contributions/executable-tutorial/miaoli-shunkang/README.md diff --git a/contributions/executable-tutorial/miaoli-shunkang/README.md b/contributions/executable-tutorial/miaoli-shunkang/README.md new file mode 100644 index 000000000..8dd4ca843 --- /dev/null +++ b/contributions/executable-tutorial/miaoli-shunkang/README.md @@ -0,0 +1,27 @@ +# Assignment Proposal + +## Title + +RAG Quality Gates: Automated Evaluation Pipelines with RAGAS + +## Names and KTH ID + + - Miao Liu (miaoli@kth.se) + - Shunkang Jia (shunkang@kth.se) + +## Deadline + +- Task 2 + +## Category + +- Executable tutorial + +## Description + +We will build a Google Colab notebook that shows how to automatically test whether a RAG (Retrieval-Augmented Generation) application's answer quality holds up before a change is merged. The reader sets up a small RAG system with LangChain and FAISS, then runs RAGAS on a provided test set of questions with reference answers (and adds a couple of their own) to score retrieval and generation quality (faithfulness, answer relevancy, context precision, context recall). The reader then deliberately weakens the retrieval step, watches the scores drop, and finally wires the evaluation into a CI step that blocks the change if the scores fall below a threshold. + +**Relevance** + +A RAG application's quality depends heavily on its retrieval setup, such as the chunking strategy, the embedding model, and the retrieval parameters. These get changed often, but unlike application code, there is no test that catches it when such a change quietly makes answers worse. RAGAS turns answer quality into something measurable, making it possible to apply the same continuous integration discipline (automated checks that gate every change) to the retrieval and generation layer of an LLM application. +