* feat(eval): add report quality evaluation module
Addresses issue #773 - How to evaluate generated report quality objectively.
This module provides two evaluation approaches:
1. Automated metrics (no LLM required):
- Citation count and source diversity
- Word count compliance per report style
- Section structure validation
- Image inclusion tracking
2. LLM-as-Judge evaluation:
- Factual accuracy scoring
- Completeness assessment
- Coherence evaluation
- Relevance and citation quality checks
The combined evaluator provides a final score (1-10) and letter grade (A+ to F).
Files added:
- src/eval/__init__.py
- src/eval/metrics.py
- src/eval/llm_judge.py
- src/eval/evaluator.py
- tests/unit/eval/test_metrics.py
- tests/unit/eval/test_evaluator.py
* feat(eval): integrate report evaluation with web UI
This commit adds the web UI integration for the evaluation module:
Backend:
- Add EvaluateReportRequest/Response models in src/server/eval_request.py
- Add /api/report/evaluate endpoint to src/server/app.py
Frontend:
- Add evaluateReport API function in web/src/core/api/evaluate.ts
- Create EvaluationDialog component with grade badge, metrics display,
and optional LLM deep evaluation
- Add evaluation button (graduation cap icon) to research-block.tsx toolbar
- Add i18n translations for English and Chinese
The evaluation UI allows users to:
1. View quick metrics-only evaluation (instant)
2. Optionally run deep LLM-based evaluation for detailed analysis
3. See grade (A+ to F), score (1-10), and metric breakdown
* feat(eval): improve evaluation reliability and add LLM judge tests
- Extract MAX_REPORT_LENGTH constant in llm_judge.py for maintainability
- Add comprehensive unit tests for LLMJudge class (parse_response,
calculate_weighted_score, evaluate with mocked LLM)
- Pass reportStyle prop to EvaluationDialog for accurate evaluation criteria
- Add researchQueries store map to reliably associate queries with research
- Add getResearchQuery helper to retrieve query by researchId
- Remove unused imports in test_metrics.py
* fix(eval): use resolveServiceURL for evaluate API endpoint
The evaluateReport function was using a relative URL '/api/report/evaluate'
which sent requests to the Next.js server instead of the FastAPI backend.
Changed to use resolveServiceURL() consistent with other API functions.
* fix: improve type accuracy and React hooks in evaluation components
- Fix get_word_count_target return type from Optional[Dict] to Dict since it always returns a value via default fallback
- Fix useEffect dependency issue in EvaluationDialog using useRef to prevent unwanted re-evaluations
- Add aria-label to GradeBadge for screen reader accessibility
* feat(web): add multi-format report export (Markdown, HTML, PDF, Word, Image)
* fix: correct import order for docx (lint error)
* fix(web): address Copilot review comments for multi-format export
- Add i18n support for dropdown menu items (en/zh)
- Add DOMPurify for HTML sanitization (XSS protection)
- Fix async handling for canvas.toBlob with Promise wrapper
- Add toast notifications for export errors
- Fix Tooltip + DropdownMenuTrigger nesting (accessibility)
- Ensure container cleanup in finally block
* fix(web): enhance markdown parsing for PDF and Word export
- Add list support (bullet and numbered) for PDF export
- Add parseInlineMarkdown helper for Word export to handle bold, italic, code, links
- Add list support for Word export (bullet and numbered)
- Address Copilot review comments from PR #756
* fix(web): address PR review feedback for multi-format export
- Extract PDF formatting magic numbers into PDF_CONSTANTS
- Add Tooltip wrapper for download dropdown button
- Reduce triggerDownload cleanup timeout from 1000ms to 100ms
- Use marked.Lexer.lexInline for robust markdown parsing
- Add console.warn for image export cleanup errors
- Add numbering config for Word document ordered lists
- Fix CSS class typo: px-5pb-20 -> px-5 pb-20
- Remove unreachable dead code in parseInlineMarkdown
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* refactor: Welcome layout and conditional rendering
Improves flex layout and spacing in ConversationStarter, and updates MessagesBlock to conditionally render ConversationStarter or MessageListView based on chat state. This streamlines the UI and removes redundant rendering logic.
* fix: replay mode
* fix: Remove unnecessary inset-0
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Add defensive checks before removeChild to prevent 'Failed to execute removeChild' error when the element has already been removed from DOM. Wrap URL.revokeObjectURL in finally block to ensure proper resource cleanup.
* fix: improve config loading resilience for non-localhost access (#510)
- Add DEFAULT_CONFIG fallback to always return valid config even if fetch fails
- Implement retry logic with exponential backoff (max 2 retries) to handle transient failures
- Add 5-second fetch timeout to prevent hanging on unreachable backends
- Improve error logging with clear messages about config fetch status
- Always return DeerFlowConfig (never null) to prevent UI rendering issues
- Add safety checks in input-box component to verify reasoning models before access
- Improve type safety: verify array length before accessing array indices
- Add comprehensive documentation in .env.example with examples for different deployment scenarios
- Document NEXT_PUBLIC_API_URL variable behavior and fallback mechanism
* Apply suggestions from code review
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* fix: add nullish coalescing to prevent TypeScript error in input-box
- Add ?? operator to handle potential undefined value when accessing reasoning[0]
- Fixes TS2322 error: Type 'string | undefined' is not assignable to type 'string | number | Date'
---------
Co-authored-by: Willem Jiang <143703838+willem-bd@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Fixes#598
* fix: parsed json with extra tokens issue
* Added unit test for json.ts
* fix the json unit test running issue
* Apply suggestions from code review
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update the code with code review suggestion
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Willem Jiang <143703838+willem-bd@users.noreply.github.com>
* fix: resolve issue #588 - react key warnings from duplicate message IDs + establish jest testing framework
* Update the makefile and workflow with the js test
* Apply suggestions from code review
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
- Backend: Convert non-string content (lists, dicts) to JSON strings in _create_event_stream_message to ensure frontend always receives string content
- Frontend: Add type guard before calling startsWith() on toolCall.result for defensive programming
This fixes the TypeError: toolCall.result.startsWith is not a function when tools return complex objects.
Fixes#570 where browser freezes when research plan has 8+ steps.
Performance optimizations:
- Add animation throttling: only animate first 10 activity items
- Reduce animation durations (0.4s → 0.3s for activities, 0.2s → 0.15s for results)
- Remove scale animations (GPU-intensive) from search results
- Limit displayed results (20 pages, 10 images max)
- Add conditional animations based on item index
- Cap animation delays to prevent excessive staggering
- Add React.memo to ActivityMessage and ActivityListItem components
These changes significantly improve performance when rendering multiple
research steps while maintaining visual appeal for smaller lists.
* fix: prevent repeated content animation during thinking streaming (#614)
- Implement chunked rendering using reasoningContentChunks
- Static content (previous chunks) renders without animation
- Only current streaming chunk animates
- Disable animation on plan content (title, thought, steps) during streaming
- Animation applies after content finishes streaming (when complete)
- Prevents visual duplication of repeated sentences in thinking process
- Changed key from question text to combination of index and question text
- Ensures unique keys even if translation has duplicate questions
- Resolves React warning: 'Each child in a list should have a unique key prop'
* feat: Implement MilvusRetriever with embedding model and resource management
* chore: Update configuration and loader files for consistency
* chore: Clean up test_milvus.py for improved readability and organization
* feat: Add tests for DashscopeEmbeddings query and document embedding methods
* feat: Add tests for embedding model initialization and example file loading in MilvusProvider
* chore: Remove unused imports and clean up test_milvus.py for better readability
* chore: Clean up test_milvus.py for improved readability and organization
* chore: Clean up test_milvus.py for improved readability and organization
* fix: replace print statements with logging in recursion limit function
* Implement feature X to enhance user experience and optimize performance
* refactor: clean up unused imports and comments in AboutTab component
* Implement feature X to enhance user experience and fix bug Y in module Z
---------
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
* feat: implment backend for adjust report style
* feat: add web part
* fix test cases
* fix: fix typing
---------
Co-authored-by: Henry Li <henry1943@163.com>
* feat: local search tool call result display
* chore: add file copyright
* fix: miss edit plan interrupt feedback
* feat: disable pasting html into input box
* feat: show repo star on site-header
* feat: add GITHUB_OAUTH_TOKEN to environment configuration
* feat: remove comment
* feat: show star counter only in website
Willem Jiang
Jiahe Wu
agoudbg
CHANGXUBO
道心坚定韩道友
johnny0120
JeffJiang
DanielWalnut
Muharrem Okutan
SToneX
Leo Hui
Henry Li
Nonoroazoro
Li Xin