deerflow2/tests/unit/tools
Willem Jiang 975b344ca7
fix: resolve issue #651 - crawl error with None content handling (#652)
* fix: resolve issue #651 - crawl error with None content handling
Fixed issue #651 by adding comprehensive null-safety checks and error handling to the crawl system.
The fix prevents the ‘TypeError: Incoming markup is of an invalid type: None’ crash by:
1. Validating HTTP responses from Jina API
2. Handling None/empty content at extraction stage
3. Adding fallback handling in Article markdown/message conversion
4. Improving error diagnostics with detailed logging
5. Adding 16 new tests with 100% coverage for critical paths

* Update src/crawler/readability_extractor.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update src/crawler/article.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-10-24 17:06:54 +08:00
..
test_crawl.py fix: resolve issue #651 - crawl error with None content handling (#652) 2025-10-24 17:06:54 +08:00
test_decorators.py feat: 1. replace black with ruff for fomatting and sort import (#489) 2025-08-17 22:57:23 +08:00
test_python_repl.py feat: 1. replace black with ruff for fomatting and sort import (#489) 2025-08-17 22:57:23 +08:00
test_search.py Fix: clarification bugs - max rounds, locale passing, and over-clarification (#647) 2025-10-24 16:43:39 +08:00
test_search_postprocessor.py feat: Add intelligent clarification feature in coordinate step for research queries (#613) 2025-10-14 13:35:57 +08:00
test_tavily_search_api_wrapper.py fix: correct image result format for OpenAI compatibility to fix #632 (#634) 2025-10-20 23:14:09 +08:00
test_tavily_search_results_with_images.py fix: resolve issue #467 - message content validation and Tavily search error handling (#645) 2025-10-23 22:08:14 +08:00
test_tools_retriever.py feat: 1. replace black with ruff for fomatting and sort import (#489) 2025-08-17 22:57:23 +08:00