PyPI: crawlee

CVE-2026-46497

Safety vulnerability ID: SFTY-20260521-65256

Safety legacy ID: pyup.io-98856

Affected versions of the crawlee package are vulnerable to Server-Side Request Forgery due to missing host and scheme validation on URLs derived from sitemap and robots.txt content. The SitemapRequestLoader, RobotsTxtFile.get_sitemaps(), and the _fetch_and_process_sitemap helper in src/crawlee/_utils/sitemap.py and src/crawlee/_utils/robots.py enqueue or directly dispatch sitemap entries through the configured HTTP client without invoking the Pydantic validate_http_url check, and the CurlImpersonateHttpClient backend additionally accepts non-http(s) schemes such as gopher, file, dict, and ftp. An attacker controlling a remote sitemap or robots.txt can coerce the crawler into issuing requests to internal hosts or to libcurl-supported non-HTTP protocols, enabling cross-host SSRF and protocol smuggling.

PythonPyPINVDNVDGitHubGHSA
Created at: Jun 10, 2026Updated at: Jun 10, 2026

Overview

Crawlee for Python: SSRF via sitemap-derived URLs

Advisory

Affected versions of the crawlee package are vulnerable to Server-Side Request Forgery due to missing host and scheme validation on URLs derived from sitemap and robots.txt content. The SitemapRequestLoader, RobotsTxtFile.get_sitemaps(), and the _fetch_and_process_sitemap helper in src/crawlee/_utils/sitemap.py and src/crawlee/_utils/robots.py enqueue or directly dispatch sitemap entries through the configured HTTP client without invoking the Pydantic validate_http_url check, and the CurlImpersonateHttpClient backend additionally accepts non-http(s) schemes such as gopher, file, dict, and ftp. An attacker controlling a remote sitemap or robots.txt can coerce the crawler into issuing requests to internal hosts or to libcurl-supported non-HTTP protocols, enabling cross-host SSRF and protocol smuggling.

Affected Package

Affecting crawlee package, versions>=1.0.0,<1.7.0

Also affects

---

How to Fix

Upgradecrawleeto1.7.0or higher.

Mitigation and Workarounds

---

Vulnerable Functions

Functions linked to known vulnerabilities.

Vulnerable function data is available for Enterprise customers

Book a call with us to see Safety in action.

References

Safety

Verified by Safety

Our Cybersecurity Intelligence Team reviewed this vulnerability. We combine public data with our own research to find issues not yet reported to public sources.

Learn more