Detecting bots
3 min read
When a request reaches Cloudflare, we run that request through a variety of detection engines to determine whether it came from a bot.
These detections run on all requests coming into the Cloudflare global network, not just for applications using Bot Management.
This means that:
- Adding bot management does not add latency to requests.
- Our bot management platform benefits from the millions of sites and trillions of requests that flow thorugh Cloudflare every day.
Heuristics
The Heuristics engine processes all requests. Cloudflare conducts a number of heuristic checks to identify automated traffic, and requests are matched against a growing database of malicious fingerprints.
The Heuristics engine immediately gives automated requests a score of 1.
Machine learning
The Machine Learning (ML) engine accounts for the majority of all detections, human and bot.
This approach leverages our global network, which proxies billions of requests daily, to identify both automated and human traffic. We constantly train the ML engine to become more accurate and adapt to new threats. Most importantly, this engine learns from traffic across all Cloudflare domains and uses these insights to score traffic while honoring our strict privacy standards.
The ML engine produces scores 2 through 99.
Anomaly detection
The Anomaly Detection (AD) engine is an optional detection engine that uses a form of unsupervised learning. Cloudflare records a baseline of your domain’s traffic and uses the baseline to intelligently detect outlier requests. This approach is user agent-agnostic and can be turned on or off by your account team.
Cloudflare does not recommend AD for domains that use SSL for SaaS or expect large amounts of API traffic. The AD engine immediately gives automated requests a score of one.
JavaScript detections
The JavaScript Detections (JSD) engine identifies headless browsers and other malicious fingerprints. This engine performs a lightweight, invisible JavaScript injection on the client side of any request while honoring our strict privacy standards. We do not collect any personally identifiable information during the process. The JSD engine either blocks, challenges, or passes requests to other engines.
JSD is enabled by default but completely optional. To adjust your settings, open the Bot Management Configuration page from Security > Bots.
___cf_bm cookie
Cloudflare uses the __cf_bm cookie to smooth out the Bot Score and reduce false positives for actual user sessions.
The Bot Management cookie measures a single user’s request pattern and applies it to the machine learning data to generate a reliable Bot Score for all of that user’s requests.
For more details, refer to Cloudflare Cookies.
Unit 3 of 3