Massive (@joinmassive) header
Massive (@joinmassive) profile picture

Massive

@joinmassive

Real-time web access for your AI

joinmassive.com/
Joined September 2018
  • Tweets255
  • Following3
  • Followers1.5K
86Media
Photo by Massive (@joinmassive)Photo by Massive (@joinmassive)Photo by Massive (@joinmassive)Photo by Massive (@joinmassive)Photo by Massive (@joinmassive)Photo by Massive (@joinmassive)Photo by Massive (@joinmassive)Photo by Massive (@joinmassive)Photo by Massive (@joinmassive)Photo by Massive (@joinmassive)
We pointed 17 scraping tools at our own server. 10 named themselves 2 said HeadlessChrome 4 looked like a browser 1 sent no user agent Only 1 of 19 setups read robots.txt. Link in the comments
Image from Massive (@joinmassive)
1
2
79
joinmassive.com/blog/web-scr…
Web Scraper User Agents: What 17 Tools Send (Tested 2026)

Web Scraper User Agents: What 17 Tools Send (Tested 2026)

We tested the user agent and robots.txt behavior of 17 web scraping tools. 10 name themselves, 2 say HeadlessChrome, and 1 of 19 setups read robots.txt.

joinmassive.com
16
We ran 10 searches through @youdotcom's keyless MCP server. 10 of 10 came back. You.com is a Massive customer. Our spotlight covers its endpoints, pricing, rate limits, and the free MCP profile.
2
1
7
277
joinmassive.com/blog/youcom-…
You.com Search API: Pricing, MCP Server, Tested

You.com Search API: Pricing, MCP Server, Tested

The You.com Search API returns LLM-ready web results for $5 per 1,000 calls. See its endpoints, pricing, keyless MCP server, and x402 payments, tested.

joinmassive.com
1
38
"Is web scraping legal?" depends on where you ask. Reading public pages is generally fine. Everything around it differs by country. How 8 countries and the EU compare:
Image from Massive (@joinmassive)
2
5
92
more replies
U.S.: public pages aren't hacking after Van Buren (2021). The open question is bypassing bot defenses: two federal courts reached different early rulings on DMCA claims in July 2026.
1
16
joinmassive.com/blog/is-web-…
Is Web Scraping Legal? 2026 Guide: 8 Countries + EU

Is Web Scraping Legal? 2026 Guide: 8 Countries + EU

Is web scraping legal? Reading public pages usually is. Privacy, copyright, contracts, and bypassing blocks decide the rest. 8 countries, the EU, 2026 rulings.

joinmassive.com
11
Cloudflare now lets sites set rules for each AI agent, not just "bots." New: a Personal Assistants analytics view, plus allow or block per agent. Its advice? Allow them.
Image from Massive (@joinmassive)
1
57
Cloudflare's post: blog.cloudflare.com/personal… Its docs recommend Allow for personal agents: developers.cloudflare.com/bo…
See the agents, set the terms: new visibility and control for personal agents

See the agents, set the terms: new visibility and control for personal agents

Personal AI agents have changed how users interact with the web. Learn how Cloudflare gives site owners visibility and control to manage personal agent access.

blog.cloudflare.com
21
An EU court adviser says chatbots reusing news articles in real time may need publishers' permission. Not training. Answers. AG opinion in Like Company v. Google, Nov 3. Not binding yet.
Image from Massive (@joinmassive)
1
65
Court of Justice press release: curia.europa.eu/site/jcms/p1…
21
Scrapling has 100,136 GitHub stars. Scrapy has 65,286. Scrapy got 4.7x the downloads last month. Stars are attention. Downloads are deployments.
Image from Massive (@joinmassive)
Image from Massive (@joinmassive)
1
3
158
17 of the 42 tools were created since 2024. They hold 58% of the stars. AI-ready crawlers, browser agents, and MCP servers are where the new attention went.
1
1
28
joinmassive.com/blog/open-so…
Open-Source Web Scraping Tools: 42 Top GitHub Repos (2026)

Open-Source Web Scraping Tools: 42 Top GitHub Repos (2026)

42 open-source web scraping tools on GitHub, ranked by stars across 8 categories, with licenses and monthly downloads. Snapshot taken November 9, 2026.

joinmassive.com
20
We downloaded the IP lists that 8 major AI bot operators publish. 14 lists, about 94,000 IPv4 addresses. ChatGPT-User's covers 69,280. GPTBot's: 2,704. A user agent is a claim. The IP list is how you check it.
Image from Massive (@joinmassive)
Image from Massive (@joinmassive)
1
1
3
160
23 AI bots from 9 operators, and how to verify each: joinmassive.com/blog/ai-craw…
AI Crawler List 2026: Verified User Agents and IP Ranges

AI Crawler List 2026: Verified User Agents and IP Ranges

AI crawler list 2026: 23 user agents from 9 operators, what each does, and the 14 published IP lists (about 94,000 IPv4 addresses) that verify them.

joinmassive.com
1
70
Wikimedia told a federal court on Oct 29: training an LLM is likely fair use. Copying pages into chatbot answers is not. News sites' robots.txt files draw the line the other way.
Image from Massive (@joinmassive)
2
2
125
more replies
The brief, in the consolidated OpenAI copyright case in New York, backs neither side, and the court hasn't ruled on it. Its fix for retrieval: follow the source's license, like Wikipedia's CC BY-SA credit. Not legal advice.
1
41
The brief: commons.wikimedia.org/wiki/F… What 49 news sites block, bot by bot: joinmassive.com/blog/should-…

File:26.10.29 WMF motion and amicus In re OAI.pdf - Wikimedia Commons

commons.wikimedia.org
37
Load more