21. founder @buildonto: your next.js site answers ai agents in clean markdown, humans get the full page. also @opendevsociety, openstock (21k github stars)
bengaluru, india
Joined June 2023
- Tweets2.3K
- Following385
- Followers956
- Likes3.9K
does this catch sites that block GPTBot at the CDN or firewall too? those look totally fine in Chrome either way
1
13
does it also check what comes back when a crawler sends Accept: text/markdown? some sites now serve a clean markdown version on the same URL, so the JS problem disappears for agents
1
2
32
do you think stores will end up keeping a separate agent-facing catalog, or just make the same product page readable for both?
1
1
12
do agents that send Accept: text/markdown to the normal page URL get the md version too, or only if they find the .md link?
1
12
did it get past App Store Connect's 2FA prompt on its own or did you have to jump in for that?
1
1
155
contribute here: github.com/Open-Dev-Society/…
1
67
is the Link header from v2 something agents actually check yet, or do they still just try /llms.txt at the root?
1
8
robots.txt can split them. block GPTBot, ClaudeBot and Google-Extended (those are the training ones) and allow OAI-SearchBot, ChatGPT-User and PerplexityBot so you still show up in answers. is the 7,400 hits one even respecting robots or is it unnamed?
45
for 18, do you check just robots.txt or also cloudflare's bot settings? those block AI crawlers by default on a lot of sites now
1
1
30
how do you think smaller d2c stores get picked by agents though? amazon can cut a deal, a shopify store with 200 products probably can't
1
45
does the Cloudflare default block also catch the bots fetching a page live when someone asks ChatGPT about it, or just the training crawlers?
18
Partly, Stop takes down the whole process group, so watchers inside that group goes too but It doesn't stop anything outside the group that might start the server again. so, Ports can say "Stopped" when the server is about to come back, but fixing that in the next build for sure !!
9
