Howdy Readers 🥰
In this newsletter, you’ll find:
🚦 A black friday runbook for slowing crawlers without losing rankings
🛒 AI shopping agents can now find products, build carts and complete payments
🏆 Ad of the day
If you’re new to ScaleUP then a hearty welcome to you. You’ve reached the right place along with 50k+ CEOs, CMOS, and marketers. Let’s get into it, shall we? Oh! Before you forget, if someone forwarded this newsletter to you, don't forget to subscribe to our newsletter so you never miss out!
🚦 A black friday runbook for slowing crawlers without losing rankings
At 9 a.m. on Black Friday, every server cycle should go to shoppers. Crawlers don’t know that.
On October 6, Search Engine Roundtable reported a notable addition to Google’s crawl rate documentation: when your server returns a 503 or 429 status code, you can include a Retry-After header telling Google’s crawlers when to come back, either as a delay in seconds or an exact UTC date and time.
The mechanism isn’t new. Google already treats 500, 503 and 429 responses as a signal to crawl less. The limits are what matter. Google says to use this for a short period, such as a couple of hours or one to two days, and warns that if Googlebot sees these codes on the same URL for multiple days, the URL may be dropped from the index.
This runbook is for stores that control their own servers or CDN, such as headless or self-hosted builds.
This month: Prepare
Measure the bot share. Split 30 days of server logs by user agent, and verify that anything claiming to be Googlebot really is, using Google’s published IP ranges.
Pick a trigger. Choose the load level that means trouble, such as response times doubling or CPU above 80%. The 80% figure is my judgment.
Build the switch now. Set up an edge or CDN rule that returns 503 with a Retry-After header to verified crawlers only. Shoppers must never see it. Test it on a staging site.
On the day: Two levels
Level 1, load rising. Throttle crawler requests to low-value URLs only: internal search results, sort and filter parameters, pagination.
Level 2, load critical. Return 503 with “Retry-After: 3600” to all crawler requests. One hour is my judgment.
Keep product pages open as long as you can at Level 1. Ad platforms check landing pages, and an unavailable one mid-sale risks your shopping ads, in my judgment.
The hard stop
Turn every rule off within 24 hours of switching it on. Never let one run past 48 hours, the edge of Google’s own one-to-two-day window. Give it a named owner.
Skip robots.txt as an emergency brake. It blocks pages outright instead of slowing visits, and Google’s emergency guidance relies on status codes.
The week after: Check
Open Crawl stats in Search Console and look for host status problems on the peak days.
Compare indexed page counts before and after.
Spot-check your top 20 product pages with URL Inspection.
Your first task
Pull last month’s logs and find what share of requests came from bots. That share decides how urgent this runbook is.
🛒 AI shopping agents can now find products, build carts and complete payments
Shopify, American Express, OpenAI and Google are expanding AI-powered shopping, introducing tools that let assistants complete purchases, verify customer instructions and surface product offers directly in AI search.
The Breakdown:
ChatGPT and Google Refine AI Shopping - ChatGPT is testing a shopping interface where users select product categories and budgets. Google’s AI Mode Shopping ads now display promotional offers, including multi-buy discounts, with expandable details.
Shopify Lets AI Agents Complete Purchases - Shopify’s Shop MCP allows compatible AI agents to search over 1 billion products, build shopping carts and complete Shop Pay checkout. Orders appear in Shopify admin like regular purchases.
Amex Adds Purchase Verification - American Express introduced Proof of Intent Tokens that record customer preferences, budgets, and merchant details. The system can flag purchases that violate instructions, such as buying Adidas when Nike was requested.
Google Tests Prescription Ads in AI Search - Google is testing branded prescription drug advertisements within AI Mode and AI Overviews in the US, extending AI search advertising into a tightly regulated category.
AI shopping is moving beyond product recommendations toward completed transactions. As agents begin selecting products and handling checkout, accurate pricing, inventory, and product descriptions become essential. Merchants increasingly need product data that AI systems can interpret and match against specific customer instructions.
🏆 Ad of the Day
What Works:
Hook Reframes Category - “Not your grandma’s banana bread” creates a playful pattern interrupt, immediately positioning the product as a modern, higher-protein alternative to a familiar comfort food.
Protein Feels Tangible - Showing the actual banana bread beside the shake and protein mix connects indulgence with function, making the 18g protein proposition feel more appetizing and believable.
Collab Adds Credibility - The Koia x Perdu’s partnership gives the creative added novelty and trust, while the warm banana palette keeps the collaboration cohesive and strengthens branding recall.
Take a nostalgic food people already understand, then reframe it around a modern benefit like protein, convenience or lower sugar to create instant message-market fit.
Advertise with Us
Wanna put out your message in front of over 50,000 best marketers and decision makers?
We are concerned about everything DTC and its winning strategies. If you liked what you read, why not join the 50k+ marketers from 13k+ DTC brands who have already subscribed? Just follow this.
At ScaleUP, we care about our readers and want to provide the best possible experience. That's why we always look for ways to improve our content and connect with our audience. If you'd like to stay in touch, be sure to follow us EVERYWHERE🥰
Thanks for your support :) We'll be back again with more such content 🥳