How Vercel built a real-time web intelligence system with Browserbase

Challenge: Building reliable web intelligence at scale

Vercel provides the developer tools and cloud infrastructure to build, scale, and secure a faster, more personalized web. Vercel also owns and maintains Next.js, the React framework for building web applications.

Vercel needs to understand the technical landscape of the web. Not just their own customers, but every major website—from funded startups to e-commerce giants. They wanted to track technology adoption, monitor performance metrics, and identify business opportunities in real-time.

Their initial approach hit major roadblocks:

• Legacy tools were unreliable: Puppeteer and similar libraries were brittle and error-prone at scale
• CDN challenges blocked access: Many sites were inaccessible due to bot detection and security measures
• Data quality issues: Existing services like BuiltWith missed critical information
• Limited parallelization: Couldn't crawl enough sites fast enough to get timely intelligence

"We tried to build this in-house with Puppeteer and all that. There are lots of challenges. Legacy libraries are brittle. We run into errors," explains Abhi Sivasailam, VP of Data.

Solution: Browserbase powers Vercel's web intelligence engine

Vercel built one of the most sophisticated internal tools out there to scale knowledge and revenue, internally named Prism. This comprehensive web intelligence system uses Browserbase to crawl the entire "Vercel-verse" and beyond.

Prism tracks all domains hosted on Vercel, top websites from Google’s Crux database, all funded startups, and many ecommerce sites.

Browserbase plays a few key roles in making this possible:

• Not your typical scraping data: Vercel isn’t just looking at site content, they need the header, the body, and the window itself, which requires a browser like Browserbase.
• CDN bypass capabilities: Many important sites and data points live in a CDN protected site. Browserbase’s Advanced Stealth feature overcomes the CDN challenge to include those sites in the data set.
• Massive parallelization: "If you're going to crawl this many domains, you need to be able to move quickly.” Browserbase allows Vercel to run thousands of browsers at one time as well as in the background and while you sleep.

Result: Real-time revenue intelligence

The impact of this system is truly impressive.

A custom churn warning system ⚠️

This program powers the revenue engine! Vercel uses this data to define Sales books of business, do propensity modeling and forecasting, and drive new pipeline. This is also key for retention! By monitoring both production and staging sites, Vercel can predict customer churn before it happens.

"We can see for Vercel customers both staging sites and production sites. If a customer is changing technologies on a staging site, we can identify that and actually do win-backs on people we've identified are about to churn."

New: A ship velocity index 🚀

Prism powers a daily Slack channel that tracks real-time technology adoption, like who is upgrading Next.js. With this data, Vercel created a unique "ship velocity index" that analyzes website code changes to determine how fast companies are shipping updates.

"We can see whether the code is changing and answer the question, how fast is the company shipping?"

Becoming the new web intelligence standard

There are no plans yet, but Vercel is considering making Prism public, potentially becoming "the new HTTP Archive" as the original Google project winds down. With Browserbase handling the infrastructure complexity, Vercel can focus on extracting valuable insights from this web data.

"It's good to have someone reliable handling the browser management and CDN challenges so we can focus on the intelligence layer," Sivasailam notes.

---

Ready to build your own web intelligence system? Join companies like Vercel who are transforming how they use the web to drive business impact.

Get started with Browserbase →