Read more about Browserbase - a web browser for your AI - the company, product, features, and news on the Browserbase blog

Browserbase Blog: Automate the web

The Browser Blog

Paul Klein

>Try our new product, [Director.ai](https://director.ai) today

In this hyper-fast world of AI, in order to compete you have to keep up. Even since starting Browserbase 16 months ago, I’ve seen the promises of AI turn into production-ready reality. 

We're proud to be part of this transformation, building the infrastructure that makes web automation accessible to everyone. What started with a thesis that AI agents would require great browser infrastructure, has turned into billion dollar companies using Browserbase to deliver better outcomes for their customers.

## Building on a foundation of primitives

The most transformative technology companies (Twilio, Stripe, Vercel) all share something in common: they're primitives. They provide the fundamental building blocks that enable developers to create powerful applications.

Browser infrastructure represents the next critical primitive. It's the foundation that will change how people interact with and automate the web, empowering developers to build software that fundamentally transforms online work.

## The opportunity ahead

Think about how often we interact with a web browser: buying products, sending messages, deploying code. Now consider all the routine online tasks, from sending emails to booking travel, that consume our time every single day.

The future of browsing is selective automation. Humans will still do the joyful, discovery-driven tasks. But repetitive, time-consuming work should be done by software. That’s what we’re building for.

This vision where AI handles the boring parts while humans focus on what matters, represents a massive business opportunity. If Browserbase can become the rails of this new era, we'll be enabling an entirely new category of applications.

And yes, maybe developers are inherently lazy, we build tools to avoid repetitive work. Browserbase is the ultimate expression of that philosophy.

## Expanding our platform

Along with our Series B announcement and lots of small product improvements, we're announcing three main platform upgrades:

1. [**Director.ai**](http://director.ai/): Browserbase has always been built for developers, and we plan to always help developers automate the web, but in this era of AI, everyone can be a developer. And that's why we're launching Director. Director is here to let anyone automate the web, generate Stagehand scripts and deploy them on Browserbase. It's empowering. And it's transformational for infrastructure companies like us, which traditionally only sold to developers, but now we can sell or support everybody. I think that's monumental.
2. **Stagehand for Python**: We're now working with two of the most popular programming languages, Python and Node.js. And I'm very excited for that. The Python version also comes with enhanced observability features built into the Browserbase dashboard, including session recordings, token usage tracking, and comprehensive logging.
3. **We're lowering our prices**: We've been hard at work improving our infrastructure to continue to offer better pricing and value. We hope to make automation a more accessible tool for all builders. Here are the key changes, but you can see all the details here. 
    - **~~Hobby~~ Developer plan**: 50% price reduction to get started, from $39 to $20/month for 100 browser hours and 1 GB of proxies.
    - **More concurrent browsers**: From 3 to 25 on the Developer plan & doubled concurrency from 25 to 50 browsers on the Startup plan.
    - **New feature**: Variable-size browsers for scale customers, choose from our pre-configured small, medium, or large instances.


 ## Series B & beyond

The product release coincides with a $40 million Series B financing round led by Notable Capital, with participation from existing investors CRV and Kleiner Perkins. This new funding marks our commitment to the long haul. Our team, investors, and most importantly, our customers share a vision of a future where people operate with unprecedented leverage through intelligent automation.

The journey ahead is substantial. Browser infrastructure still needs deep innovation—in the browser technology itself, the surrounding infrastructure, and all the supplemental features required to build powerful applications on this foundation.

The market opportunity attracts competition, which we welcome. In markets this large, innovation benefits everyone, pushing the entire ecosystem forward and delivering better solutions for customers.

Looking ahead to the next year at Browserbase, our mission remains clear: help everyone automate the web. We're building the best browser infrastructure platform, the most powerful framework, and the most accessible application to make that vision reality.

We can't wait to see what everyone builds.

We're excited to announce our Series B funding, major platform updates, and our plans for building the primitive infrastructure that will enable AI to automate the mundane work we do online every day.

Building the future of web automation

Alex Phan

Sophie Chance

PromptLoop empowers mid-market sales and go-to-market teams by enabling them to target the right accounts and boost win rates through efficient AI-powered automation. Their platform offers a seamless solution for navigating and analyzing thousands of company websites in real time, categorizing entities, and managing large-scale research tasks with ease. Leveraging Browserbase has become a pivotal move for PromptLoop, allowing them to scale web data retrieval effortlessly and focus engineering resources on core innovations.

> “I think the ease of setup is really attractive. It is really just one line of additional code.”
> 
> \- Andrew Mangan, Co-Founder, PromptLoop

**The Challenge of Scaling Web Data Retrieval**
-----------------------------------------------

For PromptLoop, maintaining reliable web data retrieval infrastructure was crucial for their AI research platform. They needed a solution that could handle thousands of simultaneous queries while keeping operational costs efficient and maintaining fast performance.

> "It was often very tricky to manage the spiky nature of our customers because often customers will come to us with very large jobs where they're analyzing 10, 20, 30, 100,000 companies in a matter of hours. And so obviously, we're trying to do that as fast as we can. If we had a big spike, oftentimes the instances wouldn't scale up properly and so jobs would get dropped."

Before Browserbase, PromptLoop faced several challenges that impacted their ability to scale effectively:

*   Managing their own browser infrastructure at scale
    
*   Processing thousands of simultaneous web queries
    
*   Keeping operational costs efficient while scaling
    

Building a Core Web Data Layer
------------------------------

Browserbase became a foundational part of PromptLoop's technology stack, serving as the backbone of their web extraction process. This transformation enabled them to focus on their core AI capabilities while maintaining reliable web automation.

> "We're iterating constantly on how we're accessing websites and how we're better retrieving data, but we're not really worrying anymore about the browser being loaded, which is super helpful. Crossing off areas that can break allows us to focus on the more impactful changes."

The impact was immediate and significant across several key areas:

**Efficient Browser Management**

The platform manages thousands of browser instances efficiently, enabling PromptLoop to process 10,000+ rows of data simultaneously without performance issues. By reducing the need to manage and troubleshoot ECS deployments, Browserbase has freed up valuable backend DevOps time, allowing the team to focus on core development priorities.

**Reliable Performance**

The implementation of Browserbase ensures consistent performance and allows PromptLoop to consistently reach over a 95% success rate for accessible websites—a significant improvement from the previous 85-90% success rate. It ensures consistent performance even with thousands of simultaneous queries, maintaining PromptLoop’s commitment to their customers

**Scalable Operations**

By providing a robust JavaScript loading layer to their web data retrieval, Browserbase helps PromptLoop maintain their efficiency advantage while processing web data at scale. The solution not only enhances accuracy but also increases capacity, allowing PromptLoop to crawl hundreds of thousands of websites monthly—something that would have been challenging to achieve with the previous system.

> ”We're now processing more jobs, like a factor, a magnitude more jobs than we were on the previous system and it definitely would have been challenging to scale that up.”

Looking Ahead: **A Collaborative Future**
-----------------------------------------

PromptLoop continues to explore additional Browserbase capabilities to enhance their platform's efficiency and capabilities.

> "We're definitely interested in exploring as we work with the team more about how we can better access these websites."

The partnership has enabled PromptLoop to build a platform that delivers on their core promise: helping teams automate research tasks at scale with zero engineering hours required. Their commitment to exploring new Browserbase features demonstrates their focus on continuous improvement and innovation.

By implementing Browserbase, PromptLoop has built a robust foundation for their AI research platform. What started as a JavaScript loading layer has become core to their technology stack, enabling them to process thousands of simultaneous queries efficiently and maintain their competitive advantage in the AI research automation space.

PromptLoop uses Browserbase to process thousands of simultaneous web queries, enabling efficient data retrieval, faster operations, and streamlined scaling

Scaling in Magnitudes: PromptLoop’s Success with Browserbase

Benny is dedicated to enhancing the lives of SNAP recipients by offering a streamlined cash-back experience. Their platform provides a much-needed solution for SNAP users, who often struggle with outdated systems and fragmented account management. By leveraging Browserbase’s powerful browser automation, Benny empowers users to link EBT accounts, track balances, and maximize cash-back rewards with minimal friction.

> “We give folks 1% cash back on their SNAP spend. They're able to link their EBT accounts with the state website and view their transactions, balances and get reminded on how they can maximize their earnings there”
> 
> \- Steven Austin, Founder, Benny

With Browserbase, Benny has not only simplified the user experience but has also transformed their internal workflows, allowing the engineering team to focus on core functionalities instead of browser maintenance.

**The Challenge of Creating an EBT-Linking Infrastructure**
-----------------------------------------------------------

Before Browserbase, Benny’s engineering team faced significant hurdles in maintaining a reliable browser automation infrastructure. As Austin explained, “There is no Plaid equivalent for EBT SNAP data.” This forced the team to develop a custom system for automating access to EBT accounts—A task that proved challenging as they tried to integrate headless browser automation, proxy solutions, CAPTCHA-solving tools, and session monitoring.

Without a unified platform, Benny encountered multiple operational difficulties:

• The team managed a patchwork of scripts and tools that required constant monitoring.

• Browser automation tasks were prone to failure, leading to costly delays and resource allocation.

• Limited visibility into browser activity made debugging and troubleshooting time-consuming.

**A Centralized Solution for Enhanced Performance**
---------------------------------------------------

Browserbase provided a unified, scalable infrastructure that alleviated Benny’s browser automation pain points. Steven highlighted that the transition allowed them to redirect engineering resources toward the company’s primary mission: delivering cash-back benefits and financial insights to SNAP users.

> “It was a lot to maintain, browser automation and infrastructure is not a core part of our offering. So we were really excited to kind of move to an all in one solution rather than us have this piecemeal approach.”

Through features like session replays and reusable session cookies, Browserbase streamlined Benny’s automation workflows, offering greater visibility and speeding up browser tasks that are critical to Benny’s user experience.

**Impact: From Patchwork Solutions to Streamlined Workflows**
-------------------------------------------------------------

The integration with Browserbase significantly boosted Benny’s performance and reliability metrics, enabling them to scale automation efforts while reducing technical overhead.

> “We measure our link success rate, that's increased around 40% and there's other things in there, but I do think us having better infrastructure in terms of automatic captcha solving, higher quality proxies, proxy switching and just the ability to debug faster is directly tied to Browserbase.”

The platform also had a measurable impact on development efficiency:

• **Session Replay for Fast Debugging**: Browserbase’s session replay feature allowed Benny’s engineers to troubleshoot issues quickly, helping them differentiate between site downtimes and actual bugs.

• **Reusable Sessions**: Persisting session cookies reduced automation lag and improved reliability, leading to a better end-user experience.

• **Enhanced Developer Experience**: The intuitive interface and session management tools in Browserbase allowed Benny’s team to operate more efficiently, with Steven describing the developer experience as

> “night and day compared to any alternative”

**Conclusion**
--------------

Browserbase has empowered Benny to transition from a fragmented, manually-maintained infrastructure to a robust, consolidated platform that enhances both their user experience and internal productivity. With Browserbase, Benny has reduced the complexity of maintaining browser automation while improving their core cash-back offering for SNAP users. As Steven affirmed:

> “I’d absolutely recommend \[Browserbase\] to others… a new startup that needs a ton of browser automation or in an AI field that wants some sort of like agent access? I would recommend you all, hands down.”

Benny’s experience showcases how Browserbase enables rapid innovation, allowing fintech companies to scale and refine their offerings without sacrificing focus on their core mission

Simplifying user experience and improving internal workflows using Browserbase

How Browserbase transformed Benny’s SNAP benefits platform

Structify develops sophisticated AI agents that navigate the web and automatically structure information into enterprise-tier datasets. Their platform helps companies efficiently collect and organize large-scale web data, transforming unstructured web content into valuable business intelligence.

### The Challenge: Scaling Browser Automation for AI Agents ### 
For Structify's team, scaling their browser automation infrastructure was a technical challenge that threatened to bottleneck their entire operation. They had mastered scaling other aspects of their infrastructure, but browser automation proved to be a unique hurdle.

>"We knew how to scale everything else: CPU workload, web balancing, GPU inference… But it turns out that when you try to open 30,000 browser tabs, things start crashing" - Alex Reichenbach, Founder, Structify

Before Browserbase, Structify faced several critical technical challenges:

Managing massive memory consumption from thousands of browser instances

Scaling up Kubernetes to handle match the tens of thousands of browsers, and being able to do it quickly and on-demand

Maintaining reliable browser instantiation and management

### From Infrastructure Headache to an API Call ### 
Browserbase transformed how Structify's AI agents interact with the web by providing a robust, scalable solution that eliminated their infrastructure challenges. The integration was straightforward and allowed them to focus on their core AI capabilities rather than browser management.

>"Anytime one of our AI agents goes out, it automatically calls Browserbase. Anytime we get a request to get information from the Internet, which is a lot of what we do, that automatically integrates with Browserbase on the backend."

The impact was substantial across several key areas:

Efficient Resource Management The implementation of Browserbase dramatically reduced their server resource consumption, particularly RAM usage, which had previously been a major bottleneck.

>"There was a measurable improvement - we went from gigabytes of RAM being eaten up by browsers in our production web server to basically zero. We wouldn’t have been able to do this without Browserbase."

Enhanced Scalability Browserbase enabled Structify to reliably scale their operations to handle thousands of concurrent browser sessions without the overhead of managing the browser infrastructure.

>"We now regularly scale to several thousand browsers. That's a lot of memory. We are able to reliably spawn that many and not worry"

Streamlined Operations By eliminating the need to manage browser infrastructure, Structify could focus on their core competencies and scaling their AI capabilities.

>"The fact that we don't have to worry about browser instantiation, keeping them alive, scaling up kubernetes to match the number of browsers, being able to do it on demand and really quick is incredible for us."

**Processing Massive Web Datasets:**

With Browserbase, Structify has been able to significantly expand their data collection capabilities, enabling them to process unprecedented volumes of web content.

>"It's allowed us to start scraping en masse larger data sets. We can scrape on the order of tens of thousands of websites for a single data set."

### Why Structify Trusts Browserbase ### 

Structify's experience with Browserbase demonstrates the value of a solution that combines ease of integration with reliable performance at scale. The platform's consistency and dependability have made it an integral part of their infrastructure.

>"I have recommended Browserbase to others. It's incredibly simple to integrate. It's one of its biggest value adds. And it just works well."

By implementing Browserbase, Structify has revolutionized their ability to process web data at scale. Their solution has fundamentally changed how they approach web automation and data collection, enabling them to handle tens of thousands of websites efficiently and reliably. This transformation has positioned Structify at the forefront of AI-powered web data structuring, helping enterprises turn massive amounts of web content into structured, actionable data sets.

How Structify uses Browserbase for AI-driven web data collection

Coframe uses AI to propose and test variations for your digital experiences.

> "Today people are running experiments manually on their websites to try to make them better in some way... Usually that's in the form of conversion rate, or some engagement, or some type of metric… We’re now building an AI-accelerated version of this."
> 
> \- Josh Payne, Co-founder, Coframe

The Struggle of Website Optimization
------------------------------------

For Coframe, managing browser infrastructure became a significant bottleneck that drained resources from their core mission of building AI-powered website optimization. What started as simple playwright scripting quickly evolved into a technical nightmare, with their team spending countless hours trying to maintain reliable browser automation. Turning what should have been straightforward website improvements into a complex infrastructure challenge.

> “I probably spent, like, 12 hours in total just banging my head against the wall with these playwright scripts... it was barely working, it wasn't even working in a docker environment like I wanted it to.”

Before finding a solution, Coframe struggled with technical and operational challenges such as:

*   Writing and maintaining complex Playwright scripts and proxies that required significant engineering effort
    
*   Running manual website experiments that demanded coordination between engineers, designers, and marketers
    
*   Spending valuable development time on browser automation infrastructure instead of improving their AI website optimization
    
*   Scaling their data collection and parallel processing for website analysis and code extraction
    
*   Managing browser-based tools that required Chrome extensions, creating a subpar user experience
    

Transforming Tedious Experiments into Self-Improving Websites
-------------------------------------------------------------

Instead of wrestling with manual Playwright scripts and proxy configurations, Browserbase became a solution that could handle complex browser operations not only for their data creation and website analysis, but for go-to-market initiatives as well.

The solution transformed how Coframe's team operates, providing a reliable and scalable approach that eliminated their browser automation headaches. This enabled them to run parallel browser tasks at scale, extract website code efficiently, and process massive amounts of data to train their own generative AI model.

> "We were able to run tons of workers at the same time instead of having to wait... and Browserbase significantly helped with that. Fast forward, I was just able to copy the documentation and have it write me what I needed and it worked out of the box... a 1000 times improvement in terms of time saved.”

This breakthrough allowed the Coframe team to channel their energy into what matters most: developing AI that makes websites self-improve, delivering automated optimization that would typically require countless hours from engineers, designers, and marketers.

The impact was immediate and significant across several key areas:

**Accelerated Data Set Creation**

The implementation of parallel processing dramatically accelerated their ability to gather, analyze, and create website data, transforming what was previously a time-consuming manual process into an efficient operation for producing new website sections.

> ”We actually worked with OpenAI to create a model that is specifically better at generating visually grounded code UI code, or website code. We extract code used to produce the sections of the websites, the styles and so forth, and use Browserbase to grab those websites do what we need to do at scale and in parallel.”

**Enhancing User Experience**

The team is now positioned to eliminate the need for Chrome extensions, creating a more seamless experience for their customers through live browser streaming capabilities.

> "People today have to use a Chrome extension to make use of our visual editor. And that's kind of a subpar experience having to download a Chrome extension. What we're really hoping is using Browserbase we can have the user go and actually use Coframe without having to download that extension."

**Reliable Solutions Drive on Browserbase**
-------------------------------------------

Delivering a clean and efficient platform allows Coframe to focus on their core mission of AI-powered website optimization. This approach to browser automation has been crucial in helping Coframe maintain momentum in expanding the fast-moving AI and website optimization space.

> "We're really excited for the multi-region coverage from Browserbase... That'll be really awesome. So, we'll be able to use it in more parts of Coframe when that happens."

**A Platform That Enables Future Growth**
-----------------------------------------

By implementing Browserbase, Coframe has revolutionized the website optimization process for businesses. Their platform has fundamentally changed how companies approach website improvements and A/B testing. What once required tedious coordination between engineers, designers, and marketers can now be automated through AI, enabling their clients to continuously improve their websites while maintaining high performance.

> "I think that it's just like a clean solution. It's like one of those great clean products. And I would certainly recommend it being kind of the product for what you guys are doing. It just makes so much sense."

This transformation has positioned Coframe to revolutionize how websites improve themselves, automating the optimization process that traditionally consumed countless hours of manual effort.

Coframe has elevated its platform’s capabilities, focusing on refining website optimization instead of battling infrastructure challenges

Why Coframe moved their automations to Browserbase

**Browserbase's Pledge to Open Source**  
At Browserbase, we've always believed in the power of open source software (OSS). It's not just a philosophy; it's a cornerstone of our technological ecosystem and a driving force behind innovation in our industry. Today, we're excited to announce our latest contribution to the open source community.

**Supporting ScrapegraphAI**  
We're thrilled to share that Browserbase is sponsoring ScrapegraphAI with a donation of $12,000 this year. This contribution represents $2,000 for each full-time developer on the Browserbase team, underscoring our commitment to giving back to the community that has given us so much.  
ScrapegraphAI is an innovative project that combines web scraping with AI-powered graph analysis, pushing the boundaries of what's possible in data extraction and interpretation. By supporting this project, we're not just investing in a tool; we're investing in the future of web technologies and AI integration.

**Why We Believe in Supporting Open Source**  
Open source software is the bedrock upon which much of modern technology is built. At Browserbase, we recognize that our success is intertwined with the health and vitality of the open source ecosystem. Here's why we believe so strongly in supporting open source developers and maintainers:

1.  **Fostering Innovation**: Open source projects are often at the forefront of technological innovation. By supporting these projects, we're helping to push the entire industry forward.
    
2.  **Building Community**: The open source community is a vibrant network of developers, thinkers, and problem-solvers. Our support helps strengthen this community and fosters collaboration.
    
3.  **Ensuring Sustainability**: Many critical open source projects are maintained by developers working in their free time. Financial support helps ensure these projects remain viable and well-maintained.
    
4.  **Giving Back**: We use open source tools every day in our work. Supporting OSS is our way of giving back to the community that has provided us with invaluable resources.
    
5.  **Promoting Transparency**: Open source embodies the principles of transparency and shared knowledge. By supporting OSS, we're endorsing these values that we believe are crucial for technological progress.
    

**  
Our Ongoing Commitment**  
Our sponsorship of ScrapegraphAI is just the first step in our ongoing commitment to open source. Looking ahead, we're committed to continuing and expanding our support for open source software. We're always on the lookout for impactful projects that align with our values and technological interests.

**Join Us in Supporting Open Source**  
We encourage other companies and individuals in the tech industry to join us in supporting open source projects. Whether through financial contributions, code contributions, or simply by using and promoting open source tools, every bit of support helps sustain this vital ecosystem.  
At Browserbase, we're proud to play our part in supporting the open source community. We believe that by working together and supporting one another, we can drive innovation, create better technologies, and build a stronger, more collaborative tech industry for everyone.

At Browserbase, we've always believed in the power of open source software (OSS). 

Browserbase's Pledge to Open Source

Sarah Chieng

Slack Scout sends a Slack notification every time your keywords are mentioned on Twitter, Hacker News, or Reddit. Get notified whenever you, your company, or topics of interest are mentioned online.

Built with Browserbase and Val Town. Inspired by f5bot.com.

![](https://framerusercontent.com/images/6g0Uo35RzV7PENESy92Ja1tCTU.gif)

### What this tutorial covers

*   Access and scrape website posts and contents using Browserbase
    
*   Write scheduled functions and APIs with Val Town
    
*   Send automated Slack messages via webhooks
    

Getting Started
===============

In this tutorial, you’ll need a

*   Browserbase API key
    
*   Valtown account
    
*   Slack Webhook URL: create it here
    

### Browserbase

Browserbase is a developer platform to run, manage, and monitor headless browsers at scale. We’ll utilize Browserbase to navigate and scrape different news sources. We’ll also use Browserbase’s Proxies to ensure we simulate authentic user interactions across multiple browser sessions.

Sign up for free to get started!

### Val Town

Val Town is a platform to write and deploy Javascript. We’ll use Val Town for three things.

1.  Create HTTP scripts that run Browserbase sessions. These Browserbase sessions will execute web automation tasks, such as navigating Hacker News and Reddit.
    
2.  Write Cron Functions (like Cron Jobs, but more flexible) that periodically run our HTTP scripts.
    
3.  Store persistent data in the Val Town provided SQLite database. This built-in database allows us to track search results, so we only send slack notifications for new, unrecorded keyword mentions.
    

Sign up for free to get started!

### Twitter (X)

For this tutorial, we’ll use the Twitter API to include Twitter post results.

_You'll need to create a new Twitter account to use the API. It costs $100 / month to have a Basic Twitter Developer account._

Once you have the `SLACK_WEBHOOK_URL`, `BROWSERBASE_API_KEY`, and `TWITTER_BEARER_TOKEN`, input all of these as Val Town Environment Variables.

Creating our APIs
=================

We’ll use a similar method to create scripts to search and scrape Reddit, Hacker News, and Twitter. First, let’s start with Reddit.

To create a new script, go to Val Town → New → HTTP Val. Our script will take in a keyword, and return all Reddit posts from the last day that include our keyword.

For each Reddit post, we want the output to include the URL, date published, and post title.

For example:

{

source: 'Reddit', // or 'Hacker News' or 'Twitter'

url: '<https://www.reddit.com/r/browsers/comments/vdhge5/browserbase\_launched/>';

date\_published: 'Aug 30, 2024';

title: 'Browserbase just launched';

}

In our new `redditSearch` script, we start by importing Puppeteer and creating a Browserbase session with proxies enabled (`enableProxy=true`). Be sure to get your `BROWSERBASE_API_KEY` from your Browserbase settings.

import { PuppeteerDeno } from "<https://deno.land/x/puppeteer@16.2.0/src/deno/Puppeteer.ts>";

const puppeteer = new PuppeteerDeno({ productName: "chrome" });

const browser = await puppeteer.connect({

browserWSEndpoint: \`wss://connect.browserbase.com?apiKey=${apiKey}&enableProxy=true\`,

ignoreHTTPSErrors: true,

});

Next, we want to

1.  Navigate to Reddit and do a keyword search
    
2.  Scrape each resulting post
    

To navigate to a Reddit URL that already has our keyword and search time frame encoded, let’s write a helper function that encodes the query and sets search parameters for data collection.

function constructSearchUrl(query: string): string {

const encodedQuery = encodeURIComponent(query).replace(/%20/g, "+");

return \`https://www.reddit.com/search/?q=${encodedQuery}&type=link&t=day\`;

}

  

const url = constructSearchUrl(query);

await page.goto(url, { waitUntil: "networkidle0" });

Once we’ve navigated to the constructed URL, we can scrape each search result. For each post, we select the `title`, `date_published`, and `url`.

const posts = document.querySelectorAll("div\[data-testid=\\"search-post-unit\\"\]");

return Array.from(posts).map(post \=> {

const titleElement = post.querySelector("a\[id^=\\"search-post-title\\"\]");

const timeElement = post.querySelector("faceplate-timeago");

return {

source: "Reddit",

title: titleElement?.textContent?.trim() || "",

url: titleElement?.href || "",

date\_published: timeElement?.textContent?.trim() || "",

};

});

  

// Example

{

source: 'Reddit', // or 'Hacker News' or 'Twitter'

url: '<https://www.reddit.com/r/browsers/comments/vdhge5/browserbase\_launched/>';

date\_published: '1 day ago';

title: 'Browserbase just launched';

}

You’ll notice that Reddit posts return the date\_published in the format of ‘1 day ago’ instead of ‘Aug 29, 2024.’ To make date handling more consistent, we create a reusable helper script, `convertRelativeDatetoString`, to convert dates to a uniform date format. We import this at the top of our redditSearch script.

import { convertRelativeDateToString } from "<https://esm.town/v/sarahxc/convertRelativeDateToString>";

const date\_published = await convertRelativeDateToString({ relativeDate: post.date\_published });

You can see the finished redditSearch code here.

We follow a similar process to create `hackerNewsSearch`, and use the Twitter API to create `twitterSearch`.

**See all three scripts here:**

_Reddit →_ redditSearch

_Hacker News →_ hackerNewsSearch

_Twitter →_ twitterSearch

Creating the Cron Function
--------------------------

For our last step, we create a slackScout cron job that calls redditSearch, hackerNewsSearch, and twitterSearch that runs every hour. To create the cron file, go to Val Town → New → Cron Val.

In our new slackScout file, let’s import our HTTP scripts.

import { hackerNewsSearch } from "https://esm.town/v/alexdphan/hackerNewsSearch";

import { twitterSearch } from "https://esm.town/v/alexdphan/twitterSearch";

import { redditSearch } from "https://esm.town/v/sarahxc/redditSearch";

  

And create helper functions that call our Reddit, Hacker News, and Twitter HTTP scripts.

// Fetch Reddit, Hacker News, and Twitter results

async function fetchRedditResults(topic: string): Promise<Website\[\]\> {

return redditSearch({ query: topic });

}

async function fetchHackerNewsResults(topic: string): Promise<Website\[\]\> {

return hackerNewsSearch({

query: topic,

pages: 2,

apiKey: Deno.env.get("BROWSERBASE\_API\_KEY") ?? "",

});

}

  

async function fetchTwitterResults(topic: string): Promise<Website\[\]\> {

return twitterSearch({

query: topic,

maxResults: 10,

daysBack: 1,

apiKey: Deno.env.get("TWITTER\_BEARER\_TOKEN") ?? "",

});

}

  

Next, to store our website results, let’s setup Val Town’s SQLite database. To do this, we import SQLite and write three helper functions.

1.  `createTable`: creates the new SQLite table
    
2.  `isURLInTable`: for each new website returned, checks if the website is already in our table
    
3.  `addWebsiteToTable`: if isURLInTable is `False` , we add the new website to our table
    

const { sqlite } = await import("https://esm.town/v/std/sqlite");

const TABLE\_NAME = "slack\_scout\_browserbase";

  

// Create an SQLite table

async function createTable(): Promise<void\> {

await sqlite.execute(\`

CREATE TABLE IF NOT EXISTS ${TABLE\_NAME} (

source TEXT NOT NULL,

url TEXT PRIMARY KEY,

title TEXT NOT NULL,

date\_published TEXT NOT NULL

)

\`);

}

async function isURLInTable(url: string): Promise<boolean\> {

const result = await sqlite.execute({

sql: \`SELECT 1 FROM ${TABLE\_NAME} WHERE url = :url LIMIT 1\`,

args: { url },

});

return result.rows.length > 0;

}

  

async function addWebsiteToTable(website: Website): Promise<void\> {

await sqlite.execute({

sql: \`INSERT INTO ${TABLE\_NAME} (source, url, title, date\_published)

VALUES (:source, :url, :title, :date\_published)\`,

args: website,

});

}

Finally, we write a function to send a Slack notification for each new website.

async function sendSlackMessage(message: string): Promise<Response\> {

const slackWebhookUrl = Deno.env.get("SLACK\_WEBHOOK\_URL");

if (!slackWebhookUrl) {

throw new Error("SLACK\_WEBHOOK\_URL environment variable is not set");

}

  

const response = await fetch(slackWebhookUrl, {

method: "POST",

headers: { "Content-Type": "application/json" },

body: JSON.stringify({

blocks: \[

{

type: "section",

text: { type: "mrkdwn", text: message },

},

\],

}),

});

  

if (!response.ok) {

throw new Error(\`Slack API error: ${response.status} ${response.statusText}\`);

}

  

return response;

}

The main function initiates our workflow, calling helper functions to fetch and process data from multiple sources.

  

export default async function(interval: Interval): Promise<void\> {

try {

await createTable();

for (const topic of KEYWORDS) {

const results = await Promise.allSettled(\[

fetchHackerNewsResults(topic),

fetchTwitterResults(topic),

fetchRedditResults(topic),

\]);

  

const validResults = results

.filter((result): result is PromiseFulfilledResult<Website\[\]> => result.status === "fulfilled")

.flatMap(result => result.value);

  

await processResults(validResults);

}

console.log("Cron job completed successfully.");

} catch (error) {

console.error("An error occurred during the cron job:", error);

}

}

And we’re done! You can see the final `slackScout` here.

And that’s it!
==============

Optionally, you can use Browserbase and Val Town to create additional HTTP scripts that can monitor additional websites like Substack, Medium, WSJ, etc. Browserbase has a list of Vals you can get started with in your own projects. If you have any questions, concerns, or feedback, please let us know :)

support@browserbase.com

Slack scout sends a slack notification every time your keywords are mentioned on Twitter, Hacker News, or Reddit

Slack Scout

Playwright (and Puppeteer) exposes its API through `Browser`, `BrowserContext`, and `Page` objects with a blurry definition of their boundaries. This article covers their definition, how to leverage them, and how to avoid common pitfalls.

What are BrowserContexts and Pages?
-----------------------------------

A typical Playwright program looks as follows:

import { chromium } from "playwright-core";

  

(async () \=> {

// 1. Get a Browser instance

const browser = await chromium.connectOverCDP(

\`wss://connect.browserbase.com?apiKey=${process.env.BROWSERBASE\_API\_KEY}\`

);

// 2. Get a BrowserContext

const defaultContext = browser.contexts()\[0\];

// 3. Get a Page

const page = defaultContext.pages()\[0\];

// 4. Act on Page await

page.goto("<https://browserbase.com>");

await page.close();

await browser.close();

})().catch((error) \=> console.error(error.message));

Let's have a closer look at the `Browser`, `BrowserContext`, and `Page` definitions by comparing them to our day-to-day Browser:

![](https://framerusercontent.com/images/ArW4b9KtfygB4Tg3n0jWggs.png)

Each concept accounts for a level of isolation and shared resources: `BrowserContext` gets a dedicated cache, later shared between `Pages`.

We can now visualize the relationship between them as follows:

![](https://framerusercontent.com/images/Fd7IPrnpZgsgCs9BN57I3m3nujo.png)

`Browser`, `BrowserContext`, and `Page` are encapsulated concepts that enable developers to control different parts of the Browser.

Let’s now see how to leverage each concept when building automation.

How to use BrowserContexts and Pages
------------------------------------

The most common use cases benefiting from using multiple BrowserContexts or Pages are parallelization and cookie management.

**Leverage BrowserContexts for parallelization**

Crawling a large number of web pages sequentially can be slow. A common practice is to create multiple pages and visit them in parallel to speed up the automation:

import { chromium } from "playwright-core";

  

(async () \=> {

const browser = await chromium.launch();

const getContent = async (url: string) \=> {

const context = await browser.newContext();

const page = await context.newPage();

try {

await page.goto(url, {

waitUntil: "domcontentloaded",

});

return await page.content();

} finally {

await page.close();

}

};

await Promise.all(

\["<https://www.google.com>", "<https://www.bing.com>"\].map(async (item) \=> {

const contents = \[\];

for (url in item.urls) {

contents.push(await getContent(url));

}

return contents;

}),

);

await browser.close();

})().catch((error) \=> console.error(error.message));

Using multiple `BrowserContext` is a good choice for parallelization as long as no authentication is involved.

However, remember that the overall performance of your automation’s parallelization will be limited by the underlying compute capacity assigned to the Browser.

**Leverage Pages for cookie management**

`Pages` sharing the same storage and cookies are a good fit for getting granular control of the navigation and history.

For example, a first page can be created to complete an authentication flow. In contrast, other pages can later be created to navigate other parts of a website with their dedicated history.

![](https://framerusercontent.com/images/4gJxe7fua7zOuegIobYeZdr8lKg.png)

Let’s now see how Browserbase lets you reuse `BrowserContext` across multiple sessions to persist authentication.

**BrowserContexts and Pages with Browserbase**

Browserbase assigns each `Browser` instance (_called a_ _Browser Session_) to a dedicated virtual machine (with isolated resources -- unlike your local browser) with a unique `BrowserContext` (by leveraging the Persistent Context API).

![](https://framerusercontent.com/images/JmVgKoeDDr99GiYhoks5dN2E9Q.png)

Combining isolated resources and a unique `BrowserContext` enables support for custom Chrome Extensions and guarantees better performance. You can use the Contexts API, to reuse `BrowserContext` across multiple sessions to handle authentication flows

This article covers how to leverage Playwright and Puppeteer, and how to avoid common pitfalls

Understand Playwright's BrowserContexts and Pages

From building automation to developing AI Agents, working with headless browsers is a craft that relies on many essential steps:

*   Building future-proof selectors
    
*   Dealing with complex dynamic UI (popups, infinite scroll)
    
*   Handling possible network latency or failure
    
*   Dodging anti-bot mechanisms
    
*   Navigating authentication flows
    

Each of those steps might require multiple iterations to debug (_by adding error handlers and_ `_console.log()_`) along with multiple complete re-runs to validate their proper functioning.

Here is an example of a Password Changer automation development flow running on thefork.com:

![](https://framerusercontent.com/images/tIQe2bt6vrcBFhzIJ9lvzoUSTok.png)

The automation faced multiple issues, such as weak selectors that required fine-tuning and a recaptcha that was finally bypassed with a custom-solving logic.

This article will cover how Browserbase, with its Stealth Browser and live Sessions access, helps achieve faster development and avoid discovering issues once deployed.

A Stealth Browser with strong defaults
--------------------------------------

Dealing with an anti-bot mechanism is one of the most time-consuming parts of developing browser automation. Websites have various tactics, from IP-based filtering to recaptcha or fingerprinting.

Some of them can be avoided by configuring libraries like `puppeteer-extra` (fingerprinting), while others require custom code (captchas) or can only be discovered once deploying your automation (IP filtering).

By developing and running your automation with Browserbase, you’ll immediately benefit from its custom-built Chromium browsers embedding crucial Stealth features such as:

*   Proxies
    
*   Automatic Captcha solving
    
*   Embedded Fingerprinting
    

You no longer need custom code or extra dependencies to handle anti-bot issues; change a single line to connect your headless framework to Browserbase:

import { chromium } from "playwright-core";

import { Browserbase } from "@browserbasehq/sdk";

const browserbase = new Browserbase();

const { id: sessionId } = await browserbase.createSession({

// provide a custom fingerprint configuration

fingerprint: { devices: \["mobile"\] }, });

const browser = await chromium.connectOverCDP(

// enable proxy

browserbase.connectUrl({

id: sessionId,

proxy: true })

);

Let’s now see how Browserbase makes dealing with complex UI and flaky selectors easier.

Live debugging of flaky selectors and complex UIs
-------------------------------------------------

Navigating complex UI and authentication flows is the primary goal of Browser automation. Building reliable and quick automation takes time, as many iterations are required to navigate the UI and test selectors.

Browserbase’s Live Session debugging provides live access to running Sessions (for example, when a selector hangs), reducing the number of slow iterations and reruns of the automation:

![](https://framerusercontent.com/images/5UqCEWkvKeekG6xC5CUvIp4CYqE.png)

Here is how to print the Live Session URL to the console as soon as a Session starts with Playwright:

import { chromium } from "playwright-core";

import { Browserbase } from "@browserbasehq/sdk"

let sessionId; (async () \=> {

const browserbase = new BrowserBase()

const { id } = await browserbase.createSession();

sessionId = id;

const browser = await chromium.connectOverCDP(

browserbase.connectUrl({ id: sessionId })

);

const defaultContext = browser.contexts()\[0\];

const page = defaultContext.pages()\[0\];

await page.goto("<https://www.browserbase.com>", {

// let's make sure the page is fully loaded before asking for the live debug URL

waitUntil: "domcontentloaded",

});

const debugUrl = await retrieveDebugConnectionURL(sessionId);

console.log(\`Session started, live debug accessible here: ${debugUrl}.\`);

await page.close();

await browser.close();

})().catch((error) \=> {

console.log(

\`Session failed, replay is accessible here: <https://www.browserbase.com/sessions/${sessionId}.\`\>,

); console.error(error.message);

});

Once your automation starts, you get immediate remote access to the underlying Browser Session running on Browserbase, with the ChromeDev Tools to inspect the page.

In case of failure (for example, a session crash) causing your automation to stop, the URL of the Session Replay grants you access to a full video and logs replay.

Optimizing resources with Session Debugger
------------------------------------------

Ensuring your automation runs as quickly as possible is essential, but identifying slow resources can be challenging.

The following snippet shows how to print a Session replay link to get access to the timeline featuring all browser events and resource timings:

import { chromium } from "playwright-core";

import { BrowserBase } from "@browserbasehq/sdk"

(async () \=> {

const browserbase = new BrowserBase()

const { id: sessionId } = await browserbase.createSession();

const browser = await chromium.connectOverCDP(

browserbase.connectUrl({ id: sessionId }) );

// ...

await page.close();

await browser.close();

console.log( \`Session replay is accessible here: <https://www.browserbase.com/sessions/${sessionId}.\`\>,

);

})().catch((error) \=> {

console.error(error.message);

});

Navigating to the printed link, and easily inspect the Session’s network requests:

![](https://framerusercontent.com/images/P1SqZMniJPqlkvQPOEZCYPHzGKE.png)

You can now easily narrow down slow or unnecessary resources and prevent them from slowing down your automation with the Playwright `route()` API:

// Abort a specific 3rd party script to load

await page.route('<https://events.framer.com/script>', route \=> route.abort());

  

// Abort images

await page.route('\*\*/\*', route \=> {

return route.request().resourceType() === 'image' ? route.abort() : route.continue();

});

Conclusion
----------

Faster browser automation development comes with less code to write, thanks to a Stealth Browser providing automatic captcha solving, proxying, and fingerprinting configuration.

Then, we leverage the Live Session View by reducing the number of iterations to fix selectors or UI navigation.

Finally, optimizing the resources to load for faster automation is achieved with the Chrome Dev Tools timeline view from the Session Debugger.

Give it a try now by starting a Browser Session from the Browserbase Playground.

From building automation to developing AI Agents, working with headless browsers is a craft that relies on many essential steps

How to Develop Browser Automation Faster

Playwright recently won the hearts of developers and became the most-used library to control headless browsers programmatically.

Its multi-language and multi-browser support makes it the perfect framework for various use cases, from testing to building automation and scraping scripts.

This article digs into its modern API with complete examples showcasing how to get the best of Playwright.

Know your fundamentals
----------------------

First, install Playwright with the `latest` tag to always benefit from the last available features:

`npm i -S playwright-core@latest`

_Note: Use the_ `_playwright-core_` _package instead of the_ `_playwright_` _one to build automation or scraping scripts._

Now, let’s go over some of Playwright's essential concepts before moving forward with essential tips.

### **Key concepts: Browser, BrowserContexts, Pages and Locators**

Playwright enables you to interact with web pages through 4 main concepts: `Browser`, `BrowserContexts`, `Pages`, and `Locators`.

The `Browser`, `BrowserContext`, and `Page` objects represent different layers of the underlying Browser. In short, a `Browser` is identical to a regular Browser process; a `BrowserContext` is an isolated context with dedicated cookies and cache, while a `Page` is an actual Browser tab.

`Locator` enables developers to locate elements on a page in a finer and more stable way than regular CSS Selectors. We will cover how to build stable locators later in this article.

Here is how those 4 pillars concepts are commonly used in code:

import { chromium } from "playwright-core";

(async () \=> {

// 1. Get a Browser instance

const browser = await chromium.launch();

// 2. Get a BrowserContext

const defaultContext = browser.contexts()\[0\];

// 3. Get a Page

const page = defaultContext.pages()\[0\];

// 4. Navigate

await page.goto("<https://browserbase.com>");

// 5. Locate an element

const cta = page.getByRole('link').filter({ hasText: "We're hiring" });

// 6. Act on the element

cta.click()

await page.close();

await browser.close();

})()

Read our Understand Playwright’s BrowserContexts and Pages article to learn how to leverage each of them.

### **The Auto-waiting mechanism**

In addition to the key concepts, Playwright also brings an innovative approach to locators with "Auto-waiting" functionality.

_“Auto-waiting”_ is a feature embedded in `Locator` that guarantees that each action is performed on a “ready to interact” element. In this case, calling `.click()` on a `Locator` will check that:

*   The locator identifies exactly one DOM element.
    
*   The element is visible, meaning it must not be hidden or transparent.
    
*   The element is stable, indicating it is not animating or it has completed its animation.
    
*   The element receives events, ensuring it is not obscured or overlaid by other elements.
    
*   The element is enabled, allowing user interactions like clicks.
    

Below is an example of how that works:

import puppeteer from 'puppeteer';

(async () \=> {

// Launch the browser and open a new blank page

const browser = await puppeteer.launch();

const page = await browser.newPage();

// Navigate the page to a URL

await page.goto('<https://developer.chrome.com/>');

// Type into search box

await page.type('.devsite-search-field', 'automate beyond recorder');

// Wait and click on first result

const searchResultSelector = '.devsite-result-item-link';

await page.waitForSelector(searchResultSelector);

await page.click(searchResultSelector);

// Locate the full title with a unique string

const textSelector = await page.waitForSelector( 'text/Customize and automate'

);

const fullTitle = await textSelector?.evaluate(el \=> el.textContent);

// Print the full title

console.log('The title of this blog post is "%s".', fullTitle);

await browser.close();

})();

As for Playwright, it removes the need to manually check that an element is ready for interaction:

import { chromium } from "playwright-core";

(async () \=> {

const browser = await chromium.launch();

const defaultContext = browser.contexts()\[0\];

const page = defaultContext.pages()\[0\];

await page.locator('.devsite-search-field').pressSequentially(

'automate beyond recorder',

{ delay: 25 }

);

await page.locator('.devsite-result-item-link').first().click();

// Locate the full title with a unique string

const textSelector = await page.locator(

'text/Customize and automate'

);

await textSelector?.waitFor()

const fullTitle = await textSelector?.innerText();

// Print the full title

console.log('The title of this blog post is "%s".', fullTitle);

await browser.close();

})();

### **Evaluating JavaScript (cautiously)**

Playwright, like Puppeteer, enables you to evaluate some JavaScript to run in the `Page`’s context.

Here, we run `page.evaluate()` as a javascript function in the context of the web page. This would bring the page results back into the environment:

import { chromium } from "playwright-core";

(async () \=> {

const browser = await chromium.launch();

const defaultContext = browser.contexts()\[0\];

const page = defaultContext.pages()\[0\];

await page.goto("<https://browserbase.com>");

const href = await page.evaluate(() \=> document.location.href);

console.log(href); // => "<https://browserbase.com>"

await page.close();

await browser.close();

})()

While this feature comes in handy to retrieve values from the page using `window` or `document`, there are some considerations to have:

**Running code within the browser is slow**

The code passed to an `evaluate()` block will run on a browser, which most likely receives lower compute capacity than your program. Also, keep in mind that the code run with `evaluate()` will impact the overall performance of the webpage (which might impact interactivity or loading time).

With this information in mind, make sure to keep `evaluate()` use for the following use cases:

*   Retrieving information private to the Page’s runtime (ex: `window`, globals)
    
*   Run interaction from the Page (ex, authenticated API calls)
    
*   Interact with the Chrome extensions
    
*   Do some Commands Batching (_covered later in this article_)
    

**How to share variables with evaluation blocks**

Can you guess the output of the following code snippet?

import { chromium } from "playwright-core";

  

(async () \=> {

const browser = await chromium.launch();

const defaultContext = browser.contexts()\[0\];

const page = defaultContext.pages()\[0\];

await page.goto("<https://browserbase.com>");

const two = 2 const sum = await page.evaluate(() \=> two + 2);

console.log(sum); // => ?

await page.close();

await browser.close();

})()

Many would expect the above code snippet to output `“4"`; however, it would print an error raised by the Page saying `"ReferenceError: two is not defined"`.

As covered earlier, the code passed to an `evaluate()` block is evaluated in the `Page`'s context. In our example, the `two` variable is not defined in the `Page` context.

This error is tricky as your IDE or TypeScript will never catch it.

Thankfully, `evaluate()` accepts a second argument to “bind” the local variables to the evaluated block. Here is our updated and fixed example:

import { chromium } from "playwright-core";

(async () \=> {

const browser = await chromium.launch();

const defaultContext = browser.contexts()\[0\];

const page = defaultContext.pages()\[0\];

await page.goto("<https://browserbase.com>");

const two = 2

const sum = await page.evaluate((two) \=> two + 2, two);

console.log(sum); // => ?

await page.close();

await browser.close();

})()

How to load pages efficiently
-----------------------------

Choosing the right page loading strategy is crucial, as trying to act on a partially loaded page will prevent locators from finding elements on the page.

The `Page`'s `goto()` method is used to navigate to a URL and offers multiple strategies to evaluate a navigation completion:

import { chromium } from "playwright-core";

(async () \=> { const browser = await chromium.launch();

const defaultContext = browser.contexts()\[0\];

const page = defaultContext.pages()\[0\];

// 4. Navigate and wait for DOM to be ready

await page.goto('<https://www.browserbase.com>', {

waitUntil: 'domcontentloaded',

});

// 5. Locate an element

const cta = page.getByRole('link').filter({ hasText: "We're hiring" });

// 6. Act on the element

cta.click()

await page.close();

await browser.close();

})()

Let’s go over the `page.goto()`'s 4 `waitUntil` possible values and their associated use cases:

`**"commit"**`**: first network request started, and the document is loading**

The `commit` event is emitted right after the document starts loading. This event is mostly an internal one that has very few use cases. This event triggers before the DOM is loaded and parsed, so it can only be used to fetch the raw HTML data from a page.

`**"domcontentloaded"**`**: the DOM is ready to be queried**

This event is based on the `DOMContentLoaded` event, fired by the browser once the `DOM` is loaded in memory and ready to be accessed by Playwright’s locators.

Using this loading strategy is a wise choice when interacting with web pages that can receive interactions without JavaScript (ex, blogs or landings but not SPA).

`**"load"**` **(default): all resources are loaded**

Based on a browser event (the load event), this loading strategy guarantees that the page and all dependent resources, such as stylesheets, scripts, iframes, and images, are fully loaded.

This default strategy (when no `waitUntil` option is given) is useful when interacting with SPAs or extracting data from a webpage containing iframes. However, you will save time by switching to `"domcontentloaded"` when extracting data from simpler web pages (ex, blogs or landings).

`**networkidle**`**: no network request in the last 500ms**

This event is discouraged by the Playwright’s documentation as it does not guarantee that the DOM is ready to be used. Performance-wise, this load page strategy can be an interesting compromise that fits between the `"domcontentloaded"` and `"load"` approaches. A good candidate for `networkidle` could be news websites, as they tend to load fast but still include some interactivity.

A good rule of thumb is to rely on the `"domcontentloaded"` strategy to extract data and rely on the `"load"` strategy when interacting with a page.

Favor Locators to Selectors
---------------------------

Selecting elements by using CSS Selectors leads to weaker scraping and automation, as they may break when a webpage is updated (especially with the rise of generated CSS classes).

Selectors have been renamed to Locators to encourage developers to use `role` attributes instead of CSS Selectors, resulting in more stable Playwright scripts.

Playwright recommends using `Locator` with a mix of `role`\-based selectors and filtering.

The following CSS Selector:

page.locator('.episodes .episode:first-child button.buttonIcon.next-episode-button').click();

Can be translated to the following more stable `Locator`:

getByRole('listitem').first().getByRole('button', { name: 'Next episode' }).click();

The `Locator` offers a rich API that surpasses the power of CSS Selectors, based on more stable selectors, by leveraging user-facing attributes (ARIA `role` or text); here are some good examples:

`**Locator**` **logical operators**

Combine multiple Locators to perform advanced checks on the web page:

// Retrieves a button element with the accessible role 'button' and the label 'New'.

const newEmail = page.getByRole('button', { name: 'New' });

  

// Retrieves an element that contains the text 'Confirm security settings'.

const dialog = page.getByText('Confirm security settings');

  

// Checks if either 'newEmail' or 'dialog' element is visible on the page and waits until one becomes visible.

await newEmail.or(dialog).toBeVisible()

  

// Checks if the 'dialog' element is currently visible on the page.

if (await dialog.isVisible()) {

// If the 'dialog' is visible, clicks the 'Dismiss' button found by its role and name.

await page.getByRole('button', { name: 'Dismiss' }).click();

}

  

// Clicks the 'newEmail' button after handling the 'dialog', or if no dialog appeared.

await newEmail.click();

`**Locator**` **parent selection**

This locator feature helps to achieve what is nearly impossible in CSS: selecting a parent.

// select an element containing the text "Hello"

const child = page.getByText('Hello');

  

// Select an element with the role="listitem" attribute and containing

// the \`child\` element

const parent = page.getByRole('listitem').filter({ has: child });

`**Locator**` **custom CSS pseudo-classes**

Finally, some selectors might be impossible to describe by using CSS or `role`\-based selectors.

The `Locator` CSS pseudo-classes can help in covering such edge-cases (_with the downside of less predictable results_).

Here, we select an element by using a combination of the `:text()` and `:right-of()` CSS pseudo-classes:

// we fill the <input> positioned at the right of the "Username" text (label)

await page.locator('input:right-of(:text("Username"))').fill('value');

Explore how to write Locators based on layout assumptions in this Playwright guide.

Better performance with the Route API, Commands batching, and Parallelization
-----------------------------------------------------------------------------

Playwright programs are slow, spending most of their time waiting for pages to load and elements to be ready for interaction.

In the first section, we saw how to speed up the initial load of a page by leveraging the `waitUntil` option. Let's now explore other tactics to speed up your Playwright program.

**Batching commands**

A complex use case might result in a chain of Locators and actions as follows:

// 3 roundtrips with the underlying browser instance

const btn = await page.locator('.myButton')

await btn.click();

const buttonText = await btn.allInnerTexts()\[0\];

Each `await` Locator call results in a roundtrip with the Browser instance, resulting in latency.

When facing such a scenario, a good practice is to group related selectors and actions in `evaluate()` blocks as follows:

// 1 roundtrip with the underlying browser instance

const buttonText = await page.evaluate(() \=> {

const btn = document.querySelector('.myButton');

btn.click();

return btn.innerText;

});

_Command batching should be used cautiously, as selectors defined in an_ `_evaluate()_` _block won’t benefit from the “Auto-waiting” mechanism covered earlier._

**Filter out unnecessary HTTP requests and resources**

Another good way to speed up a Playwright program is to prevent any heavy or useless resources from loading. Resources such as videos or images are rarely useful for programs and slow down the initial `"load"` event of the browser.

Fortunately, the Playwright `Route` API is a unique feature that makes it easy to filter out some specific or pattern of network requests:

// Abort a specific 3rd party script to load

await page.route('<https://events.framer.com/script>', route \=> route.abort());

  

// Abort images

await page.route('\*\*/\*', route \=> {

return route.request().resourceType() === 'image' ? route.abort() : route.continue();

});

**Navigate through multiple pages in parallel**

Some scenarios navigate an array of pages that, if processed sequentially, can run for hours upon completion.

It is a common practice to try to parallelize repetitive scenarios by leveraging parallel navigations.

Leveraging multiple `BrowserContext` with a pool of Promises is a viable approach for navigating static or simple web pages such as Wikipedia or blogs.

However, applying this pattern to webpages using JavaScript (therefore being resources-intensive) will cap the parallelization effect to the browser’s allocated resources.

For this reason, a performant and efficient way to process many web pages relies on a good headless architecture. Let’s dive into this topic in our next section.

Using reliable headless browsers
--------------------------------

As covered with the essential tips shared in this article, writing performant and stable Playwright programs is a craft. Still, it is only the tip of the iceberg, as ensuring that the underlying browser will run reliably and undetected is another craft that would take another full article to cover.

Browserbase lets you focus on your code and manage your headless browsers. Browserbase comes with several robust features, including:

*   **Enhanced Observability**: Monitor your browser sessions in detail with the **Session Inspector**.
    
*   **Stealth Operations**: Benefit from a **Stealth browser** that handles captchas automatically and integrates proxies for smoother browsing.
    
*   **Advanced Capabilities:** Support for advanced features such as **Custom Extensions**, \*\*File downloads,\*\* and long-running Sessions.
    
*   **Flexible Integration**: Easily integrate APIs to access features like a **live view of sessions** and **retrieve session logs and recordings**.
    
*   **Robust Infrastructure:** A secure and scalable infrastructure with **fair and transparent pricing**.
    

Get started in under 5 minutes with Playwright with our ready-to-use Github template: https://github.com/browserbase/quickstart-playwright-js.

This article digs into its modern API with complete examples showcasing how to get the best of Playwright

The Browser Blog

Building the future of web automation

Building the future of web automation

Read Article

Scaling in Magnitudes: PromptLoop’s Success with Browserbase

Read Article

How Browserbase transformed Benny’s SNAP benefits platform

Read Article

How Structify uses Browserbase for AI-driven web data collection

Read Article

Why Coframe moved their automations to Browserbase

Read Article

Browserbase's Pledge to Open Source

Read Article

Slack Scout

Read Article

Understand Playwright's BrowserContexts and Pages

Read Article

How to Develop Browser Automation Faster

Read Article

Get the best of Playwright

Read Article

Browserbase

Stagehand

Director

MCP

The Browser Blog

Building the future of web automation

Building the future of web automation

Read Article

Scaling in Magnitudes: PromptLoop’s Success with Browserbase

Read Article

How Browserbase transformed Benny’s SNAP benefits platform

Read Article

How Structify uses Browserbase for AI-driven web data collection

Read Article

Why Coframe moved their automations to Browserbase

Read Article

Browserbase's Pledge to Open Source

Read Article

Slack Scout

Read Article

Understand Playwright's BrowserContexts and Pages

Read Article

How to Develop Browser Automation Faster

Read Article

Get the best of Playwright

Read Article