Get the best of Playwright

Playwright recently won the hearts of developers and became the most-used library to control headless browsers programmatically.

Its multi-language and multi-browser support makes it the perfect framework for various use cases, from testing to building automation and scraping scripts.

This article digs into its modern API with complete examples showcasing how to get the best of Playwright.

Know your fundamentals

First, install Playwright with the latest tag to always benefit from the last available features:

npm i -S playwright-core@latest

Note: Use the playwright-core package instead of the playwright one to build automation or scraping scripts.

Now, let’s go over some of Playwright's essential concepts before moving forward with essential tips.

Key concepts: Browser, BrowserContexts, Pages and Locators

Playwright enables you to interact with web pages through 4 main concepts: Browser, BrowserContexts, Pages, and Locators.

The Browser, BrowserContext, and Page objects represent different layers of the underlying Browser. In short, a Browser is identical to a regular Browser process; a BrowserContext is an isolated context with dedicated cookies and cache, while a Page is an actual Browser tab.

Locator enables developers to locate elements on a page in a finer and more stable way than regular CSS Selectors. We will cover how to build stable locators later in this article.

Here is how those 4 pillars concepts are commonly used in code:

import { chromium } from "playwright-core"; 
(async () => { 
  // 1. Get a Browser instance 
  const browser = await chromium.launch(); 
  
  // 2. Get a BrowserContext 
  const defaultContext = browser.contexts()[0]; 
  
  // 3. Get a Page 
  const page = defaultContext.pages()[0]; 
  
  // 4. Navigate 
  await page.goto("<https://browserbase.com>"); 
  
  // 5. Locate an element 
  const cta = page.getByRole('link').filter({ hasText: "We're hiring" }); 
  
  // 6. Act on the element 
  cta.click() 
    
  await page.close(); 
  await browser.close(); 
})()

Read our Understand Playwright’s BrowserContexts and Pages article to learn how to leverage each of them.

The Auto-waiting mechanism

In addition to the key concepts, Playwright also brings an innovative approach to locators with "Auto-waiting" functionality.

“Auto-waiting” is a feature embedded in Locator that guarantees that each action is performed on a “ready to interact” element. In this case, calling .click() on a Locator will check that:

• The locator identifies exactly one DOM element.

• The element is visible, meaning it must not be hidden or transparent.

• The element is stable, indicating it is not animating or it has completed its animation.

• The element receives events, ensuring it is not obscured or overlaid by other elements.

• The element is enabled, allowing user interactions like clicks.

Below is an example of how that works:

import puppeteer from 'puppeteer'; 
(async () => { 
  // Launch the browser and open a new blank page 
  const browser = await puppeteer.launch(); 
  const page = await browser.newPage(); 
  
  // Navigate the page to a URL 
  await page.goto('<https://developer.chrome.com/>'); 
  
  // Type into search box 
  await page.type('.devsite-search-field', 'automate beyond recorder'); 
  
  // Wait and click on first result 
  const searchResultSelector = '.devsite-result-item-link'; 
  await page.waitForSelector(searchResultSelector); 
  await page.click(searchResultSelector); 
  
  // Locate the full title with a unique string 
  const textSelector = await page.waitForSelector( 'text/Customize and automate' 
  ); 
  const fullTitle = await textSelector?.evaluate(el => el.textContent); 
  
  // Print the full title 
  console.log('The title of this blog post is "%s".', fullTitle); 
  await browser.close();
})();

As for Playwright, it removes the need to manually check that an element is ready for interaction:

import { chromium } from "playwright-core"; 
(async () => { 
  const browser = await chromium.launch(); 
  const defaultContext = browser.contexts()[0]; 
  const page = defaultContext.pages()[0]; 
  
  await page.locator('.devsite-search-field').pressSequentially( 
    'automate beyond recorder', 
    { delay: 25 } 
  ); 
  
  await page.locator('.devsite-result-item-link').first().click(); 
  
  // Locate the full title with a unique string 
  const textSelector = await page.locator( 
    'text/Customize and automate' 
  ); 
  await textSelector?.waitFor() 
    
  const fullTitle = await textSelector?.innerText(); 
  
  // Print the full title 
  console.log('The title of this blog post is "%s".', fullTitle); 
  await browser.close(); 
})();

Evaluating JavaScript (cautiously)

Playwright, like Puppeteer, enables you to evaluate some JavaScript to run in the Page’s context.

Here, we run page.evaluate() as a javascript function in the context of the web page. This would bring the page results back into the environment:

import { chromium } from "playwright-core"; 
(async () => { 
  const browser = await chromium.launch(); 
  const defaultContext = browser.contexts()[0]; 
  const page = defaultContext.pages()[0]; 
  await page.goto("<https://browserbase.com>"); 
  const href = await page.evaluate(() => document.location.href); 
  console.log(href); // => "<https://browserbase.com>" 
  
  await page.close(); 
  await browser.close(); 
})()

While this feature comes in handy to retrieve values from the page using window or document, there are some considerations to have:

Running code within the browser is slow

The code passed to an evaluate() block will run on a browser, which most likely receives lower compute capacity than your program. Also, keep in mind that the code run with evaluate() will impact the overall performance of the webpage (which might impact interactivity or loading time).

With this information in mind, make sure to keep evaluate() use for the following use cases:

• Retrieving information private to the Page’s runtime (ex: window, globals)

• Run interaction from the Page (ex, authenticated API calls)

• Interact with the Chrome extensions

• Do some Commands Batching (covered later in this article)

How to share variables with evaluation blocks

Can you guess the output of the following code snippet?

import { chromium } from "playwright-core"; 

(async () => { 
  const browser = await chromium.launch(); 
  const defaultContext = browser.contexts()[0]; 
  const page = defaultContext.pages()[0]; 
  await page.goto("<https://browserbase.com>"); 
  const two = 2 const sum = await page.evaluate(() => two + 2); 
  console.log(sum); // => ? 
  
  await page.close(); 
  await browser.close(); 
})()

Many would expect the above code snippet to output “4"; however, it would print an error raised by the Page saying "ReferenceError: two is not defined".

As covered earlier, the code passed to an evaluate() block is evaluated in the Page's context. In our example, the two variable is not defined in the Page context.

This error is tricky as your IDE or TypeScript will never catch it.

Thankfully, evaluate() accepts a second argument to “bind” the local variables to the evaluated block. Here is our updated and fixed example:

import { chromium } from "playwright-core"; 
(async () => { 
  const browser = await chromium.launch(); 
  const defaultContext = browser.contexts()[0]; 
  const page = defaultContext.pages()[0]; 
  
  await page.goto("<https://browserbase.com>"); 
  
  const two = 2 
    const sum = await page.evaluate((two) => two + 2, two); 
  console.log(sum); // => ? 
  
  await page.close(); 
  await browser.close(); 
})()

How to load pages efficiently

Choosing the right page loading strategy is crucial, as trying to act on a partially loaded page will prevent locators from finding elements on the page.

The Page's goto() method is used to navigate to a URL and offers multiple strategies to evaluate a navigation completion:

import { chromium } from "playwright-core"; 
(async () => { const browser = await chromium.launch(); 
    const defaultContext = browser.contexts()[0]; 
    const page = defaultContext.pages()[0]; 
              
    // 4. Navigate and wait for DOM to be ready 
    await page.goto('<https://www.browserbase.com>', { 
      waitUntil: 'domcontentloaded', 
    }); 
              
    // 5. Locate an element 
    const cta = page.getByRole('link').filter({ hasText: "We're hiring" }); 
              
    // 6. Act on the element 
    cta.click() 
              
    await page.close(); 
    await browser.close(); 
})()

Let’s go over the page.goto()'s 4 waitUntil possible values and their associated use cases:

"commit": first network request started, and the document is loading

The commit event is emitted right after the document starts loading. This event is mostly an internal one that has very few use cases. This event triggers before the DOM is loaded and parsed, so it can only be used to fetch the raw HTML data from a page.

"domcontentloaded": the DOM is ready to be queried

This event is based on the DOMContentLoaded event, fired by the browser once the DOM is loaded in memory and ready to be accessed by Playwright’s locators.

Using this loading strategy is a wise choice when interacting with web pages that can receive interactions without JavaScript (ex, blogs or landings but not SPA).

"load" (default): all resources are loaded

Based on a browser event (the load event), this loading strategy guarantees that the page and all dependent resources, such as stylesheets, scripts, iframes, and images, are fully loaded.

This default strategy (when no waitUntil option is given) is useful when interacting with SPAs or extracting data from a webpage containing iframes. However, you will save time by switching to "domcontentloaded" when extracting data from simpler web pages (ex, blogs or landings).

networkidle: no network request in the last 500ms

This event is discouraged by the Playwright’s documentation as it does not guarantee that the DOM is ready to be used. Performance-wise, this load page strategy can be an interesting compromise that fits between the "domcontentloaded" and "load" approaches. A good candidate for networkidle could be news websites, as they tend to load fast but still include some interactivity.

A good rule of thumb is to rely on the "domcontentloaded" strategy to extract data and rely on the "load" strategy when interacting with a page.

Favor Locators to Selectors

Selecting elements by using CSS Selectors leads to weaker scraping and automation, as they may break when a webpage is updated (especially with the rise of generated CSS classes).

Selectors have been renamed to Locators to encourage developers to use role attributes instead of CSS Selectors, resulting in more stable Playwright scripts.

Playwright recommends using Locator with a mix of role-based selectors and filtering.

The following CSS Selector:

page.locator('.episodes .episode:first-child button.buttonIcon.next-episode-button').click();

Can be translated to the following more stable Locator:

getByRole('listitem').first().getByRole('button', { name: 'Next episode' }).click();

The Locator offers a rich API that surpasses the power of CSS Selectors, based on more stable selectors, by leveraging user-facing attributes (ARIA role or text); here are some good examples:

Locator logical operators

Combine multiple Locators to perform advanced checks on the web page:

// Retrieves a button element with the accessible role 'button' and the label 'New'. 
const newEmail = page.getByRole('button', { name: 'New' }); 

// Retrieves an element that contains the text 'Confirm security settings'. 
const dialog = page.getByText('Confirm security settings'); 

// Checks if either 'newEmail' or 'dialog' element is visible on the page and waits until one becomes visible. 
await newEmail.or(dialog).toBeVisible() 

// Checks if the 'dialog' element is currently visible on the page. 
if (await dialog.isVisible()) { 
// If the 'dialog' is visible, clicks the 'Dismiss' button found by its role and name. 
await page.getByRole('button', { name: 'Dismiss' }).click(); 
} 

// Clicks the 'newEmail' button after handling the 'dialog', or if no dialog appeared. 
await newEmail.click();

Locator parent selection

This locator feature helps to achieve what is nearly impossible in CSS: selecting a parent.

// select an element containing the text "Hello" 
const child = page.getByText('Hello'); 

// Select an element with the role="listitem" attribute and containing 
// the `child` element 
const parent = page.getByRole('listitem').filter({ has: child });

Locator custom CSS pseudo-classes

Finally, some selectors might be impossible to describe by using CSS or role-based selectors.

The Locator CSS pseudo-classes can help in covering such edge-cases (with the downside of less predictable results).

Here, we select an element by using a combination of the :text() and :right-of() CSS pseudo-classes:

// we fill the <input> positioned at the right of the "Username" text (label) 
await page.locator('input:right-of(:text("Username"))').fill('value');

Explore how to write Locators based on layout assumptions in this Playwright guide.

Better performance with the Route API, Commands batching, and Parallelization

Playwright programs are slow, spending most of their time waiting for pages to load and elements to be ready for interaction.

In the first section, we saw how to speed up the initial load of a page by leveraging the waitUntil option. Let's now explore other tactics to speed up your Playwright program.

Batching commands

A complex use case might result in a chain of Locators and actions as follows:

// 3 roundtrips with the underlying browser instance 
const btn = await page.locator('.myButton') 
await btn.click(); 
const buttonText = await btn.allInnerTexts()[0];

Each await Locator call results in a roundtrip with the Browser instance, resulting in latency.

When facing such a scenario, a good practice is to group related selectors and actions in evaluate() blocks as follows:

// 1 roundtrip with the underlying browser instance 
const buttonText = await page.evaluate(() => { 
  const btn = document.querySelector('.myButton'); 
  btn.click(); 
  return btn.innerText; 
});

Command batching should be used cautiously, as selectors defined in an evaluate() block won’t benefit from the “Auto-waiting” mechanism covered earlier.

Filter out unnecessary HTTP requests and resources

Another good way to speed up a Playwright program is to prevent any heavy or useless resources from loading. Resources such as videos or images are rarely useful for programs and slow down the initial "load" event of the browser.

Fortunately, the Playwright Route API is a unique feature that makes it easy to filter out some specific or pattern of network requests:

// Abort a specific 3rd party script to load 
await page.route('<https://events.framer.com/script>', route => route.abort()); 

// Abort images 
await page.route('**/*', route => { 
  return route.request().resourceType() === 'image' ? route.abort() : route.continue(); 
});

Navigate through multiple pages in parallel

Some scenarios navigate an array of pages that, if processed sequentially, can run for hours upon completion.

It is a common practice to try to parallelize repetitive scenarios by leveraging parallel navigations.

Leveraging multiple BrowserContext with a pool of Promises is a viable approach for navigating static or simple web pages such as Wikipedia or blogs.

However, applying this pattern to webpages using JavaScript (therefore being resources-intensive) will cap the parallelization effect to the browser’s allocated resources.

For this reason, a performant and efficient way to process many web pages relies on a good headless architecture. Let’s dive into this topic in our next section.

Using reliable headless browsers

As covered with the essential tips shared in this article, writing performant and stable Playwright programs is a craft. Still, it is only the tip of the iceberg, as ensuring that the underlying browser will run reliably and undetected is another craft that would take another full article to cover.

Browserbase lets you focus on your code and manage your headless browsers. Browserbase comes with several robust features, including:

• Enhanced Observability: Monitor your browser sessions in detail with the Session Inspector.

• Stealth Operations: Benefit from a Stealth browser that handles captchas automatically and integrates proxies for smoother browsing.

• Advanced Capabilities: Support for advanced features such as Custom Extensions, **File downloads,** and long-running Sessions.

• Flexible Integration: Easily integrate APIs to access features like a live view of sessions and retrieve session logs and recordings.

• Robust Infrastructure: A secure and scalable infrastructure with fair and transparent pricing.

Get started in under 5 minutes with Playwright with our ready-to-use Github template: https://github.com/browserbase/quickstart-playwright-js.