How to: Chrome extension (manifest v3) - capture network logs using webRequest API

Introduction

One of the first tasks I was assigned to at my new workplace ( ✨Deep BlueDot ✨) was to find a way to fetch console logs (info, errors, warnings, asserts etc) and HAR network logs(Network requests and responses) of a webpage and access it through chrome extension.

Visit Devign here:

Devign - A Data-based documentation tool

Super easy browser extension ready for instant disposal. Build a robust team library, communicate based on single source of data-point, and report issues with the right context. Launch more perfect products, faster. Loved by teams that transform good ideas into great products. Team productivity is all about being on the same page Minimize multiple channel communications.

https://www.devign.app/

Devign provides the essential context and data for developers such as console logs, network logs, dev environment, user flow, user footprints, and more—to resolve the issue.

This article shines light upon how the Devign chrome extension fetches the network log information (HAR - HTTP Archive) using Chrome’s webRequest API which is useful for debugging.

⚠️ Warning

HAR logs capture all network information including personal information such as cookies or passwords. Be careful about whom you share the log information with!

How the Chrome Extension webRequest API works

Manifest

Your manifest.json must include webRequest permission in order to access webRequests API

Side Note

The web request API presents an abstraction of the network stack to the extension.

Internally, one URL request can be split into several HTTP requests (for example to fetch individual byte ranges from a large file) or can be handled by the network stack without communicating with the network.

Background (in version 3, referred to as `service workers`)

What service workers do: listen for and respond to events in order to enhance the end user's experience.

webRequest API events used:

onBeforeSendHeaders

Fired before sending an HTTP request, once the request headers are available. This may occur after a TCP connection is made to the server, but before any HTTP data is sent.


chrome.webRequest.onBeforeSendHeaders.addListener(
  callback: function,
  filter: RequestFilter,
  extraInfoSpec?: OnBeforeSendHeadersOptions[],
)

onHeadersReceived

Fired when HTTP response headers of a request have been received.


chrome.webRequest.onHeadersReceived.addListener(
  callback: function,
  filter: RequestFilter,
  extraInfoSpec?: OnHeadersReceivedOptions[],
)

`src/background/index.ts`

Note: The following example stores the network log data in an array but Devign stores the data in a queue.

Ignoring requests initiated by the extension with :if(details.initiator?.includes('chrome-extension://')) return;

Storing requestTimeStamp and responseTimeStamp separately

Example of the Request, Response Header structures (Variable - depends on the individual request type)

Request Headers

Response Headers

`src/background/api.ts` - Post the collected network log data to backend server

Content Scripts

This is the part of the extension which can access the Document Object Model (DOM) of the webpage and tells the service workers when to start collecting the network log data.

Devign extension uses a toggle button to decide whether to collect logs or not. When toggled on, the content script sends a message to the background to capture the network logs using the following code: