running the blog extraction fails with Ollama client and Qwen3-8b, wondering the open source model fails so bad #1318
Description
i am running an extraction task with a very simple sample blog page - https://dummy-press-releases.surge.sh/news , the AI model is based on (qwen3:8b) with ollama. The code is running well but the results is really bad,
import { Stagehand } from "@browserbasehq/stagehand";
import { z } from "zod";
import { AISdkClient } from "./external_clients/aisdk";
import { createOllama } from 'ollama-ai-provider-v2';
const ollama = createOllama( { baseURL:'http://localhost:11434/api' });
const stagehand = new Stagehand({
env: "LOCAL",
verbose: 2,
cacheDir: "act-cache", // Specify a cache directory
domSettleTimeout: 15000, // Longer wait for stability
llmClient: new AISdkClient({
model: ollama('qwen3:8b')
}),
});
await stagehand.init();
const pages = stagehand.context.pages();
await news( stagehand , ),
export async function news( stagehand:Stagehand , page: Page) {
await page.goto("https://dummy-press-releases.surge.sh/news", {
waitUntil: "domcontentloaded",
});
const elements = await stagehand.observe("find all news listings");
console.log("Found elements:", elements.map(e => e.description));
await new Promise((resolve) => setTimeout(resolve, 5000));
await stagehand.act("scroll to the bottom of the page", { page: page });
const rawResult = await stagehand.extract(
"extract the title and corresponding publish date of EACH AND EVERY press releases on this page. DO NOT MISS ANY PRESS RELEASES.",
z.object({
items: z.array(
z.object({
title: z.string().describe("The title of the press release"),
publish_date: z
.string()
.describe("The date the press release was published"),
}),
),
}),
);
console.log( rawResult );
}
The debugging shown as blew
[2025-11-29 07:14:44.739 +0800] INFO: Launching local browser
category: "init"
[2025-11-29 07:15:10.258 +0800] INFO: starting observation
category: "observation"
instruction: "find all news listings"
[2025-11-29 07:15:12.866 +0800] INFO: Got accessibility tree data
category: "observation"
Found elements: [
'Brad Lander for Comptroller',
'An Unassuming Liberal Makes a Rapid Ascent to Power Broker',
'Sign up for campaign updates and news about Brad',
'Form for FastAction',
'Content info and navigation links',
'Paid for by Lander 2025'
]
[2025-11-29 07:22:45.774 +0800] INFO: found elements
category: "observation"
elements: [
{
"description": "Brad Lander for Comptroller",
"method": "click",
"arguments": [
"https://www.landerfornyc.com"
],
"selector": "xpath=/html[1]/body[1]/div[1]/div[3]/div[1]/main[1]/div[2]/div[1]/div[1]/div[1]/div[2]/div[1]/div[1]/div[1]/div[1]/div[28]/div[1]/a[1]"
},
{
"description": "An Unassuming Liberal Makes a Rapid Ascent to Power Broker",
"method": "click",
"arguments": [
"https://www.nytimes.com/2014/01/23/nyregion/brad-lander-an-unassuming-liberal-makes-a-rapid-ascent-to-power-broker.html"
],
"selector": "xpath=/html[1]/body[1]/div[1]/div[3]/div[1]/main[1]/div[2]/div[1]/div[1]/div[1]/div[2]/div[1]/div[1]/div[1]/div[1]/div[29]/div[1]/a[1]"
},
{
"description": "Sign up for campaign updates and news about Brad",
"method": "click",
"arguments": [
"https://www.landerfornyc.com"
],
"selector": "xpath=/html[1]/body[1]/div[1]/div[3]/div[1]/main[1]/div[2]/div[1]/div[1]/div[1]/div[3]/div[1]/div[1]/div[1]/section[1]"
},
{
"description": "Form for FastAction",
"method": "submit",
"arguments": [
"POSTAL CODE",
"EMAIL",
"Remember me so that I can use FastAction next time."
],
"selector": "xpath=/html[1]/body[1]/div[1]/div[3]/div[1]/main[1]/div[2]/div[1]/div[1]/div[1]/div[3]/div[1]/div[1]/div[1]/section[1]/form[1]"
},
{
"description": "Content info and navigation links",
"method": "click",
"arguments": [
"https://www.landerfornyc.com"
],
"selector": "xpath=/html[1]/body[1]/div[1]/div[3]/div[1]/footer[1]"
},
{
"description": "Paid for by Lander 2025",
"method": "click",
"arguments": [
"https://www.landerfornyc.com"
],
"selector": "xpath=/html[1]/body[1]/div[1]/div[3]/div[1]/footer[1]/div[1]/div[1]/div[1]/div[4]/div[1]"
}
]
[2025-11-29 07:23:27.873 +0800] DEBUG: final tail
category: "deep-hop"
frameId: "191A3347D0EF42D4824C7BC49215C7BB"
finalSelector: "xpath=/html[1]/body[1]/div[1]/div[3]/div[1]/main[1]/div[2]/div[1]/div[1]/div[1]/div[3]/div[1]/div[1]"
[2025-11-29 07:23:27.883 +0800] DEBUG: performing understudy method
category: "action"
xpath: "xpath=/html[1]/body[1]/div[1]/div[3]/div[1]/main[1]/div[2]/div[1]/div[1]/div[1]/div[3]/div[1]/div[1]"
method: "GET"
url: "https://dummy-press-releases.surge.sh/news"
[2025-11-29 07:23:27.883 +0800] INFO: chosen method is invalid
category: "action"
method: "GET"
[2025-11-29 07:23:27.883 +0800] INFO: error performing method
category: "action"
trace: "UnderstudyCommandException: Method GET not supported\n
method: "GET"
xpath: "xpath=/html[1]/body[1]/div[1]/div[3]/div[1]/main[1]/div[2]/div[1]/div[1]/div[1]/div[3]/div[1]/div[1]"
error: "Method GET not supported"
[2025-11-29 07:23:27.884 +0800] INFO: Error performing action. Reprocessing the page and trying again
category: "action"
action: {
"description": "A section containing a form for signing up for campaign updates and news about Brad Lander.",
"method": "GET",
"arguments": [],
"selector": "xpath=/html[1]/body[1]/div[1]/div[3]/div[1]/main[1]/div[2]/div[1]/div[1]/div[1]/div[3]/div[1]/div[1]"
}
error: "Method GET not supported"
[2025-11-29 07:24:27.777 +0800] DEBUG: final tail
category: "deep-hop"
frameId: "191A3347D0EF42D4824C7BC49215C7BB"
finalSelector: "xpath=/html[1]/body[1]/div[1]/div[3]/div[1]/main[1]/div[2]/div[1]/div[1]/div[1]/div[3]/div[1]/div[1]"
[2025-11-29 07:24:27.920 +0800] DEBUG: performing understudy method
category: "action"
xpath: "xpath=/html[1]/body[1]/div[1]/div[3]/div[1]/main[1]/div[2]/div[1]/div[1]/div[1]/div[3]/div[1]/div[1]"
method: "GET"
url: "https://dummy-press-releases.surge.sh/news"
[2025-11-29 07:24:27.920 +0800] INFO: chosen method is invalid
category: "action"
method: "GET"
[2025-11-29 07:24:27.920 +0800] INFO: error performing method
category: "action"
trace: "UnderstudyCommandException: Method GET not supported\n
method: "GET"
xpath: "xpath=/html[1]/body[1]/div[1]/div[3]/div[1]/main[1]/div[2]/div[1]/div[1]/div[1]/div[3]/div[1]/div[1]"
error: "Method GET not supported"
[2025-11-29 07:24:28.314 +0800] INFO: Starting extraction using a11y snapshot
category: "extraction"
instruction: "extract the title and corresponding publish date of EACH AND EVERY press releases on this page. DO NOT MISS ANY PRESS RELEASES."
{
items: [
{
title: "Brad Lander's Campaign",
publish_date: '2024-04-15T12:00:00Z'
},
{
title: 'A Fiscal Watchdog in a City of $99 Billion',
publish_date: '2024-04-15T12:00:00Z'
},
{
title: 'From Housing Advocate to Power Broker',
publish_date: '2024-04-15T12:00:00Z'
},
{
title: 'An Unassuming Liberal Makes a Rapid Ascent to Power Broker',
publish_date: '2014-01-23T12:00:00Z'
},
{
title: 'Sign up for campaign updates and news about Brad',
publish_date: '2024-04-15T12:00:00Z'
},
{
title: 'Sign up for campaign updates and news about Brad!',
publish_date: '2024-04-15T12:00:00Z'
}
]
}
[2025-11-29 07:36:17.314 +0800] INFO: Extraction completed successfully
category: "extraction"
prompt_tokens: "4548"
completion_tokens: "263"
inference_time_ms: "708887"
As shown in the debugging ,
Firstly in the observation stage,
Found elements: [
'Brad Lander for Comptroller',
'An Unassuming Liberal Makes a Rapid Ascent to Power Broker',
'Sign up for campaign updates and news about Brad',
'Form for FastAction',
'Content info and navigation links',
'Paid for by Lander 2025'
]
the elements it found are not what it is expected as the instruction of getting the listing of the news.
Secondly it tried to do the action on the element of '/html[1]/body[1]/div[1]/div[3]/div[1]/main[1]/div[2]/div[1]/div[1]/div[1]/div[3]/div[1]/div[1]', I think that is not the xpath for selecting the news listing , and with a lot of Method GET not supported exceptions .
Thirdly, the final result points to a list of news with merely 5 items ( the real results contains more than 20 news ), the 5 items contains only one true item, the other 4 items are either fake or pointing to the final form
items: [
{
title: "Brad Lander's Campaign", // fake
publish_date: '2024-04-15T12:00:00Z'
},
{
title: 'A Fiscal Watchdog in a City of $99 Billion', // fake
publish_date: '2024-04-15T12:00:00Z'
},
{
title: 'From Housing Advocate to Power Broker', // fake
publish_date: '2024-04-15T12:00:00Z'
},
{
title: 'An Unassuming Liberal Makes a Rapid Ascent to Power Broker', // only works
publish_date: '2014-01-23T12:00:00Z'
},
{
title: 'Sign up for campaign updates and news about Brad', // the final form
publish_date: '2024-04-15T12:00:00Z'
},
{
title: 'Sign up for campaign updates and news about Brad!', // the final form
publish_date: '2024-04-15T12:00:00Z'
}
]
any body ever testing on any open source model that works well , at least now I tried the Qwen3-8b failed almost 90% of cases.