Back

Show HN: Tarsier – Vision utilities for web interaction agents

171 points22 hoursgithub.com

Hey HN! I built a tool that gives LLMs the ability to understand the visual structure of a webpage even if they don't accept image input. We've found that unimodal GPT-4 + Tarsier's textual webpage representation consistently beats multimodal GPT-4V/4o + webpage screenshot by 10-20%, probably because multimodal LLMs still aren't as performant as they're hyped to be.

Over the course of experimenting with pruned HTML, accessibility trees, and other perception systems for web agents, we've iterated on Tarsier's components to maximize downstream agent/codegen performance.

Here's the Tarsier pipeline in a nutshell:

1. tag interactable elements with IDs for the LLM to act upon & grab a full-sized webpage screenshot

2. for text-only LLMs, run OCR on the screenshot & convert it to whitespace-structured text (this is the coolest part imo)

3. map LLM intents back to actions on elements in the browser via an ID-to-XPath dict

Humans interact with the web through visually-rendered pages, and agents should too. We run Tarsier in production for thousands of web data extraction agents a day at Reworkd (https://reworkd.ai).

By the way, we're hiring backend/infra engineers with experience in compute-intensive distributed systems!

https://reworkd.ai/careers

wyclif1 hour ago

Hey! I'm actually in the Philippines now, and I've spent a lot of time on the island of Bohol, which has the world's greatest concentration of tarsiers. In fact, I visited the Tarsier Wildlife Sanctuary on the island of Bohol, Philippines with my wife, which is the world's main tarsier sanctuary. So I was instantly intrigued by the name of the app.

https://flickr.com/photos/wyclif/3271137617/in/album-7215761...

bckmn22 hours ago

Reminds me of [Language as Intermediate Representation](https://chrisvoncsefalvay.com/posts/lair/) - LLMs are optimized for language, so translate an image into language and they'll do better at modeling it.

KhoomeiK21 hours ago

Cool connection, hadn't seen this before but feels intuitively correct! I also formulate similar (but a bit more out-there) philosophical thoughts on word-meaning as being described by the topological structure of its corresponding images in embedding space, in Section 5.3 of my undergrad thesis [1].

[1] https://arxiv.org/abs/2305.16328

abrichr20 hours ago

Congratulations on shipping!

In https://github.com/OpenAdaptAI/OpenAdapt/blob/main/openadapt... we use FastSAM to first segment the UI elements, then have the LLM describe each segment individually. This seems to work quite well; see https://twitter.com/OpenAdaptAI/status/1789430587314336212 for a demo.

More coming soon!

jackienotchan19 hours ago

Looking at OpenAdapt, I'm wondering why they didn't integrate Tarsier into AgentGPT, which is their flagship github repo but doesn't seem to be under active development anymore.

KhoomeiK19 hours ago

We have a lot more powerful use-cases for Tarsier in web data extraction at the moment. Stay tuned for a broader launch soon!

davedx22 hours ago

How do you make sure the tagging of elements is robust? With regular browser automation it's quite hard to write selectors that will keep working after webpages get updated; often when writing E2E testing teams end up putting [data] attributes into the elements to aid with selection. Using a numerical identifier seems quite fragile.

KhoomeiK22 hours ago

Totally agreed—this is a design choice that basically comes from our agent architecture, and the codegen-based architecture that we think will likely proliferate for web agent tasks in the future. We provide Tarsier's text/screenshot to an LLM and have it write code with generically written selectors rather than the naive selectors that Tarsier assigns to each element.

It's sort of like when you (as a human) write a web scraper and visually click on individual elements to look at the surrounding HTML structure / their selectors, but then end up writing code with more general selectors—not copypasting the selectors of the elements you clicked.

davedx20 hours ago

Ooh that's a very neat approach, great idea! Chains of thought across abstraction layers. Definitely worth a blog post I reckon.

Good luck!

KhoomeiK20 hours ago

Thanks! We might put out a paper about it with some Carnegie Mellon collaborators this summer.

bryanrasmussen8 hours ago

You might like to look at https://tsigalko18.github.io/assets/pdf/2016-Leotta-JSEP.pdf

ROBULA+: An Algorithm for Generating Robust XPath Locators for Web Testing.

ghxst22 hours ago

Great question, also situations where you have multiple CTAs with similar names/contexts on a page is still something I see LLM based automation struggle with.

KhoomeiK22 hours ago

Hm, not sure I follow why those situations would be especially difficult? Regarding website changes, the nice thing about using LLMs is that we can simply provide the previous scraper as context and have it regenerate the scraper to "self-heal" when significant website changes are detected.

dbish22 hours ago

Very cool. We do something similar by combining OCR along with accessiblity data and other data (speech reco et. al.) for desktop based screensharing understanding, but evaluation compared to multi-modal LLMs has not been easy. How are you evaluating to come up with this number "consistently beats multimodal GPT-4V/4o + webpage screenshot by 10-20%,"?

fwiw so far we've seen that Azure has the best OCR for screenshot type data across the proprietary and open source models, though we are far more focused on grabbing data from desktop based applications then web pages so ymmv

KhoomeiK22 hours ago

Yup, evals can definitely be tough. We basically have a suite of several hundred web data extraction evals in a tool we built called Bananalyzer [1]. It's made it pretty straightforward for us to benchmark how accurately our agent generates code when it uses Tarsier-text (+ GPT-4) for perception v.s. Tarsier-screenshot (+ GPT-4V/o).

Will have to look into supporting Azure OCR in Tarsier then—thanks for the tip!

[1] https://github.com/reworkd/bananalyzer

dbish20 hours ago

Awesome, will take a look at this. thank you

timabdulla20 hours ago

Neat. Do you have the Bananalyzer eval results for Tarsier published somewhere?

KhoomeiK20 hours ago

We're hoping to release an evals paper about Bananalyzer this summer and compare Tarsier to a variety of other perception systems in it. The hard part with evaluating a perception/context system though is that it's very intertwined with the agent's architecture, and that's not something we're comfortable fully open-sourcing yet. We'll have to think of interesting ways to decouple the perception system and eval them with Bananalyzer.

SomaticPirate21 hours ago

Surprised to hear Azure beats AWS Textract. I found it to be the best OCR offering but that was when I was doing documents.

navanchauhan20 hours ago

In my experience Azure is probably the best OCR offering right now. They are also the only ones to be able to recognize my terrible handwriting.

dbish20 hours ago

Yes, Textract does not work as well for desktop screenshots from our testing

reidbarber16 hours ago

Neat! Been building something similar to the tagging feature in Typescript: https://github.com/reidbarber/webmarker

The Python API on this is really nice though.

bravura15 hours ago

A few questions:

Does this work in headless mode?

Are you getting a screenshot of the whole webpage including scrolling? Or just the visible part. The whole page, like singlepage.js would be great and is much more useful in many circumstances, although I'm not sure sure how to handle infinite scrolling. (If not, clean simple APIs for scrolling that don't require fiddling and experimentation would be great.)

Instead of Google OCR (the only OCR), what about Apple's native OCR? That would be amazing.

KhoomeiK14 hours ago

Yes it does work headless and we do grab a fullpage screenshot including scrolling (by resizing viewport to content height). We haven’t had to deal with infinite scrolling much but that’s an interesting feature we’d appreciate a PR for.

We haven’t tried Apple’s OCR but hopefully will integrate Azure OCR soon based on others’ advice.

bravura13 hours ago

By Apple OCR, I mean instead of calling an external cloud API which requires tokens, etc. I simply mean vision which runs on OSX. It can be done in about 30 lines of Swift code.

savy9120 hours ago

Am I wrong thinking this could very well be the backbone of an alternative to the Rabbit AI? Where you basically end up having possibly infinite tools for your LLM assistant to use to reach a goal without having to build api integrations.

KhoomeiK20 hours ago

Yup, it could! There are a lot of players in the generalist personal web agent space but I personally think that use-case will be eaten by big players since fundamental foundation model improvements are required. That being said, Tarsier is a great place to start for building an open-source web agent for automating cool little tasks.

At Reworkd, we're focused on web agents for data extraction at scale, which isn't as hyped as the generalist agents but we find provides a lot of value and already works pretty well.

pk1923822 hours ago

this is such a creative solution. reminds me of how a team rendered wolfenstein into ASCII characters and fine tuned mistral to successfully play it.

KhoomeiK22 hours ago

Thanks! Yeah, it seems like a lot can be done with just text while we wait for multimodal models to catch up. The recent Platonic Representation Hypothesis [1] also suggests that different models, regardless of modality, build the same internal representations of the world.

[1] https://arxiv.org/abs/2405.07987

shekhar10121 hours ago

Tangential - I just want a decent (financial transaction) Table to text conversion that can retain the table structure well enough (e.g. merged cells) and have tried everything under the sun short of fine tuning my own model, including all the multimodal LLMs. None of them work very well without a lot of prompt engineering on case by case basis. Can this help? How can I set it up with a large number of pdfs that are sorted by type and extract tabular information? Any other suggestions?

derefr20 hours ago

Or how about the opposite? Give me a CLI tool to pipe implicitly-tabular space-padded text into — a smart cut(1) — where I can say "give me column 3" and it understands how to analyze the document as a whole (or at least a running sample of a dozen lines or so), to model the correct column boundaries, to extract the contents of that column. (Which would also include trimming off any space-padding from the content. I want the data, not a fixed-width field containing it!)

For that matter, give me a CLI tool that takes in an entire such table, and lets me say "give me rows 4-6 of column Foo" — and it reads the table's header (even through fancy box-drawing line-art) to determine which column is Foo, ignores any horizontal dividing lines, etc.

I'm not sure whether these tasks actually require full-on ML — probably just a pile of heuristics would work. Anything would be better than the low-level tools we have today.

KhoomeiK21 hours ago

That's an interesting problem—Tarsier probably isn't the best solution here since it's focused on webpage perception rather than any kind of OCR. But one could try adapting the `format_text` function in tarsier/text_format.py to convert any set of OCR annotations to a whitespace-structured string. Curious to see if that works.

vikp20 hours ago

This isn't specifically tuned for tables (more for general pdf to markdown), but it's worked for some people with similar use-cases - https://github.com/VikParuchuri/marker

Oras21 hours ago

Have you tried AWS textract for table extraction then LLM to format the data?

davedx20 hours ago

Azure have a decent set of offerings for this too, they work quite well: https://learn.microsoft.com/en-gb/azure/ai-services/document...

davedx20 hours ago

I'm having decent success with GPT4o on this. Have you given it a try? It probably varies from table structure to table structure.

shodai8020 hours ago

How do you know, for a specific webelement, what label it is associated with for a textbox or select?

For instance, I might want to tag as you did where elements are, but I still need an association with a label, quite often, to determine what the actual context of the textbox or select is.

awtkns20 hours ago

Tarsier provides a mapping of element number (eg: [23]) to xpath. So for any tagged item we're able to map it back to the actual element in the DOM, allowing for easy interaction with the elements on the page.

shodai8020 hours ago

I understand that, I assume you are tagging the node and making a basic xpath to the node/attribute with your tag id. Understood. But how relevant is tagging a node when I have no idea what the node is actually for?

EX: Given a simple login form, I may not know if the label is above or below the username textbox. A password box would be below it. I have a hard time understanding the relevance to tagging without context.

Tagging is basically irrelevant to any automated task if we do not know the context. I am not trying to diminish your great work, don't get me wrong, but if you don't have context I don't see much relevance. Youre doing something that is easily scripted with xpath templates which I've done for over a decade.

awtkns20 hours ago

This is where a LLM comes it. In a typical pipeline would tag a page, transform it into a textual representation and then pass it to an llm which would be able to reason about which field(s) are the one you're looking for much like a human.

+1
shodai8020 hours ago
jumploops20 hours ago

How does the performance compare to VimGPT[0]?

I assume the screenshot-based approach is similar, whereas the text approach should be improved?

Very cool either way!

[0] https://github.com/ishan0102/vimGPT

KhoomeiK20 hours ago

VimGPT couples the perception to a specific LLM/agent whereas Tarsier is solely a perception system that you can use for any uni/multi-modal web agent. So it's hard to compare, but you could say that VimGPT's performance probably lies somewhere in the middle of Tarsier's performance distribution (which varies as a function of your specific agent/prompt system).

esha_manideep15 hours ago

Great work guys! How did you benchmark traiser's 10-20% better? Would love to see exactly how each method scored

KhoomeiK14 hours ago

Great question! See this thread:

https://news.ycombinator.com/item?id=40369713

v3ss0n20 hours ago

Since it is just a wrapper around hosted API if Google , can't be ran as local fully opensource

KhoomeiK20 hours ago

More OCR providers are on the roadmap and we'd love for you to contribute any local OCR models you think could be useful! I wouldn't call it a wrapper though :)

jadbox13 hours ago

Anything like this for nodejs? (This is py)

jackienotchan20 hours ago

Why was the Show HN text removed? Too much self promotion? You're a YC company, so I'm surprised the mods would do that.

https://hn.algolia.com/?dateRange=pastYear&page=0&prefix=tru...

> Hey HN! I built a tool that gives LLMs the ability to understand the visual structure of a webpage even if they don't accept image input. We've found that unimodal GPT-4 + Tarsier's textual webpage representation consistently beats multimodal GPT-4V/4o + webpage screenshot by 10-20%, probably because multimodal LLMs still aren't as performant as they're hyped to be. Over the course of experimenting with pruned HTML, accessibility trees, and other perception systems for web agents, we've iterated on Tarsier's components to maximize downstream agent/codegen performance.

Here's the Tarsier pipeline in a nutshell:

1. tag interactable elements with IDs for the LLM to act upon & grab a full-sized webpage screenshot

2. for text-only LLMs, run OCR on the screenshot & convert it to whitespace-structured text (this is the coolest part imo)

3. map LLM intents back to actions on elements in the browser via an ID-to-XPath dict

Humans interact with the web through visually-rendered pages, and agents should too. We run Tarsier in production for thousands of web data extraction agents a day at Reworkd (https://reworkd.ai).

By the way, we're hiring backend/infra engineers with experience in compute-intensive distributed systems!

reworkd.ai/careers

dang20 hours ago

Not sure what happened there! I've restored the text now.

KhoomeiK20 hours ago

Thanks for pointing this out! Yeah, it's pretty strange. We thought including Show HN text was encouraged to engage with the community?

jackienotchan20 hours ago

Did you delete it or the mods?

KhoomeiK20 hours ago

Must have been the mods, I spent quite a bit of time on the content lol