Would an easier way to scrape 100s of websites be useful to you?

In the process of building AI agents, we’ve found that what we built could eventually be good at dynamically scraping data across a variety of websites (10s to 100s of different sites at a time)

Our understanding is that existing web scraping tools are bad at this because you need to write custom scraping configurations per site. Not only that, but when a site changes styling, it might completely break your automation. With agents however, you can provide a high level natural language overview of the data you’d like from a website or class of websites, and the agent system will deal with the details of traversing a page and fetching data automatically.

We’re curious how useful this might be for people. If you’ve experienced issues that this might solve or have already explored the space, I’d love to hear from you

submitted by /u/asim-shrestha
[link] [comments]

September 23, 2023
Read More >>

Friday Squid Blogging: New Squid Species

An ancient squid:

New research on fossils has revealed that a vampire-like ancient squid haunted Earth’s oceans 165 million years ago. The study, published in June edition of the journal Papers in Palaeontology, says the creature had a bullet-shaped body with luminous organs, eight arms and sucker attachments. The discovery was made by scientists in France, who used modern imaging technique to analyse the previously discovered fossils. The ancient squid has been named Vampyrofugiens atramentum, which stands for the “fleeing vampire”. The researchers said that these features have never been recorded before…

September 23, 2023
Read More >>