Detect bots/crawlers/spiders using the user agent string
Detect bots/crawlers/spiders using the user agent string
Teach daily is web crawl from dev.to, freecodecamp, medium, hashnode
Teach daily is web crawl from dev.to, freecodecamp, medium, hashnode
Project crawl data with golang
Project crawl data with golang
Crawler for Revista dos Tribunais
Crawler for Revista dos Tribunais
Rapid Smart Contract Crawler
Rapid Smart Contract Crawler
A sample & efficient web crawler for golang
A sample & efficient web crawler for golang
Crawls the website and finds broken social media links that can be hijacked
Crawls the website and finds broken social media links that can be hijacked
A crawler that tests HTML forms for reflection
A crawler that tests HTML forms for reflection
A crawler that tests HTML forms for reflection
A crawler that tests HTML forms for reflection
Go Site Crawler - a simple application written in go that can fetch content from a url endpoint
Go Site Crawler - a simple application written in go that can fetch content from a url endpoint
Remake of the original sqlifinder but in GOlang, and allows for listed targets, domain crawling, and tor connections
Remake of the original sqlifinder but in GOlang, and allows for listed targets, domain crawling, and tor connections
Pholcus: a distributed high-concurrency crawler software written in pure golang
Pholcus: a distributed high-concurrency crawler software written in pure golang
Crawler for StandNews, a news-network shut down by the Chinese Goverment
Crawler for StandNews, a news-network shut down by the Chinese Goverment
Pagser: a simple and deserialize html page to struct based on goquery and struct tags for golang crawler
Pagser: a simple, extensible, configurable parse and deserialize html page to struct based on goquery and struct tags for golang crawler
Elegant Scraper and Crawler Framework for Golang
Elegant Scraper and Crawler Framework for Golang
Geziyor: a fast web crawling & scraping framework for Go. Supports JS rendering
Geziyor: a fast web crawling & scraping framework for Go. Supports JS rendering
A lightweight distributed friendly Golang crawler framework
A lightweight distributed friendly Golang crawler framework
Antch, a fast, powerful and extensible web crawling & scraping framework for Go
Antch, a fast, powerful and extensible web crawling & scraping framework for Go
Go spider: A crawler of vertical communities achieved by GOLANG
Go spider: A crawler of vertical communities achieved by GOLANG
Creeper: The Next Generation Crawler Framework (Go)
Creeper: The Next Generation Crawler Framework (Go)
Distributed web crawler admin platform for spiders management regardless of languages and frameworks
Distributed web crawler admin platform for spiders management regardless of languages and frameworks
Verify IP addresses of respectful crawlers like Googlebot by reverse dns and forward dns lookups
Verify IP addresses of respectful crawlers like Googlebot by reverse dns and forward dns lookups
Curry is a crawler written in Golang with the purpose of verifying the exchange value of USD(Dollar) x BRL(Real) in stores in Paraguay
Curry is a crawler written in Golang with the purpose of verifying the exchange value of USD(Dollar) x BRL(Real) in stores in Paraguay
Take a list of domains and scan for endpoints, secrets, api keys, file extensions, tokens and more
Take a list of domains and scan for endpoints, secrets, api keys, file extensions, tokens and more
A polite, slim and concurrent web crawler written in Go
A polite, slim and concurrent web crawler written in Go
A libp2p DHT crawler that also monitors the liveness and availability of peers
A libp2p DHT crawler that also monitors the liveness and availability of peers
A web crawler for Golang
A web crawler for Golang
IMDb Crawler With Golang
IMDb Crawler With Golang
A small script written in Go to crawl all the scores of the National High School Exam-2021
A small script written in Go to crawl all the scores of the National High School Exam-2021
A Unix-style personal search engine and web crawler for your digital footprint
A Unix-style personal search engine and web crawler for your digital footprint
A simple crawler sending Telegram notification when Refurbished Macbook Air / Pro in stock
A simple crawler sending Telegram notification when Refurbished Macbook Air / Pro in stock
An API wrapper for a webcrawler that finds free items off of freecycle.com and returns the in JSON format
An API wrapper for a webcrawler that finds free items off of freecycle.com and returns the in JSON format
A simple crawling study that obtained the new houses for sale in Wuhan, China
A simple crawling study that obtained the new houses for sale in Wuhan, China
Multiplexer: HTTP-Server & URL Crawler
Multiplexer: HTTP-Server & URL Crawler
Simple content crawler for joyreactor.cc
Simple content crawler for joyreactor.cc
New World Auction House Crawler In Golang
New World Auction House Crawler In Golang
A Unix-style personal search engine and web crawler for your digital footprint
Interactive Terraform visualization, State and configuration explorer
A very useful and productive web crawlign framework
A very useful and productive web crawlign framework
A PCPartPicker crawler for Golang
A PCPartPicker crawler for Golang
A simple cralwer with golang
A simple cralwer with golang
Gets joke from icanhazdadjoke.com, prints it
Gets joke from icanhazdadjoke.com, prints it
A companion tool to your existing web scraper that instantly makes it scalable, maintainable, and more unblockable
A companion tool to your existing web scraper that instantly makes it scalable, maintainable, and more unblockable
A Fast, highly configurable, cloud native dark web crawler Written in go
Fast, highly configurable, cloud native dark web crawler.
Web crawling by using golang
Construct a product price comparison service that enables customers to search and compare prices from two online shopping platforms (Amazon and Walmart) simultaneously.
A simple web crawler in golang
A simple web crawler in golang
A crawler/scraper based on golang + colly, configurable via JSON
This a very thin layer on top of Colly which allows configuration from a JSON file. The output is JSONL which is ready to be imported into Typesense.
A IMDb Crawler Made using Go + Colly
Want to know which celebrities have a common birthday with yours? Get the full data about them. Made using Go + Colly
GoSpider - Fast web spider written in Go
GoSpider - Fast web spider written in Go