Skip to content

@TeamHG-Memex TeamHG

Change the repository type filter

All

    Repositories list

    74 repositories

    • scrash-pageuploader

      Public
      [UNMAINTAINED] S3 Uploader pipelines for HTML and screenshots rendered by Splash
      Python
      •1•1•4•0•Updated Mar 19, 2026Mar 19, 2026

    • Formasaurus

      Public
      Formasaurus tells you the type of an HTML form and its fields using machine learning
      HTML
      •45•121•15•0•Updated Mar 19, 2026Mar 19, 2026

    • broadcrawl

      Public
      Middleware that limits number of internal/external links during broad crawl
      Python
      •2•2•4•0•Updated Mar 19, 2026Mar 19, 2026

    • crazy-form-submitter

      Public
      A library to collect data from search forms
      Python
      •3•0•4•0•Updated Mar 19, 2026Mar 19, 2026

    • autologin

      Public
      A project to attempt to automatically login to a website given a single seed
      Python
      •
      Apache License 2.0
      •41•129•13•5•Updated Mar 19, 2026Mar 19, 2026

    • scrapy-login

      Public
      [UNMAINTAINED] A middleware that provides continuous site login facility
      Python
      •1•4•4•0•Updated Feb 23, 2026Feb 23, 2026

    • page-compare

      Public
      Simple heuristic for measuring web page similarity (& data set)
      HTML
      •16•91•4•0•Updated Feb 23, 2026Feb 23, 2026

    • trec-dd.org

      Public
      web site for TREC Dynamic Domain
      HTML
      •
      MIT License
      •5•1•0•0•Updated Feb 23, 2026Feb 23, 2026

    • fortia

      Public
      [UNMAINTAINED] Firefox addon for Scrapely
      JavaScript
      •4•5•6•0•Updated Feb 23, 2026Feb 23, 2026

    • scrapy-dockerhub

      Public
      [UNMAINTAINED] Deploy, run and monitor your Scrapy spiders.
      Python
      •
      Apache License 2.0
      •12•12•3•0•Updated Feb 23, 2026Feb 23, 2026

    • tldextract

      Public
      Accurately separate the TLD from the registered domain and subdomains of a URL, using the Public Suffix List.
      Python
      •
      Other
      •212•2•0•0•Updated Feb 23, 2026Feb 23, 2026

    • Datawake

      Public
      Browser extension and backend services aimed at enhancing Internet search with domain specific knowledge, collaboration, and analysis.
      JavaScript
      •
      Apache License 2.0
      •5•5•0•0•Updated Feb 23, 2026Feb 23, 2026

    • scrash-lua-examples

      Public
      A collection of example LUA scripts and JS utilities
      JavaScript
      •3•7•4•0•Updated Feb 23, 2026Feb 23, 2026

    • sitehound

      Public
      This is the facade for installation and access to the individual components
      Shell
      •
      Apache License 2.0
      •6•16•3•0•Updated Feb 10, 2026Feb 10, 2026

    • scrapy-kafka-export

      Public
      Scrapy extension which writes crawled items to Kafka
      Python
      •
      MIT License
      •8•31•5•0•Updated Feb 10, 2026Feb 10, 2026

    • privoxy

      Public
      Privoxy HTTP Proxy based on jess/privoxy
      Dockerfile
      •
      Other
      •2•7•3•0•Updated Feb 10, 2026Feb 10, 2026

    • url-summary

      Public
      Show summary of a large number of URLs in a Jupyter Notebook
      Python
      •
      MIT License
      •7•19•3•0•Updated Feb 10, 2026Feb 10, 2026

    • proxy-middleware

      Public
      Scrapy middleware that reads proxy config from settings
      Python
      •
      MIT License
      •4•4•3•0•Updated Feb 10, 2026Feb 10, 2026

    • tor-proxy

      Public
      a tor socks proxy docker image
      Dockerfile
      •
      Other
      •4•12•3•0•Updated Feb 10, 2026Feb 10, 2026

    • sitehound-backend

      Public
      Sitehound's backend
      HTML
      •
      Apache License 2.0
      •4•7•3•0•Updated Feb 10, 2026Feb 10, 2026

    • sitehound-frontend

      Public
      Site Hound (previously THH) is a Domain Discovery Tool
      topicsdomain-discovery
      topicsdomain-discovery
      HTML
      •
      Apache License 2.0
      •9•24•5•1•Updated Feb 10, 2026Feb 10, 2026

    • program-index

      Public
      A list of memex-related tools and their repository URLs
      MIT License
      •57•6•0•0•Updated Feb 10, 2026Feb 10, 2026

    • scrapy-crawl-once

      Public
      Scrapy middleware which allows to crawl only new content
      scrapy
      scrapy
      Python
      •
      MIT License
      •21•80•7•2•Updated Feb 10, 2026Feb 10, 2026

    • fuzzyset

      Public
      A simple fuzzy matching set for python strings
      Python
      •48•1•0•0•Updated Feb 10, 2026Feb 10, 2026

    • extract-html-diff

      Public
      extract difference between two html pages
      htmldiff
      htmldiff
      HTML
      •
      MIT License
      •5•33•5•0•Updated Feb 10, 2026Feb 10, 2026

    • scrapy-rotating-proxies

      Public
      use multiple proxies with Scrapy
      proxyscrapy
      proxyscrapy
      Python
      •
      MIT License
      •158•774•48•5•Updated Feb 10, 2026Feb 10, 2026

    • linkdepth

      Public
      [UNMAINTAINED] scrapy spider to check link depth over time
      Python
      •1•4•3•0•Updated Feb 10, 2026Feb 10, 2026

    • memex-cdr

      Public
      This repository hosts code and schema information related to the Memex Crawl Data Repository (CDR)
      Python
      •8•1•0•0•Updated Feb 10, 2026Feb 10, 2026

    • tensorboard_logger

      Public
      Log TensorBoard events without touching TensorFlow
      Python
      •
      MIT License
      •48•629•12•1•Updated Feb 10, 2026Feb 10, 2026

    • hh-deep-deep

      Public
      THH ↔ deep-deep integration
      public-wip
      public-wip
      Python
      •
      MIT License
      •2•3•2•0•Updated Feb 10, 2026Feb 10, 2026
    ProTip! When viewing an organization's repositories, you can use the props. filter to filter by custom property.