safety: Malware Domains is dead

[dgw]

Description

A cybersec firm bought Malware Domains and shut down its information feeds as of November 30, 2020.

Partly, this means that safety no longer works. But it does support checking against VirusTotal, right? (If one acquires an API key.) Problem is that safety won't even load now that it can't fetch Malware Domains' list.

Reproduction steps

1. Start Sopel with the safety plugin enabled (which it is by default)
2. safety throws an exception during setup and fails to load

Expected behavior

safety at least continues to operate using the last-fetched malwaredomains.txt and/or VirusTotal API access (if configured).

Logs

[2020-12-22 13:35:56,136] sopel.modules.safety INFO     - Downloading malwaredomains db from https://mirror1.malwaredomains.com/files/justdomains
[2020-12-22 13:35:56,332] sopel.bot            ERROR    - Error in safety setup: HTTP Error 404: Not Found
Traceback (most recent call last):
  File "/mnt/c/Users/dgw/github/sopel/sopel/bot.py", line 310, in setup_plugins
    plugin.setup(self)
  File "/mnt/c/Users/dgw/github/sopel/sopel/plugins/handlers.py", line 265, in setup
    self._module.setup(bot)
  File "/mnt/c/Users/dgw/github/sopel/sopel/modules/safety.py", line 98, in setup
    _download_malwaredomains_db(loc)
  File "/mnt/c/Users/dgw/github/sopel/sopel/modules/safety.py", line 116, in _download_malwaredomains_db
    urlretrieve(url, path)
  File "/usr/lib/python3.6/urllib/request.py", line 248, in urlretrieve
    with contextlib.closing(urlopen(url, data)) as fp:
  File "/usr/lib/python3.6/urllib/request.py", line 223, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.6/urllib/request.py", line 532, in open
    response = meth(req, response)
  File "/usr/lib/python3.6/urllib/request.py", line 642, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python3.6/urllib/request.py", line 570, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.6/urllib/request.py", line 650, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 404: Not Found

Environment

• Sopel .version: master or 7.0.x
• Sopel installed via: pip (-e or from PyPI)
• Python version: 3.6.9 or 3.8.2
• Operating system: Any
• Relevant plugins: safety

Notes

I created a question on the Security Stack Exchange asking for Malware Domains replacement options. Stack Exchange is so useless; they close just about everything as off-topic 😡

At minimum, this issue should be resolved by a PR that removes malwaredomains.txt fetching. Ideally that PR would replace it with an equivalent service, and add error handling/logging for if fetching the new list fails.

If Malware Domains cannot be replaced, and VirusTotal's API becomes the only option, remove the safety plugin to its own package (core plugins MUST NOT require an API key to function).

[kwaaak]

There are a couple of candidates on https://github.com/StevenBlack/hosts and on https://firebog.net/ ("Malicious")

[dgw]

@kwaaak Thanks for those. StevenBlack looked the most promising, so I used it for #2012. Eventually safety should probably have a configurable list provider, but this is good enough for 7.1.