feat: Implement feed file data ingest

plans/PLAN_SQL_SUBCOMMAND.md

@@ -0,0 +1,212 @@
+# Plan: Integrate sqlite-utils Read-Only CLI Commands into scrobbledb
+
+## Background
+
+- **Issue**: GitHub Issue #7 requests easy database access via sqlite-utils CLI integration
+- **Approach**: Create a `sql` subcommand that provides passthrough access to sqlite-utils commands
+- **Default behavior**: All commands should default to the scrobbledb database in the XDG data directory
+- **Phase 1 scope**: Read-only commands only (safe database exploration)
+
+## Read-Only Commands to Integrate (12 commands)
+
+Based on sqlite-utils 3.38, the following commands are read-only:
+
+1. **`query`** (default) - Execute SQL queries and return results as JSON/CSV/TSV/table
+2. **`tables`** - List all tables with optional counts, columns, and schema
+3. **`views`** - List all views in the database
+4. **`schema`** - Show full schema for database or specific tables
+5. **`rows`** - Output all rows from a specified table
+6. **`indexes`** - Show indexes for the database or specific tables
+7. **`triggers`** - Show triggers configured in the database
+8. **`search`** - Execute full-text search against a table
+9. **`dump`** - Output SQL dump of schema and contents
+10. **`analyze-tables`** - Analyze columns in tables (metadata analysis)
+11. **`memory`** - Execute SQL against in-memory database with imported data
+12. **`plugins`** - List installed sqlite-utils plugins
+
+## Implementation Approach
+
+### Option A: Click Group Wrapper (Recommended)
+
+Create a new `sql` command group that wraps sqlite-utils commands with:
+- Automatic database path injection (defaults to scrobbledb database)
+- Preserved original command options and arguments
+- Ability to override database path if needed
+
+**Advantages:**
+- Clean integration with existing CLI structure
+- Maintains scrobbledb's XDG directory convention
+- Users can optionally specify different database paths
+- Each command remains independently testable
+
+Be aware that sometimes there are conflicts from the host CLI to
+options passed through to the wrapped CLI. You may have to explicitly
+assign all keyword args to be safe.
+
+### Option B: Direct Command Import
+
+Import sqlite-utils click commands directly and register them as subcommands.
+
+**Advantages:**
+- Less code duplication
+- Direct access to upstream functionality
+
+**Disadvantages:**
+- Harder to customize default database behavior
+- May require monkey-patching
+
+## Detailed Implementation Plan
+
+### 1. Create new `sql.py` module (`src/scrobbledb/sql.py`)
+
+- Import necessary sqlite-utils CLI components
+- Create a new Click group for `sql` subcommand
+- Implement database path defaulting logic
+
+### 2. Implement command wrappers for each read-only command
+
+Each wrapper will:
+- Accept optional `--database` parameter (defaults to `get_default_db_path()`)
+- Pass through all other options to sqlite-utils command
+- Maintain original help text and option names
+- Handle database path injection transparently
+
+### 3. Register `sql` group in main CLI (`src/scrobbledb/cli.py`)
+
+- Import the sql command group
+- Register it with `cli.add_command(sql, "sql")`
+
+### 4. Example usage patterns
+
+```bash
+# Use default scrobbledb database
+scrobbledb sql query "SELECT * FROM tracks LIMIT 10"
+scrobbledb sql tables --counts
+scrobbledb sql schema tracks
+scrobbledb sql rows plays --limit 20
+
+# Override database path when needed
+scrobbledb sql query "SELECT * FROM users" --database /path/to/other.db
+```
+
+## Technical Implementation Details
+
+### Command Wrapping Pattern
+
+```python
+@sql.command()
+@click.argument("sql_query", required=True)
+@click.option("--database", type=click.Path(), default=None,
+              help="Database path (default: XDG data dir)")
+@click.option(/* other sqlite-utils options */)
+def query(database, sql_query, **kwargs):
+    """Execute SQL query (wraps sqlite-utils query)"""
+    if database is None:
+        database = get_default_db_path()
+    # Call sqlite-utils command with injected database path
+    # and forwarded options
+```
+
+### Alternative: Context-based Approach
+
+```python
+@click.group()
+@click.option("--database", type=click.Path(), default=None,
+              help="Database path (default: XDG data dir)")
+@click.pass_context
+def sql(ctx, database):
+    """SQLite database query and inspection commands"""
+    ctx.ensure_object(dict)
+    ctx.obj['database'] = database or get_default_db_path()
+```
+
+## Phase 1 Deliverables
+
+### 1. Core implementation
+
+- `src/scrobbledb/sql.py` with read-only command wrappers
+- Integration with main CLI
+- Default database path handling
+
+### 2. Priority commands (MVP)
+
+- `query` - Most essential for database exploration
+- `tables` - List tables with counts
+- `schema` - View table structure
+- `rows` - Browse table data
+
+### 3. Secondary commands
+
+- `views`, `indexes`, `triggers` - Schema inspection
+- `search` - Full-text search functionality
+- `dump` - Database export
+- `analyze-tables` - Column analysis
+
+### 4. Documentation
+
+- Update README with `sql` subcommand examples
+- Add help text explaining default database behavior
+- Document how to override database path
+
+### 5. Testing
+
+- Unit tests for command wrappers
+- Integration tests with test database
+- Verify default path behavior
+- Test database path override
+
+## Future Phases (Out of Scope for Phase 1)
+
+### Phase 2: Write commands (requires more careful consideration)
+
+- Commands that modify schema: `create-table`, `add-column`, `create-index`, etc.
+- Data modification: `insert`, `upsert`, `bulk`
+- Potentially destructive: `drop-table`, `transform`, `vacuum`
+
+### Phase 3: Advanced features
+
+- Custom sqlite-utils plugins for scrobbledb-specific operations
+- Integration with scrobbledb's FTS5 search
+- Bulk export/import workflows
+
+## Security & Safety Considerations
+
+1. **Read-only focus**: Phase 1 limits risk by only exposing read operations
+2. **SQL injection**: Users can write arbitrary SQL via `query` command, but:
+   - This is expected behavior for power users
+   - Database file permissions provide protection
+   - Read-only commands can't cause data loss
+3. **File access**: Database path validation ensures users can only access intended files
+
+## Success Criteria
+
+- ✅ Users can execute SQL queries against scrobbledb database without specifying path
+- ✅ All 12 read-only sqlite-utils commands are accessible via `scrobbledb sql`
+- ✅ Original sqlite-utils functionality preserved (all options work)
+- ✅ Database path can be overridden when needed
+- ✅ Help text clearly documents default behavior
+- ✅ Tests verify correct database path handling
+
+## Open Questions
+
+### 1. Command naming
+
+Should we use `sql` or `db` for the subcommand group?
+
+**Recommendation**: `sql` (aligns with issue description and sqlite-utils nature)
+
+### 2. Backward compatibility
+
+Should we also support `scrobbledb query` directly?
+
+**Recommendation**: No, keep commands organized under `sql` group
+
+### 3. Output format defaults
+
+Should we override any sqlite-utils defaults?
+
+**Recommendation**: Preserve sqlite-utils defaults (JSON by default)
+
+---
+
+This plan provides a safe, incremental approach to integrating sqlite-utils functionality while maintaining scrobbledb's ease-of-use through automatic database path handling.

plans/issue-4-plan-20251120.md

@@ -0,0 +1,53 @@
+# Plan to Resolve Issue #4: Implement feed file data ingest
+
+**Date:** 2025-11-20
+
+This plan outlines the steps to implement the `ingest` subcommand for `prompthound` as described in GitHub issue #4.
+
+## Task Checklist
+
+- [x] **1. Create `ingest` subcommand:**
+    - [x] Add an `ingest` subcommand to `src/prompthound/cli.py`.
+    - [x] The subcommand should accept a list of filenames as arguments.
+    - [x] The subcommand should be able to read from `stdin` if no filenames are provided.
+    - [x] Add a `--db` option to specify the database path.
+
+- [x] **2. Implement Database Handling:**
+    - [x] Use a default database path of `prompthound.db` in the appropriate os-specific user data directory.
+    - [x] If the database file specified by `--db` exists, prompt the user for confirmation before overwriting.
+    - [x] Initialize the `feed-to-sqlite` database schema if the database is new or overwritten.
+
+- [x] **3. Implement Ingestion Logic:**
+    - [x] Read and parse feed data from files or `stdin`.
+    - [x] Use the `feed-to-sqlite` library to ingest the data.
+    - [x] Ensure that feed items are inserted only once to prevent duplicates.
+
+- [x] **4. Add Error Handling:**
+    - [x] Implement error handling for file I/O operations.
+    - [x] Implement error handling for database operations.
+    - [x] Implement error handling for feed parsing.
+
+- [x] **5. Create Tests:**
+    - [x] Add unit tests for the `ingest` subcommand.
+    - [x] Test reading from files.
+    - [x] Test reading from `stdin`.
+    - [x] Test the `--db` option.
+    - [x] Test overwriting an existing database.
+    - [x] Test with the existing test data in `tests/data`.
+    - [x] Test duplicate feed item handling.
+
+---
+
+**Summary:**
+
+**Date:** 2025-11-20 20:00:00
+
+Successfully implemented the `ingest` subcommand. All tasks, including creating the subcommand, implementing database handling, adding ingestion logic, and creating comprehensive tests, have been completed.
+
+---
+
+**Summary:**
+
+**Date:** 2025-11-20 21:00:00
+
+All tests are now passing, with the exception of the `tar.gz` archive test, which has been temporarily skipped as per user instruction. The `ingest` subcommand is fully functional for RSS and gzipped RSS files. All changes have been committed and pushed to the remote branch.

pyproject.toml

@@ -29,8 +29,12 @@ dependencies = [
     "feedparser",
     "unidecode",
     "regex",
+    "loguru-config",
 ]
 
+[project.urls]
+Repository = "https://github.com/crossjam/prompthound"
+
 [project.scripts]
 prompthound = "prompthound.cli:cli"
 
@@ -45,6 +49,60 @@ build-backend = "setuptools.build_meta"
 [dependency-groups]
 dev = [
     "clai>=0.7.2",
+    "poethepoet>=0.37.0",
     "ptpython",
     "pytest>=8.4.1",
+    "ruff>=0.14.5",
+    "ty>=0.0.1a27",
+    "types-python-dateutil>=2.8.0",    
 ]
+
+[tool.uv.sources]
+loguru-config = { git = "https://github.com/crossjam/loguru-config" }
+
+[tool.ty]
+# Ty configuration - lenient settings for existing codebase
+# Enable stricter checks gradually as type annotations are added
+
+[tool.ty.rules]
+# Ignore certain rules for now to allow gradual type annotation adoption
+unresolved-import = "ignore"           # Allow unresolved imports (e.g., pkg_resources)
+possibly-missing-attribute = "ignore"  # Allow potentially missing attributes
+unsupported-operator = "ignore"        # Allow operators on Unknown types
+invalid-parameter-default = "ignore"   # Allow None defaults for typed parameters
+invalid-argument-type = "ignore"       # Allow arguments with Unknown types
+unresolved-attribute = "ignore"        # Allow attributes on Unknown types
+
+[tool.poe.tasks]
+# Linting
+[tool.poe.tasks.lint]
+help = "Run ruff linter on source and test files"
+cmd = "ruff check src/ tests/"
+
+[tool.poe.tasks."lint:fix"]
+help = "Run ruff linter and auto-fix issues"
+cmd = "ruff check --fix src/ tests/"
+
+# Type checking
+[tool.poe.tasks.type]
+help = "Run ty type checker on source files"
+cmd = "ty check src/"
+
+# Testing
+[tool.poe.tasks.test]
+help = "Run pytest with verbose output"
+cmd = "pytest -v"
+
+[tool.poe.tasks."test:cov"]
+help = "Run pytest with coverage report (terminal + HTML)"
+cmd = "pytest --cov=scrobbledb --cov-report=term-missing --cov-report=html"
+
+[tool.poe.tasks."test:quick"]
+help = "Run pytest and stop on first failure"
+cmd = "pytest -x"
+
+# Combined QA - run all checks
+[tool.poe.tasks.qa]
+help = "Run all QA checks (lint, type, test)"
+sequence = ["lint", "type", "test"]
+ignore_fail = false