Merge pull request #66 from motherduckdb/ducklake-improvements

[ducklake] filter out ducklake metadata tables from metadata/fields, add readme

Author: hrl20

README.md

@@ -59,6 +59,16 @@ JOIN (SELECT * from '/Users/you/movies/title.ratings.parquet') y ON x.tconst = y
 ORDER BY averageRating * numVotes DESC
 ```
 
+## Ducklake
+
+Starting from driver version 1.4.1.0, you can configure the DuckDB data source to point to a ducklake database by setting the database file field to `ducklake:/path/to/db_name.ducklake`. This will also create a folder `/path/to/db_name.ducklake.files`, where the parquet files are stored.
+
+Right now, specifying alternative data path for a brand new ducklake database, like `ATTACH 'ducklake:my_other_ducklake.ducklake' AS my_other_ducklake (DATA_PATH 'some/other/path/');` is not natively supported. But you can first initialize the ducklake in SQL, using another duckdb client or within the Metabase SQL interface, with the target data path, then create the data source attaching the ducklake database already initialized with the target data path. 
+
+### MotherDuck-hosted Ducklake
+If you're using a ducklake database on MotherDuck, it can be attached like a regular MotherDuck database, e.g. `md:my_ducklake_database`. 
+
+
 ## Docker
 
 Unfortunately, DuckDB plugin doesn't work in the default Alpine based Metabase docker container out of the box due to some glibc problems. But we provide a Dockerfile to create a Docker image of Metabase based on Debian where the DuckDB plugin does work.

src/metabase/driver/duckdb.clj

@@ -396,13 +396,13 @@
   (let
    [database_file (get (get database :details) :database_file)
     database_file (first (database-file-path-split database_file))  ;; remove additional options in connection string
-    get_tables_query (str "select * from information_schema.tables "
+    get_tables_query (str "select * from information_schema.tables where table_catalog not like '__ducklake_metadata%' "
                                ;; Additionally filter by db_name if connecting to MotherDuck, since
                                ;; multiple databases can be attached and information about the
                                ;; non-target database will be present in information_schema.
                           (if (is_motherduck database_file)
                             (let [db_name_without_md (motherduck_db_name database_file)]
-                              (format "where table_catalog = '%s' " db_name_without_md))
+                              (format "and table_catalog = '%s' " db_name_without_md))
                             ""))]
     {:tables
      (sql-jdbc.execute/do-with-connection-with-options
@@ -420,7 +420,7 @@
         database_file (first (database-file-path-split database_file))  ;; remove additional options in connection string
         get_columns_query (str
                            (format
-                            "select * from information_schema.columns where table_name = '%s' and table_schema = '%s'"
+                            "select * from information_schema.columns where table_name = '%s' and table_schema = '%s' and table_catalog not like '__ducklake_metadata%%' "
                             table_name schema)
                                   ;; Additionally filter by db_name if connecting to MotherDuck, since
                                   ;; multiple databases can be attached and information about the