ADBC driver doesn't work?

[yutannihilation]

Describe the bug

As of GDAL 3.11, ADBC driver is available. But, it seems it doesn't work with sf. I investigated if this is a problem on GDAL's side (OSGeo/gdal#10887 (comment)), but GDAL seems fine.

Note that, while I can just use the CRAN version on Window, macOS still uses GDAL 3.8.5, and ubuntugis-unstable PPA is also still 3.10.3. If you are on macOS or Ubuntu, you need to prepare the latest GDAL libary and compile sf from source.

To Reproduce

# check if ADBC driver is available
sf::st_drivers() |> dplyr::filter(name == "ADBC")
#>      name                   long_name write  copy is_raster is_vector   vsi
#> ADBC ADBC Arrow Database Connectivity FALSE FALSE     FALSE      TRUE FALSE

# download a test file
dir.create("gdal-tmp", showWarnings = FALSE)
download.file("https://github.com/OSGeo/gdal/raw/refs/heads/master/autotest/ogr/data/parquet/poly.parquet", "gdal-tmp/poly.parquet", mode = "wb")

# make sure the file is a valid parquet format
nanoparquet::read_parquet_info("gdal-tmp/poly.parquet")
#> # A data frame: 1 × 7
#>   file_name           num_cols num_rows num_row_groups file_size parquet_version
#>   <chr>                  <int>    <dbl>          <int>     <dbl>           <int>
#> 1 gdal-tmp/poly.parq…        8       10              1     10227               2
#> # ℹ 1 more variable: created_by <chr>

# download DuckDB (I'm on Windows, so downloads a Windows binary)
download.file("https://github.com/duckdb/duckdb/releases/download/v1.1.3/libduckdb-windows-amd64.zip", "gdal-tmp/libduckdb.zip", mode = "wb")
unzip("gdal-tmp/libduckdb.zip", exdir = "gdal-tmp/libduckdb")

sf::st_read("ADBC:gdal-tmp/poly.parquet", layer = "poly", options = c(
  'PRELUDE_STATEMENTS="load spatial"',
  "ADBC_DRIVER=gdal-tmp/libduckdb/duckdb.dll" # I'm on Windows
))
#> options:        PRELUDE_STATEMENTS="load spatial" ADBC_DRIVER=gdal-tmp/libduckdb/duckdb.dll
#> Warning in CPL_read_ogr(dsn, layer, query, as.character(options), quiet, : GDAL
#> Error 1: AdbcLoadDriver() failed:
#> Error: Cannot open "ADBC:gdal-tmp/poly.parquet"; Check connection parameters.

^{Created on 2025-06-22 with reprex v2.1.1}

If reporting a change from previous versions

no

Additional context
n/a

sessionInfo():

R version 4.5.0 (2025-04-11 ucrt)
Platform: x86_64-w64-mingw32/x64
Running under: Windows 11 x64 (build 26100)

Matrix products: default
  LAPACK version 3.12.1

locale:
[1] LC_COLLATE=Japanese_Japan.utf8  LC_CTYPE=Japanese_Japan.utf8    LC_MONETARY=Japanese_Japan.utf8 LC_NUMERIC=C                    LC_TIME=Japanese_Japan.utf8    

time zone: Asia/Tokyo
tzcode source: internal

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
 [1] vctrs_0.6.5        cli_3.6.5          knitr_1.50         xfun_0.52          rlang_1.1.6        processx_3.8.6     DBI_1.2.3          KernSmooth_2.23-26 generics_0.1.4    
[10] sf_1.0-21          glue_1.8.0         clipr_0.8.0        htmltools_0.5.8.1  e1071_1.7-16       ps_1.9.1           rmarkdown_2.29     grid_4.5.0         evaluate_1.0.3    
[19] classInt_0.4-11    tibble_3.3.0       fastmap_1.2.0      yaml_2.3.10        lifecycle_1.0.4    nanoparquet_0.4.2  compiler_4.5.0     dplyr_1.1.4        fs_1.6.6          
[28] Rcpp_1.0.14        pkgconfig_2.0.3    rstudioapi_0.17.1  digest_0.6.37      R6_2.6.1           class_7.3-23       reprex_2.1.1       tidyselect_1.2.1   utf8_1.2.6        
[37] pillar_1.10.2      callr_3.7.6        magrittr_2.0.3     tools_4.5.0        proxy_0.4-27       withr_3.0.2        units_0.8-7   

sf::sf_extSoftVersion():

          GEOS           GDAL         proj.4 GDAL_with_GEOS     USE_PROJ_H           PROJ 
      "3.13.1"       "3.11.0"        "9.6.0"         "true"         "true"        "9.6.0" 

[yutannihilation]

It seems the ADBC-DuckDB integration currently doesn't work on ARM platforms. I'm still not sure if this is supposed to work on Windows.

OSGeo/gdal#10887 (comment)

[paleolimbot]

As long as the DuckDB R package works we can probably load the driver...I did a POC for this when I was working on the ADBC GDAL stuff:

main...paleolimbot:sf:adbc-experiments

...that version also used a modified copy of adbcdrivermanager whose branch I can't find right now, but you could special-case DuckDB there, too.

[yutannihilation]

Oh, thanks for sharing! The problem seems that, even if GDAL is build with ADBC-support, it doesn't necessarily mean it supports ADBC driver manager. You can check the driver metadata, HAS_ADBC_DRIVER_MANAGER. If it is set to 'YES', GDAL can use ADBC driver manager. If not, I think your POC won't work, and it seems it's the most common case, unfortunately.

from osgeo import gdal

drv = gdal.GetDriverByName("ADBC")
drv.GetMetadataItem("HAS_ADBC_DRIVER_MANAGER")

So, how useful is ADBC driver without ADBC driver manager? GDAL's document suggests that ADBC driver can work without ADBC driver manager if you implement your own loading function. So, this might be easier to you.

https://gdal.org/en/stable/drivers/vector/adbc.html

or it has not, in which case applications embedding GDAL must use GDALSetAdbcLoadDriverOverride() as detailed in a below paragraph. Note that use of that function can also be done even if the driver has been built against the adbc-driver-manager library.

cf. https://github.com/OSGeo/gdal/blob/2cd76a17ba60ddb3246e259e9fcc7ed3c717529c/ogr/ogrsf_frmts/adbc/ogradbcdrivercore.cpp#L104-L106

Now, I'm suggesting to implement a minimal loading function for DuckDB on GDAL's side. As you are very familiar with ADBC, could you verify if my idea is valid?

OSGeo/gdal#12631

[paleolimbot]

GDAL's document suggests that ADBC driver can work without ADBC driver manager if you implement your own loading function.

That's what main...paleolimbot:sf:adbc-experiments implements! (that particular feature is my fault for this express use case 🙂 ).

[yutannihilation]

Ah, sorry, makes sense now. So, are you suggesting to implement the same loading function using the duckdb R package just like you did with the adbcdrivermanager R package?

you could special-case DuckDB there, too.

[paleolimbot]

I found the other half!

apache/arrow-adbc@main...paleolimbot:arrow-adbc:r-adbcdrivermanager-init . I'm not sure that the driver manager is strictly necessary (i.e., we could maybe just load duckdb_adbc_init from sf too).

Now, I'm suggesting to implement a minimal loading function for DuckDB on GDAL's side. As you are very familiar with ADBC, could you verify if my idea is valid?

I think this is OK, although probably better to use the driver embedded in the R build (to avoid two copies of DuckDB!). It may take me a bit to be able to circle back and clean up those two PRs (but probably this is still faster than waiting for a GDAL release).

[yutannihilation]

Thanks!

this is still faster than waiting for a GDAL release

Totally agreed. To be clear

• my pull request on GDAL is not only for R, but more general one (e.g. gdal CLI). So, I think it's still useful even after sf gets the ability to use DuckDB without it.
• on the other hand, I want some faster solution for R

[yutannihilation]

Here's the version that I think is minimum to use DuckDB.

main...yutannihilation:poc/minimal-adbc-duckdb

It works just fine 🎉 But, I'm getting to feel it's better to wait for your proper implementation utilizing the adbcdrivermanager pacakge because

• if we use GDALSetAdbcLoadDriverOverride() it needs to handle all drivers because there's no fallback mechanism
• the implementation anyway needs to know the definition of a few types (e.g. AdbcDriverInitFunc) defined in ADBC-related header files.

# download a test file
dir.create("gdal-tmp", showWarnings = FALSE)
download.file("https://github.com/OSGeo/gdal/raw/refs/heads/master/autotest/ogr/data/parquet/poly.parquet", "gdal-tmp/poly.parquet", mode = "wb")

x <- sf::st_read("ADBC:gdal-tmp/poly.parquet", layer = "poly", options = "ADBC_DRIVER=duckdb")
#> options:        ADBC_DRIVER=duckdb 
#> Reading layer `poly' from data source `ADBC:gdal-tmp/poly.parquet' using driver `ADBC'
#> Simple feature collection with 10 features and 3 fields
#> Geometry type: POLYGON
#> Dimension:     XY
#> Bounding box:  xmin: 478315.5 ymin: 4762880 xmax: 481645.3 ymax: 4765610
#> Projected CRS: OSGB36 / British National Grid

x
#> Simple feature collection with 10 features and 3 fields
#> Geometry type: POLYGON
#> Dimension:     XY
#> Bounding box:  xmin: 478315.5 ymin: 4762880 xmax: 481645.3 ymax: 4765610
#> Projected CRS: OSGB36 / British National Grid
#>           AREA EAS_ID  PRFEDEA                       geometry
#> 1   215229.266    168 35043411 POLYGON ((479819.8 4765180,...
#> 2   247328.172    179 35043423 POLYGON ((480035.3 4765558,...
#> 3   261752.781    171 35043414 POLYGON ((479819.8 4765180,...
#> 4   547597.188    173 35043416 POLYGON ((479014.9 4765148,...
#> 5    15775.758    172 35043415 POLYGON ((479029.7 4765110,...
#> 6   101429.977    169 35043412 POLYGON ((480083 4765050, 4...
#> 7   268597.625    166 35043409 POLYGON ((480389.7 4764950,...
#> 8  1634833.375    158 35043369 POLYGON ((480701.1 4764738,...
#> 9  -596610.313    165 35043408 POLYGON ((479750.7 4764702,...
#> 10    5268.813    170 35043413 POLYGON ((479750.7 4764702,...

^{Created on 2025-06-24 with reprex v2.1.1}

[paleolimbot]

Awesome!

I'll try to resurrect that branch in the next week or two...ADBC just added a new mechanism to refer to drivers using .toml files and I'll be adding support for that as well.

[yutannihilation]

Thanks! I hope I can help you on getting this ready.

Currently, it seems my implementation doesn't work on Windows for some reason. I'll investigate and report the findings here.

[paleolimbot]

In addition to using the dll path, you should also be able to use duckdb::duckdb_adbc()$driver_init_func, which is an external pointer to the init function there. Not sure if that will fix Windows or not but that's how we load/test DuckDB via ADBC otherwise.

typedef int AdbcStatusCode;
struct AdbcError;
typedef AdbcStatusCode (*AdbcDriverInitFunc)(int version, void* driver,
                                             struct AdbcError* error);

auto driver_init_func =
      reinterpret_cast<AdbcDriverInitFunc>(R_ExternalPtrAddrFn(driver_init_func_xptr));

[yutannihilation]

Sorry, it turned out the problem was simply that my Rtools was a bit old (GDAL 3.10.2)...

However, it seems there's another problem. Even if I use duckdb::duckdb_adbc()$driver_init_func, the error messages are the same.

dir.create("gdal-tmp", showWarnings = FALSE)
download.file("https://github.com/OSGeo/gdal/raw/refs/heads/master/autotest/ogr/data/parquet/poly.parquet", "gdal-tmp/poly.parquet", mode = "wb")

x <- sf::st_read("ADBC:gdal-tmp/poly.parquet", layer = "poly", options = "ADBC_DRIVER=duckdb")
#> options:        ADBC_DRIVER=duckdb
#> Warning in CPL_read_ogr(dsn, layer, query, as.character(options), quiet, : GDAL
#> Error 6: For decimal field, only width 128 and 256 are supported
#> Reading layer `poly' from data source `ADBC:gdal-tmp/poly.parquet' using driver `ADBC'
#> Warning in CPL_read_ogr(dsn, layer, query, as.character(options), quiet, : GDAL
#> Error 1: Cannot find OGR field for Arrow array AREA
#> Re-reading with feature count reset from 10 to 0
#> Warning in CPL_read_ogr(dsn, layer, query, as.character(options), quiet, : GDAL
#> Error 1: Cannot find OGR field for Arrow array AREA
#> Simple feature collection with 0 features and 2 fields
#> Bounding box:  xmin: NA ymin: NA xmax: NA ymax: NA
#> Projected CRS: OSGB36 / British National Grid

x
#> Simple feature collection with 0 features and 2 fields
#> Bounding box:  xmin: NA ymin: NA xmax: NA ymax: NA
#> Projected CRS: OSGB36 / British National Grid
#> [1] EAS_ID   PRFEDEA  geometry
#> <0 rows> (or 0-length row.names)

^{Created on 2025-06-25 with reprex v2.1.1}

[paleolimbot]

Hmm! Are the versions of DuckDB the same that you're using successfully on another machine?

[yutannihilation]

Thanks for the hint, no, I found the duckdb version on my Windows machine was 1.3.0 while that on macOS machine was 1.3.1. If I update the package to 1.3.1, it succeeds!

I'm curious about the cause. I quickly checked the NEWS and the GitHub release, but I couldn't find the changes that might relate to this.

[yutannihilation]

I tried the same code with duckdb v1.3.0 on macOS, and got the same error as Windows. So, it's probably that something is wrong with duckdb v1.3.0.