library(sf) causes high memory commit

[arthurgailes]

Describe the bug
library(sf) causes a high commit memory even with low usage. Furthermore, this seems to matriculate to individual cores in parallel (future) workers

To Reproduce
I start with a fresh installation of R 4.4.3 (I have also confirmed this in 4.3.1 and 4.3.3). The issue does not occur in 4.2.1. I then install sf

Additional context
Confirmed on Windows 10,11, and server 2016. sf version 1.0-20 and 1.0-19.

Edit: adding the reprex of the issue.
Edit 2: clarify that library(sf) also triggers memory commit.

library(ps)

library(sf)
#> Linking to GEOS 3.11.2, GDAL 3.8.2, PROJ 9.3.1; sf_use_s2() is TRUE

mem_info <- ps::ps_memory_info()

bytes <- as.numeric(mem_info)
mem_df <- data.frame(
  metric = names(mem_info),
  bytes = bytes,
  GB = round(bytes / 1024^3, 2),
  stringsAsFactors = FALSE
)

mem_df
#>                 metric      bytes   GB
#> 1      num_page_faults      46373 0.00
#> 2            peak_wset  156405760 0.15
#> 3                 wset  156405760 0.15
#> 4      peak_paged_pool     773592 0.00
#> 5           paged_pool     556296 0.00
#> 6  peak_non_paged_pool      28200 0.00
#> 7       non_paged_pool      23672 0.00
#> 8             pagefile 2323931136 2.16
#> 9        peak_pagefile 2323931136 2.16
#> 10         mem_private 2323931136 2.16
#> 11                 rss  156405760 0.15
#> 12                 vms 2323931136 2.16

^{Created on 2025-04-08 with reprex v2.1.1}

[rsbivand]

And in a regular terminal R session on Windows? Isn't this a user error anyway?

[arthurgailes]

It reproduces in Rgui with the script below (as a saved file) but not in R.exe.

I'm confused by "user error" in this context. Typing the characters sf:: is not an error. It's how one types eg sf::st_as_sf

pt1 <- sf::st_point(c(0,1))
pt2 <- sf::st_point(c(1,1))
sf::st_sfc(pt1, pt2)
d <- data.frame(a = 1:2)
d$geom <- sf::st_sfc(pt1, pt2)
df <- sf::st_as_sf(d)

[rsbivand]

Windows 10, R 4.4.3, nothing I can see entering sf::sf_extSoftVersion() in RGui using perfmon. How are you detecting or measuring apparent memory allocation (I only ever use Windows or macOS to try to answer questions, I have no idea how to access anything process-specific like top in Linux.

The first load of sf caused by the first sf::<exported object name> calls .on.load()

sf/R/init.R

Lines 34 to 44 in ac49ec2

	.onLoad = function(libname, pkgname) {
	if (getRversion() < as.numeric_version("3.6")) { # nocov start
	pathGrob <<- function(..., pathId.lengths) {
	grid::pathGrob(...)
	}
	} # nocov end
	load_gdal()
	if ((s2 <- Sys.getenv("_SF_USE_S2")) != "")
	options(sf_use_s2 = s2 != "false")
	FULL_bbox_ <<- st_set_crs(FULL_bbox_, "OGC:CRS84")
	}

and in that load_gdal

sf/R/init.R

Lines 92 to 108 in ac49ec2

	load_gdal <- function() {
	if (!identical(Sys.getenv("R_SF_USE_PROJ_DATA"), "true")) {
	if (file.exists(prj <- system.file("proj", package = "sf")[1])) {
	# nocov start
	if (! sf_proj_search_paths(prj)) { # if TRUE, uses C API to set path, leaving PROJ_LIB / PROJ_DATA alone
	save_and_replace("PROJ_LIB", prj, .sf_cache)
	save_and_replace("PROJ_DATA", prj, .sf_cache)
	}
	# CPL_use_proj4_init_rules(1L)
	# nocov end
	}
	if (file.exists(gdl <- system.file("gdal", package = "sf")[1]))
	save_and_replace("GDAL_DATA", gdl, .sf_cache)
	}
	CPL_gdal_init()
	register_all_s3_methods() # dynamically registers non-imported pkgs (tidyverse)
	}

. This calls into the static GDAL library built into sf.dll

sf/src/gdal.cpp

Lines 71 to 77 in ac49ec2

	void CPL_gdal_init()
	{
	CPLSetErrorHandler((CPLErrorHandler)__err_handler);
	CPLSetConfigOption("GDAL_NETCDF_REPORT_EXTRA_DIM_VALUES", "YES");
	GDALAllRegister();
	OGRRegisterAll();
	}

.

Was sf installed as a CRAN binary package?

[arthurgailes]

OK. I can now reproduce this in the R terminal as well. Reprex below. Windows, yes as CRAN package.

vms is the relevant value here - committed memory. This is neither actively used by R nor freed for Windows to use.

library(ps)
library(dplyr)
#> 
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#> 
#>     filter, lag
#> The following objects are masked from 'package:base':
#> 
#>     intersect, setdiff, setequal, union

sf::st_as_sf
#> function (x, ...) 
#> UseMethod("st_as_sf")
#> <bytecode: 0x0000024aa4d674b0>
#> <environment: namespace:sf>

mem_info <- ps::ps_memory_info()
mem_df <- data.frame(
  metric = names(mem_info),
  bytes = as.numeric(mem_info),
  GB = round(as.numeric(mem_info) / 1024^3, 2)
)

mem_df
#>                 metric      bytes   GB
#> 1      num_page_faults      99662 0.00
#> 2            peak_wset  162435072 0.15
#> 3                 wset  162435072 0.15
#> 4      peak_paged_pool     773592 0.00
#> 5           paged_pool     560704 0.00
#> 6  peak_non_paged_pool      28288 0.00
#> 7       non_paged_pool      24352 0.00
#> 8             pagefile 2330132480 2.17
#> 9        peak_pagefile 2330132480 2.17
#> 10         mem_private 2330132480 2.17
#> 11                 rss  162435072 0.15
#> 12                 vms 2330132480 2.17

^{Created on 2025-04-08 with reprex v2.1.1}

Edit: the reprex without sf

library(ps)
library(dplyr)
#> 
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#> 
#>     filter, lag
#> The following objects are masked from 'package:base':
#> 
#>     intersect, setdiff, setequal, union


mem_info <- ps::ps_memory_info()
mem_df <- data.frame(
  metric = names(mem_info),
  bytes = as.numeric(mem_info),
  GB = round(as.numeric(mem_info) / 1024^3, 2)
)

mem_df
#>                 metric     bytes   GB
#> 1      num_page_faults     41239 0.00
#> 2            peak_wset 141852672 0.13
#> 3                 wset 141852672 0.13
#> 4      peak_paged_pool    773592 0.00
#> 5           paged_pool    358952 0.00
#> 6  peak_non_paged_pool     28200 0.00
#> 7       non_paged_pool     19184 0.00
#> 8             pagefile 163037184 0.15
#> 9        peak_pagefile 163037184 0.15
#> 10         mem_private 163037184 0.15
#> 11                 rss 141852672 0.13
#> 12                 vms 163037184 0.15

^{Created on 2025-04-08 with reprex v2.1.1}

[rsbivand]

Not ps, please. Provide the name of a Windows app, not a Posit package. It has to run outside R, because it could be part of the problem rather than the solution.

[arthurgailes]

Not ps, please. Provide the name of a Windows app, not a Posit package. It has to run outside R, because it could be part of the problem rather than the solution.

Task manager. Please see the screenshot at the top, noting that it matches the ps output.

[rsbivand]

Note also that my Windows system speaks Norwegian. What happens without attaching dplyr?

[arthurgailes]

Without dplyr:

library(ps)

sf::st_as_sf
#> function (x, ...) 
#> UseMethod("st_as_sf")
#> <bytecode: 0x000001ffab9a95b8>
#> <environment: namespace:sf>


mem_info <- ps::ps_memory_info()

bytes <- as.numeric(mem_info)
mem_df <- data.frame(
  metric = names(mem_info),
  bytes = bytes,
  GB = round(bytes / 1024^3, 2),
  stringsAsFactors = FALSE
)

mem_df
#>                 metric      bytes   GB
#> 1      num_page_faults      47097 0.00
#> 2            peak_wset  156282880 0.15
#> 3                 wset  156282880 0.15
#> 4      peak_paged_pool     773592 0.00
#> 5           paged_pool     556296 0.00
#> 6  peak_non_paged_pool      28200 0.00
#> 7       non_paged_pool      23672 0.00
#> 8             pagefile 2323800064 2.16
#> 9        peak_pagefile 2323800064 2.16
#> 10         mem_private 2323800064 2.16
#> 11                 rss  156282880 0.15
#> 12                 vms 2323800064 2.16

^{Created on 2025-04-08 with reprex v2.1.1}

[arthurgailes]

An edit to this issue: I'm able to reproduce the commit with library(sf) as well. I think RStudio must run a proxy behind the scenes when you type sf::, which also triggers the commit.

[rsbivand]

With both the Rtools44 console and the regular Windows console, the underlying R process goes from about 45MB to about 70MB when I enter loadNamespace("sf") only.

With Rgui, the same numbers, but with ps:

> mem_info <- ps::ps_memory_info()
> mem_info
    num_page_faults           peak_wset                wset     peak_paged_pool 
              17603            65347584            65343488              328384 
         paged_pool peak_non_paged_pool      non_paged_pool            pagefile 
             328208               17672               17672            90992640 
      peak_pagefile         mem_private                 rss                 vms 
           90992640            90992640            65343488            90992640 
> loadNamespace("sf")
<environment: namespace:sf>
> mem_info <- ps::ps_memory_info()
> mem_info
    num_page_faults           peak_wset                wset     peak_paged_pool 
              25792            98484224            98480128              536528 
         paged_pool peak_non_paged_pool      non_paged_pool            pagefile 
             536352               22024               22024           387584000 
      peak_pagefile         mem_private                 rss                 vms 
          387584000           387584000            98480128           387584000 

I have no idea what pools or other variables refer to.

Please report the output of: packageVersion("sf") and how sfwas installed.

If CRAN binary, then the versions of external software may indicate which (sub) version of Rtools44 was used for that build. This could be something specific for your systems, since I cannot replicate, or could be a particular version of external libraries or their MXE build trains and/or Rtools. sf got an extra line in src/gdal.cpp in CPL_gdal_init January 7 this year (@edzer), but why it should play out differently for different users is unknown.

Please also try loadNamespace for terra, vapour, and gdalraster which all initialise GDAL in different ways.

[arthurgailes]

packageVersion("sf")
[1] ‘1.0.20’

Installed with CRAN.

So your output indicates a commit of 380MB (vms) - high but more manageable. I think this has something to do with available system memory. On a computer with 32GB of RAM, my commit is 2GB. On a second computer with 1TB memory, it's 5GB.

When you say "underlying process", are you referring to memory or commit memory (aka vms, virtual memory reserved for the process)? This is important distinction - the memory (paged_pool) is not an issue here.

In Norwegian, under details of the task manager, you'd be looking for "Forpliktet," not "Minne" according to ChatGPT.

However, you might be right in hypothesizing a gdal issue; see the vms growth when loading similar packages:

ps_mem_df <- function(){
  mem_info <- ps::ps_memory_info()
  
  bytes <- as.numeric(mem_info)
  mem_df <- data.frame(
    metric = names(mem_info),
    bytes = bytes,
    GB = round(bytes / 1024^3, 2),
    stringsAsFactors = FALSE
  )
  return(mem_df)
}

ps_mem_df()
#>                 metric     bytes   GB
#> 1      num_page_faults     31029 0.00
#> 2            peak_wset 100036608 0.09
#> 3                 wset  99860480 0.09
#> 4      peak_paged_pool    773592 0.00
#> 5           paged_pool    355704 0.00
#> 6  peak_non_paged_pool     28200 0.00
#> 7       non_paged_pool     18232 0.00
#> 8             pagefile 119812096 0.11
#> 9        peak_pagefile 120111104 0.11
#> 10         mem_private 119812096 0.11
#> 11                 rss  99860480 0.09
#> 12                 vms 119812096 0.11

loadNamespace("vapour")
#> <environment: namespace:vapour>
ps_mem_df()
#>                 metric      bytes   GB
#> 1      num_page_faults      37254 0.00
#> 2            peak_wset  124198912 0.12
#> 3                 wset  124198912 0.12
#> 4      peak_paged_pool     773592 0.00
#> 5           paged_pool     550296 0.00
#> 6  peak_non_paged_pool      28200 0.00
#> 7       non_paged_pool      22448 0.00
#> 8             pagefile 2293510144 2.14
#> 9        peak_pagefile 2293510144 2.14
#> 10         mem_private 2293510144 2.14
#> 11                 rss  124198912 0.12
#> 12                 vms 2293510144 2.14
loadNamespace("gdalraster")
#> <environment: namespace:gdalraster>
ps_mem_df()
#>                 metric      bytes   GB
#> 1      num_page_faults      47569 0.00
#> 2            peak_wset  166215680 0.15
#> 3                 wset  166215680 0.15
#> 4      peak_paged_pool     773592 0.00
#> 5           paged_pool     741488 0.00
#> 6  peak_non_paged_pool      28200 0.00
#> 7       non_paged_pool      25440 0.00
#> 8             pagefile 4484964352 4.18
#> 9        peak_pagefile 4484964352 4.18
#> 10         mem_private 4484964352 4.18
#> 11                 rss  166215680 0.15
#> 12                 vms 4484964352 4.18
loadNamespace("sf")
#> <environment: namespace:sf>
ps_mem_df()
#>                 metric      bytes   GB
#> 1      num_page_faults      51092 0.00
#> 2            peak_wset  180453376 0.17
#> 3                 wset  180453376 0.17
#> 4      peak_paged_pool     933832 0.00
#> 5           paged_pool     931976 0.00
#> 6  peak_non_paged_pool      29384 0.00
#> 7       non_paged_pool      28840 0.00
#> 8             pagefile 6646800384 6.19
#> 9        peak_pagefile 6646800384 6.19
#> 10         mem_private 6646800384 6.19
#> 11                 rss  180453376 0.17
#> 12                 vms 6646800384 6.19

^{Created on 2025-04-08 with reprex v2.1.1}

[edzer]

Does it help if you set the GDAL_CACHEMAX environment variable before loading any package that uses GDAL? See here - it seems to take 5% of the available memory by default.

[arthurgailes]

Does it help if you set the GDAL_CACHEMAX environment variable before loading any package that uses GDAL? See here - it seems to take 5% of the available memory by default.

Thank you, that does seem to be the issue given the variance between systems with different memory. However, setting the config in R doesn't seem to help the issue. Set GDAL_CACHEMAX=16 in .Renviron - it comes through but GDAL doesn't seem to respect it.

Sys.getenv("GDAL_CACHEMAX")
#> [1] "16"
ps_mem_df <- function(){
  mem_info <- ps::ps_memory_info()
  
  bytes <- as.numeric(mem_info)
  mem_df <- data.frame(
    metric = names(mem_info),
    bytes = bytes,
    GB = round(bytes / 1024^3, 2),
    stringsAsFactors = FALSE
  )
  return(mem_df)
}

ps_mem_df()
#>                 metric     bytes   GB
#> 1      num_page_faults     31043 0.00
#> 2            peak_wset 100114432 0.09
#> 3                 wset  99983360 0.09
#> 4      peak_paged_pool    773592 0.00
#> 5           paged_pool    355704 0.00
#> 6  peak_non_paged_pool     28360 0.00
#> 7       non_paged_pool     18232 0.00
#> 8             pagefile 119877632 0.11
#> 9        peak_pagefile 120135680 0.11
#> 10         mem_private 119877632 0.11
#> 11                 rss  99983360 0.09
#> 12                 vms 119877632 0.11

loadNamespace("gdalraster")
#> <environment: namespace:gdalraster>
ps_mem_df()
#>                 metric      bytes   GB
#> 1      num_page_faults      44494 0.00
#> 2            peak_wset  153808896 0.14
#> 3                 wset  153808896 0.14
#> 4      peak_paged_pool     773592 0.00
#> 5           paged_pool     518296 0.00
#> 6  peak_non_paged_pool      28360 0.00
#> 7       non_paged_pool      22856 0.00
#> 8             pagefile 2322747392 2.16
#> 9        peak_pagefile 2322747392 2.16
#> 10         mem_private 2322747392 2.16
#> 11                 rss  153808896 0.14
#> 12                 vms 2322747392 2.16

loadNamespace("sf")
#> <environment: namespace:sf>
ps_mem_df()
#>                 metric      bytes   GB
#> 1      num_page_faults      47692 0.00
#> 2            peak_wset  166711296 0.16
#> 3                 wset  166711296 0.16
#> 4      peak_paged_pool     773592 0.00
#> 5           paged_pool     708784 0.00
#> 6  peak_non_paged_pool      28360 0.00
#> 7       non_paged_pool      26392 0.00
#> 8             pagefile 4483170304 4.18
#> 9        peak_pagefile 4483170304 4.18
#> 10         mem_private 4483170304 4.18
#> 11                 rss  166711296 0.16
#> 12                 vms 4483170304 4.18

^{Created on 2025-04-08 with reprex v2.1.1}

Edit: neither setting the environment variable nor via the gdarlc config file make any change to the issue.