Use arduino-cli backend

CHANGELOG.md

@@ -7,14 +7,31 @@ and this project adheres to [Semantic Versioning](http://semver.org/).
 
 ## [Unreleased]
 ### Added
+- Special handling of attempts to run the `arduino_ci.rb` CI script against the ruby library instead of an actual Arduino project
+- Explicit checks for attemping to test `arduino_ci` itself as if it were a library, resolving a minor annoyance to this developer.
+- Code coverage tooling
+- Explicit check and warning for library directory names that do not match our guess of what the library should/would be called
+- Symlink tests for `Host`
 
 ### Changed
+- Arduino backend is now `arduino-cli` version `0.13.0`
+- `ArduinoCmd` is now `ArduinoBackend`
+- `CppLibrary` now relies largely on `ArduinoBackend` instead of making its own judgements about libraries (metadata, includes, and examples)
+- `ArduinoBackend` functionality related to `CppLibrary` now lives in `CppLibrary`
+- `CppLibrary` now works in an installation-first manner for exposure to `arduino-cli`'s logic -- without installation, there is no ability to reason about libraries
+- `CppLibrary` forces just-in-time recursive dependency installation in order to work sensibly
+- `ArduinoBackend` maintains the central "best guess" logic on what a library (on disk) might be named
 
 ### Deprecated
+- `arduino_ci_remote.rb` CLI switch `--skip-compilation`
+- Deprecated `arduino_ci_remote.rb` in favor of `arduino_ci.rb`
 
 ### Removed
+- `ARDUINO_CI_SKIP_SPLASH_SCREEN_RSPEC_TESTS` no longer affects any tests because there are no longer splash screens since switching to `arduino-cli`
 
 ### Fixed
+- Mismatches between library names in `library.properties` and the directory names, which can cause cryptic failures
+- `LibraryProperties` skips over parse errors instead of crashing: only lines with non-empty keys and non-nil values are recorded
 
 ### Security
 

CONTRIBUTING.md

@@ -31,7 +31,6 @@ See `SampleProjects/TestSomething/test/*.cpp` for the existing tests (run by rsp
 
 To speed up testing by targeting only the files you're working on, you can set several environment variables that `bundle exec rspec` will respond to:
 
-* `ARDUINO_CI_SKIP_SPLASH_SCREEN_RSPEC_TESTS`: if set, this will avoid any rspec test that calls the arduino executable (and as such, causes the splash screen to pop up).
 * `ARDUINO_CI_SKIP_RUBY_RSPEC_TESTS`: if set, this will skip all tests against ruby code (useful if you are not changing Ruby code).
 * `ARDUINO_CI_SKIP_CPP_RSPEC_TESTS`: if set, this will skip all tests against the `TestSomething` sample project (useful if you are not changing C++ code).
 * `ARDUINO_CI_SELECT_CPP_TESTS=<glob>`: if set, this will skip all C++ unit tests whose filenames don't match the provided glob (executed in the tests directory)

Gemfile

@@ -4,3 +4,10 @@ git_source(:github) { |repo_name| "https://github.com/#{repo_name}" }
 
 # Specify your gem's dependencies in arduino_ci.gemspec
 gemspec
+
+gem "bundler", "> 1.15", require: false, group: :test
+gem "keepachangelog_manager", "~> 0.0.2", require: false, group: :test
+gem "rspec", "~> 3.0", require: false, group: :test
+gem 'rubocop', '~>0.59.0', require: false, group: :test
+gem 'simplecov', require: false, group: :test
+gem 'yard', '~>0.9.11', require: false, group: :test

README.md

@@ -1,6 +1,6 @@
 
-# ArduinoCI Ruby gem (`arduino_ci`) 
-[![Gem Version](https://badge.fury.io/rb/arduino_ci.svg)](https://rubygems.org/gems/arduino_ci) 
+# ArduinoCI Ruby gem (`arduino_ci`)
+[![Gem Version](https://badge.fury.io/rb/arduino_ci.svg)](https://rubygems.org/gems/arduino_ci)
 [![Documentation](http://img.shields.io/badge/docs-rdoc.info-blue.svg)](http://www.rubydoc.info/gems/arduino_ci/0.4.0)
 [![Gitter](https://badges.gitter.im/Arduino-CI/arduino_ci.svg)](https://gitter.im/Arduino-CI/arduino_ci?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge)
 
@@ -36,6 +36,9 @@ For a bare-bones example that you can copy from, see [SampleProjects/DoSomething
 
 The complete set of C++ unit tests for the `arduino_ci` library itself are in the [SampleProjects/TestSomething](SampleProjects/TestSomething) project.  The [test files](SampleProjects/TestSomething/test/) are named after the type of feature being tested.
 
+> Arduino expects all libraries to be in a specific `Arduino/libraries` directory on your system.  If your library is elsewhere, `arduino_ci` will _automatically_ create a symbolic link in the `libraries` directory that points to the directory of the project being tested.  This simplifieds working with project dependencies, but **it can have unintended consequences on Windows systems** because [in some cases deleting a folder that contains a symbolic link to another folder can cause the _entire linked folder_ to be removed instead of just the link itself](https://superuser.com/a/306618).
+>
+> If you use a Windows system **it is recommended that you only run `arduino_ci` from project directories that are already inside the `libraries` directory**
 
 ### You Need Ruby and Bundler
 

SampleProjects/ExcludeSomething/library.properties

@@ -1,4 +1,4 @@
-name=TestSomething
+name=ExcludeSomething
 version=0.1.0
 author=Ian Katz <[email protected]>
 maintainer=Ian Katz <[email protected]>

SampleProjects/NetworkLib/library.properties

@@ -1,4 +1,4 @@
-name=Ethernet
+name=NetworkLib
 version=0.1.0
 author=James Foster <[email protected]>
 maintainer=James Foster <[email protected]>

arduino_ci.gemspec

@@ -27,10 +27,4 @@ Gem::Specification.new do |spec|
 
   spec.add_dependency "os", "~> 1.0"
   spec.add_dependency "rubyzip", "~> 1.2"
-
-  spec.add_development_dependency "bundler", "> 1.15"
-  spec.add_development_dependency "keepachangelog_manager", "~> 0.0.2"
-  spec.add_development_dependency "rspec", "~> 3.0"
-  spec.add_development_dependency 'rubocop', '~>0.59.0'
-  spec.add_development_dependency 'yard', '~>0.9.11'
 end

exe/arduino_ci.rb

@@ -9,6 +9,7 @@
 
 @failure_count = 0
 @passfail = proc { |result| result ? "✓" : "✗" }
+@backend = nil
 
 # Use some basic parsing to allow command-line overrides of config
 class Parser
@@ -29,11 +30,6 @@ def self.parse(options)
         output_options[:skip_unittests] = p
       end
 
-      opts.on("--skip-compilation", "Don't compile example sketches (deprecated)") do |p|
-        puts "The option --skip-compilation has been deprecated in favor of --skip-examples-compilation"
-        output_options[:skip_compilation] = p
-      end
-
       opts.on("--skip-examples-compilation", "Don't compile example sketches") do |p|
         output_options[:skip_compilation] = p
       end
@@ -68,11 +64,11 @@ def self.parse(options)
 def terminate(final = nil)
   puts "Failures: #{@failure_count}"
   unless @failure_count.zero? || final
-    puts "Last message: #{@arduino_cmd.last_msg}"
+    puts "Last message: #{@backend.last_msg}"
     puts "========== Stdout:"
-    puts @arduino_cmd.last_out
+    puts @backend.last_out
     puts "========== Stderr:"
-    puts @arduino_cmd.last_err
+    puts @backend.last_err
   end
   retcode = @failure_count.zero? ? 0 : 1
   exit(retcode)
@@ -161,7 +157,7 @@ def file_is_hidden_somewhere?(path)
 # print out some files
 def display_files(pathname)
   # `find` doesn't follow symlinks, so we should instead
-  realpath = pathname.symlink? ? pathname.readlink : pathname
+  realpath = Host.symlink?(pathname) ? Host.readlink(pathname) : pathname
 
   # suppress directories and dotfile-based things
   all_files = realpath.find.select(&:file?)
@@ -172,25 +168,33 @@ def display_files(pathname)
   non_hidden.each { |p| puts "#{margin}#{p}" }
 end
 
-def install_arduino_library_dependencies(aux_libraries)
-  aux_libraries.each do |l|
-    if @arduino_cmd.library_present?(l)
-      inform("Using pre-existing library") { l.to_s }
+# @return [Array<String>] The list of installed libraries
+def install_arduino_library_dependencies(library_names, on_behalf_of, already_installed = [])
+  installed = already_installed.clone
+  library_names.map { |n| @backend.library_of_name(n) }.each do |l|
+    if installed.include?(l)
+      # do nothing
+    elsif l.installed?
+      inform("Using pre-existing dependency of #{on_behalf_of}") { l.name }
     else
-      assure("Installing aux library '#{l}'") { @arduino_cmd.install_library(l) }
+      assure("Installing dependency of #{on_behalf_of}: '#{l.name}'") do
+        next nil unless l.install
+
+        l.name
+      end
     end
+    installed << l.name
+    installed += install_arduino_library_dependencies(l.arduino_library_dependencies, l.name, installed)
   end
+  installed
 end
 
-def perform_unit_tests(file_config)
+def perform_unit_tests(cpp_library, file_config)
   if @cli_options[:skip_unittests]
     inform("Skipping unit tests") { "as requested via command line" }
     return
   end
   config = file_config.with_override_config(@cli_options[:ci_config])
-  cpp_library = ArduinoCI::CppLibrary.new(Pathname.new("."),
-                                          @arduino_cmd.lib_dir,
-                                          config.exclude_dirs.map(&Pathname.method(:new)))
 
   # check GCC
   compilers = config.compilers_to_use
@@ -214,10 +218,25 @@ def perform_unit_tests(file_config)
 
   # iterate boards / tests
   if !cpp_library.tests_dir.exist?
-    inform_multiline("Skipping unit tests; no tests dir at #{cpp_library.tests_dir}") do
-      puts "  In case that's an error, this is what was found in the library:"
-      display_files(cpp_library.tests_dir.parent)
-      true
+    # alert future me about running the script from the wrong directory, instead of doing the huge file dump
+    # otherwise, assume that the user might be running the script on a library with no actual unit tests
+    if Pathname.new(__dir__).parent == Pathname.new(Dir.pwd)
+      inform_multiline("arduino_ci seems to be trying to test itself") do
+        [
+          "arduino_ci (the ruby gem) isn't an arduino project itself, so running the CI test script against",
+          "the core library isn't really a valid thing to do... but it's easy for a developer (including the",
+          "owner) to mistakenly do just that.  Hello future me, you probably meant to run this against one of",
+          "the sample projects in SampleProjects/ ... if not, please submit a bug report; what a wild case!"
+        ].each { |l| puts "  #{l}" }
+        false
+      end
+      exit(1)
+    else
+      inform_multiline("Skipping unit tests; no tests dir at #{cpp_library.tests_dir}") do
+        puts "  In case that's an error, this is what was found in the library:"
+        display_files(cpp_library.tests_dir.parent)
+        true
+      end
     end
   elsif cpp_library.test_files.empty?
     inform_multiline("Skipping unit tests; no test files were found in #{cpp_library.tests_dir}") do
@@ -228,7 +247,7 @@ def perform_unit_tests(file_config)
   elsif config.platforms_to_unittest.empty?
     inform("Skipping unit tests") { "no platforms were requested" }
   else
-    install_arduino_library_dependencies(config.aux_libraries_for_unittest)
+    install_arduino_library_dependencies(config.aux_libraries_for_unittest, "<unittest/libraries>")
 
     config.platforms_to_unittest.each do |p|
       config.allowable_unittest_files(cpp_library.test_files).each do |unittest_path|
@@ -256,39 +275,12 @@ def perform_unit_tests(file_config)
   end
 end
 
-def perform_compilation_tests(config)
+def perform_example_compilation_tests(cpp_library, config)
   if @cli_options[:skip_compilation]
     inform("Skipping compilation of examples") { "as requested via command line" }
     return
   end
 
-  # index the existing libraries
-  attempt("Indexing libraries") { @arduino_cmd.index_libraries } unless @arduino_cmd.libraries_indexed
-
-  # initialize library under test
-  installed_library_path = attempt("Installing library under test") do
-    @arduino_cmd.install_local_library(Pathname.new("."))
-  end
-
-  if !installed_library_path.nil? && installed_library_path.exist?
-    inform("Library installed at") { installed_library_path.to_s }
-  else
-    assure_multiline("Library installed successfully") do
-      if installed_library_path.nil?
-        puts @arduino_cmd.last_msg
-      else
-        # print out the contents of the deepest directory we actually find
-        @arduino_cmd.lib_dir.ascend do |path_part|
-          next unless path_part.exist?
-
-          break display_files(path_part)
-        end
-        false
-      end
-    end
-  end
-  library_examples = @arduino_cmd.library_examples(installed_library_path)
-
   # gather up all required boards for compilation so we can install them up front.
   # start with the "platforms to unittest" and add the examples
   # while we're doing that, get the aux libraries as well
@@ -297,6 +289,7 @@ def perform_compilation_tests(config)
   aux_libraries = Set.new(config.aux_libraries_for_build)
   # while collecting the platforms, ensure they're defined
 
+  library_examples = cpp_library.example_sketches
   library_examples.each do |path|
     ovr_config = config.from_example(path)
     ovr_config.platforms_to_build.each do |platform|
@@ -317,35 +310,36 @@ def perform_compilation_tests(config)
   # do that, set the URLs, and download the packages
   all_packages = example_platform_info.values.map { |v| v[:package] }.uniq.reject(&:nil?)
 
+  builtin_packages, external_packages = all_packages.partition { |p| config.package_builtin?(p) }
+
   # inform about builtin packages
-  all_packages.select { |p| config.package_builtin?(p) }.each do |p|
+  builtin_packages.each do |p|
     inform("Using built-in board package") { p }
   end
 
   # make sure any non-builtin package has a URL defined
-  all_packages.reject { |p| config.package_builtin?(p) }.each do |p|
+  external_packages.each do |p|
     assure("Board package #{p} has a defined URL") { board_package_url[p] }
   end
 
   # set up all the board manager URLs.
   # we can safely reject nils now, they would be for the builtins
-  all_urls = all_packages.map { |p| board_package_url[p] }.uniq.reject(&:nil?)
+  all_urls = external_packages.map { |p| board_package_url[p] }.uniq.reject(&:nil?)
 
   unless all_urls.empty?
     assure("Setting board manager URLs") do
-      @arduino_cmd.board_manager_urls = all_urls
+      @backend.board_manager_urls = all_urls
     end
   end
 
-  all_packages.each do |p|
+  external_packages.each do |p|
     assure("Installing board package #{p}") do
-      @arduino_cmd.install_boards(p)
+      @backend.install_boards(p)
     end
   end
 
-  install_arduino_library_dependencies(aux_libraries)
+  install_arduino_library_dependencies(aux_libraries, "<compile/libraries>")
 
-  last_board = nil
   if config.platforms_to_build.empty?
     inform("Skipping builds") { "no platforms were requested" }
     return
@@ -356,46 +350,58 @@ def perform_compilation_tests(config)
     return
   end
 
-  attempt("Setting compiler warning level")  { @arduino_cmd.set_pref("compiler.warning_level", "all") }
-
-  # switching boards takes time, so iterate board first
-  # _then_ whichever examples match it
-  examples_by_platform = library_examples.each_with_object({}) do |example_path, acc|
+  library_examples.each do |example_path|
     ovr_config = config.from_example(example_path)
     ovr_config.platforms_to_build.each do |p|
-      acc[p] = [] unless acc.key?(p)
-      acc[p] << example_path
-    end
-  end
-
-  examples_by_platform.each do |platform, example_paths|
-    board = example_platform_info[platform][:board]
-    assure("Switching to board for #{platform} (#{board})") { @arduino_cmd.use_board(board) } unless last_board == board
-    last_board = board
-
-    example_paths.each do |example_path|
+      board = example_platform_info[p][:board]
       example_name = File.basename(example_path)
-      attempt("Verifying #{example_name}") do
-        ret = @arduino_cmd.verify_sketch(example_path)
+      attempt("Compiling #{example_name} for #{board}") do
+        ret = @backend.compile_sketch(example_path, board)
         unless ret
           puts
-          puts "Last command: #{@arduino_cmd.last_msg}"
-          puts @arduino_cmd.last_err
+          puts "Last command: #{@backend.last_msg}"
+          puts @backend.last_err
         end
         ret
       end
     end
   end
-
 end
 
 # initialize command and config
 config = ArduinoCI::CIConfig.default.from_project_library
 
-@arduino_cmd = ArduinoCI::ArduinoInstallation.autolocate!
-inform("Located Arduino binary") { @arduino_cmd.binary_path.to_s }
+@backend = ArduinoCI::ArduinoInstallation.autolocate!
+inform("Located arduino-cli binary") { @backend.binary_path.to_s }
+
+# initialize library under test
+cpp_library_path = Pathname.new(".")
+cpp_library = assure("Installing library under test") do
+  @backend.install_local_library(cpp_library_path)
+end
+
+assumed_name = @backend.name_of_library(cpp_library_path)
+ondisk_name = cpp_library_path.realpath.basename
+if assumed_name != ondisk_name
+  inform("WARNING") { "Installed library named '#{assumed_name}' has directory name '#{ondisk_name}'" }
+end
+
+if !cpp_library.nil?
+  inform("Library installed at") { cpp_library.path.to_s }
+else
+  # this is a longwinded way of failing, we aren't really "assuring" anything at this point
+  assure_multiline("Library installed successfully") do
+    puts @backend.last_msg
+    false
+  end
+end
+
+install_arduino_library_dependencies(
+  cpp_library.arduino_library_dependencies,
+  "<#{ArduinoCI::CppLibrary::LIBRARY_PROPERTIES_FILE}>"
+)
 
-perform_unit_tests(config)
-perform_compilation_tests(config)
+perform_unit_tests(cpp_library, config)
+perform_example_compilation_tests(cpp_library, config)
 
 terminate(true)

exe/arduino_library_location.rb

@@ -2,6 +2,6 @@
 require 'arduino_ci'
 
 # locate and/or forcibly install Arduino, keep stdout clean
-@arduino_cmd = ArduinoCI::ArduinoInstallation.autolocate!($stderr)
+@backend = ArduinoCI::ArduinoInstallation.autolocate!($stderr)
 
-puts @arduino_cmd.lib_dir
+puts @backend.lib_dir

lib/arduino_ci/arduino_backend.rb

@@ -0,0 +1,218 @@
+require 'fileutils'
+require 'pathname'
+require 'json'
+
+# workaround for https://github.com/arduino/Arduino/issues/3535
+WORKAROUND_LIB = "USBHost".freeze
+
+module ArduinoCI
+
+  # To report errors that we can't resolve or possibly even explain
+  class ArduinoExecutionError < StandardError; end
+
+  # Wrap the Arduino executable.  This requires, in some cases, a faked display.
+  class ArduinoBackend
+
+    # We never even use this in code, it's just here for reference because the backend is picky about it. Used for testing
+    # @return [String] the only allowable name for the arduino-cli config file.
+    CONFIG_FILE_NAME = "arduino-cli.yaml".freeze
+
+    # the actual path to the executable on this platform
+    # @return [Pathname]
+    attr_accessor :binary_path
+
+    # If a custom config is deired (i.e. for testing), specify it here.
+    # Note https://github.com/arduino/arduino-cli/issues/753 : the --config-file option
+    # is really the director that contains the file
+    # @return [Pathname]
+    attr_accessor :config_dir
+
+    # @return [String] STDOUT of the most recently-run command
+    attr_reader   :last_out
+
+    # @return [String] STDERR of the most recently-run command
+    attr_reader   :last_err
+
+    # @return [String] the most recently-run command
+    attr_reader   :last_msg
+
+    # @return [Array<String>] Additional URLs for the boards manager
+    attr_reader   :additional_urls
+
+    def initialize(binary_path)
+      @binary_path        = binary_path
+      @config_dir         = nil
+      @additional_urls    = []
+      @last_out           = ""
+      @last_err           = ""
+      @last_msg           = ""
+    end
+
+    def _wrap_run(work_fn, *args, **kwargs)
+      # do some work to extract & merge environment variables if they exist
+      has_env = !args.empty? && args[0].class == Hash
+      env_vars = has_env ? args[0] : {}
+      actual_args = has_env ? args[1..-1] : args  # need to shift over if we extracted args
+      custom_config = @config_dir.nil? ? [] : ["--config-file", @config_dir.to_s]
+      full_args = [binary_path.to_s, "--format", "json"] + custom_config + actual_args
+      full_cmd = env_vars.empty? ? full_args : [env_vars] + full_args
+
+      shell_vars = env_vars.map { |k, v| "#{k}=#{v}" }.join(" ")
+      @last_msg = " $ #{shell_vars} #{full_args.join(' ')}"
+      work_fn.call(*full_cmd, **kwargs)
+    end
+
+    # build and run the arduino command
+    def run_and_output(*args, **kwargs)
+      _wrap_run((proc { |*a, **k| Host.run_and_output(*a, **k) }), *args, **kwargs)
+    end
+
+    # run a command and capture its output
+    # @return [Hash] {:out => String, :err => String, :success => bool}
+    def run_and_capture(*args, **kwargs)
+      ret = _wrap_run((proc { |*a, **k| Host.run_and_capture(*a, **k) }), *args, **kwargs)
+      @last_err = ret[:err]
+      @last_out = ret[:out]
+      ret
+    end
+
+    def capture_json(*args, **kwargs)
+      ret = run_and_capture(*args, **kwargs)
+      ret[:json] = JSON.parse(ret[:out])
+      ret
+    end
+
+    # Get a dump of the entire config
+    # @return [Hash] The configuration
+    def config_dump
+      capture_json("config", "dump")[:json]
+    end
+
+    # @return [String] the path to the Arduino libraries directory
+    def lib_dir
+      Pathname.new(config_dump["directories"]["user"]) + "libraries"
+    end
+
+    # Board manager URLs
+    # @return [Array<String>] The additional URLs used by the board manager
+    def board_manager_urls
+      config_dump["board_manager"]["additional_urls"] + @additional_urls
+    end
+
+    # Set board manager URLs
+    # @return [Array<String>] The additional URLs used by the board manager
+    def board_manager_urls=(all_urls)
+      raise ArgumentError("all_urls should be an array, got #{all_urls.class}") unless all_urls.is_a? Array
+
+      @additional_urls = all_urls
+    end
+
+    # check whether a board is installed
+    # we do this by just selecting a board.
+    #   the arduino binary will error if unrecognized and do a successful no-op if it's installed
+    # @param boardname [String] The board to test
+    # @return [bool] Whether the board is installed
+    def board_installed?(boardname)
+      # capture_json("core", "list")[:json].find { |b| b["ID"] == boardname } # nope, this is for the family
+      run_and_capture("board", "details", "--fqbn", boardname)[:success]
+    end
+
+    # install a board by name
+    # @param name [String] the board name
+    # @return [bool] whether the command succeeded
+    def install_boards(boardfamily)
+      result = run_and_capture("core", "install", boardfamily)
+      result[:success]
+    end
+
+    # @return [Hash] information about installed libraries via the CLI
+    def installed_libraries
+      capture_json("lib", "list")[:json]
+    end
+
+    # @param path [String] The sketch to compile
+    # @param boardname [String] The board to use
+    # @return [bool] whether the command succeeded
+    def compile_sketch(path, boardname)
+      ext = File.extname path
+      unless ext.casecmp(".ino").zero?
+        @last_msg = "Refusing to compile sketch with '#{ext}' extension -- rename it to '.ino'!"
+        return false
+      end
+      unless File.exist? path
+        @last_msg = "Can't compile Sketch at nonexistent path '#{path}'!"
+        return false
+      end
+      ret = run_and_capture("compile", "--fqbn", boardname, "--warnings", "all", "--dry-run", path.to_s)
+      ret[:success]
+    end
+
+    # Guess the name of a library
+    # @param path [Pathname] The path to the library (installed or not)
+    # @return [String] the probable library name
+    def name_of_library(path)
+      src_path = path.realpath
+      properties_file = src_path + CppLibrary::LIBRARY_PROPERTIES_FILE
+      return src_path.basename.to_s unless properties_file.exist?
+      return src_path.basename.to_s if LibraryProperties.new(properties_file).name.nil?
+
+      LibraryProperties.new(properties_file).name
+    end
+
+    # Create a handle to an Arduino library by name
+    # @param name [String] The library "real name"
+    # @return [CppLibrary] The library object
+    def library_of_name(name)
+      raise ArgumentError, "name is not a String (got #{name.class})" unless name.is_a? String
+
+      CppLibrary.new(name, self)
+    end
+
+    # Create a handle to an Arduino library by path
+    # @param path [Pathname] The path to the library
+    # @return [CppLibrary] The library object
+    def library_of_path(path)
+      # the path must exist... and if it does, brute-force search the installed libs for it
+      realpath = path.realpath  # should produce error if the path doesn't exist to begin with
+      entry = installed_libraries.find { |l| Pathname.new(l["library"]["install_dir"]).realpath == realpath }
+      probable_name = entry["real_name"].nil? ? realpath.basename.to_s : entry["real_name"]
+      CppLibrary.new(probable_name, self)
+    end
+
+    # install a library from a path on the local machine (not via library manager), by symlink or no-op as appropriate
+    # @param path [Pathname] library to use
+    # @return [CppLibrary] the installed library, or nil
+    def install_local_library(path)
+      src_path         = path.realpath
+      library_name     = name_of_library(path)
+      cpp_library      = library_of_name(library_name)
+      destination_path = cpp_library.path
+
+      # things get weird if the sketchbook contains the library.
+      # check that first
+      if cpp_library.installed?
+        # maybe the project has always lived in the libraries directory, no need to symlink
+        return cpp_library if destination_path == src_path
+
+        uhoh = "There is already a library '#{library_name}' in the library directory (#{destination_path})"
+        # maybe it's a symlink? that would be OK
+        if Host.symlink?(destination_path)
+          current_destination_target = Host.readlink(destination_path)
+          return cpp_library if current_destination_target == src_path
+
+          @last_msg = "#{uhoh} and it's symlinked to #{current_destination_target} (expected #{src_path})"
+          return nil
+        end
+
+        @last_msg = "#{uhoh}.  It may need to be removed manually."
+        return nil
+      end
+
+      # install the library
+      libraries_dir = destination_path.parent
+      libraries_dir.mkpath unless libraries_dir.exist?
+      Host.symlink(src_path, destination_path)
+      cpp_library
+    end
+  end
+end

lib/arduino_ci/arduino_downloader.rb

@@ -1,4 +1,5 @@
 require 'fileutils'
+require 'pathname'
 require 'net/http'
 require 'open-uri'
 require 'zip'
@@ -10,10 +11,10 @@ module ArduinoCI
   # Manage the OS-specific download & install of Arduino
   class ArduinoDownloader
 
-    # @param desired_ide_version [string] Version string e.g. 1.8.7
+    # @param desired_version [string] Version string e.g. 1.8.7
     # @param output [IO] $stdout, $stderr, File.new(/dev/null, 'w'), etc. where console output will be sent
-    def initialize(desired_ide_version, output = $stdout)
-      @desired_ide_version = desired_ide_version
+    def initialize(desired_version, output = $stdout)
+      @desired_version = desired_version
       @output = output
     end
 
@@ -30,7 +31,7 @@ def prepare
 
     # The autolocated executable of the installation
     #
-    # @return [string] or nil
+    # @return [Pathname] or nil
     def self.autolocated_executable
       # Arbitrarily, I'm going to pick the force installed location first
       # if it exists.  I'm not sure why we would have both, but if we did
@@ -39,70 +40,54 @@ def self.autolocated_executable
       locations.find { |loc| !loc.nil? && File.exist?(loc) }
     end
 
-    # The autolocated directory of the installation
-    #
-    # @return [string] or nil
-    def self.autolocated_installation
-      # Arbitrarily, I'm going to pick the force installed location first
-      # if it exists.  I'm not sure why we would have both, but if we did
-      # a force install then let's make sure we actually use it.
-      locations = [self.force_install_location, self.existing_installation]
-      locations.find { |loc| !loc.nil? && File.exist?(loc) }
+    # The executable Arduino file in an existing installation, or nil
+    # @return [Pathname]
+    def self.existing_executable
+      self.must_implement(__method__)
     end
 
-    # The path to the directory of an existing installation, or nil
+    # The local file (dir) name of the desired IDE package (zip/tar/etc)
     # @return [string]
-    def self.existing_installation
-      self.must_implement(__method__)
+    def package_file
+      self.class.must_implement(__method__)
     end
 
-    # The executable Arduino file in an existing installation, or nil
+    # The local filename of the extracted IDE package (zip/tar/etc)
     # @return [string]
-    def self.existing_executable
+    def self.extracted_file
       self.must_implement(__method__)
     end
 
     # The executable Arduino file in a forced installation, or nil
-    # @return [string]
+    # @return [Pathname]
     def self.force_installed_executable
-      self.must_implement(__method__)
+      Pathname.new(ENV['HOME']) + self.extracted_file
     end
 
     # The technology that will be used to complete the download
     # (for logging purposes)
     # @return [string]
-    def downloader
+    def self.downloader
       "open-uri"
     end
 
     # The technology that will be used to extract the download
     # (for logging purposes)
     # @return [string]
-    def extracter
-      "Zip"
-    end
-
-    # The URL of the desired IDE package (zip/tar/etc) for this platform
-    # @return [string]
-    def package_url
-      "https://downloads.arduino.cc/#{package_file}"
+    def self.extracter
+      self.must_implement(__method__)
     end
 
-    # The local file (dir) name of the desired IDE package (zip/tar/etc)
-    # @return [string]
-    def package_file
-      self.class.must_implement(__method__)
+    # Extract the package_file to extracted_file
+    # @return [bool] whether successful
+    def self.extract(_package_file)
+      self.must_implement(__method__)
     end
 
-    # The local filename of the extracted IDE package (zip/tar/etc)
+    # The URL of the desired IDE package (zip/tar/etc) for this platform
     # @return [string]
-    def extracted_file
-      self.class.must_implement(__method__)
-    end
-
-    # @return [String] The location where a forced install will go
-    def self.force_install_location
-      File.join(ENV['HOME'], 'arduino_ci_ide')
+    def package_url
+      "https://github.com/arduino/arduino-cli/releases/download/#{@desired_version}/#{package_file}"
     end
 
     # Download the package_url to package_file
@@ -130,26 +115,10 @@ def download
       @output.puts "\nArduino force-install failed downloading #{package_url}: #{e}"
     end
 
-    # Extract the package_file to extracted_file
-    # @return [bool] whether successful
-    def extract
-      Zip::File.open(package_file) do |zip|
-        batch_size = [1, (zip.size / 100).to_i].max
-        dots = 0
-        zip.each do |file|
-          @output.print "." if (dots % batch_size).zero?
-          file.restore_permissions = true
-          file.extract { accept_all }
-          dots += 1
-        end
-      end
-    end
-
-    # Move the extracted package file from extracted_file to the force_install_location
+    # Move the extracted package file from extracted_file to the force_installed_executable
     # @return [bool] whether successful
     def install
-      # Move only the content of the directory
-      FileUtils.mv extracted_file, self.class.force_install_location
+      FileUtils.mv self.class.extracted_file.to_s, self.class.force_installed_executable.to_s
     end
 
     # Forcibly install Arduino on linux from the web
@@ -161,40 +130,40 @@ def execute
         return false
       end
 
-      arduino_package = "Arduino #{@desired_ide_version} package"
+      arduino_package = "Arduino #{@desired_version} package"
       attempts = 0
 
       loop do
-        if File.exist? package_file
-          @output.puts "#{arduino_package} seems to have been downloaded already" if attempts.zero?
+        if File.exist?(package_file)
+          @output.puts "#{arduino_package} seems to have been downloaded already at #{package_file}" if attempts.zero?
           break
         elsif attempts >= DOWNLOAD_ATTEMPTS
           break @output.puts "After #{DOWNLOAD_ATTEMPTS} attempts, failed to download #{package_url}"
         else
-          @output.print "Attempting to download #{arduino_package} with #{downloader}"
+          @output.print "Attempting to download #{arduino_package} with #{self.class.downloader}"
           download
           @output.puts
         end
         attempts += 1
       end
 
-      if File.exist? extracted_file
-        @output.puts "#{arduino_package} seems to have been extracted already"
-      elsif File.exist? package_file
-        @output.print "Extracting archive with #{extracter}"
-        extract
+      if File.exist?(self.class.extracted_file)
+        @output.puts "#{arduino_package} seems to have been extracted already at #{self.class.extracted_file}"
+      elsif File.exist?(package_file)
+        @output.print "Extracting archive with #{self.class.extracter}"
+        self.class.extract(package_file)
         @output.puts
       end
 
-      if File.exist? self.class.force_install_location
-        @output.puts "#{arduino_package} seems to have been installed already"
-      elsif File.exist? extracted_file
+      if File.exist?(self.class.force_installed_executable)
+        @output.puts "#{arduino_package} seems to have been installed already at #{self.class.force_installed_executable}"
+      elsif File.exist?(self.class.extracted_file)
         install
       else
-        @output.puts "Could not find extracted archive (tried #{extracted_file})"
+        @output.puts "Could not find extracted archive (tried #{self.class.extracted_file})"
       end
 
-      File.exist? self.class.force_install_location
+      File.exist?(self.class.force_installed_executable)
     end
 
   end

lib/arduino_ci/arduino_downloader_linux.rb

@@ -1,7 +1,5 @@
 require "arduino_ci/arduino_downloader"
 
-USE_BUILDER = false
-
 module ArduinoCI
 
   # Manage the linux download & install of Arduino
@@ -10,13 +8,25 @@ class ArduinoDownloaderLinux < ArduinoDownloader
     # The local filename of the desired IDE package (zip/tar/etc)
     # @return [string]
     def package_file
-      "#{extracted_file}-linux64.tar.xz"
+      "arduino-cli_#{@desired_version}_Linux_64bit.tar.gz"
+    end
+
+    # The local file (dir) name of the extracted IDE package (zip/tar/etc)
+    # @return [string]
+    def self.extracted_file
+      "arduino-cli"
+    end
+
+    # The executable Arduino file in an existing installation, or nil
+    # @return [string]
+    def self.existing_executable
+      Host.which("arduino-cli")
     end
 
     # Make any preparations or run any checks prior to making changes
     # @return [string] Error message, or nil if success
     def prepare
-      reqs = [extracter]
+      reqs = [self.class.extracter]
       reqs.each do |req|
         return "#{req} does not appear to be installed!" unless Host.which(req)
       end
@@ -26,62 +36,14 @@ def prepare
     # The technology that will be used to extract the download
     # (for logging purposes)
     # @return [string]
-    def extracter
+    def self.extracter
       "tar"
     end
 
     # Extract the package_file to extracted_file
     # @return [bool] whether successful
-    def extract
-      system(extracter, "xf", package_file)
-    end
-
-    # The local file (dir) name of the extracted IDE package (zip/tar/etc)
-    # @return [string]
-    def extracted_file
-      "arduino-#{@desired_ide_version}"
-    end
-
-    # The path to the directory of an existing installation, or nil
-    # @return [string]
-    def self.existing_installation
-      exe = self.existing_executable
-      return nil if exe.nil?
-
-      File.dirname(exe) # it's not really this
-      # but for this platform it doesn't really matter
-    end
-
-    # The executable Arduino file in an existing installation, or nil
-    # @return [string]
-    def self.existing_executable
-      if USE_BUILDER
-        # builder_name = "arduino-builder"
-        # cli_place = Host.which(builder_name)
-        # unless cli_place.nil?
-        #   ret = ArduinoCmdLinuxBuilder.new
-        #   ret.base_cmd = [cli_place]
-        #   return ret
-        # end
-      end
-      Host.which("arduino")
-    end
-
-    # The executable Arduino file in a forced installation, or nil
-    # @return [string]
-    def self.force_installed_executable
-      if USE_BUILDER
-        # forced_builder = File.join(ArduinoCmdLinuxBuilder.force_install_location, builder_name)
-        # if File.exist?(forced_builder)
-        #   ret = ArduinoCmdLinuxBuilder.new
-        #   ret.base_cmd = [forced_builder]
-        #   return ret
-        # end
-      end
-      forced_arduino = File.join(self.force_install_location, "arduino")
-      return forced_arduino if File.exist? forced_arduino
-
-      nil
+    def self.extract(package_file)
+      system(extracter, "xf", package_file, extracted_file)
     end
 
   end

lib/arduino_ci/arduino_downloader_osx.rb

@@ -8,54 +8,42 @@ class ArduinoDownloaderOSX < ArduinoDownloader
     # The local filename of the desired IDE package (zip/tar/etc)
     # @return [string]
     def package_file
-      "arduino-#{@desired_ide_version}-macosx.zip"
+      "arduino-cli_#{@desired_version}_macOS_64bit.tar.gz"
     end
 
     # The local file (dir) name of the extracted IDE package (zip/tar/etc)
     # @return [string]
-    def extracted_file
-      "Arduino.app"
+    def self.extracted_file
+      "arduino-cli"
     end
 
-    # @return [String] The location where a forced install will go
-    def self.force_install_location
-      # include the .app extension
-      File.join(ENV['HOME'], 'Arduino.app')
-    end
-
-    # An existing Arduino directory in one of the given directories, or nil
-    # @param Array<string> a list of places to look
+    # The executable Arduino file in an existing installation, or nil
     # @return [string]
-    def self.find_existing_arduino_dir(paths)
-      paths.find(&File.method(:exist?))
+    def self.existing_executable
+      Host.which("arduino-cli")
     end
 
-    # An existing Arduino file in one of the given directories, or nil
-    # @param Array<string> a list of places to look for the executable
-    # @return [string]
-    def self.find_existing_arduino_exe(paths)
-      paths.find do |path|
-        exe = File.join(path, "MacOS", "Arduino")
-        File.exist? exe
+    # Make any preparations or run any checks prior to making changes
+    # @return [string] Error message, or nil if success
+    def prepare
+      reqs = [self.class.extracter]
+      reqs.each do |req|
+        return "#{req} does not appear to be installed!" unless Host.which(req)
       end
+      nil
     end
 
-    # The path to the directory of an existing installation, or nil
+    # The technology that will be used to extract the download
+    # (for logging purposes)
     # @return [string]
-    def self.existing_installation
-      self.find_existing_arduino_dir(["/Applications/Arduino.app"])
+    def self.extracter
+      "tar"
     end
 
-    # The executable Arduino file in an existing installation, or nil
-    # @return [string]
-    def self.existing_executable
-      self.find_existing_arduino_exe(["/Applications/Arduino.app"])
-    end
-
-    # The executable Arduino file in a forced installation, or nil
-    # @return [string]
-    def self.force_installed_executable
-      self.find_existing_arduino_exe([self.force_install_location])
+    # Extract the package_file to extracted_file
+    # @return [bool] whether successful
+    def self.extract(package_file)
+      system(extracter, "xf", package_file, extracted_file)
     end
 
   end

lib/arduino_ci/arduino_downloader_windows.rb

@@ -10,19 +10,6 @@ module ArduinoCI
   # Manage the POSIX download & install of Arduino
   class ArduinoDownloaderWindows < ArduinoDownloader
 
-    # Make any preparations or run any checks prior to making changes
-    # @return [string] Error message, or nil if success
-    def prepare
-      nil
-    end
-
-    # The technology that will be used to complete the download
-    # (for logging purposes)
-    # @return [string]
-    def downloader
-      "open-uri"
-    end
-
     # Download the package_url to package_file
     # @return [bool] whether successful
     def download
@@ -35,29 +22,28 @@ def download
       @output.puts "\nArduino force-install failed downloading #{package_url}: #{e}"
     end
 
-    # Move the extracted package file from extracted_file to the force_install_location
-    # @return [bool] whether successful
-    def install
-      # Move only the content of the directory
-      FileUtils.mv extracted_file, self.class.force_install_location
-    end
-
     # The local filename of the desired IDE package (zip/tar/etc)
     # @return [string]
     def package_file
-      "#{extracted_file}-windows.zip"
+      "arduino-cli_#{@desired_version}_Windows_64bit.zip"
+    end
+
+    # The executable Arduino file in an existing installation, or nil
+    # @return [string]
+    def self.existing_executable
+      Host.which("arduino-cli")
     end
 
     # The technology that will be used to extract the download
     # (for logging purposes)
     # @return [string]
-    def extracter
+    def self.extracter
       "Expand-Archive"
     end
 
     # Extract the package_file to extracted_file
     # @return [bool] whether successful
-    def extract
+    def self.extract(package_file)
       Zip::File.open(package_file) do |zip|
         zip.each do |file|
           file.extract(file.name)
@@ -67,36 +53,8 @@ def extract
 
     # The local file (dir) name of the extracted IDE package (zip/tar/etc)
     # @return [string]
-    def extracted_file
-      "arduino-#{@desired_ide_version}"
-    end
-
-    # The path to the directory of an existing installation, or nil
-    # @return [string]
-    def self.existing_installation
-      exe = self.existing_executable
-      return nil if exe.nil?
-
-      File.dirname(exe)
-    end
-
-    # The executable Arduino file in an existing installation, or nil
-    # @return [string]
-    def self.existing_executable
-      arduino_reg = 'SOFTWARE\WOW6432Node\Arduino'
-      Win32::Registry::HKEY_LOCAL_MACHINE.open(arduino_reg).find do |reg|
-        path = reg.read_s('Install_Dir')
-        exe = File.join(path, "arduino_debug.exe")
-        File.exist? exe
-      end
-    rescue
-      nil
-    end
-
-    # The executable Arduino file in a forced installation, or nil
-    # @return [string]
-    def self.force_installed_executable
-      File.join(self.force_install_location, "arduino_debug.exe")
+    def self.extracted_file
+      "arduino-cli.exe"
     end
 
   end

lib/arduino_ci/arduino_installation.rb

@@ -1,15 +1,11 @@
 require 'pathname'
 require "arduino_ci/host"
-require "arduino_ci/arduino_cmd_osx"
-require "arduino_ci/arduino_cmd_linux"
-require "arduino_ci/arduino_cmd_windows"
-require "arduino_ci/arduino_cmd_linux_builder"
+require "arduino_ci/arduino_backend"
 require "arduino_ci/arduino_downloader_osx"
 require "arduino_ci/arduino_downloader_linux"
-
 require "arduino_ci/arduino_downloader_windows" if ArduinoCI::Host.os == :windows
 
-DESIRED_ARDUINO_IDE_VERSION = "1.8.6".freeze
+DESIRED_ARDUINO_CLI_VERSION = "0.13.0".freeze
 
 module ArduinoCI
 
@@ -23,80 +19,22 @@ class << self
       # attempt to find a workable Arduino executable across platforms
       #
       # Autolocation assumed to be an expensive operation
-      # @return [ArduinoCI::ArduinoCmd] an instance of the command or nil if it can't be found
+      # @return [ArduinoCI::ArduinoBackend] an instance of the command or nil if it can't be found
       def autolocate
-        ret = nil
-        case Host.os
-        when :osx then
-          ret = autolocate_osx
-        when :linux then
-          loc = ArduinoDownloaderLinux.autolocated_executable
-          return nil if loc.nil?
-
-          ret = ArduinoCmdLinux.new
-          ret.base_cmd = [loc]
-          ret.binary_path = Pathname.new(loc)
-        when :windows then
-          loc = ArduinoDownloaderWindows.autolocated_executable
-          return nil if loc.nil?
-
-          ret = ArduinoCmdWindows.new
-          ret.base_cmd = [loc]
-          ret.binary_path = Pathname.new(loc)
+        downloader_class = case Host.os
+        when :osx     then ArduinoDownloaderOSX
+        when :linux   then ArduinoDownloaderLinux
+        when :windows then ArduinoDownloaderWindows
         end
-        ret
-      end
 
-      # @return [ArduinoCI::ArduinoCmdOSX] an instance of the command or nil if it can't be found
-      def autolocate_osx
-        osx_root = ArduinoDownloaderOSX.autolocated_installation
-        return nil if osx_root.nil?
-        return nil unless File.exist? osx_root
+        loc = downloader_class.autolocated_executable
+        return nil if loc.nil?
 
-        launchers = [
-          # try a hack that skips splash screen
-          # from https://github.com/arduino/Arduino/issues/1970#issuecomment-321975809
-          [
-            "java",
-            "-cp",
-            "#{osx_root}/Contents/Java/*",
-            "-DAPP_DIR=#{osx_root}/Contents/Java",
-            "-Dfile.encoding=UTF-8",
-            "-Dapple.awt.UIElement=true",
-            "-Xms128M",
-            "-Xmx512M",
-            "processing.app.Base",
-          ],
-          # failsafe way
-          [File.join(osx_root, "Contents", "MacOS", "Arduino")]
-        ]
-
-        # create return and find a command launcher that works
-        ret = ArduinoCmdOSX.new
-        launchers.each do |launcher|
-          # test whether this method successfully launches the IDE
-          # note that "successful launch" involves a command that will fail,
-          # because that's faster than any command which succeeds.  what we
-          # don't want to see is a java error.
-          args = launcher + ["--bogus-option"]
-          result = Host.run_and_capture(*args)
-
-          # NOTE: Was originally searching for "Error: unknown option: --bogus-option"
-          #           but also need to find "Erreur: option inconnue : --bogus-option"
-          #           and who knows how many other languages.
-          # For now, just search for the end of the error and hope that the java-style
-          #  launch of this won't include a similar string in it
-          next unless result[:err].include? ": --bogus-option"
-
-          ret.base_cmd = launcher
-          ret.binary_path = Pathname.new(osx_root)
-          return ret
-        end
-        nil
+        ArduinoBackend.new(loc)
       end
 
       # Attempt to find a workable Arduino executable across platforms, and install it if we don't
-      # @return [ArduinoCI::ArduinoCmd] an instance of a command
+      # @return [ArduinoCI::ArduinoBackend] an instance of a command
       def autolocate!(output = $stdout)
         candidate = autolocate
         return candidate unless candidate.nil?
@@ -109,7 +47,7 @@ def autolocate!(output = $stdout)
 
       # Forcibly install Arduino from the web
       # @return [bool] Whether the command succeeded
-      def force_install(output = $stdout, version = DESIRED_ARDUINO_IDE_VERSION)
+      def force_install(output = $stdout, version = DESIRED_ARDUINO_CLI_VERSION)
         worker_class = case Host.os
         when :osx then ArduinoDownloaderOSX
         when :windows then ArduinoDownloaderWindows

lib/arduino_ci/host.rb

@@ -6,6 +6,13 @@ module ArduinoCI
 
   # Tools for interacting with the host machine
   class Host
+    # TODO: this came from https://stackoverflow.com/a/22716582/2063546
+    #       and I'm not sure if it can be replaced by self.os == :windows
+    WINDOWS_VARIANT_REGEX = /mswin32|cygwin|mingw|bccwin/
+
+    # e.g. 11/27/2020  01:02 AM    <SYMLINKD>     ExcludeSomething [C:\projects\arduino-ci\SampleProjects\ExcludeSomething]
+    DIR_SYMLINK_REGEX = %r{\d+/\d+/\d+\s+[^<]+<SYMLINKD?>\s+(.*) \[([^\]]+)\]}
+
     # Cross-platform way of finding an executable in the $PATH.
     # via https://stackoverflow.com/a/5471032/2063546
     #   which('ruby') #=> /usr/bin/ruby
@@ -38,21 +45,69 @@ def self.os
       return :windows if OS.windows?
     end
 
+    # Cross-platform symlinking
     # if on windows, call mklink, else self.symlink
     # @param [Pathname] old_path
     # @param [Pathname] new_path
     def self.symlink(old_path, new_path)
-      return FileUtils.ln_s(old_path.to_s, new_path.to_s) unless RUBY_PLATFORM =~ /mswin32|cygwin|mingw|bccwin/
+      # we would prefer `new_path.make_symlink(old_path)` but "symlink function is unimplemented on this machine" with windows
+      return new_path.make_symlink(old_path) unless needs_symlink_hack?
 
-      # https://stackoverflow.com/a/22716582/2063546
+      # via https://stackoverflow.com/a/22716582/2063546
       # windows mklink syntax is reverse of unix ln -s
       # windows mklink is built into cmd.exe
       # vulnerable to command injection, but okay because this is a hack to make a cli tool work.
-      orp = old_path.realpath.to_s.tr("/", "\\") # HACK DUE TO REALPATH BUG where it
-      np = new_path.to_s.tr("/", "\\")           # still joins windows paths with '/'
+      orp = pathname_to_windows(old_path.realpath)
+      np  = pathname_to_windows(new_path)
 
       _stdout, _stderr, exitstatus = Open3.capture3('cmd.exe', "/C mklink /D #{np} #{orp}")
       exitstatus.success?
     end
+
+    # Hack for "realpath" which on windows joins paths with slashes instead of backslashes
+    # @param path [Pathname] the path to render
+    # @return [String] A path that will work on windows
+    def self.pathname_to_windows(path)
+      path.to_s.tr("/", "\\")
+    end
+
+    # Hack for "realpath" which on windows joins paths with slashes instead of backslashes
+    # @param str [String] the windows path
+    # @return [Pathname] A path that will be recognized by pathname
+    def self.windows_to_pathname(str)
+      Pathname.new(str.tr("\\", "/"))
+    end
+
+    # Whether this OS requires a hack for symlinks
+    # @return [bool]
+    def self.needs_symlink_hack?
+      RUBY_PLATFORM =~ WINDOWS_VARIANT_REGEX
+    end
+
+    # Cross-platform is-this-a-symlink function
+    # @param [Pathname] path
+    # @return [bool] Whether the file is a symlink
+    def self.symlink?(path)
+      return path.symlink? unless needs_symlink_hack?
+
+      !readlink(path).nil?
+    end
+
+    # Cross-platform "read link" function
+    # @param [Pathname] path
+    # @return [Pathname] the link target
+    def self.readlink(path)
+      return path.readlink unless needs_symlink_hack?
+
+      the_dir  = pathname_to_windows(path.parent)
+      the_file = path.basename.to_s
+
+      stdout, _stderr, _exitstatus = Open3.capture3('cmd.exe', "/c dir /al #{the_dir}")
+      symlinks = stdout.lines.map { |l| DIR_SYMLINK_REGEX.match(l) }.compact
+      our_link = symlinks.find { |m| m[1] == the_file }
+      return nil if our_link.nil?
+
+      windows_to_pathname(our_link[2])
+    end
   end
 end

lib/arduino_ci/library_properties.rb

@@ -11,12 +11,22 @@ class LibraryProperties
     # @param path [Pathname] The path to the library.properties file
     def initialize(path)
       @fields = {}
-      File.foreach(path) do |line|
+      File.foreach(path) do |line_with_delim|
+        line = line_with_delim.chomp
         parts = line.split("=", 2)
-        @fields[parts[0]] = parts[1].chomp unless parts.empty?
+        next if parts[0].nil?
+        next if parts[0].empty?
+        next if parts[1].nil?
+
+        @fields[parts[0]] = parts[1] unless parts[1].empty?
       end
     end
 
+    # @return [Hash] the properties as a hash, all strings
+    def to_h
+      @fields.clone
+    end
+
     # Enable a shortcut syntax for library property accessors, in the style of `attr_accessor` metaprogramming.
     # This is used to create a named field pointing to a specific property in the file, optionally applying
     # a specific formatting function.

spec/arduino_cmd_spec.rb → spec/arduino_backend_spec.rb

@@ -6,106 +6,95 @@ def get_sketch(dir, file)
 end
 
 
-RSpec.describe ArduinoCI::ArduinoCmd do
+RSpec.describe ArduinoCI::ArduinoBackend do
   next if skip_ruby_tests
-  next if skip_splash_screen_tests
 
-  arduino_cmd = ArduinoCI::ArduinoInstallation.autolocate!
+  backend = ArduinoCI::ArduinoInstallation.autolocate!
 
   after(:each) do |example|
     if example.exception
-      puts "Last message: #{arduino_cmd.last_msg}"
+      puts "Last message: #{backend.last_msg}"
       puts "========== Stdout:"
-      puts arduino_cmd.last_out
+      puts backend.last_out
       puts "========== Stderr:"
-      puts arduino_cmd.last_err
+      puts backend.last_err
     end
   end
 
   context "initialize" do
     it "sets base vars" do
-      expect(arduino_cmd.base_cmd).not_to be nil
-      expect(arduino_cmd.prefs.class).to be Hash
+      expect(backend.binary_path).not_to be nil
     end
   end
 
   context "board_installed?" do
     it "Finds installed boards" do
-      uno_installed = arduino_cmd.board_installed? "arduino:avr:uno"
+      backend.install_boards("arduino:avr") # we used to assume this was installed... not the case for arduino-cli
+      uno_installed = backend.board_installed? "arduino:avr:uno"
       expect(uno_installed).to be true
       expect(uno_installed).not_to be nil
     end
 
     it "Doesn't find bogus boards" do
-      bogus_installed = arduino_cmd.board_installed? "eggs:milk:wheat"
+      bogus_installed = backend.board_installed? "eggs:milk:wheat"
       expect(bogus_installed).to be false
       expect(bogus_installed).not_to be nil
     end
   end
 
   context "installation of boards" do
     it "installs and sets boards" do
-      expect(arduino_cmd.install_boards("arduino:sam")).to be true
-      expect(arduino_cmd.use_board("arduino:sam:arduino_due_x")).to be true
+      expect(backend.install_boards("arduino:sam")).to be true
     end
   end
 
   context "libraries" do
     it "knows where to find libraries" do
-      fake_lib = "_____nope"
-      expected_dir = Pathname.new(arduino_cmd.lib_dir) + fake_lib
-      expect(arduino_cmd.library_path(fake_lib)).to eq(expected_dir)
-      expect(arduino_cmd.library_present?(fake_lib)).to be false
+      fake_lib_name = "_____nope"
+      expected_dir = Pathname.new(backend.lib_dir) + fake_lib_name
+      fake_lib = backend.library_of_name(fake_lib_name)
+      expect(fake_lib.path).to eq(expected_dir)
+      expect(fake_lib.installed?).to be false
     end
   end
 
-  context "set_pref" do
-
-    it "Sets key to what it was before" do
-      upload_verify = arduino_cmd.get_pref("upload.verify")
-      result = arduino_cmd.set_pref("upload.verify", upload_verify)
-      expect(result).to be true
-    end
-  end
-
-
   context "board_manager" do
     it "Reads and writes board_manager URLs" do
       fake_urls = ["http://foo.bar", "http://arduino.ci"]
-      existing_urls = arduino_cmd.board_manager_urls
+      existing_urls = backend.board_manager_urls
 
       # try to ensure maxiumum variability in the test
       test_url_sets = (existing_urls.empty? ? [fake_urls, []] : [[], fake_urls]) + [existing_urls]
 
       test_url_sets.each do |urls|
-        arduino_cmd.board_manager_urls = urls
-        expect(arduino_cmd.board_manager_urls).to match_array(urls)
+        backend.board_manager_urls = urls
+        expect(backend.board_manager_urls).to match_array(urls)
       end
     end
   end
 
 
-  context "verify_sketch" do
+  context "compile_sketch" do
 
     sketch_path_ino = get_sketch("FakeSketch", "FakeSketch.ino")
     sketch_path_pde = get_sketch("FakeSketch", "FakeSketch.pde")
     sketch_path_mia = get_sketch("NO_FILE_HERE", "foo.ino")
     sketch_path_bad = get_sketch("BadSketch", "BadSketch.ino")
 
     it "Rejects a PDE sketch at #{sketch_path_pde}" do
-      expect(arduino_cmd.verify_sketch(sketch_path_pde)).to be false
+      expect(backend.compile_sketch(sketch_path_pde, "arduino:avr:uno")).to be false
     end
 
     it "Fails a missing sketch at #{sketch_path_mia}" do
-      expect(arduino_cmd.verify_sketch(sketch_path_mia)).to be false
+      expect(backend.compile_sketch(sketch_path_mia, "arduino:avr:uno")).to be false
     end
 
     it "Fails a bad sketch at #{sketch_path_bad}" do
-      expect(arduino_cmd.verify_sketch(sketch_path_bad)).to be false
+      expect(backend.compile_sketch(sketch_path_bad, "arduino:avr:uno")).to be false
     end
 
     it "Passes a simple INO sketch at #{sketch_path_ino}" do
-      expect(arduino_cmd.verify_sketch(sketch_path_ino)).to be true
+      expect(backend.compile_sketch(sketch_path_ino, "arduino:avr:uno")).to be true
     end
   end
 end

spec/arduino_downloader_spec.rb

@@ -7,18 +7,15 @@
     it "has correct class properties" do
       ad = ArduinoCI::ArduinoDownloader
 
-      expect{ad.autolocated_executable}.to raise_error(NotImplementedError)
-      expect{ad.autolocated_installation}.to raise_error(NotImplementedError)
-      expect{ad.existing_installation}.to raise_error(NotImplementedError)
       expect{ad.existing_executable}.to raise_error(NotImplementedError)
-      expect{ad.force_installed_executable}.to raise_error(NotImplementedError)
-      expect(ad.force_install_location).to eq(File.join(ENV['HOME'], 'arduino_ci_ide'))
+      expect{ad.extracted_file}.to raise_error(NotImplementedError)
+      expect{ad.extracter}.to raise_error(NotImplementedError)
+      expect{ad.extract("foo")}.to raise_error(NotImplementedError)
     end
 
     it "has correct instance properties" do
       ad = ArduinoCI::ArduinoDownloader.new(DESIRED_VERSION)
       expect(ad.prepare).to be nil
-      expect{ad.package_url}.to raise_error(NotImplementedError)
       expect{ad.package_file}.to raise_error(NotImplementedError)
     end
   end
@@ -29,23 +26,20 @@
   context "Basics" do
     it "has correct class properties" do
       ad = ArduinoCI::ArduinoDownloaderLinux
-      # these will vary with CI.  Don't test them.
-      # expect(ad.autolocated_executable).to be nil
-      # expect(ad.autolocated_installation).to be nil
-      # expect(ad.existing_installation).to be nil
+      # these can vary with CI.  Don't test them.
       # expect(ad.existing_executable).to be nil
+      # expect(ad.autolocated_executable).to be nil
       # expect(ad.force_installed_executable).to be nil
 
-      expect(ad.force_install_location).to eq(File.join(ENV['HOME'], 'arduino_ci_ide'))
+      expect(ad.downloader).to eq("open-uri")
+      expect(ad.extracter).to eq("tar")
     end
 
     it "has correct instance properties" do
       ad = ArduinoCI::ArduinoDownloaderLinux.new(DESIRED_VERSION)
       expect(ad.prepare).to be nil
-      expect(ad.downloader).to eq("open-uri")
-      expect(ad.extracter).to eq("tar")
-      expect(ad.package_url).to eq("https://downloads.arduino.cc/arduino-rhubarb-linux64.tar.xz")
-      expect(ad.package_file).to eq("arduino-rhubarb-linux64.tar.xz")
+      expect(ad.package_url).to eq("https://github.com/arduino/arduino-cli/releases/download/rhubarb/arduino-cli_rhubarb_Linux_64bit.tar.gz")
+      expect(ad.package_file).to eq("arduino-cli_rhubarb_Linux_64bit.tar.gz")
     end
   end
 end
@@ -55,23 +49,48 @@
   context "Basics" do
     it "has correct class properties" do
       ad = ArduinoCI::ArduinoDownloaderOSX
-      # these will vary with CI.  Don't test them.
-      # expect(ad.autolocated_executable).to be nil
-      # expect(ad.autolocated_installation).to be nil
-      # expect(ad.existing_installation).to be nil
+      # these can vary with CI.  Don't test them.
       # expect(ad.existing_executable).to be nil
+      # expect(ad.autolocated_executable).to be nil
       # expect(ad.force_installed_executable).to be nil
 
-      expect(ad.force_install_location).to eq(File.join(ENV['HOME'], 'Arduino.app'))
+      expect(ad.downloader).to eq("open-uri")
+      expect(ad.extracter).to eq("tar")
     end
 
     it "has correct instance properties" do
       ad = ArduinoCI::ArduinoDownloaderOSX.new(DESIRED_VERSION)
       expect(ad.prepare).to be nil
-      expect(ad.downloader).to eq("open-uri")
-      expect(ad.extracter).to eq("Zip")
-      expect(ad.package_url).to eq("https://downloads.arduino.cc/arduino-rhubarb-macosx.zip")
-      expect(ad.package_file).to eq("arduino-rhubarb-macosx.zip")
+      expect(ad.package_url).to eq("https://github.com/arduino/arduino-cli/releases/download/rhubarb/arduino-cli_rhubarb_macOS_64bit.tar.gz")
+      expect(ad.package_file).to eq("arduino-cli_rhubarb_macOS_64bit.tar.gz")
+    end
+  end
+end
+
+
+if ArduinoCI::Host.os == :windows
+  RSpec.describe ArduinoCI::ArduinoDownloaderWindows do
+    next if skip_ruby_tests
+    context "Basics" do
+      it "has correct class properties" do
+        ad = ArduinoCI::ArduinoDownloaderWindows
+        # these will vary with CI.  Don't test them.
+        # expect(ad.autolocated_executable).to be nil
+        # expect(ad.existing_executable).to be nil
+        # expect(ad.force_installed_executable).to be nil
+
+        expect(ad.downloader).to eq("open-uri")
+        expect(ad.extracter).to eq("Expand-Archive")
+      end
+
+      it "has correct instance properties" do
+        ad = ArduinoCI::ArduinoDownloaderWindows.new(DESIRED_VERSION)
+        expect(ad.prepare).to be nil
+        expect(ad.package_url).to eq("https://github.com/arduino/arduino-cli/releases/download/rhubarb/arduino-cli_rhubarb_Windows_64bit.zip")
+        expect(ad.package_file).to eq("arduino-cli_rhubarb_Windows_64bit.zip")
+      end
     end
   end
+
+
 end

spec/arduino_installation_spec.rb

@@ -2,7 +2,6 @@
 
 RSpec.describe ArduinoCI::ArduinoInstallation do
   next if skip_ruby_tests
-  next if skip_splash_screen_tests
 
   context "autolocate" do
     it "doesn't fail" do
@@ -11,10 +10,10 @@
   end
 
   context "autolocate!" do
-    arduino_cmd = ArduinoCI::ArduinoInstallation.autolocate!
+    backend = ArduinoCI::ArduinoInstallation.autolocate!
     it "doesn't fail" do
-      expect(arduino_cmd.base_cmd).not_to be nil
-      expect(arduino_cmd.lib_dir).not_to be nil
+      expect(backend.binary_path).not_to be nil
+      expect(backend.lib_dir).not_to be nil
     end
   end
 
@@ -24,11 +23,10 @@
       output.rewind
       expect(output.read.empty?).to be true
       # install a bogus version to save time downloading
-      arduino_cmd = ArduinoCI::ArduinoInstallation.force_install(output, "BOGUS VERSION")
+      backend = ArduinoCI::ArduinoInstallation.force_install(output, "BOGUS VERSION")
       output.rewind
       expect(output.read.empty?).to be false
     end
   end
 
 end
-

spec/ci_config_spec.rb

@@ -1,6 +1,8 @@
 require "spec_helper"
 require "pathname"
 
+require "fake_lib_dir"
+
 RSpec.describe ArduinoCI::CIConfig do
   next if skip_ruby_tests
   context "default" do
@@ -156,21 +158,40 @@
   end
 
   context "allowable_unittest_files" do
+
+    # we will need to install some dummy libraries into a fake location, so do that on demand
+    fld = FakeLibDir.new
+    backend = fld.backend
     cpp_lib_path = Pathname.new(__dir__) + "fake_library"
-    cpp_library = ArduinoCI::CppLibrary.new(cpp_lib_path, Pathname.new("my_fake_arduino_lib_dir"), [])
+
+    around(:example) { |example| fld.in_pristine_fake_libraries_dir(example) }
+    before(:each) { @cpp_library = backend.install_local_library(cpp_lib_path) }
 
     it "starts with a known set of files" do
-      expect(cpp_library.test_files.map { |f| File.basename(f) }).to match_array([
+      expect(cpp_lib_path.exist?).to be(true)
+      expect(@cpp_library).to_not be(nil)
+      expect(@cpp_library.path.exist?).to be(true)
+      expect(@cpp_library.test_files.map(&:basename).map(&:to_s)).to match_array([
         "sam-squamsh.cpp",
         "yes-good.cpp",
         "mars.cpp"
       ])
     end
 
     it "filters that set of files" do
+      expect(cpp_lib_path.exist?).to be(true)
+      expect(@cpp_library).to_not be(nil)
+      expect(@cpp_library.test_files.map(&:basename).map(&:to_s)).to match_array([
+        "sam-squamsh.cpp",
+        "yes-good.cpp",
+        "mars.cpp"
+      ])
+
       override_file = File.join(File.dirname(__FILE__), "yaml", "o1.yaml")
       combined_config = ArduinoCI::CIConfig.default.with_override(override_file)
-      expect(combined_config.allowable_unittest_files(cpp_library.test_files).map { |f| File.basename(f) }).to match_array([
+      expect(combined_config.unittest_info[:testfiles][:select]).to match_array(["*-*.*"])
+      expect(combined_config.unittest_info[:testfiles][:reject]).to match_array(["sam-squamsh.*"])
+      expect(combined_config.allowable_unittest_files(@cpp_library.test_files).map(&:basename).map(&:to_s)).to match_array([
         "yes-good.cpp",
       ])
     end

spec/cpp_library_spec.rb

@@ -1,59 +1,70 @@
 require "spec_helper"
 require "pathname"
+require 'tmpdir'
+
+require 'fake_lib_dir'
 
 sampleproj_path = Pathname.new(__dir__).parent + "SampleProjects"
 
-def get_relative_dir(sampleprojects_tests_dir)
-  base_dir = sampleprojects_tests_dir.ascend do |path|
-    break path if path.split[1].to_s == "SampleProjects"
-  end
-  sampleprojects_tests_dir.relative_path_from(base_dir)
+def verified_install(backend, path)
+  ret = backend.install_local_library(path)
+  raise "backend.install_local_library from '#{path}' failed: #{backend.last_msg}" if ret.nil?
+  ret
 end
 
-
 RSpec.describe "ExcludeSomething C++" do
   next if skip_cpp_tests
 
-  cpp_lib_path = sampleproj_path + "ExcludeSomething"
+  # we will need to install some dummy libraries into a fake location, so do that on demand
+  fld = FakeLibDir.new
+  backend = fld.backend
+  test_lib_name = "ExcludeSomething"
+  cpp_lib_path = sampleproj_path + test_lib_name
+  around(:example) { |example| fld.in_pristine_fake_libraries_dir(example) }
+  before(:each) do
+    @base_dir = fld.libraries_dir
+    @cpp_library = verified_install(backend, cpp_lib_path)
+  end
+
   context "without excludes" do
-    cpp_library = ArduinoCI::CppLibrary.new(cpp_lib_path,
-                                            Pathname.new("my_fake_arduino_lib_dir"),
-                                            [])
     context "cpp_files" do
       it "finds cpp files in directory" do
+        expect(@cpp_library).to_not be(nil)
         excludesomething_cpp_files = [
           Pathname.new("ExcludeSomething/src/exclude-something.cpp"),
           Pathname.new("ExcludeSomething/src/excludeThis/exclude-this.cpp")
         ]
-        relative_paths = cpp_library.cpp_files.map { |f| get_relative_dir(f) }
+        relative_paths = @cpp_library.cpp_files.map { |f| f.relative_path_from(@base_dir) }
         expect(relative_paths).to match_array(excludesomething_cpp_files)
       end
     end
 
     context "unit tests" do
       it "can't build due to files that should have been excluded" do
-        config = ArduinoCI::CIConfig.default.from_example(cpp_lib_path)
-        path = config.allowable_unittest_files(cpp_library.test_files).first
-        compiler = config.compilers_to_use.first
-        result = cpp_library.build_for_test_with_configuration(path,
-                                                              [],
-                                                              compiler,
-                                                              config.gcc_config("uno"))
+        @cpp_library = verified_install(backend, cpp_lib_path)
+        config       = ArduinoCI::CIConfig.default.from_example(cpp_lib_path)
+        path         = config.allowable_unittest_files(@cpp_library.test_files).first
+        compiler     = config.compilers_to_use.first
+        result       = @cpp_library.build_for_test_with_configuration(path,
+                                                                      [],
+                                                                      compiler,
+                                                                      config.gcc_config("uno"))
         expect(result).to be nil
       end
     end
   end
 
   context "with excludes" do
-    cpp_library = ArduinoCI::CppLibrary.new(cpp_lib_path,
-                                            Pathname.new("my_fake_arduino_lib_dir"),
-                                            ["src/excludeThis"].map(&Pathname.method(:new)))
+
     context "cpp_files" do
       it "finds cpp files in directory" do
+        @cpp_library = verified_install(backend, cpp_lib_path)
+        @cpp_library.exclude_dirs = ["src/excludeThis"].map(&Pathname.method(:new))
+
         excludesomething_cpp_files = [
           Pathname.new("ExcludeSomething/src/exclude-something.cpp")
         ]
-        relative_paths = cpp_library.cpp_files.map { |f| get_relative_dir(f) }
+        relative_paths = @cpp_library.cpp_files.map { |f| f.relative_path_from(@base_dir) }
         expect(relative_paths).to match_array(excludesomething_cpp_files)
       end
     end
@@ -68,6 +79,7 @@ def get_relative_dir(sampleprojects_tests_dir)
   answers = {
     DoSomething: {
       one_five: false,
+      library_properties: true,
       cpp_files: [Pathname.new("DoSomething") + "do-something.cpp"],
       cpp_files_libraries: [],
       header_dirs: [Pathname.new("DoSomething")],
@@ -80,6 +92,7 @@ def get_relative_dir(sampleprojects_tests_dir)
     },
     OnePointOhDummy: {
       one_five: false,
+      library_properties: false,
       cpp_files: [
         "OnePointOhDummy/YesBase.cpp",
         "OnePointOhDummy/utility/YesUtil.cpp",
@@ -96,6 +109,7 @@ def get_relative_dir(sampleprojects_tests_dir)
     },
     OnePointFiveMalformed: {
       one_five: false,
+      library_properties: false,
       cpp_files: [
         "OnePointFiveMalformed/YesBase.cpp",
         "OnePointFiveMalformed/utility/YesUtil.cpp",
@@ -110,6 +124,7 @@ def get_relative_dir(sampleprojects_tests_dir)
     },
     OnePointFiveDummy: {
       one_five: true,
+      library_properties: true,
       cpp_files: [
         "OnePointFiveDummy/src/YesSrc.cpp",
         "OnePointFiveDummy/src/subdir/YesSubdir.cpp",
@@ -129,6 +144,7 @@ def get_relative_dir(sampleprojects_tests_dir)
   # easier to construct this one from the other test cases
   answers[:DependOnSomething] = {
     one_five: true,
+    library_properties: true,
     cpp_files: ["DependOnSomething/src/YesDeps.cpp"].map { |f| Pathname.new(f) },
     cpp_files_libraries: answers[:OnePointOhDummy][:cpp_files] + answers[:OnePointFiveDummy][:cpp_files],
     header_dirs: ["DependOnSomething/src"].map { |f| Pathname.new(f) }, # this is not recursive!
@@ -141,32 +157,52 @@ def get_relative_dir(sampleprojects_tests_dir)
   answers.freeze
 
   answers.each do |sampleproject, expected|
+
+    # we will need to install some dummy libraries into a fake location, so do that on demand
+    fld = FakeLibDir.new
+    backend = fld.backend
+
     context "#{sampleproject}" do
       cpp_lib_path = sampleproj_path + sampleproject.to_s
-      cpp_library = ArduinoCI::CppLibrary.new(cpp_lib_path, sampleproj_path, [])
-      dependencies = cpp_library.arduino_library_dependencies.nil? ? [] : cpp_library.arduino_library_dependencies
+      around(:example) { |example| fld.in_pristine_fake_libraries_dir(example) }
+      before(:each) do
+        @base_dir = fld.libraries_dir
+        @cpp_library = verified_install(backend, cpp_lib_path)
+      end
+
+      it "is a sane test env" do
+        expect(sampleproject.to_s).to eq(@cpp_library.name)
+      end
 
       it "detects 1.5 format" do
-        expect(cpp_library.one_point_five?).to eq(expected[:one_five])
+        expect(@cpp_library.one_point_five?).to eq(expected[:one_five])
+      end
+
+      it "detects library.properties" do
+        expect(@cpp_library.library_properties?).to eq(expected[:library_properties])
       end
 
+
       context "cpp_files" do
         it "finds cpp files in directory" do
-          relative_paths = cpp_library.cpp_files.map { |f| get_relative_dir(f) }
+          relative_paths = @cpp_library.cpp_files.map { |f| f.relative_path_from(@base_dir) }
           expect(relative_paths.map(&:to_s)).to match_array(expected[:cpp_files].map(&:to_s))
         end
       end
 
       context "cpp_files_libraries" do
         it "finds cpp files in directories of dependencies" do
-          relative_paths = cpp_library.cpp_files_libraries(dependencies).map { |f| get_relative_dir(f) }
+          @cpp_library.all_arduino_library_dependencies!  # side effect: installs them
+          dependencies = @cpp_library.arduino_library_dependencies.nil? ? [] : @cpp_library.arduino_library_dependencies
+          dependencies.each { |d| verified_install(backend, sampleproj_path + d) }
+          relative_paths = @cpp_library.cpp_files_libraries(dependencies).map { |f| f.relative_path_from(@base_dir) }
           expect(relative_paths.map(&:to_s)).to match_array(expected[:cpp_files_libraries].map(&:to_s))
         end
       end
 
       context "header_dirs" do
         it "finds directories containing h files" do
-          relative_paths = cpp_library.header_dirs.map { |f| get_relative_dir(f) }
+          relative_paths = @cpp_library.header_dirs.map { |f| f.relative_path_from(@base_dir) }
           expect(relative_paths.map(&:to_s)).to match_array(expected[:header_dirs].map(&:to_s))
         end
       end
@@ -176,56 +212,64 @@ def get_relative_dir(sampleprojects_tests_dir)
           # since we don't know where the CI system will install this stuff,
           # we need to go looking for a relative path to the SampleProjects directory
           # just to get our "expected" value
-          relative_path = get_relative_dir(cpp_library.tests_dir)
+          relative_path = @cpp_library.tests_dir.relative_path_from(@base_dir)
           expect(relative_path.to_s).to eq("#{sampleproject}/test")
         end
       end
 
       context "test_files" do
         it "finds cpp files in directory" do
-          relative_paths = cpp_library.test_files.map { |f| get_relative_dir(f) }
+          relative_paths = @cpp_library.test_files.map { |f| f.relative_path_from(@base_dir) }
           expect(relative_paths.map(&:to_s)).to match_array(expected[:test_files].map(&:to_s))
         end
       end
 
       context "arduino_library_src_dirs" do
         it "finds src dirs from dependent libraries" do
           # we explicitly feed in the internal dependencies
-          relative_paths = cpp_library.arduino_library_src_dirs(dependencies).map { |f| get_relative_dir(f) }
+          dependencies = @cpp_library.arduino_library_dependencies.nil? ? [] : @cpp_library.arduino_library_dependencies
+          dependencies.each { |d| verified_install(backend, sampleproj_path + d) }
+          relative_paths = @cpp_library.arduino_library_src_dirs(dependencies).map { |f| f.relative_path_from(@base_dir) }
           expect(relative_paths.map(&:to_s)).to match_array(expected[:arduino_library_src_dirs].map(&:to_s))
         end
       end
     end
   end
 
   context "test" do
+
+    # we will need to install some dummy libraries into a fake location, so do that on demand
+    fld = FakeLibDir.new
+    backend = fld.backend
     cpp_lib_path = sampleproj_path + "DoSomething"
-    cpp_library = ArduinoCI::CppLibrary.new(cpp_lib_path, Pathname.new("my_fake_arduino_lib_dir"), [])
     config = ArduinoCI::CIConfig.default
 
+    around(:example) { |example| fld.in_pristine_fake_libraries_dir(example) }
+    before(:each) { @cpp_library = verified_install(backend, cpp_lib_path) }
+
     after(:each) do |example|
       if example.exception
-        puts "Last command: #{cpp_library.last_cmd}"
+        puts "Last command: #{@cpp_library.last_cmd}"
         puts "========== Stdout:"
-        puts cpp_library.last_out
+        puts @cpp_library.last_out
         puts "========== Stderr:"
-        puts cpp_library.last_err
+        puts @cpp_library.last_err
       end
     end
 
     it "is going to test more than one library" do
-      test_files = cpp_library.test_files
+      test_files = @cpp_library.test_files
       expect(test_files.empty?).to be false
     end
 
-    test_files = cpp_library.test_files
+    test_files = Pathname.glob(Pathname.new(cpp_lib_path) + "test" + "*.cpp")
     test_files.each do |path|
       expected = path.basename.to_s.include?("good")
       config.compilers_to_use.each do |compiler|
         it "tests #{File.basename(path)} with #{compiler} expecting #{expected}" do
-          exe = cpp_library.build_for_test_with_configuration(path, [], compiler, config.gcc_config("uno"))
+          exe = @cpp_library.build_for_test_with_configuration(path, [], compiler, config.gcc_config("uno"))
           expect(exe).not_to be nil
-          expect(cpp_library.run_test_file(exe)).to eq(expected)
+          expect(@cpp_library.run_test_file(exe)).to eq(expected)
         end
       end
     end

spec/fake_lib_dir.rb

@@ -0,0 +1,56 @@
+require "arduino_ci"
+
+class FakeLibDir
+
+  attr_reader :config_dir
+  attr_reader :config_file
+  attr_reader :backend
+  attr_reader :arduino_dir
+  attr_reader :libraries_dir
+
+  def initialize
+    # we will need to install some dummy libraries into a fake location, so do that on demand
+    @config_dir = Pathname.new(Dir.pwd).realpath
+    @config_file = @config_dir + ArduinoCI::ArduinoBackend::CONFIG_FILE_NAME
+    @backend = ArduinoCI::ArduinoInstallation.autolocate!
+    @backend.config_dir = @config_dir
+  end
+
+  # designed to be called by rspec's "around" function
+  def in_pristine_fake_libraries_dir(example)
+    d = Dir.mktmpdir
+    begin
+      # write a yaml file containing the current directory
+      dummy_config = { "directories" => { "user" => d.to_s } }
+      @arduino_dir = Pathname.new(d)
+      @libraries_dir = @arduino_dir + "libraries"
+      Dir.mkdir(@libraries_dir)
+
+      f = File.open(@config_file, "w")
+      begin
+        f.write dummy_config.to_yaml
+        f.close
+        example.run
+      ensure
+        begin
+          File.unlink(@config_file)
+        rescue Errno::ENOENT
+          # cool, already done
+        end
+      end
+    ensure
+      if ArduinoCI::Host.needs_symlink_hack?
+        stdout, stderr, exitstatus = Open3.capture3('cmd.exe', "/c rmdir /s /q #{ArduinoCI::Host.pathname_to_windows(d)}")
+        unless exitstatus.success?
+          puts "====== rmdir of #{d} failed"
+          puts stdout
+          puts stderr
+        end
+      else
+        FileUtils.remove_entry(d)
+      end
+    end
+  end
+
+
+end

spec/host_spec.rb

@@ -0,0 +1,53 @@
+require "spec_helper"
+require 'tmpdir'
+
+
+def idempotent_delete(path)
+  path.delete
+rescue Errno::ENOENT
+end
+
+# creates a dir at <path> then deletes it after block executes
+# this will DESTROY any existing entry at that location in the filesystem
+def with_tmpdir(path)
+  begin
+    idempotent_delete(path)
+    path.mkpath
+    yield
+  ensure
+    idempotent_delete(path)
+  end
+end
+
+
+RSpec.describe ArduinoCI::Host do
+  next if skip_ruby_tests
+
+  context "symlinks" do
+    it "creates symlinks that we agree are symlinks" do
+      our_dir = Pathname.new(__dir__)
+      foo_dir = our_dir + "foo_dir"
+      bar_dir = our_dir + "bar_dir"
+
+      with_tmpdir(foo_dir) do
+        foo_dir.unlink # we just want to place something at this location
+        expect(foo_dir.exist?).to be_falsey
+
+        with_tmpdir(bar_dir) do
+          expect(bar_dir.exist?).to be_truthy
+          expect(bar_dir.symlink?).to be_falsey
+
+          ArduinoCI::Host.symlink(bar_dir, foo_dir)
+          expect(ArduinoCI::Host.symlink?(bar_dir)).to be_falsey
+          expect(ArduinoCI::Host.symlink?(foo_dir)).to be_truthy
+          expect(ArduinoCI::Host.readlink(foo_dir).realpath).to eq(bar_dir.realpath)
+        end
+      end
+
+      expect(foo_dir.exist?).to be_falsey
+      expect(bar_dir.exist?).to be_falsey
+
+    end
+  end
+
+end

spec/library_properties_spec.rb

@@ -3,7 +3,7 @@
 RSpec.describe ArduinoCI::LibraryProperties do
 
   context "property extraction" do
-    library_properties = ArduinoCI::LibraryProperties.new(Pathname.new(__dir__) + "properties/example.library.properties")
+    library_properties = ArduinoCI::LibraryProperties.new(Pathname.new(__dir__) + "properties" + "example.library.properties")
 
     expected = {
       string: {
@@ -36,10 +36,34 @@
       end
     end
 
+    it "reads full_paragraph" do
+      expect(library_properties.full_paragraph).to eq ([
+        expected[:string][:sentence],
+        expected[:string][:paragraph]
+      ].join(" "))
+    end
+
     it "doesn't crash on nonexistent fields" do
       expect(library_properties.dot_a_linkage).to be(nil)
     end
   end
 
+  context "Input handling" do
+    malformed_examples = [
+      "extra_blank_line.library.properties",
+      "just_equals.library.properties",
+      "no_equals.library.properties",
+      "no_key.library.properties",
+      "no_value.library.properties",
+    ].map { |e| Pathname.new(__dir__) + "properties" + e }
+
+    malformed_examples.each do |e|
+      quirk = e.basename.to_s.split(".library.").first
+      it "reads a properties file with #{quirk}" do
+        expect { ArduinoCI::LibraryProperties.new(e) }.to_not raise_error
+      end
+    end
+  end
+
 
 end

spec/properties/extra_blank_line.library.properties

@@ -0,0 +1,3 @@
+name=ExtraBlank
+
+sentence=We put the blank line in the middle so overzealous text editors dont trim it

spec/properties/just_equals.library.properties

@@ -0,0 +1,3 @@
+name=JustEquals
+sentence=Bad file with just an equals on a line
+=

spec/properties/no_equals.library.properties

@@ -0,0 +1,3 @@
+name=NoEquals
+sentence=Bad file with no equals on a line
+wat

spec/properties/no_key.library.properties

@@ -0,0 +1,3 @@
+name=NoKey
+sentence=Bad file with no key on a line
+=profit

spec/properties/no_value.library.properties

@@ -0,0 +1,3 @@
+name=NoValue
+sentence=Bad file with no value on a line
+seriously_why_do_we_even_have_this_line=

spec/spec_helper.rb

@@ -1,3 +1,7 @@
+require 'simplecov'
+SimpleCov.start do
+  add_filter %r{^/spec/}
+end
 require "bundler/setup"
 require "arduino_ci"
 
@@ -13,10 +17,6 @@
   end
 end
 
-def skip_splash_screen_tests
-  !ENV["ARDUINO_CI_SKIP_SPLASH_SCREEN_RSPEC_TESTS"].nil?
-end
-
 def skip_ruby_tests
   !ENV["ARDUINO_CI_SKIP_RUBY_RSPEC_TESTS"].nil?
 end

spec/testsomething_unittests_spec.rb

@@ -1,40 +1,48 @@
 require "spec_helper"
 require "pathname"
 
-sampleproj_path = Pathname.new(__dir__).parent + "SampleProjects"
-
-def get_relative_dir(sampleprojects_tests_dir)
-  base_dir = sampleprojects_tests_dir.ascend do |path|
-    break path if path.split[1].to_s == "SampleProjects"
-  end
-  sampleprojects_tests_dir.relative_path_from(base_dir)
-end
+require 'fake_lib_dir'
 
+sampleproj_path = Pathname.new(__dir__).parent + "SampleProjects"
 
 RSpec.describe "TestSomething C++" do
   next if skip_cpp_tests
-  cpp_lib_path = sampleproj_path + "TestSomething"
-  cpp_library = ArduinoCI::CppLibrary.new(cpp_lib_path,
-                                          Pathname.new("my_fake_arduino_lib_dir"),
-                                          ["src/excludeThis"].map(&Pathname.method(:new)))
+
+  # we will need to install some dummy libraries into a fake location, so do that on demand
+  fld = FakeLibDir.new
+  backend = fld.backend
+  test_lib_name = "TestSomething"
+  cpp_lib_path = sampleproj_path + test_lib_name
+
   context "cpp_files" do
+    around(:example) { |example| fld.in_pristine_fake_libraries_dir(example) }
+    before(:each) do
+      @base_dir = fld.libraries_dir
+      @cpp_library = backend.install_local_library(cpp_lib_path)
+    end
+
     it "finds cpp files in directory" do
       testsomething_cpp_files = [Pathname.new("TestSomething/src/test-something.cpp")]
-      relative_paths = cpp_library.cpp_files.map { |f| get_relative_dir(f) }
+      relative_paths = @cpp_library.cpp_files.map { |f| f.relative_path_from(@base_dir) }
       expect(relative_paths).to match_array(testsomething_cpp_files)
     end
   end
   config = ArduinoCI::CIConfig.default.from_example(cpp_lib_path)
 
   context "unit tests" do
+    around(:example) { |example| fld.in_pristine_fake_libraries_dir(example) }
+    before(:each) do
+      @base_dir = fld.libraries_dir
+      @cpp_library = backend.install_local_library(cpp_lib_path)
+    end
 
     it "is going to test more than one library" do
-      test_files = cpp_library.test_files
+      test_files = @cpp_library.test_files
       expect(test_files.empty?).to be false
     end
 
     it "has some allowable test files" do
-      allowed_files = config.allowable_unittest_files(cpp_library.test_files)
+      allowed_files = config.allowable_unittest_files(@cpp_library.test_files)
       expect(allowed_files.empty?).to be false
     end
 
@@ -46,11 +54,12 @@ def get_relative_dir(sampleprojects_tests_dir)
       expect(config.platforms_to_unittest.length.zero?).to be(false)
     end
 
+    cpp_library = backend.install_local_library(cpp_lib_path)
     test_files = config.allowable_unittest_files(cpp_library.test_files)
 
     # filter the list based on a glob, if provided
     unless ENV["ARDUINO_CI_SELECT_CPP_TESTS"].nil?
-      Dir.chdir(cpp_library.tests_dir) do
+      Dir.chdir(@cpp_library.tests_dir) do
         globbed = Pathname.glob(ENV["ARDUINO_CI_SELECT_CPP_TESTS"])
         test_files.select! { |p| globbed.include?(p.basename) }
       end
@@ -61,19 +70,21 @@ def get_relative_dir(sampleprojects_tests_dir)
       config.compilers_to_use.each do |compiler|
 
         context "file #{tfn} (using #{compiler})" do
+          around(:example) { |example| fld.in_pristine_fake_libraries_dir(example) }
 
           before(:all) do
-            @exe = cpp_library.build_for_test_with_configuration(path, [], compiler, config.gcc_config("uno"))
+            @cpp_library = backend.install_local_library(cpp_lib_path)
+            @exe = @cpp_library.build_for_test_with_configuration(path, [], compiler, config.gcc_config("uno"))
           end
 
           # extra debug for c++ failures
           after(:each) do |example|
             if example.exception
-              puts "Last command: #{cpp_library.last_cmd}"
+              puts "Last command: #{@cpp_library.last_cmd}"
               puts "========== Stdout:"
-              puts cpp_library.last_out
+              puts @cpp_library.last_out
               puts "========== Stderr:"
-              puts cpp_library.last_err
+              puts @cpp_library.last_err
             end
           end
 
@@ -82,7 +93,7 @@ def get_relative_dir(sampleprojects_tests_dir)
           end
           it "#{tfn} passes tests" do
             skip "Can't run the test program because it failed to build" if @exe.nil?
-            expect(cpp_library.run_test_file(@exe)).to_not be_falsey
+            expect(@cpp_library.run_test_file(@exe)).to_not be_falsey
           end
         end
       end