Skip to content

Support custom python libraries in dev shell nixland #153

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 12 commits into
base: main
Choose a base branch
from

Conversation

drbh
Copy link
Collaborator

@drbh drbh commented Jun 12, 2025

This PR supersedes #150 and avoids adding the extra libraries into the build.toml.

example flake

{
  description = "Flake for megablocks_moe kernel";
  
  inputs = {
    kernel-builder.url = "github:huggingface/kernel-builder/support-custom-python-libraries-in-dev-shell-nixland";
    # Add libraries as inputs
    composer = {
      url = "github:mosaicml/composer";
      flake = false;
    };
    stk = {
      url = "github:stanford-futuredata/stk";
      flake = false;
    };
  };
  
  outputs = {
    self,
    kernel-builder,
    composer,
    stk,
  }:
    kernel-builder.lib.genFlakeOutputs {
      path = ./.;
      rev = self.shortRev or self.dirtyShortRev or self.lastModifiedDate;
      
      # Map custom packages to their sources
      customPythonPackages = {
        composer = composer;
        stk = stk;
      };
      
      pythonTestDeps = [
        "tqdm"
        "py-cpuinfo"
        "importlib-metadata"
        "torchmetrics"
        "composer"
        "stk" 
      ];
    };
}

Note: I was unclear on how to best include the custom Python package in an idiomatic Nix way and settled on the current solution. If there is an easier way to resolve Git repositories as libraries, that would be preferred.

@danieldk
Copy link
Member

I think this can be closed in favor of #155?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants