Support for private/enterprise patterns

[sdboyer]

Today, dep's primary target is Go's public, FLOSS ecosystem. The public ecosystem is the necessary shared baseline use case that must be satisfied.

Necessary...but not sufficient. Supporting private ecosystems and workflows not only makes things easier for Go users, but it can make it easier to convert private code to open source down the line.

There are a number of things enterprise organizations, or really any private organizations, may want:

• Private code hosting (related: Support for private repositories e.g. Github Enterprise #174, Central packaging registry #175, Workflow: local cache servers for remote packages #263)
• Policies to ban/bless projects, in their entirety or just a subset of their versions - Allow defining project {black,white}list policies  #285
• Seamless integration with monorepos
• Integration with higher-order build systems (bazel, pants, etc.)

It doesn't seem like these changes should be a blocker for initially merging dep into the go toolchain, though they should probably be in good shape by the time the go toolchain ships a version with dep in it.

[sdboyer]

/cc @pbennett this would be a great place for you to summarize some of the requirements you have that we've discussed :)

[thejerf]

Apologies if this is covered elsewhere; I've had a look around and I couldn't find it.

The two big use cases I have as at least a semi-enterprise user that often seem to be poorly covered for me is:

1. Being able to internally mirror repos and pull from those, even though we still use the "real" URL in the source itself.
2. Being able to put a patch on top of an external repo for whatever reason (security, integration, etc.) and use this patched repo in preference to the "real" one.

Given the way Go works, I think it would suffice for both these use cases to be able to specify an "alias" or something for a package, i.e., "I want github.com/go/AwesomePackage but please pull source from internalgit.enterprise.org/gits/go/AwesomePackage" or something like that. That would be really useful for me.

I tried to read the source code to see if this is already there, but I don't think it is? I see "overrides" in the manifest but that appears to be related to relaxing version constraints.

[sdboyer]

@thejerf it is, but no worries :) Have a look around here - #174 (comment). The source field in the manifest will allow you to individually specify an alternate location from which to pull a given dep. Roughly, an alias.

Doing them as more than one-offs is trickier, as described in the linked comment.

[lancehit]

#148 is one of the main issues I've been running up against using dep in my current project.

[sdboyer]

@lancehit #148 is one of those things that we may end up needing to devise an alternate approach for, rather than trying to support it more directly. Not sure yet, but it's possible.

[aajtodd]

Take this all with a grain of salt, I'm still wrapping my head around the Go import model.

We have a SVN monorepo and are looking at how we should structure Go code within it and how imports will work.

I'm curious what support for a monorepo would look like? Could you elaborate?

For example if we have this example monorepo structure:

trunk/
    pkg1/
    project/
        pkg2/

How do we get to the desired GOPATH structure:

$GOPATH/
    src/
        company/
            pkg1/
            pkg2/

I've already fought with go get and subversion and it seems more hassle than it's worth. Besides we want clean import paths such as company/pkg1 not svn.company.org:8000/trunk/ (which I realize is not a valid import path). Perhaps vanity imports or canonical import paths are of some use but I haven't spent enough time investigating.

Go doesn't really care how the code got into the $GOPATH so my current thought process is to checkout each project/pkg into $GOPATH/src/company as needed. This is obviously slightly annoying though. Each package that has internal dependencies will have to know which packages to check out.

I guess it woud be nice to have the manifest reference a local relative path in the monorepo with a different import name. Something like

{
    "dependencies": {
        "company/pkg1": {
            "source": "../../path/inside/monorepo/pkg1"
        },
    }
}

[steveklabnik]

@sdboyer hey hey, this isn't exactly about go, but I bet you'd find some of the conversation in rust-lang/rust-roadmap-2017#12 very helpful. We're currently investigating this stuff in rust-land too.

Good luck!

[sdboyer]

@steveklabnik oh man, there's a wealth of useful information there - thanks!

[quentinmit]

Grr, GitHub mobile moved the button into where the text field was just as I was tapping, sorry :)

@aajtodd If you have a monorepo, you will do best if you have a single directory that can be checked out into GOPATH. For example, svn.company.com:8000/trunk/go -> $GOPATH/SRC/go.company.com. It is greatly confusing when the directory structure in the repo does not match the package structure. (At that point, you don't have a monorepo, you have a bunch of separate repos that just happen to be stored in the same place.)

[aajtodd]

@quentinmit That's what I was leaning towards. And yes you are correct that it is currently more a bunch of separate repos/projects stored in one place. Not my choice, such is life.