hcl.Unmarshal adding spurious arrays

[jantman]

I'm not entirely sure if this is a bug or not. However, I was using https://github.com/kvz/json2hcl in reverse mode to convert a HCL example to JSON, and stumbled on this...

HCL Template

Taken directly from https://vagrantcloud.com/help/vault/vsi/configuration

environment "aws" {
}

vault {
  address = "https://vault.service.consul:8200"
  mount_path = "auth/aws"
}

serve "file" {
  path = "/ramdisk/vault-token"
}

Expected behavior

What should have happened?

{
  "environment": {
    "aws": {}
  },
  "serve": {
    "file": {
      "path": "/ramdisk/vault-token"
    }
  },
  "vault": {
    "address": "https://vault.service.consul:8200",
    "mount_path": "auth/aws"
  }
}

Actual behavior

An invalid configuration, according to VSI:

{
  "environment": [
    {
      "aws": [
        {}
      ]
    }
  ],
  "serve": [
    {
      "file": [
        {
          "path": "/ramdisk/vault-token"
        }
      ]
    }
  ],
  "vault": [
    {
      "address": "https://vault.service.consul:8200",
      "mount_path": "auth/aws"
    }
  ]
}

This JSON has an extra level of arrays inside each top-level object, which is detected as invalid.

Steps to reproduce

I was using https://github.com/kvz/json2hcl for this, which is a CLI wrapper around hcl.

1. Clone and build https://github.com/kvz/json2hcl
2. Write the hcl template example to a file
3. cat example.hcl | json2hcl -reverse

References

• Arrays where they shouldn't be kvz/json2hcl#3 (comment)

[sethvargo]

This is the expected behavior (and the correct hcl to json equivalent) from what I can see.

The reason it is an array is because it's totally valid to specify environment "aws" {} multiple times. It may not be valid in your use case, but it's valid HCL:

environment "aws" {
  a = b
}

environment "aws" {
  c = d
}

environment "aws" {
  e = f
}

Maybe your application wants to merge those values, or maybe it wants to always use the one last-defined.

which is detected as invalid.

Could you elaborate on this a bit more? What makes it invalid? This seems like valid JSON to me.

[jantman]

I thought it looked right as well. Sorry about the vague "invalid"...

jantman@phoenix:pts/12:~/tmp$ ./vault-secure-intro -version
Vault Secure Introduction Client v0.1.0-alpha ('a730993ff7fdb24f97c7aeb7b4fb8db7aed3547a')
jantman@phoenix:pts/12:~/tmp$ cat example.hcl 
environment "aws" {
}

vault {
  address = "https://vault.service.consul:8200"
  mount_path = "auth/aws"
}

serve "file" {
  path = "/ramdisk/vault-token"
}
jantman@phoenix:pts/12:~/tmp$ ./vault-secure-intro -config=example.hcl 
[ERR] server/file: error opening temp file at /ramdisk for writing: open /ramdisk/vault-token.tmp.030468548: no such file or directory

this is correct and expected; the config parsed right, but /ramdisk doesn't exist

jantman@phoenix:pts/12:~/tmp$ cat example.hcl | json2hcl -reverse > example.json 
jantman@phoenix:pts/12:~/tmp$ cat example.json 
{
  "environment": [
    {
      "aws": [
        {}
      ]
    }
  ],
  "serve": [
    {
      "file": [
        {
          "path": "/ramdisk/vault-token"
        }
      ]
    }
  ],
  "vault": [
    {
      "address": "https://vault.service.consul:8200",
      "mount_path": "auth/aws"
    }
  ]
}
jantman@phoenix:pts/12:~/tmp$ ./vault-secure-intro -config=example.json 
Error loading configuration: error parsing 'environment': environment type not specified

I suppose it's possible that the bug is actually in VSI, but my assumption was that since VSI uses the hcl library, this should work...

Looking at the above, it seems that VSI is choking on "environment" being an array instead of a hash... and I'm not sure if that's the vault of hcl or vsi. But this feels to me like a bug in HCL, if it's supposed to support JSON but the JSON generated from HCL isn't treated as equivalent by apps...

[sethvargo]

Hmm yea - this is going to be a tough one, since the mapping between HCL<->JSON isn't 1:1. Specifically, there is more than one way to represent JSON in HCL and visa-versa.

[lukevers]

Hey all,

I've had this same problem as I started using viper recently and tried out HCL for the first time--it's great, I love it!

It was super annoying that every map was being surrounded by a slice. I was playing around with the source and figured out a quick fix for it: lukevers@90f1452.

I haven't done any testing except for my config tests for an IRC bot I have which is setup like this and works perfectly with the patch I added:

type Config struct {
    Servers map[string]Server
}

type Server struct {
    Nick      string
    User      string
    Name      string
    Host      string
    Port      int
    TLS       bool
    Reconnect bool
    Channels  []string
    Debug     bool
    Plugins   map[string]struct {
        Enabled bool
        Folder  string
        Pattern string
        Events  []string
    }
}

Just thought I'd point this out.

[erick-thompson]

I understand that JSON <-> HCL isn't one-to-one, but it would be nice if a given HCL resulted in valid JSON for terraform. I wouldn't expect a mapping backwards would result in the same HCL, but right now the inability to get terraform compatible JSON from valid HCL is causing me major headaches.