Handling configuration transformations

[Blacksmoke16]

Is your feature request related to a problem? Please describe.

I'm wanting to store the configuration for an application, but the format its stored in is hex encoded JSON. Ideally I would like to be able to have chezmoi track the JSON, but then when applying the changes, have it transform it into the required format. This way it's easier to manage.

I currently have it working by modeling it after the example in https://www.chezmoi.io/user-guide/use-scripts-to-perform-actions/#clear-the-state-of-run_once_-scripts via:

#!/bin/bash

# profile.json hash: {{ include "private_dot_config/private_Typora/profile.json" | sha256sum }}
cat {{ joinPath .chezmoi.sourceDir "/private_dot_config/private_Typora/profile.json" }} | oq -c . | tr -d '\n' | xxd -p | tr -d '\n' > $HOME/.config/Typora/profile.data

But wanted to bring this up to see if there's either a better existing way to handle this, or maybe is room for a new feature to better handle it.

Describe the solution you'd like

Basing a solution off of typora/typora-issues#2353 (comment), I was thinking of having some sort of transform_ prefix script. Like it would allow having profile.data in the source state with the JSON contents. Then if you had another file transform_profile.data it would pass the contents of profile.data to the script, and write its output to the target's profile.data file. This way it keeps the contents and the transformations separate.

Describe alternatives you've considered

Is possible that the run_once_ script approach is fine. The > $HOME/.config/Typora/profile.data part just felt a bit hacky.

Additional context

I don't feel too strongly about my proposed solution, so feel free to close if its not something worth adding.

[halostatue]

I’ve changed this into a discussion, because I think that it would be an interesting concept to explore. The difficulty that would be present is indicating what the transformation mechanisms would end up looking like.

At least some of this could probably be done now as your file:

{{-/* private_dot_config/Typora/profile.data */-}}
{{- .typora | toJson | hexEncode -}}

The hexEncode function doesn’t exist right now, as far as I know. It would mean maintaining all of your Typora configuration in your Chezmoi config under .data.typora.

As a note, your script could be simplified some by using xxd -p -c0 instead of needing the tr -d '\n' pipes.

Back to your original idea, though, some sort of transform capability could be very interesting for these cases, as that could also be used for diff/merge operations so that you’re not working solely with this.

Some of this would lean toward to ideas that @bradenhilton originally suggested for template substitutions. Maybe there’s a .transforms chezmoi configuration key (shown in yaml format here):

transforms:
  private_dot_config/Typora/profile.data:
    encode: jq -S -c $source | xxd -p -c0
    decode: xxd -r -p $target | jq -S .

That way, your private_dot_config/Typora/profile.data would be stored as unencoded formatted JSON (the lack of .json can’t be helped because of Chezmoi’s mapping rules), but would be written as hex-encoded JSON via encode. Diffs and merges could be piped through the decode scripts for comparison.

[halostatue]

hexEncode and hexDecode now exist in chezmoi 2.29 (thanks @twpayne!).

[Blacksmoke16]

That approach would deff work for me! Making it part of the configuration would deff be simpler than needing to manage a dedicated extra file per file. For more complex requirements could possibly still allow a dedicated file by pointing it at a script file leveraging the interpreters feature.

Another thought I had would be like using some magic "header"/string in the file. Something along the lines of {{- /* chezmoi:modify-template */ -}}.

{{- /* chezmoi:encode jq -S -c $source | xxd -p -c0 */ -}}
{{- /* chezmoi:dencode xxd -r -p $target | jq -S . */ -}}
hexencodedstringhere

Not a big fan of having it just magically work due to that tho. Being a bit more explicit/decoupled would prob be the better option.

[twpayne]

Note also that you can use .chezmoidata.<format> to store the Typora config in a separate file (i.e. not in your main config file). This would also have the side effect of allowing you to store your Typora config in a more human-friendly format than JSON (e.g. YAML or TOML).

Another thought I had would be like using some magic "header"/string in the file.

This is an interesting idea. I prefer the idea approach as it maintains a file's specific configuration inside the file itself, which means that there's only one file to edit when you make a change. However, the usability aspects are tricky: the encode/decode step would have to be performed by chezmoi edit (which forces you to use chezmoi edit) and chezmoi's template syntax interacts poorly with some formats, notably JSON.

[halostatue]

Two things:

{{- /* chezmoi:encode jq -S -c $source | xxd -p -c0 */ -}}
{{- /* chezmoi:dencode xxd -r -p $target | jq -S . */ -}}
hexencodedstringhere

That should actually be:

// chezmoi:encode: jq -S -c $source | xxd -p -c0
// chezmoi:decode: xxd -r -p $target | jq -S .
{
  "json": {
    "encoded": {
      "data": "lives here"
    }
  }
}

A chezmoi.encode directive or transforms.*.encode configuration represent a pipeline (more on that shortly) that takes the $source file and encodes it for writing to the $target by chezmoi. That means that the file is JSON encoded. Or, if you prefer what @twpayne suggested:

# chezmoi:encode: yaml2json $source | jq -S -c | xxd -p c0
# chezmoi:decode: xxd -r -p $target | jq -S . | json2yaml
yaml:
  encoded:
    data: "lives here"

Similarly, the decode configuration / directive represents a pipeline that reads from the $target for either chezmoi diff or chezmoi merge operations.

I see these as similar to clean/smudge filter configurations as in .gitconfig and .gitattributes.

This is an interesting idea. I prefer the idea approach as it maintains a file's specific configuration inside the file itself, which means that there's only one file to edit when you make a change. However, the usability aspects are tricky: the encode/decode step would have to be performed by chezmoi edit (which forces you to use chezmoi edit) and chezmoi's template syntax interacts poorly with some formats, notably JSON.

You wouldn’t need to use chezmoi edit if the encode directives are only used only apply operations and decode directives are only used for diff and merge operations. The bigger problem with the use of directives is that you have to read the file in order to know what filters to apply on the file. And using pipelines as such sort of requires using a compatible shell syntax so that they can be executed. There are alternate ways to consider the construction of a filter pipeline if it’s in an external file:

transforms:
  .config/Typora/profile.data:
    encode:
      - jq -S -c .
      - xxd -p -c0
    decode:
      - xxd -r -p
      - jq -S .

These would mean the same thing, but would become external commands that would be executed in series by chezmoi on data that is more or less in memory (it would pass the current buffer as stdin and receive stdout as the new buffer and repeat until the list of items is complete or there is an error in processing). This could be done with a directive pipeline (split the command on | and execute the commands in order the same way), but that approach would not work with PowerShell (something I don’t use, personally) or even with bash / zsh / fish builtins.

Ultimately, I’m not sure that there’s enough of a need for something like this to justify the implementation effort, but it is neat. If one has a lot of JSON configuration files — I only see ~2 in my regular configuration files and I rarely touch them — I think that this might be of more benefit.