trim_newlines replaces ^M with line break

[sblask]

FAQ

• I have checked the FAQ and it didn't resolve my problem.

Issues

• I have checked existing issues and there are no issues with the same problem.

Neovim Version

NVIM v0.8.3 Build type: Release LuaJIT 2.1.0-beta3 Compiled by [email protected] Features: +acl +iconv +tui See ":help feature-compile" system vimrc file: "$VIM/sysinit.vim" fall-back for $VIM: "/opt/homebrew/Cellar/neovim/0.8.3/share/nvim" Run :checkhealth for more info

Dev Version?

• I am using a stable Neovim release version, or if I am using a dev version of Neovim I have confirmed that my issue is reproducible on a stable version.

Operating System

MacOS

Minimal Config

-- this template is borrowed from nvim-lspconfig
local on_windows = vim.loop.os_uname().version:match("Windows")

local function join_paths(...)
    local path_sep = on_windows and "\\" or "/"
    local result = table.concat({ ... }, path_sep)
    return result
end

vim.g.loaded_remote_plugins = ""
vim.cmd([[set runtimepath=$VIMRUNTIME]])

local temp_dir = vim.loop.os_getenv("TEMP") or "/tmp"

vim.cmd("set packpath=" .. join_paths(temp_dir, "nvim", "site"))

local package_root = join_paths(temp_dir, "nvim", "site", "pack")
local install_path = join_paths(package_root, "packer", "start", "packer.nvim")
local compile_path = join_paths(install_path, "plugin", "packer_compiled.lua")

local null_ls_config = function()
    local null_ls = require("null-ls")
    -- add only what you need to reproduce your issue
    null_ls.setup({
        sources = {
            null_ls.builtins.formatting.trim_newlines,
        },
        debug = true,
    })
end

local function load_plugins()
    -- only add other plugins if they are necessary to reproduce the issue
    require("packer").startup({
        {
            "wbthomason/packer.nvim",
            {
                "jose-elias-alvarez/null-ls.nvim",
                requires = { "nvim-lua/plenary.nvim" },
                config = null_ls_config,
            },
        },
        config = {
            package_root = package_root,
            compile_path = compile_path,
        },
    })
end

if vim.fn.isdirectory(install_path) == 0 then
    vim.fn.system({ "git", "clone", "https://github.com/wbthomason/packer.nvim", install_path })
    load_plugins()
    require("packer").sync()
else
    load_plugins()
    require("packer").sync()
end

Steps to Reproduce

• nvim --clean -u minimal_init.lua
• edit this file: https://github.com/sblask/dotfiles/blob/bf86261ae0006e8672ab258a77cc6499700ac397/dotfiles_shared/.config/nvim.symlink/vim/commands.vim
• :lua vim.lsp.buf.format()

Reproducibility Check

• I confirm that my minimal config is based on the minimal_init.lua template and that my issue is reproducible by running nvim --clean -u minimal_init.lua and following the steps above.

Expected Behavior

The file should remain unchanged.

Actual Behavior

In https://github.com/sblask/dotfiles/blob/bf86261ae0006e8672ab258a77cc6499700ac397/dotfiles_shared/.config/nvim.symlink/vim/commands.vim#L53 the ^M get replaces by a new line resulting in a broken config

Debug Log

[TRACE Mon 24 Apr 11:48:37 2023] ...te/pack/packer/start/null-ls.nvim/lua/null-ls/client.lua:97: starting null-ls client
[TRACE Mon 24 Apr 11:48:37 2023] .../site/pack/packer/start/null-ls.nvim/lua/null-ls/rpc.lua:102: received LSP request for method initialize
[DEBUG Mon 24 Apr 11:48:37 2023] ...te/pack/packer/start/null-ls.nvim/lua/null-ls/client.lua:152: unable to notify client for method textDocument/didOpen (client not active): {
textDocument = {
uri = "file:///Users/sebastianblask/Code/personal/dotfiles/dotfiles_shared/.config/nvim.symlink/vim/commands.vim"
}
}
[TRACE Mon 24 Apr 11:48:37 2023] .../site/pack/packer/start/null-ls.nvim/lua/null-ls/rpc.lua:127: received LSP notification for method initialized
[TRACE Mon 24 Apr 11:48:37 2023] .../site/pack/packer/start/null-ls.nvim/lua/null-ls/rpc.lua:127: received LSP notification for method textDocument/didOpen
[TRACE Mon 24 Apr 11:48:37 2023] ...ack/packer/start/null-ls.nvim/lua/null-ls/generators.lua:21: running generators for method NULL_LS_DIAGNOSTICS_ON_OPEN
[DEBUG Mon 24 Apr 11:48:37 2023] ...ack/packer/start/null-ls.nvim/lua/null-ls/generators.lua:24: no generators available
[TRACE Mon 24 Apr 11:48:45 2023] .../site/pack/packer/start/null-ls.nvim/lua/null-ls/rpc.lua:102: received LSP request for method textDocument/formatting
[TRACE Mon 24 Apr 11:48:45 2023] ...ack/packer/start/null-ls.nvim/lua/null-ls/generators.lua:21: running generators for method NULL_LS_FORMATTING
[DEBUG Mon 24 Apr 11:48:45 2023] ...t/null-ls.nvim/lua/null-ls/helpers/generator_factory.lua:320: spawning command "awk" at /Users/sebastianblask/Code/personal/dotfiles with args { 'NF{print s $0; s=""; next} {s=s ORS}' }
[TRACE Mon 24 Apr 11:48:45 2023] ...t/null-ls.nvim/lua/null-ls/helpers/generator_factory.lua:204: error output: nil
[TRACE Mon 24 Apr 11:48:45 2023] ...t/null-ls.nvim/lua/null-ls/helpers/generator_factory.lua:205: output: if has('python3')
py3file ~/.config/nvim/vim/commands.py
endif

function! RunWithErrorHandling(command) abort
let view = winsaveview()
let errorOutputFile = tempname()

" fake change so undo does not jump to top of file
normal! ix
normal! "_x

let shellredir_save = &shellredir
let &shellredir = '>%s 2>' . errorOutputFile
silent execute a:command
let &shellredir = shellredir_save

if v:shell_error != 0
silent undo
for error in readfile(errorOutputFile)
echoerr error
endfor
endif

call delete(errorOutputFile)
call winrestview(view)
endfunction

command! JsonFormat call RunWithErrorHandling('% ! json-format')
command! JsonFormat2 call RunWithErrorHandling('% ! json-format --indent 2')
command! JsonFormatSorted call RunWithErrorHandling('% ! json-format --sorted')
command! JsonFormatSorted2 call RunWithErrorHandling('% ! json-format --indent 2 --sorted')
command! JsonToDict call RunWithErrorHandling('% ! python -c "import json; import sys; as_dict = json.load(sys.stdin); print(as_dict)"')
command! PrettyDict call RunWithErrorHandling('% ! python -c "import pprint; import sys; as_dict = eval(sys.stdin); pprint.pprint(as_dict)"')
command! YamlFormat call RunWithErrorHandling('% ! python -c "import yaml; import sys; as_dict = yaml.load(sys.stdin); print(yaml.dump(as_dict, default_flow_style=False))"')
command! YamlToDict call RunWithErrorHandling('% ! python -c "import yaml; import sys; as_dict = yaml.load(sys.stdin); print(as_dict)"')

command! -range Sort ,!sort
command! -range SortCommaSeparated ,python3 apply_on_one_line_selection(sort_comma_separated)
command! -range SortWhitespaceSeparated ,python3 apply_on_one_line_selection(sort_whitespace_separated)

command! Base64Decode % python3 base64_decode()
command! Base64Encode % python3 base64_encode()
command! DictToJson % python3 dict_to_json()

command! FoldDisable set nofoldenable | set foldcolumn=0
command! FoldEnable set foldenable | set foldcolumn=3
command! IgnoreWhitespaceOff set diffopt-=iwhite
command! IgnoreWhitespaceOn set diffopt+=iwhite
command! RelativeNumber set relativenumber!

command! ReloadConfig source $MYVIMRC
command! SudoWrite w !sudo tee % > /dev/null
command! Todo Gcd | Ack! "TODO|FIXME"
command! WindowsToLinux :%s/
$//

function! JumpInList(count, isLocationList, isBackward) abort
if ! QuickFixCurrentNumber#Next(a:count, a:isLocationList, a:isBackward) && ! QuickFixCurrentNumber#Border(a:count, a:isLocationList, a:isBackward)
if ingo#err#IsSet()
call ingo#msg#ErrorMsg(ingo#err#Get())
else
call ingo#msg#WarningMsg('Nothing in list')
endif
endif
endfunction

nnoremap ]q :call JumpInList(v:count1, 0, 0)
nnoremap [q :call JumpInList(v:count1, 0, 1)
nnoremap ]l :call JumpInList(v:count1, 1, 0)
nnoremap [l :call JumpInList(v:count1, 1, 1)

[DEBUG Mon 24 Apr 11:48:45 2023] ...ack/packer/start/null-ls.nvim/lua/null-ls/formatting.lua:89: received edits from generators
[TRACE Mon 24 Apr 11:48:45 2023] ...ack/packer/start/null-ls.nvim/lua/null-ls/formatting.lua:90: {
newText = "\n$//",
range = {
["end"] = {
character = 32,
line = 54
},
start = {
character = 28,
line = 54
}
},
rangeLength = 4
}
[TRACE Mon 24 Apr 11:48:45 2023] .../site/pack/packer/start/null-ls.nvim/lua/null-ls/rpc.lua:127: received LSP notification for method textDocument/didChange
[TRACE Mon 24 Apr 11:48:45 2023] ...ack/packer/start/null-ls.nvim/lua/null-ls/generators.lua:21: running generators for method NULL_LS_DIAGNOSTICS
[DEBUG Mon 24 Apr 11:48:45 2023] ...ack/packer/start/null-ls.nvim/lua/null-ls/generators.lua:24: no generators available
[TRACE Mon 24 Apr 11:48:48 2023] .../site/pack/packer/start/null-ls.nvim/lua/null-ls/rpc.lua:102: received LSP request for method shutdown
[TRACE Mon 24 Apr 11:48:48 2023] .../site/pack/packer/start/null-ls.nvim/lua/null-ls/rpc.lua:127: received LSP notification for method exit

Help

No

Implementation Help

No response

Requirements

• I have read and followed the instructions above and understand that my issue will be closed if I did not provide the required information.

[jose-elias-alvarez]

On my end this results in a slightly different error, though it does indeed break the file:

diff --git a/dotfiles_shared/.config/nvim.symlink/vim/commands.vim b/dotfiles_shared/.config/nvim.symlink/vim/commands.vim
index 0312555..3c6ff96 100644
--- a/dotfiles_shared/.config/nvim.symlink/vim/commands.vim
+++ b/dotfiles_shared/.config/nvim.symlink/vim/commands.vim
@@ -52,7 +52,8 @@ command! RelativeNumber      set relativenumber!
 command! ReloadConfig source $MYVIMRC
 command! SudoWrite w !sudo tee % > /dev/null
 command! Todo Gcd | Ack! "TODO|FIXME"
-command! WindowsToLinux :%s/
$//
+command! WindowsToLinux :%s/
+$//
 
 
 function! JumpInList(count, isLocationList, isBackward) abort

Either way, the formatter invokes a simple awk one-liner. If you're able to find an equivalent that handles ^M line endings, feel free to put in a PR.

[sblask]

I saw the awk one-liner, I have no idea how it works though. Seems pretty magic to me. This works though: perl -0pe 's|\n+\z|\n|' test.txt with test.txt being:

foobar

prints:

foobar

The flags are also not super obvious, but -0 makes perl read the whole file, -p prints the result and -e executes the "program".

[sblask]

But that also handles ^M in a weird way...

[sblask]

It's the printing that causes the problem. This produces the same as the perl command:

python -c 'import sys; import re; sys.stdout.write(re.sub("\n+\Z", "\n", sys.stdin.read()))' < ./personal/dotfiles/dotfiles_shared/.config/nvim.symlink/vim/commands.vim

but

python -c 'import sys; import re; open("foo", "w").write(re.sub("\n+\Z", "\n", sys.stdin.read()))' < ./personal/dotfiles/dotfiles_shared/.config/nvim.symlink/vim/commands.vim

produces a file foo that looks right.

Not sure how the null-ls formatter works, if it goes through stdout it might always be a problem.