I prefer the extension to start disabled, and toggle it in and out via Ctrl-Alt-V for when I want to use it. To do this, in settings.json:

    "vim.disableExtension": true

And the keybinding in keybindings.json

    {
      "key": "ctrl+alt+v",
      "command": "toggleVim"
    },