I prefer the extension to start disabled, and toggle it in and out via Ctrl-Alt-V for when I want to use it. To do this, in settings.json:
    "vim.disableExtension": true
And the keybinding in keybindings.json
    {
      "key": "ctrl+alt+v",
      "command": "toggleVim"
    },