= Indexing, caching & performance

How Projectile discovers, lists, orders and caches the files in a project, and how to tune all of that for speed (including over TRAMP).

== Project indexing method

Projectile has three modes of operation - one is portable and is
implemented in Emacs Lisp (therefore it's _native_ to Emacs and is
known as the `native indexing method`) and the other two (`hybrid` and
`alien`) rely on external commands like `find`, `git`, etc to
obtain the list of files in a project.

The `alien` indexing method trades features for raw speed: Projectile does no
processing or sorting of the files returned by the external commands, so you get
the maximum performance possible.  This behaviour makes a lot of sense for most people, as
they'd typically be putting ignores in their VCS config (e.g. `.gitignore`) and
won't care about any additional ignores/unignores/sorting that Projectile might
also provide.

NOTE: By default the `alien` method is used on all operating systems except Windows.

To force the
use of native indexing in all operating systems:

[source,elisp]
----
(setq projectile-indexing-method 'native)
----

To force the use of hybrid indexing in all operating systems:

[source,elisp]
----
(setq projectile-indexing-method 'hybrid)
----

To force the use of alien indexing in all operating systems:

[source,elisp]
----
(setq projectile-indexing-method 'alien)
----

Alien indexing is significantly faster, especially on big projects, but it
relies on external Unix utilities. That's why it isn't the default on Windows;
if you run into trouble there, fall back to the `native` indexing method.

=== Alien indexing

The alien indexing works in a pretty simple manner - it simply shells
out to a command that returns the list of files within a project.
For version-controlled projects by default Projectile will use the
VCS itself to obtain the list of files. As an example, here is the
command that Projectile uses for Git projects:

----
git ls-files -zco --exclude-standard
----

For every supported VCS there's a matching Projectile defcustom holding the command
to invoke for it:

|===
| Variable | VCS

| `projectile-git-command`
| Git

| `projectile-hg-command`
| Mercurial

| `projectile-svn-command`
| Subversion

| `projectile-bzr-command`
| Bazaar

| `projectile-darcs-command`
| Darcs

| `projectile-fossil-command`
| Fossil

| `projectile-pijul-command`
| Pijul

| `projectile-sapling-command`
| Sapling

| `projectile-jj-command`
| Jujutsu
|===

There are also two Git-specific commands for listing submodules and ignored files:

[source,elisp]
----
;; Command to list git submodules (set to nil to disable)
(setq projectile-git-submodule-command
  "git submodule --quiet foreach 'echo $displaypath' | tr '\\n' '\\0'")

;; Command to get git-ignored files
(setq projectile-git-ignored-command "git ls-files -zcoi --exclude-standard")
----

WARNING: If you ever decide to tweak those keep in mind that the command should always be returning
 the list of files **relative** to the project root and the resulting file list should be 0-delimited
 (as opposed to newline delimited).

For non-VCS projects Projectile will invoke whatever is in `projectile-generic-command`. The default chooses `fd` when it's installed and falls back to `find`:

[source,elisp]
----
;; Effective default value of projectile-generic-command, picked at load time:
;;   when fd is on PATH:
"fd . -0 --type f --color=never --strip-cwd-prefix"
;;   otherwise:
"find . -type f | cut -c3- | tr '\\n' '\\0'"
----

TIP: It's a great idea to install https://github.com/sharkdp/fd[fd] which is much faster than `find`.
 If `fd` is found, projectile will use it as a replacement for `find` for non-VCS projects.

WARNING: The `find` fallback does *not* exclude common build/cache directories
 (`.git`, `node_modules`, `target`, `build`, …) itself, so it walks the whole
 tree even though Projectile then drops the ignored entries from its output (see
 xref:#alien-ignores[below]).  Install `fd` or override
 `projectile-generic-command` with a tighter recipe if that walk is slow.

By default, `fd` is also used inside Git repositories (instead of `git ls-files`),
because `git ls-files` has the limitation that it lists deleted files until the
deletions are staged. With `fd`, deleted files disappear from the listing
immediately; with `git ls-files`, Projectile post-filters the listing against
`git ls-files -zd` to hide deletions until they're staged. You can control this
with `projectile-git-use-fd`:

[source,elisp]
----
;; Disable fd in git repos (use git ls-files instead)
(setq projectile-git-use-fd nil)
----

You can also customize the path to the `fd` executable and the arguments passed to it:

[source,elisp]
----
;; Use a custom fd path
(setq projectile-fd-executable "/usr/local/bin/fd")

;; Customize the fd arguments used in git repos
(setq projectile-git-fd-args "-H -0 -E .git -tf --strip-cwd-prefix -c never")
----

=== Hybrid indexing

The `hybrid` method runs the same external command as `alien` and then
post-processes the result with Projectile's filtering rules.  In other
words, it's `alien` plus a second pass that applies:

* `.projectile` (dirconfig) ignore/keep/ensure entries
* `projectile-globally-ignored-files`,
  `projectile-globally-ignored-directories` and
  `projectile-globally-ignored-file-suffixes`
* `projectile-globally-unignored-files` and
  `projectile-globally-unignored-directories`
* The configured sort order from `projectile-sort-order`

This is the right choice when your VCS lists more files than you want
Projectile to surface (e.g. you want to drop `*.elc` even though they're
checked in), or when you rely on `.projectile` to keep/ignore subtrees
inside an otherwise large repository.  It's slower than `alien` because
of the extra pass, but it remains substantially faster than `native` for
big projects since the file list itself is still produced by the
external tool.

=== Indexing method comparison

Not all Projectile features apply to every indexing method.  The table
below summarises which configuration knobs take effect under which mode:

[cols="3,1,1,1", options="header"]
|===
| Feature | `native` | `hybrid` | `alien`

| Project file listing speed
| Slow (Elisp)
| Fast
| Fastest

| Honors `.projectile` (dirconfig) `-` ignore rules
| Yes
| Yes
| Yes

| Honors `.projectile` (dirconfig) `+` keep rules
| Yes
| Yes
| Yes

| Honors `.projectile` (dirconfig) `!` unignore rules
| Yes
| Yes
| Yes

| Honors `projectile-globally-ignored-files` / `-directories` / `-file-suffixes`
| Yes
| Yes
| Yes

| Honors `projectile-global-ignore-file-patterns` (Emacs regexps)
| Yes
| No
| No

| Honors `projectile-globally-unignored-*`
| Yes
| Yes
| Yes

| Re-adds files the VCS itself ignores (`projectile-globally-unignored-*`, `!` entries)
| n/a
| Yes
| No

| Sorts via `projectile-sort-order`
| Yes
| Yes
| No (returned in the external tool's order)

| File caching enabled by default
| Yes
| No
| No

| Works on Windows out of the box
| Yes
| Requires Unix utilities
| Requires Unix utilities
|===

[#alien-ignores]
=== How `alien` applies the ignore rules

`alien` honors the ignore rules without giving up its speed: instead of
filtering the tool's output in Emacs, it hands the rules to the tool as
exclusion arguments. Which tools can do that varies, so the ones that can't get
their output filtered in Emacs Lisp after all. Every path applies the same
gitignore rules to the same pattern list, so the resulting file set is the same
- the difference is only where the work happens:

[cols="1,2,2",options="header"]
|===
| Indexing command | How the rules are applied | Cost

| `git ls-files`
| `:(exclude,glob)` pathspecs
| In the tool

| `fd` (including `projectile-git-use-fd`)
| repeated `--exclude` globs
| In the tool

| `hg locate`, `sl locate`, `jj file list`
| output filtered in Emacs Lisp
| One pass over the file list

| `svn`, `fossil`, `bzr`, `darcs`, `pijul`
| output filtered in Emacs Lisp
| One pass over the file list

| the plain `find` fallback
| output filtered in Emacs Lisp
| One pass over the file list
|===

Set `projectile-alien-honors-ignores` to nil to get the raw listing back and
defer entirely to the tool's own ignore rules (`.gitignore` and friends).

NOTE: An exclusion argument can't be taken back, so a project whose dirconfig
has `!` ensure entries is filtered in Emacs Lisp under `alien` too, rather than
pushed down - the rules are the same, only the fast path is given up. Dirconfig
`+` keep entries are passed to the tool as path arguments where it takes them,
and applied to its output where it doesn't.

NOTE: No indexing method can add back files the external tool never listed in
the first place (because `.gitignore` excludes them); only `hybrid` does that,
via its VCS-aware unignore pass.

NOTE: Ignore matching is case-sensitive under every indexing method. With
`.elc` in `projectile-globally-ignored-file-suffixes`, a file named `BUILD.ELC`
is *not* ignored - spell out both cases if you want both. Before Projectile 3.3
`native` and `hybrid` folded case here (a side effect of the Emacs functions
they matched with), which `alien` had no way to reproduce in the external tools.

NOTE: Since Projectile 3.3 every ignore option speaks gitignore patterns and
every indexing method matches them the same way, so a bare entry in
`projectile-globally-ignored-directories` matches at any depth everywhere. It
used to be top-level-only under `hybrid`, and a `*` prefix was a marker rather
than a wildcard; both quirks are gone.

TIP: For the details of each ignore mechanism (`.projectile`, the indexing
tools, and the global ignore/unignore variables), see
xref:ignoring.adoc[Ignoring files].

== Sorting

You can choose how Projectile sorts files by customizing `projectile-sort-order`.

NOTE: If Alien indexing is set, files are not sorted by Projectile at all.

The default is to not sort files:

[source,elisp]
----
(setq projectile-sort-order 'default)
----

To sort files by recently opened:

[source,elisp]
----
(setq projectile-sort-order 'recentf)
----

To sort files by recently active buffers and then recently opened files:

[source,elisp]
----
(setq projectile-sort-order 'recently-active)
----

// These URLs below are in HTML so that the parentheses in the URL fragments are properly recognised.

To sort files by https://en.wikipedia.org/wiki/MAC_times#Modification_time_(mtime)[modification time] (mtime):

[source,elisp]
----
(setq projectile-sort-order 'modification-time)
----

To sort files by https://en.wikipedia.org/wiki/MAC_times#Access_time_(atime)[access time] (atime):

[source,elisp]
----
(setq projectile-sort-order 'access-time)
----

Finally, `projectile-sort-order` can be set to a function that receives
the list of project files (as relative paths) and returns them in the
desired order (added in Projectile 3.1). For example, to list files
with shorter paths first:

[source,elisp]
----
(setq projectile-sort-order
      (lambda (files) (seq-sort-by #'length #'< files)))
----

== File ranking (frecency)

Since Projectile 3.1, `projectile-find-file` (and its dwim variant)
rank the files you actually work with first: Projectile records file
visits per project and orders completion candidates by "frecency", a
combination of visit frequency and recency (frequent visits decay with
a half-life of two weeks). Files you've never visited keep their
original order after the ranked ones.

Unlike `projectile-sort-order`, the ranking is applied through
completion metadata (`display-sort-function`), so it works with any
completion UI that honors it (the default completion UI, Vertico,
Icomplete, ...) and under *every* indexing method, including `alien`.

The feature is enabled by default. To turn it off:

[source,elisp]
----
(setq projectile-enable-frecency nil)
----

The visit history is persisted across sessions in
`projectile-frecency-file` and capped per project by
`projectile-frecency-max-files` (200 by default). Concurrent Emacs
sessions merge their histories on save rather than overwriting each
other. Remote (TRAMP) projects are not tracked.

NOTE: For tracked files the frecency order takes precedence over
`projectile-sort-order`, since completion UIs apply the metadata sort
to whatever candidate order they receive.

== Caching

=== Project files

Since indexing a big project is not exactly quick (especially in Emacs
Lisp), Projectile supports caching of the project's files. The caching
is enabled by default whenever native indexing is enabled.

To enable caching unconditionally use this snippet of code:

[source,elisp]
----
(setq projectile-enable-caching t)
----

At this point you can try out a Projectile command such as kbd:[s-p f] (kbd:[M-x] `projectile-find-file` kbd:[RET]).

Running kbd:[C-u s-p f] will invalidate the cache prior to
prompting you for a file to jump to.

Pressing kbd:[s-p z] will add the currently visited file to the
cache for current project. Generally files created outside Emacs will
be added to the cache automatically the first time you open them.

A file the project ignores isn't added, whether it's your `.projectile` and
the global ignore settings that exclude it or - under `alien` and `hybrid`
indexing, where the file list comes from the VCS - your `.gitignore`.
Otherwise opening, say, a compiled artifact would put it in a cache that
indexing would never have produced.

Normally the cache lasts for the duration of your Emacs session.
If you want to cache to persist between Emacs sessions you
should set this option to `'persistent`.

[source,elisp]
----
(setq projectile-enable-caching 'persistent)
----

Now the project cache is persistent and will be preserved during Emacs restarts.
Each project gets its own cache file, that will be placed in the root folder of the
project. The name of the cache file is `.projectile-cache.eld` by default, but you can tweak it
if you want to:

[source,elisp]
----
(setq projectile-cache-file "foo.eld")
----

The cache file will be loaded automatically in memory the first time you trigger
a "find file" operation for the project it belongs to.

You can purge an individual file from the cache with `M-x projectile-purge-file-from-cache` or an
entire directory with `M-x projectile-purge-dir-from-cache`.

If the caches of many projects have gone stale at once (something you'd
typically notice with commands operating on several projects, like
`projectile-find-file-in-known-projects`), you can invalidate them all in one
go with `M-x projectile-invalidate-cache-all`. Remote (TRAMP) projects are
skipped to avoid a potentially slow connection round-trip per project;
invalidate those individually with `projectile-invalidate-cache`.

NOTE: Prior to Projectile 2.9 the cache for all projects was serialized to the same file.
In Projectile 2.9 this was changed and now each project has its own cache file relative to
the project's root directory.

When `projectile-mode` is enabled Projectile will auto-update the project cache when files
within are added or deleted from within Emacs (via file hooks). This behavior
can be disabled like this:

[source,elisp]
----
(setq projectile-auto-update-cache nil)
----

You can also set the project files cache to expire after a given number of
seconds:

[source,elisp]
----
;; Expire the project files cache after 5 minutes
(setq projectile-files-cache-expire (* 5 60))
----

By default `projectile-files-cache-expire` is `nil`, meaning the cache never
expires automatically.

One last thing - the project cache will be auto-invalidated if you're using
`.projectile` and its last modification time is more recent than the time at
which the cache file was last updated.

=== Automatic cache updates

NOTE: This feature is experimental and was added in Projectile 3.2.

The file hooks described above only see what happens inside Emacs. Files
created, deleted or renamed by external tools (a `git pull`, a code
generator, another editor) still leave the cache stale until you run
`projectile-invalidate-cache` by hand. If you'd rather have Projectile
notice such changes itself, it can watch the directories of your cached
projects using filesystem notifications:

[source,elisp]
----
(setq projectile-auto-update-cache-with-watches t)
----

Whenever a project's file list is cached, Projectile then registers a
`file-notify` watch for each of the project's directories and applies the
resulting events to the cached list: new files are run through the usual
ignore filtering before being added, deleted files (and directories) are
removed, renames are handled as both. Events are debounced, so bursts
(say, a branch switch touching hundreds of files) are folded into a single
update. Anything that can't be applied incrementally with confidence falls
back to invalidating the project's cache, which simply gets rebuilt on the
next file listing.

Be aware of the trade-offs before enabling it:

* Emacs file notifications are not recursive, so a watched project costs
  one watch, and typically one file descriptor, per directory. Projects
  spanning more directories than `projectile-watch-directory-limit`
  (512 by default) are not watched at all; set `projectile-verbose` to be
  told when that happens.
* Remote (TRAMP) projects are never watched.
* Watched directories are derived from the cached file list, so a
  directory that contains no cached files (e.g. an empty directory) isn't
  watched until the next full re-index.
* VCS-level ignores (e.g. `.gitignore`) are not consulted when adding
  newly created files, only Projectile's own ignore settings and your
  `.projectile` file. Under `alien` indexing a watched project may thus
  temporarily pick up files your VCS would have excluded. (Opening such a
  file doesn't cache it - that path does consult the VCS - but a watch
  batch can cover hundreds of files at once, which is too many processes
  to ask git about one at a time.)

Watches are dropped automatically when a project's cache is invalidated
(and re-armed when it's next filled), when the option is disabled via
Customize or `setopt`, when `projectile-mode` is turned off (turning the
mode back on re-arms them for all cached projects), when a project is
removed from the known projects and on Emacs shutdown.

=== Background indexing

The first "find file" in a large or remote project has to index it. With the
`alien` and `hybrid` indexing methods Projectile does this without freezing
Emacs: the indexing command runs asynchronously and is awaited in a way that
keeps redisplay going and leaves kbd:[C-g] live, so you can abort a slow
index (for instance one stuck on an unresponsive remote host) instead of
staring at a frozen editor. The resulting file list is exactly what the
synchronous indexer would produce.

This is on by default and controlled by `projectile-async-indexing`:

[source,elisp]
----
;; index synchronously, as in older Projectile versions
(setq projectile-async-indexing nil)
----

NOTE: It has no effect under `native` indexing (the Emacs Lisp directory walk
can't run off the main thread), in batch mode, or while a keyboard macro is
executing - those index synchronously.

The result of the indexing command reaches Projectile from the process's
sentinel, and Emacs doesn't guarantee it runs that promptly while a command is
waiting on the process. If it hasn't run `projectile-async-index-sentinel-timeout`
seconds after the command finished, Projectile parses the output itself instead
of waiting any longer. You shouldn't ever need to touch that option; it exists
because waiting forever was once possible.

You can also warm the cache ahead of time, without blocking, using kbd:[M-x]
`projectile-index-project-async`:

[source,elisp]
----
;; index the current project in the background
(projectile-index-project-async)
----

It runs the project's indexing command via an asynchronous process and
populates the same cache that a regular `projectile-find-file` reads, so a
later "find file" finds the cache already warm instead of indexing on the
spot. You could, for instance, warm a project right after switching to it:

[source,elisp]
----
(add-hook 'projectile-after-switch-project-hook
          #'projectile-index-project-async)
----

NOTE: Background indexing only works with the `alien` and `hybrid` indexing
methods - the `native` (pure Emacs Lisp) walk can't run off the main thread.
It also requires caching to be enabled, since the warmed result is stored in
the project files cache.

The asynchronous building blocks it's made of -
`projectile-files-via-ext-command-async` and
`projectile-dir-files-alien-async` - are public, so a streaming file finder
(for example one built on top of `consult`) can drive Projectile's indexing
command asynchronously and feed results into its own UI. See the next
section.

=== Integrating a streaming file finder

If you'd rather have your own asynchronous/streaming UI (à la `fzf`,
`consult` or `affe`) list a project's files instead of `projectile-find-file`,
`projectile-project-files-producer` hands you everything you need to run
Projectile's own indexing command yourself:

[source,elisp]
----
(projectile-project-files-producer)
;; => (:directory "/path/to/project/"
;;     :vcs git
;;     :command "git ls-files -zco --exclude-standard"
;;     :separator "\0")
----

Run `:command` in `:directory` and split its output on `:separator`. A
minimal (non-streaming) example built on the public asynchronous runner:

[source,elisp]
----
(defun my/project-find-file ()
  "Pick a file from the current project, indexing without blocking."
  (interactive)
  (let* ((producer (projectile-project-files-producer))
         (default-directory (plist-get producer :directory)))
    (projectile-files-via-ext-command-async
     default-directory (plist-get producer :command)
     (lambda (files _err)
       (find-file (expand-file-name (completing-read "File: " files)
                                    default-directory))))))
----

A real streaming finder would instead consume the process output
incrementally (filtering as you type) rather than collecting it. For git
projects, note that `:command` lists the main worktree only; if you want
byte-for-byte the same set as `projectile-find-file` (submodule files folded
in, deleted-but-unstaged files removed) drive `projectile-dir-files-alien-async`
instead.

==== Bundled Consult integration

Projectile ships such a finder for https://github.com/minad/consult[Consult]
in `projectile-consult.el`. It's an optional module with a soft dependency on
`consult` (Projectile core never loads it), so you opt in by requiring it:

[source,elisp]
----
(require 'projectile-consult)
(define-key projectile-command-map (kbd "f") #'projectile-consult-find-file)
----

`projectile-consult-find-file` lists the project with Projectile's own
indexing command and streams the candidates into Consult as they arrive, so
there's no wait for the whole project to be indexed before you can start
typing. Because it goes through `projectile-project-files-producer`, it stays
VCS-aware and honours the project's indexing configuration. The module needs
Consult 2.0+, which in turn requires Emacs 29.1+.

=== File exists cache

Projectile does many file existence checks since that is how it identifies a
project root. Normally this is fine, however in some situations the file system
speed is much slower than usual and can make Emacs "freeze" for extended
periods of time when opening files and browsing directories.

The most common example would be interfacing with remote systems using
TRAMP/ssh. By default all remote file existence checks are cached.

To disable the remote file-exists cache, use this snippet of code:

[source,elisp]
----
(setq projectile-file-exists-remote-cache-expire nil)
----

To change the remote file exists cache expire to 10 minutes use this snippet
of code:

[source,elisp]
----
(setq projectile-file-exists-remote-cache-expire (* 10 60))
----

You can also enable the cache for local file systems, that is normally not
needed but possible:

[source,elisp]
----
(setq projectile-file-exists-local-cache-expire (* 5 60))
----

[#_tramp_performance]
== TRAMP performance

Projectile is TRAMP-aware and the hot paths cache aggressively, so
once the caches are warm, navigating a remote project should feel
similar to a local one.  The first command on a fresh remote project
pays a one-time setup cost (one VCS-detection listing, one project-root
walk, one indexing shell-out); after that everything is served from
memory until you invalidate it.

The following knobs are most relevant for remote use:

* `projectile-indexing-method` - leave at the default (`alien`) or set
  to `hybrid`.  *Don't* use `native` over TRAMP: native indexing walks
  the tree with one `directory-files` call per subdirectory, which is
  one remote round-trip per subdirectory.  `hybrid` and `alien` use a
  single `git ls-files` / `fd` shell-out.
* `projectile-enable-caching` - enables the per-project file list
  cache.  Set to `'persistent` if you want the cache to survive Emacs
  restarts, so reopening a remote project doesn't re-shell-out.
* `projectile-file-exists-remote-cache-expire` (default 5 minutes) -
  TTL for cached remote file-existence results.  Lower it if you
  frequently create new project markers (`.projectile`, `.git`, ...)
  on the remote and want them detected sooner; raise it if your
  remotes are stable and you'd rather avoid the round-trip.

After creating, removing, or moving a project marker on a remote, run
`projectile-discard-root-cache` (or `projectile-invalidate-cache`) -
both also clear the file-existence cache, so the new state takes
effect on the next call instead of waiting for the TTL.

A few things that don't require configuration but are worth knowing:

* `fd` is auto-detected per remote host (cached for the session).  The
  `projectile-fd-executable` defcustom is the *local* detection result
  and isn't used remotely - whatever Projectile finds at
  `executable-find` time on the remote (`fd` or `fdfind`) is what runs
  there.  No `fd` on the remote means the indexing path falls back to
  `git ls-files`.
* If indexing fails (e.g. `git` or `fd` isn't on the remote `PATH`),
  Projectile signals a `user-error` pointing at
  `+*projectile-files-errors*+` instead of returning an empty file
  list.  Older versions silently returned `nil`, which manifested as
  unexplained empty completion lists or `stringp, nil` crashes
  downstream.
