bash or zsh HISTSIZE vs HISTFILESIZE

Understanding the nuances of your shell environment is crucial for efficient command-line usage. Two important variables in Bash and Zsh (often referred to as shell history) are HISTSIZE and HISTFILESIZE. While they both govern the shell history, they serve distinct purposes: HISTSIZE controls the number of commands remembered during a shell session, while HISTFILESIZE determines how many commands are saved to the history file on disk after the session ends. Confusing these two can lead to unexpected history behavior, such as losing commands you thought were safely stored. This article will delve into the differences between HISTSIZE and HISTFILESIZE, explaining how they work, how to configure them, and why mastering them is essential for any serious command-line user.

Understanding HISTSIZE: The Session History

HISTSIZE defines the maximum number of commands stored in the shell’s memory during a session. Think of it as the working memory for your command history. As you type commands, they are added to this in-memory list. Once the number of commands exceeds HISTSIZE, the oldest commands are discarded to make room for new ones. This variable is incredibly useful for quickly recalling recently used commands within the current terminal session using the up and down arrow keys or searching with Ctrl+R. Setting an appropriate HISTSIZE ensures that you have enough recent commands readily available without consuming excessive memory.

For example, if you set HISTSIZE=500, your shell will remember the last 500 commands you entered. If you then type 501 commands, the first command you entered will be removed from the in-memory history. Changes to HISTSIZE only affect the current and subsequent shell sessions. Existing sessions will continue to use the HISTSIZE value that was in effect when they were started. To check your current HISTSIZE value, simply type echo $HISTSIZE in your terminal.

It’s important to note that HISTSIZE is independent of HISTFILESIZE. Even if you have a large HISTFILESIZE, if your HISTSIZE is small, you won’t be able to access older commands within the current session. Adjusting HISTSIZE involves considering your memory constraints and the typical length of your shell sessions. According to a study by IBM, a well-configured command-line interface can improve developer productivity by up to 30% IBM Research.

Understanding HISTFILESIZE: The Persistent History

HISTFILESIZE specifies the maximum number of commands saved to the history file (typically .bash_history or .zsh_history) when you close a shell session. This file acts as a persistent record of your command history, allowing you to access commands from previous sessions. When a session ends, the shell writes the last HISTFILESIZE commands from the in-memory history (up to HISTSIZE) to the history file. This ensures that your most recent and relevant commands are preserved for future use. The HISTFILESIZE variable is crucial for maintaining a comprehensive and useful history across multiple sessions.

Imagine you have HISTFILESIZE=1000. When you close your terminal, the shell will save the last 1000 commands from your current session (assuming you entered at least 1000 commands and your HISTSIZE was also at least 1000) to your history file. If the history file already contains commands from previous sessions, the shell will append the new commands, potentially truncating the file to ensure it doesn’t exceed HISTFILESIZE in total entries. To view your current HISTFILESIZE, type echo $HISTFILESIZE in your terminal.

A larger HISTFILESIZE allows you to retain a more extensive command history, which can be invaluable for debugging, auditing, and simply recalling commands you used weeks or even months ago. However, a very large HISTFILESIZE can also lead to a larger history file, which might take longer to load and search. Therefore, it’s essential to strike a balance between retaining enough history and maintaining performance. Proper management of HISTFILESIZE directly influences your ability to leverage past work and maintain a clear record of your command-line activities. GNU Bash Manual offers extensive documentation on these variables.

Configuring HISTSIZE and HISTFILESIZE

Configuring HISTSIZE and HISTFILESIZE is straightforward. You can set these variables in your shell’s configuration file, typically .bashrc or .zshrc in your home directory. This ensures that the settings are applied automatically each time you start a new shell session. To modify these variables, open your configuration file in a text editor (e.g., nano ~/.bashrc or nano ~/.zshrc) and add or modify the lines that define HISTSIZE and HISTFILESIZE.

For example, to set HISTSIZE to 1000 and HISTFILESIZE to 2000, you would add the following lines to your .bashrc or .zshrc file:

HISTSIZE=1000 HISTFILESIZE=2000

After saving the changes, you need to either close and reopen your terminal or source the configuration file (e.g., source ~/.bashrc or source ~/.zshrc) to apply the new settings to your current session. Experiment with different values to find the settings that best suit your workflow and system resources. Remember that these settings are user-specific, so each user on a system can have their own unique HISTSIZE and HISTFILESIZE configurations. Check out our guide for more shell configuration tips.

Best Practices and Troubleshooting

When working with HISTSIZE and HISTFILESIZE, consider these best practices to optimize your command-line experience:

Set HISTSIZE appropriately: Choose a value that balances memory usage and the number of commands you typically need to recall within a session.
Set HISTFILESIZE higher than HISTSIZE: This ensures that all commands from your session can be saved to the history file.
Periodically clean your history file: Remove sensitive information or irrelevant commands to maintain a clean and manageable history.

Here are some troubleshooting tips:

History not saving: Ensure that HISTFILESIZE is set and that you have write permissions to your history file.
History not accessible in current session: Make sure you’ve sourced your configuration file after making changes.
Large history file slowing down shell: Consider reducing HISTFILESIZE or cleaning up the history file.

To prevent duplicate entries in your history, you can add these lines to your .bashrc or .zshrc:

HISTCONTROL=ignoredups:erasedups

This will prevent duplicate commands from being saved and remove any duplicates from the history file. Many seasoned developers prefer setting HISTSIZE to a moderate value (e.g., 500-1000) and HISTFILESIZE significantly higher (e.g., 2000-5000) to preserve a comprehensive history while maintaining session responsiveness.

Optimizing History Search

Beyond the basic configuration, you can enhance your history usage with powerful search capabilities. Bash and Zsh offer reverse incremental search (usually triggered by Ctrl+R), which allows you to search for commands containing specific strings. The effectiveness of this search is directly impacted by the size and quality of your history. A well-maintained and appropriately sized history significantly improves the speed and accuracy of your command retrieval.

Consider these steps to optimize your history search:

Use descriptive commands: Opt for clear and descriptive commands to make them easier to find later.
Leverage aliases: Create aliases for frequently used commands to simplify and standardize your command history.
Regularly review and clean your history: Remove obsolete or irrelevant commands to improve search efficiency.

FAQ: Common Questions About HISTSIZE and HISTFILESIZE

What happens if HISTSIZE is larger than HISTFILESIZE?: If HISTSIZE is larger than HISTFILESIZE, you'll have more commands in memory during your session than will be saved to the history file. When the session ends, only the last HISTFILESIZE commands from the in-memory history will be saved, potentially losing older commands from your session.
Can I disable history saving entirely?: Yes, you can disable history saving by setting HISTFILESIZE=0. This will prevent the shell from writing any commands to the history file.
Where is the history file located?: The history file is typically located in your home directory and is named .bash\_history for Bash and .zsh\_history for Zsh. The exact location can be customized by setting the HISTFILE variable.

Understanding HISTSIZE and HISTFILESIZE is a cornerstone of efficient command-line usage. By mastering these variables, you can optimize your workflow, enhance your productivity, and maintain a valuable record of your command-line activities. Remember that HISTSIZE governs the commands remembered during a shell session, while HISTFILESIZE determines how many commands are saved to disk after the session. Properly configuring these variables allows you to find frequently used commands quickly, improving your daily workflow.

Take some time to experiment with different settings and discover what works best for you. Consider sharing your configurations and tips with others in the community to further enhance collective knowledge and best practices. Now that you understand the power and control these variables give you, go forth and conquer the command line. Check out our other articles on shell scripting and command-line productivity to further enhance your skills.

Question & Answer :
What is the difference in HISTSIZE vs. HISTFILESIZE?

They are used to extend bash history beyond the default 500 lines.

There seems to be a lack of clarity here and in other forums about why they are both needed. (Example 1, Example 2, Example 3).

Is this the same for zsh?

Short answer:

HISTSIZE is the number of lines or commands that are stored in memory in a history list while your bash session is ongoing.

HISTFILESIZE is the number of lines or commands that (a) are allowed in the history file at startup time of a session, and (b) are stored in the history file at the end of your bash session for use in future sessions.

Notice the distinction between file: on disk - and list: in memory.

(NOTE: as noted by this, in zsh HISTFILESIZE should be SAVEHIST)

Long answer:

All the info above + some examples:

Example 1: HISTFILESIZE=10 and HISTSIZE=10

You start your session.
Your HISTFILE (file that stores your bash command history), is truncated to contain HISTFILESIZE=10 lines.
You write 50 lines.
At the end of your 50 commands, only commands 41 to 50 are in your history list, whose size is determined by HISTSIZE=10.
You end your session.
Assuming histappend is not enabled, commands 41 to 50 are saved to your HISTFILE which now has the 10 commands it held at the beginning plus the 10 newly written commands.
Your HISTFILE is truncated to contain HISTFILESIZE=10 lines.
You now have 10 commands in your history - the last 10 that you just typed in the session you just finished.
When you start a new session, you start over at step 1 with a HISTFILE of HISTFILESIZE=10.

Example 2: HISTFILESIZE=10 and HISTSIZE=5

You start your session.
Your HISTFILE (file that stores your bash command history), is truncated to contain at most HISTFILESIZE=10 lines.
You write 50 lines.
At the end of your 50 commands, only commands 46 to 50 are in your history list, whose size is determined by HISTSIZE=5.
You end your session.
Assuming histappend is not enabled, commands 46 to 50 are saved to your HISTFILE which now has the 10 commands it held at the beginning plus the 5 newly written commands.
Your HISTFILE is truncated to contain HISTFILESIZE=10 lines.
You now have 10 commands in your history - 5 from a previous session and the last 5 that you just typed in the session you just finished.
When you start a new session, you start over at step 1 with a HISTFILE of HISTFILESIZE=10.

Example 3: HISTFILESIZE=5 and HISTSIZE=10

You start your session.
Your HISTFILE (file that stores your bash command history), is truncated to contain at most HISTFILESIZE=5 lines.
You write 50 lines.
At the end of your 50 commands, only commands 41 to 50 are in your history list, whose size is determined by HISTSIZE=10.
You end your session.
Assuming histappend is not enabled, commands 41 to 50 are saved to your HISTFILE which now has the 5 commands it held at the beginning plus the 10 newly written commands.
Your HISTFILE is truncated to contain HISTFILESIZE=5 lines.
You now have 5 commands in your history - the last 5 that you just typed in the session you just finished.
When you start a new session, you start over at step 1 with a HISTFILE of HISTFILESIZE=5.

Info from elixir_sinari:

The history “file” is not updated as you type the commands. The commands get stored in a “list” separately (accessed by the history command). The number of these stored commands is controlled by HISTSIZE value. When the shell (interactive) exits, the last $HISTSIZE lines are copied/appended to $HISTFILE from that “list”. If HISTFILESIZE is set, then after this operation, it is ensured that only $HISTFILESIZE lines (latest) exist in $HISTFILE . And when the shell starts, the “list” is initialized from $HISTFILE up to a maximum of $HISTSIZE commands.

And from the man bash page:

The value of the HISTSIZE variable is used as the number of commands to save in a history list. The text of the last HISTSIZE commands (default 500) is saved. (…)

On startup, the history is initialized from the file named by the variable HISTFILE (default ~/.bash_history). The file named by the value of HISTFILE is truncated, if necessary, to contain no more than the number of lines specified by the value of HISTFILESIZE. (…) When an interactive shell exits, the last $HISTSIZE lines are copied from the history list to $HISTFILE. If the histappend shell option is enabled (see the description of shopt under SHELL BUILTIN COMMANDS below), the lines are appended to the history file, otherwise the history file is overwritten. If HISTFILE is unset, or if the history file is unwritable, the history is not saved. (…) After saving the history, the history file is truncated to contain no more than HISTFILESIZE lines. If HISTFILESIZE is not set, no truncation is performed.