This post is originally published as a Github Gist last year at 2023-07-22. I re-posted it with modifications here simply because I feel like documenting it here.

As a part of our company workflow, we requires a tmux setup that would automatically open specific file locations with a certain layout when vscode terminal is open.

Let’s implement them.

Issues

Our company requires us work with several project directories. For simplification, we assume that each project directories have unique directory names. Below are example arbitrary directory path under which we will have to work on:

• ~/App/core/ for backend repository
• ~/App/fe/ for frontend repository
• ~/App/deployment/ for deployment repository

We are working with our codes using vscode, and normally we access its built-in terminal to run various tools such as artisan, composer, npm, or even yarn under our project directories. Sometimes we want the session to persist, even when vscode fails. Even better, we should also be able to connect to the vscode terminal session from an external terminal emulator. This is usually done because we can get wider and cleaner view using a dedicated terminal emulator, rather than using built-in vscode terminal.

How do we satisfy those specific use cases?

Design Overview

Since we are working with multiple directories, and we want to have named sessions we can attach to from either vscode built-in terminal or a proper terminal emulator, we have to use some form of session persistence. There are plenty of tools that allow persistent terminal sessions, such as screen and tmux. In this article, we are going to use tmux, a modern terminal multiplexer that allows persistence of terminal sessions.

Since we also have multiple directories, ideally each of those directories have their own dedicated named sessions. For simplicity, we will just have to use the directory’s basename for their session names. Therefore we will have to have the following sessions: core, fe, and deployment.

We will have one default tmux configuration file that would actually show us useful information. Then for each of those sessions, we would have to craft their own config files depending on their use cases and specific requirements. Since the configuration has to work wherever we decided to attach the sessions from, the sessions would have to be made whenever we run any terminal from the first time, in a detached mode. The config file would also take care of the working directory.

At last, we want to have the persistent session as default behavior in our vscode terminal. We don’t want to just fired up vscode, open its terminal, and forgetting to set up tmux, and then run an important program that requires persistence. Therefore, if we fire up the vscode terminal, we must also automatically be attached to a tmux session.

Pseudocode

To satisfy the requirements in both of our use case and design overview, we can actually simplify our tasks in just six steps.

1. On startup, check if we’re already under tmux.
2. If we are not under tmux, check if there are existing session with a specific name.
3. If the specified named tmux session is not present, then we make a new detached named session.
4. From within the created detached name session, we send keys to it to load its custom configuration files. The config files would also take care of their working directory.
5. Finally, we check if we are running under vscode.
6. If we are, then we will connect to a tmux session with the same name as our working vscode directory.

A sane default tmux configuration

tmux actually works out of the box, and can be used right away. However, the sane configuration can be very esoteric for the uninitiated. I have to shift through a lot of documentations, just to get a sane default configuration. I started from Archlinux Wiki, and then from Practical Tmux, and I managed to compile my current default configuration.

The gist of my current configuration is as follows:

Keys separated by a plus sign (‘+’) must be pressed together, while the one separated with a space means perform the key on the left first, then release and press the next key in the sequence.

I also set a minimalistic but useful tmux status bar, that will show you the current session name, current session group, current tmux windows (that changes color depending on whether they are the active window or not), your username and machine name, and current machine time.

Allowing multiple clients sharing the same session

Per our requirements, we want the sessions to be accessible from both the vscode terminal AND any external terminal emulator. Even better, pair programming is actually possible by having a remote user accessing the local machine via ssh and access the same terminal session.

To better achieve that, we can spawn two sessions, with the second session is attached and synchronized to the first main session. Therefore, any new clients can attach to the first, be it a vscode terminal, a terminal emulator, or even a remote user via ssh. To prevent cluttering of ‘zombie’ sessions, that is client sessions that are no longer attached, we have to do clean up on the sessions that are connected to the main session.

There are many solutions to achieve the aforementioned specifications. First there is one from Practical Tmux article, and there are two alternative versions in Archlinux Wiki. The one in Archlinux Wiki is cleaner, and actually handles cleaning up of zombie sessions better than the one in Practical Tmux article. Both uses current datetime stamp for children sessions, and the alternative implementation from Archlinux Wiki uses a bash function stored in our ~/.bashrc file uses unix timestamp to differentiate parent session and child session.

However I am not satisfied with them for cosmetic reasons. Therefore I modified the tmx script implementation from both Practical Tmux article and Archlinux Wiki one, with one major difference: I use the substring of md5 hash of child session creation unix timestamp as the child session name. I do that because my tmux configuration shows both the session name and the session group name for brevity.

You can see my tmx implementation here.

To use the script, you have to add that script in your path and make it executable. It can easily be done by exporting your script directory to your path. For example in my case, to add ~/.local/bin in our $PATH only if it is not yet appended:

test -z "$(echo $PATH|grep "$HOME/\.local/bin")" \
    && PATH="$HOME/.local/bin:$PATH"

Then to make the script executable, you can do:

chmod +x ~/.local/bin/tmx

To make the changes persistent, add them to your .bashrc or .zshrc.

Therefore, with tmx script, we can create a new named session on demand, or connect to an existing named session if it is already present. It is useful since in the next part we will discuss about the creation of multiple detached named sessions, each with custom configuration files of the same name located under ~/.tmux.conf.d/sessions/. After they are automatically generated, we can just connect them from any terminal in our machine just by running tmx <session_name>. That including automated deployment of a tmux session from vscode at later stages of this article.

A convenience script to generate named sessions

Basically, step 1-4 of our pseudocode can be done in just a single convenience script. Here we will make a script called set_tmux and put it somewhere in our path. In my case, I put it under ~/.local/bin/

The content of the script is as follows (read the comments to see the workflow):

#!/bin/sh
# Sets named tmux sessions
# usage: set_tmux <session>
# Assumes ~/.tmux.conf.d/sessions/<session>.conf
# file exist

# Loads session $session, if $session is empty,
# then load a session with the same name as the
# first argument. If the first argument is empty,
# then $session name is 'main'
tmux_load_config(){
test -z "${session}" && session="$1"
test -z "${session}" && session="main"
local config_file="${HOME}/.tmux.conf.d/sessions/${session}.conf"
test -f "$config_file" && \
    tmux send-keys -t "${session}" \
        "tmux source-file ${config_file}" C-m
}

# If session $1 is not present, make one
configure_tmux(){
local session="$1"
if test -z "$TMUX" ; then
  tmux ls 2>/dev/null | grep -q "^$session":
  if test $? -gt 0 ; then
    tmux new-session -d -s "$session" && \
    tmux_load_config
  fi
fi
}
# Loop through session names and generates named
# sessions
for x in "$@" ;
do
  configure_tmux "$x"
done

The script is made with the assumption that we have more than one working directories. For example, if we need one for each of ~/App/core, ~/App/fe (front-end directory), and ~/App/deployment, then we can just have to create three configuration files under ~/.tmux.conf.d/sessions/:

• core.conf
• fe.conf
• deployment.conf

We will discuss the tmux config files later in this article.

The for-loop section basically passess all arguments as session names. Then for each of them, it would run configure_tmux() by taking the session name as the argument. The configure_tmux() function would check with tmux ls command and piping the output to grep to determine whether the session of that name is already run or not. If it is not run already, only then would the function launches a new session of that name. It would then run tmux_load_config that basically checks if the config file for that particular session name is present or not. If it is present, then the config file would be loaded, otherwise it would be ignored and only default tmux configuration is loaded.

Then in our shell configurations (either ~/.bashrc, ~/.zshrc, or both), we just have to set the following lines:

set_tmux core fe deployment

Note that you must ensure that set_tmux is within your path. See the previous section that discusses tmx script on how to add this file to your path and make it executable.

Custom tmux configuration files

In this article we assume that you put your session configuration files under ~/.tmux.conf.d/sessions/. For the purpose of this article, we assume that the session name is the same as the configuration file name. For example, tmux session core would have a configuration file under ~/.tmux.conf.d/sessions/core.conf.

This article also assumes that the named tmux sessions are created with set_tmux script called from our ~/.bashrc or ~/.zshrc as described in the previous section. This is important, because set_tmux actually checks whether configuration file for that it creates is present in ~/.tmux.conf.d/sessions/<session>.conf. If it is present, then the script will load the configuration file. Therefore the config file will only be run once during the creation of those detached sessions.

Since most of our desired configurations are already set in our default tmux configuration (mine is located at ~/.config/tmux/tmux.conf), we would only need custom configurations for our own projects. In my case it is usually automated creation of workspace layout. It can be of windows, or to autorun certain commands during tmux session startup.

The simplest use case might be to automatically open two named windows in our tmux session: git and npm. To do that, we basically have to rename the first tmux window to git, and then we create a new named window npm. Here’s a simple configuration file that implements that:

# File ~/.tmux.conf.d/sessions/fe.conf
# Rename window :0
rename-window -t :0 'git'

# Make a new named window
new-window -t :1 -n 'npm'

# Select first window
selectw -t 0

However the working directory would still be the working directory of the spawned terminal. If the first terminal we open is the one in vscode, it is very likely that its working directory is the project directory, which is exactly what we want to do. However if you open another terminal emulator, that might not be the case. Instead it is likely that the working directory is from your home directory ($HOME or ~/).

We can actually circumvent that by creating new named windows with a specified working directory with the following command under the config file:

new-window -t xadf:1 -n 'git' -c ~/Documents/xadf

However when we make a new tmux session, there will always be one default window, usually <session_name>:0 (eg. main:0 or simply :0). The command above would only affect the working directory of new tmux windows. If we want to change the current working directory of the first window, or any tmux windows if that matters, is simply by sending key sequences to that specific tmux window:

# Target is a named tmux session 'xadf'

# Change directory to ~/Documents/xadf, enter,
# and clear the screen
send-keys -t xadf:0 'cd ~/Documents/xadf' C-m C-l

# Clear the screen, and then display current
# branch's name and status, then enter
send-keys -t xadf:1 'clear;git status -sb' C-m

In fact we can send any arbitrary commands to a tmux window with that method. Note that if we don’t include C-m, the command would just sit in the terminal as if you press those keys on the window. It is not run yet, as we haven’t send the ENTER key to execute it. The C-m sequence is equivalent to pressin ENTER. Likewise, the C-l sequence is the same as pressing Ctrl+l or running clear command.

Revisiting our fe.conf, we can then have the following to consistently open a detached named tmux session fe during a terminal start if it is not present already and have it shows the correct working directory for both windows:

# File ~/.tmux.conf.d/sessions/fe.conf
# Working directory is ~/App/fe

# Rename window :0
rename-window -t :0 'git'

# Send 'cd ~/App/fe' to window :0 and clear
send-keys -t :0 'cd ~/App/fe' C-m C-l

# Make a new named window
new-window -t :1 -n 'npm' -c ~/App/fe

# Select first window
selectw -t 0

Then we can create similar configuration files for core and deployment. We can determine how many tmux windows would each of them be, and what to name (or not) each of the tmux windows. We can also send and run arbitrary commands automatically (such as npm run dev, git pull, or even git log --oneline --graph) in the desired tmux windows.

Automatically run tmux in vscode terminal

The last part that we need to implement are step 5-6 of our pseudocode. We want to test whether we’re running under vscode or not. Surprisingly, the solution to this is fairly trivial. You can read it from StackOverflow or AskUbuntu.

Basically we want to check whether or not $TERM_PROGRAM environment variable is set to vscode. If that is true, then we are in vscode, and we can just connect to a session with the same name as our working directory using our tmx script. The implementation of those statements that can be run under bash and zsh is:

[[ "$TERM_PROGRAM" == "vscode" ]] \
    && tmx "$(basename "$PWD")" || true

The last true command is only run if we are not in vscode. I added it there because in some terminal prompt style, we get a certain feedback if the last run command gives nonzero exit status. For example in a default oh-my-zsh installation, the prompt indicator might turn from green to red the first time I run a terminal from outside of vscode. That is a minor annoyance for me, especially if that line is appended at the end of our .bashrc and/or .zshrc.

The true statement would always return exit status 0, which guarantees a clean exit status at the beginning of our interactive terminal prompt. That is, assuming that the tmx script does not terminate with a nonzero exit status.

Conclusion

To improve our developer workflows, and addresses the issues explored in this article, we implemented the following measures:

1. We implemented a sane, minimalistic, and usable default tmux configuration in ~/.config/tmux/tmux.conf. It can be downloaded from gitlab.
2. To allow a persistant terminal session that can be accessed from any terminal or ssh connection in our developer’s machine, we implemented a script to generate a new named tmux session on demand, or connect to an existing named tmux session if already present. The script can be obtained from gitlab. To use it, add the file to your path and make it executable.
3. To automatically launch arbitrary amount of named tmux sessions on terminal start up if they are not run already, and load the associated tmux configuration files under ~/tmux.conf.d/sessions/, we implemented a set_tmux script. The script can be obtained from gitlab. To use it, add the file to your path and make it executable.
4. Developers are encouraged to create their own tmux config files and place it under ~/.tmux.conf.d/sessions/ with the same name as their project directory and file extension .conf, as every developer have their own specific workflows and preferences.
5. In ~/.bashrc or ~/.zshrc, adds the attached snippets (Appendix) to enable our setup. Modify as you see fit.

Appendix

# Add ~/.local/bin/ to $PATH if not already
# present. It should contains tmx and set_tmux
# script for our setup
test -z "$(echo $PATH|grep "$HOME/\.local/bin")" \
    && PATH="$HOME/.local/bin:$PATH"

# Change the tmux session names according to your
# needs. Also prepare custom configurations under
# ~/.tmux.conf.d/sessions/ if desired
set_tmux core fe deployment

# Check if we are under vscode. If yes, run tmux
# and/or attach to an existing tmux session
[[ "$TERM_PROGRAM" == "vscode" ]] \
    && tmx "$(basename "$PWD")" || true

Yes, Eloquent relationships are more succinct when we have implemented it to our models. In the following sections I will try to demonstrate the difference between the two and how eloquent creates cleaner code.

Before Eloquent - Query Builder

What we did to fetch a post was originally like the following:

use App\Models\Post;

// Fetch published posts of a certain type,
// the amount of which is limited
$type = 'news';
$limit = 15;

Post::select('posts.id',
           'posts.created_at as created',
           'post_types.code as type',
           'post_types.name as type_name',
           'posts.slug',
           'posts.title',
           'users.name as author',
           'posts.body')
    ->where('is_published',1)
    ->where('post_types.code',$type)
    ->orderBy('created','desc')
    ->join('users',
           'users.id',
           '=','posts.users_id')
    ->join('post_types',
           'post_types.id',
           '=','posts.post_types_id')
    ->limit($limit)
    ->get();

// Fetch a particular post by selecting its
// ID
$id='put-post-id-here';
Post::select('posts.id',
    'posts.created_at as created',
    'post_types.code as type',
    'posts.slug',
    'posts.title',
    'users.name as author',
    'users.id as author_id',
    'posts.body',
    'posts.is_published')
        ->where('posts.id',$id)
        ->join('users',
               'users.id',
               '=','posts.users_id')
        ->join('post_types',
               'post_types.id',
               '=','posts.post_types_id')
        ->firstOrFail();

The joins are necessary for me to be able to access values of its post types and author information.

Sets up Eloquent Relationships

Now let us implement Eloquent, by first adding the required functions to our models. First we want to access posts from our User model.

namespace App\Models;

use Illuminate\Database\Eloquent\Model;
use Illuminate\Database\Eloquent\Relations\HasMany;

class User extends Authenticatable
{
    /**
     * Get the posts for this user
     */
    public function posts(): HasMany
    {
        return $this->hasMany(
                    Post::class,
                    'users_id');
    }
}

Now we also want to do the same from our PostType model.

namespace App\Models;

use Illuminate\Database\Eloquent\Relations\HasMany;
use Illuminate\Database\Eloquent\Model;

class PostType extends Model
{
    /**
     * Get the posts for this PostType
     */
    public function posts(): HasMany
    {
        return $this->hasMany(
                    Post::class,
                    'post_types_id');
    }
}

Note that we use the hasMany methods to define that each users and post types can have many posts. We also supplied our foreign keys users_id and post_types_id because we used nonstandard column names in our table migrations. Eloquent would normally use snake case of our model name and append it with _id. So for example, if in our posts table, our foreign key for post type is post_type_id, we can just return $this->hasMany(Post::class); and the method would just work.

Now we need to define the inverse of hasMany in our Post model. To define the inverse of hasMany we can use belongsTo method:

namespace App\Models;

use Illuminate\Database\Eloquent\Relations\BelongsTo;
use Illuminate\Database\Eloquent\Model;

class Post extends Model
{
    /**
     * Get the owner of a post
     */
    public function user(): BelongsTo
    {
        return $this->belongsTo(
                    User::class,
                    'users_id');
    }

    /**
     * Get the type of a post
     */
    public function type(): BelongsTo
    {
        return $this->belongsTo(
                    PostType::class,
                    'post_types_id');
    }
}

Accessing related models with Eloquent

Now we can access our post’s user and type effortlessly:

use App\Models\Post;

// Get the model of the post's owner
Post::first()->user;

//Get the model of our post type
Post::first()->type

Then if we want to say, get the name of the post’s author, or if we want to get the code of our post type, we can do:

use App\Models\Post;

// Get the name of the post's author
Post::first()->user->name;

// Get the code of the post's type
Post::first()->type->code;

Or even accessing posts created by a particular user:

use App\Models\User;

User::first()->posts;

Or get posts of a certain post type:

use App\Models\PostType;

PostType::first()->posts;

We can even use query builder alongside with Eloquent. The following achieves similar results as our earlier two query builder examples.

use App\Models\PostType;
use App\Models\Post;

// Fetch published posts of a certain type,
// the amount of which is limited
$type = 'news';
$limit = 15;
PostType::where('code',$type)
    ->first()
    ->posts()
    ->where('is_published',true)
    ->orderBy('created_at','desc')
    ->limit($limit)
    ->get();

// Fetch a particular post by selecting its
// ID
$id='put-post-id-here';
Post::where('id',$id)
    ->firstOrFail();

The primary difference is that now we don’t have to perform joins. The difference would be on how we access the attributes.

When using query builder, we aliased certain columns for easy accessing. For example users.name is aliased to author when users table is joined to the posts table in our previous query builder example. Therefore assume that we store a particular Post model after the join operation into a variable $post, we can access the author name attribute by $post->author.

If the same post model is implementing eloquent relationships like we described above, there is no need to alias users.id to author, all we have to do is to get the user of a post, and access the name attribute: $post->user->name. Therefore we also made modifications to our controller methods to reflect the changes. We are also adjusting our blade templates, as some still directly access models for displaying contents.

Conclusion

The change is done with minimal differences from the perspective of casual website viewers. However behind the scene the code is being tidied out. Our code can be cleaner as we no longer have to manually join database tables.

In the future we wish to implement user roles using eloquent relationship, and therefore allowing us to conveniently differentiate user’s permissions based on their role that is accessible directly from their user model.

Problem

Due to the need to interact with some electronics such as speakers and file transfer to printing places, I have to maintain a number of flash drives formatted as FAT drives. Despite its rather popular usage with significantly wide support, FAT is not exactly the most robust filesystem in the wild. Therefore it is not uncommon to find filesystem errors when we have to use FAT.

Since I have been using Linux for many years (from at least 2011), today all of my computers are using Linux only. Therefore I have the need to be able to diagnose and deal with damaged FAT drives. Now how do we do that in Linux?

Check FAT filesystem

There are several options, but I mainly use this askubuntu answer. Meanwhile another answer suggests this filesystem troubleshooting article. The tool in question is available in Arch Linux repository as dosfstools.

To install it from Arch Linux repository, we can run the following command:

sudo pacman -S dosfstools

After installing the package, we will have access to the tool fsck.vfat. For example if we want to check the filesystem inside /dev/sdb1 (change to the appropriate device name), we can simply run the following:

sudo fsck.vfat -tawl /dev/sdb1

The options I use are:

• -t: mark unreadable clusters as bad
• -a: automatically fix errors
• -w: write to disk immediately
• -l: list processed filenames

Check for fake flash drives

Sometimes the problem is not because of a broken filesystem, but due to a fake flash drive. Essentially a fake flash drive claims that they have more storage space than they actually have. When interacting with such drive, it would at first appear fine if you just store files smaller than its actual storage capacity. Once you went beyond the actual storage capacity (because the device claims it still has more free spaces), it will start overwriting existing files.

For that we will also need a tool called f3 - Fight Flash Fraud. On Arch Linux, you can obtain the tool from the following AUR package: f3

Then to check whether the announced disk size is true or not (in this example we use /dev/sdb), we can use:

f3probe -t /dev/sdb

Note that here we did not specify the partition number (eg. /dev/sdb1 or /dev/sdb2), but rather, the block device itself (/dev/sdb).

Reformat the flash drive

Finally, to reformat the partition with FAT:

sudo mkfs.vfat -n MYFLASHDRIVE /dev/sdb1

The -n MYFLASHDRIVE option allows us to label the newly formatted partition as MYFLASHDRIVE.

If we want to specify the FAT type:

sudo mkfs.vfat -F 32 -n MYFLASHDRIVE /dev/sdb1

Conclusion

With this article, now you can:

1. Check and repair damages of your FAT drives using fsck.vfat.
2. Check if the device you have is announcing their true storage size or not using f3probe.
3. Format the device with FAT filesystem using mkfs.vfat.

1. Use of SSH Proxy as a replacement of VPN
2. Connect to home computer via ssh
3. Forward services from home to office
4. Access resources on local LAN from a shared remote
5. Check which ports are used for reverse SSH tunnel
6. My personal setup

Use of SSH Proxy as a replacement of VPN

Our browser is google-chrome, and we want it to connect to our cloud server via a proxy. Therefore we can configure google-chrome to connect to the internet via the configured proxy. The configuration should make google-chrome dependent on the proxy to be present, otherwise it should not be able to connect to the internet.

Read more in this article.

Connect to home computer via ssh

The problem with a normal home network is that it is hidden behind a NAT. Moreover setting up port forwarding is not exactly trivial for end-user. Therefore, since we have a readily accessible remote server, we can leverage it for our purpose.

Read more in this article.

Forward services from home to office

We have some services that we would like to be made available at office for convenience reason. For security purposes we have to use ssh tunnels instead of exposing the port from our home network’s port forwarding. It takes one cloud device we control that we utilized as a bridge between our home computer and office computer.

Read more in this article.

Access resources on local LAN from a shared remote

We have certain resources accessible only from home LAN, and we want to connect to them from anywhere. Given that we have a high availability cloud server, we can use it as a reverse proxy to access specific services we wanted to access. A custom configuration is required, and will be outlined in the article below.

Read more in this article.

Check which ports are used for reverse SSH tunnel

Following this answer, we can check which TCP ports are in state LISTEN and used by sshd using lsof:

$ sudo lsof -iTCP -sTCP:LISTEN | grep sshd

It should output something like the following:

user@remotehost:~$ sudo lsof -iTCP -sTCP:LISTEN|grep sshd
sshd   753  root   3u  IPv4  19286  0t0  TCP *:ssh (LISTEN)
sshd   753  root   4u  IPv6  19298  0t0  TCP *:ssh (LISTEN)
sshd  2039  user   7u  IPv6  24888  0t0  TCP localhost:2222 (LISTEN)
sshd  2039  user   9u  IPv4  24889  0t0  TCP localhost:2222 (LISTEN)
sshd  2039  user  10u  IPv6  24892  0t0  TCP localhost:8385 (LISTEN)
sshd  2039  user  11u  IPv4  24893  0t0  TCP localhost:8385 (LISTEN)
sshd  8141  user   7u  IPv6  60294  0t0  TCP localhost:2223 (LISTEN)
sshd  8141  user   9u  IPv4  60295  0t0  TCP localhost:2223 (LISTEN)

The first two (of user root) are the SSH Daemon, while the rest (of normal user user) are TCP tunnels, each listed twice: one for IPv6 and another for IPv4.

My personal setup

Since the basics are already explained on the previous sections, the next thing we would like to address is how we actually configure our SSH tunneling infrastructure.

homecomputer

First we set up file ~/.ssh/config as follows:

Host cloud-tun
    User                user
    HostName            remotehost
    # connects to ssh at mobile or work
    LocalForward        2223 localhost:2223
    # ssh server
    RemoteForward       2222 localhost:22
    # home syncthing
    RemoteForward       8385 localhost:8384
    # router
    RemoteForward       8090 192.168.100.1:80
    DynamicForward      8257
    ServerAliveInterval 120

Host cloud-tun is configured with four parameters. First we configures SSH tunnels at port 2223 that receives traffic to remote system port 2223. Afterwards we configured a reverse tunnel that forwards traffic to our port 22 from remote system port 2222. Followed with another reverse tunnel that forwards traffic to our port 8384 from remote system port 8385. Then we opened an encrypted SOCKS port 8257 that we can use with our applications.

Then we set up a systemd user service:

# file ~/.config/systemd/user/cloud-tun.service
[Unit]
Description=SSH tunnel to remotehost

[Service]
Type=simple
Restart=always
RestartSec=10
ExecStart=/usr/bin/ssh -F %h/.ssh/config -N cloud-tun

[Install]
WantedBy=default.target

We activate them by enabling and starting our systemd user service:

$ systemctl --user enable cloud-tun
$ systemctl --user start cloud-tun

The systemd user service would run the command specified in ExecStart, and would restart after ten seconds of disconnection. Therefore ensures our service to be available nearly continuously as long as we are logged in.

workcomputer

First we set up file ~/.ssh/config as follows:

# File ~/.ssh/config
Host cloud-tun
    User                user
    HostName            remotehost
    # home syncthing
    LocalForward        8485 localhost:8385
    # connects to home
    LocalForward        8022 localhost:2222
    # ssh server
    RemoteForward       2223 localhost:22
    # router
    RemoteForward       8091 192.168.100.1:80
    DynamicForward      8257
    ServerAliveInterval 120

Host home
    User                pete
    HostName            localhost
    Port                8022
    ServerAliveInterval 120

Host xph
    HostName            xenophone
    Port                8022

Host cloud-tun is configured with four parameters. First we configures SSH tunnels at port 8485 that receives traffic to remote system port 8385. Then another SSH tunnel at port 8022 that connects to remote system port 2222. Afterwards we configured a reverse tunnel that forwards traffic to our port 22 from remote system port 2223. At last we set up an encrypted SOCKS port 8257 that we can use with our applications.

Then we configured a systemd user service:

# file ~/.config/systemd/user/tunnel.service
[Unit]
Description=SSH tunnel to remotehost

[Service]
Type=simple
Restart=always
RestartSec=10
ExecStart=/usr/bin/ssh -F %h/.ssh/config -N cloud-tun

[Install]
WantedBy=default.target

Afterwards we enable and start the service. We activate them by enabling and starting our systemd user service:

$ systemctl --user enable tunnel
$ systemctl --user start tunnel

The systemd user service would run the command specified in ExecStart, and would restart after ten seconds of disconnection. Therefore ensures our service to be available nearly continuously as long as we are logged in.

remotehost

There is no specific configuration needed at remotehost but a working ssh server. Ideally it must only use public key authentication to prevent password brute force hacking.

Problem

Certain resources, such as a wireless internet router configuration, is typically only accessible under LAN. Exposing those resources directly to the internet is impractical because it has no static IP. Another alternative would be to have an accessible, reachable local device that can access the LAN resources to act as a proxy to access them on our behalf.

Solution

There are two known possible configurations to connect to a resource available only in a certain LAN from outside.

1. The first one is to let another local device to perform reverse SSH tunnel that maps a remote port to a certain device in its LAN. Then we configure a reverse proxy in the remote to make it accessible from the remote’s webserver.
2. The second one is just like the first one, except that instead of configuring a reverse proxy at the remote, we set up a local forwarding from another machine to the remote machine, mapping the exposed remote port to another port in the local machine (outside the LAN).

We will explore both configurations as they shared the same initial setup. Therefore this solution section will be divided into three parts: the first part is where we configure the machine inside our target LAN that we can control, the second part is where we configure a reverse proxy in our remote server, and the third part is where we configure external devices to access the LAN services from outside.

The Device From Within Target LAN

It is actually quite straightforward, as all we need to do is to prepare a reverse SSH tunnel targeting a locally available service, and make it available on a port in the remote server. The setup is already being described in great length on this article.

For example if we want to access our home router, we want to use a computer inside the same network as our home router, and set up a reverse SSH tunnel:

$ ssh -TNv -R 8090:192.168.100.1:80 user@remotehost

Basically we make 192.168.100.1:80 in our local network be accessible from remotehost at an arbitrary port (in this example, we use port 8090). Remember this port as we are going to simply write it as is in the following sections.

Alternatively, we can just add that to our SSH config file ~/.ssh/config:

# file ~/.ssh/config

Host cloud-tun
    User                user
    HostName            remotehost
    RemoteForward       8090 192.168.100.1:80
    ServerAliveInterval 120

Therefore we can just issue:

$ ssh -TNv cloud-tun

The Reverse Proxy On Remote Server

On our server, we can have something like the following configuration:

# Redirect specific sites to use https
# Courtesy of:
# https://serversforhackers.com/c/redirect-http-to-https-nginx
server {
    listen 80;
    listen [::]:80;

    server_name myrouter.domain.tld;

    return 301 https://$server_name$request_uri;
}

# Separate server block for http and https
# The block below only accepts ssl connection
# The block above basically redirects to this block
# Courtesy of:
# https://bobcares.com/blog/nginx-multiple-domains-ssl/
server {
    listen [::]:443 ssl;
    listen 443 ssl;

    server_name myrouter.domain.tld;

    include certs_params_domain.tld_wildcard;

    location / {
        proxy_pass http://127.0.0.1:8090;
        include proxy_params;
    }
}

Then we just have to point our browser to our domain, for example https://myrouter.domain.tld and our router is accessible there.

The Device From Outside Target LAN

It is actually quite straightforward, as all we need to do is to prepare a SSH tunnel connecting a certain local port to a specified port at remote. Again, the setup is already being described in great length on this article.

For example if we want to access our home router and we already configured the device within the same network as our target service, we can connect to the service with any device as long as we configured the following SSH tunnel:

$ ssh -TNv -L 8080:localhost:8090 user@remotehost

Note that the service we want to use in this article is being made available at port 8090 of server remotehost.

Alternatively, we can just add that to our SSH config file ~/.ssh/config:

# file ~/.ssh/config

Host cloud-tun
    User                user
    HostName            remotehost
    LocalForward        8080 localhost:8090
    ServerAliveInterval 120

Therefore we can just issue:

$ ssh -TNv cloud-tun

After we set up the tunnel, we can just point our browser to http://localhost:8080.

The problem with a normal home network is that it is hidden behind a NAT. Moreover setting up port forwarding is not exactly trivial for end-user. Therefore, since we have a readily accessible remote server, we can leverage it for our purpose.

Solution

Basically now there are three devices under our control: - homecomputer - mymobile - cloudserver

We can then configure cloudserver as a bridge, as we are in control with it. That way we can, configure a ssh reverse tunnel homecomputer > cloudserver connection, and also another ssh reverse tunnel to mymobile > cloudserver . Then we should also create their corresponding ssh tunnels: homecomputer < cloudserver and mymobile < cloudserver.

SSH tunnel from home and phone to cloud

From homecomputer we can issue:

ssh -R 2222:localhost:22 \
  -L 2223:localhost:2223 \
  user@cloudserver

Here’s how it works:

• The -R argument tells homecomputer to: expose port 22 on localhost, and any connection from cloudserver port 2222 should be accepted by our exposed port.
• The -L argument tells homecomputer to: tunnel any connection made to localhost:2223 into port 2223 on cloudserver.

Then we just have to configure the reverse from mymobile:

ssh -R 2223:localhost:8022 \
  -L 8023:localhost:2222 \
  user@cloudserver

SSH tunnel between home and phone with cloud as a bridge

This way, from homecomputer to mymobile, I just have to run:

ssh -p 2223 localhost

And from mymobile to homecomputer, I just have to run:

ssh -p 8023 user@localhost

Even better, we should consider also adding -TNv argument to ssh.

Automatic configuration on home

Like the one we write in SSH proxy as VPN replacement, we can actually configure them as a systemd/user service:

# file ~/.config/systemd/user/remote-tunnel.service
[Unit]
Description=SSH tunnel to domain.tld

[Service]
Type=simple
Restart=always
RestartSec=10
ExecStart=/usr/bin/ssh -F %h/.ssh/config -N remote-tunnel

[Install]
WantedBy=default.target

This service will be run after user logs in to a session. It will be killed after the last session of that user is closed. If we want it to start at boot and would remain be open after the last session of a user is closed, we can let our service to linger:

loginctl enable-linger

Then we write a configuration file as follows:

# file ~/.ssh/config
# Equivalent to:
# ssh \
#   -R 2222:localhost:22 \
#   -L 2223:localhost:2223 \
#   -D 8257 \
#   user@cloudserver
Host remote-tunnel
    User                user
    HostName            domain.tld
    LocalForward        2223 localhost:2223
    RemoteForward       2222 localhost:22
    DynamicForward      8257
    ServerAliveInterval 120

It would then enable us to establish the tunnel (bonus the proxy port) as soon as we login into our account. Then connection to phone is as simple as:

ssh -p 2223 localhost

Automatic configuration on phone

I use termux on my phone. From my phone, we can also set a configuration file:

# ssh -R 2223:localhost:8022 \
#   -L 8023:localhost:2222 \
#   user@domain.tld
Host remote-tunnel
    User                user
    HostName            domain.tld
    LocalForward        8023 localhost:2222
    RemoteForward       2223 localhost:8022
    ServerAliveInterval 120

Therefore we can just do this to initiate connection:

ssh -TNv remote-tunnel

It can be done with the same tmux setup described in the SSH proxy as VPN replacement article. Then to connect to our computer, we invoke:

ssh -p 8023 user@localhost

Conclusion

The pattern is then:

ssh -R port:localhost:hostport \
  -L localport:localhost:remoteport \
  [user@]remotehost

Note that localhost in -L part is actually local to remotehost, not our localhost. Then for [user@]remotehost part, it means that the username is optional. If the username is not given, and only remote host is specified, ssh client will attempt to connect to the remotehost with the same username as the local computer’s username.

The map from homecomputer is then:

localhost:22 listens from cloudserver:2222
localhost:2223 forwarded to cloudserver:2223

The map from mymobile is then:

localhost:8022 listens from cloudserver:2223
localhost:8023 forwarded to cloudserver:2222

The map from cloudserver is then:

localhost:2222 to homecomputer:22
localhost:2223 to mymobile:8022
localhost:2223 from homecomputer:2223
localhost:2222 from mymobile:8023

Appendix

In an afterthought, we realized that we don’t really need to connect to and from our phone with this method most of the time. At home, we would prefer to connect to the phone via our home’s local network, simply because it’s much faster. Conversely, at work, we would prefer to connect to the phone via our work’s local network.

We would only need to connect to our home computer from our phone when we’re outside and our phone is not in the same network as our work computer. Even then when it does happen, our work computer would not be normally connected to our home computer.

It means our phone really only need to connect to home computer when we explicitly told it so. Therefore, our work computer can use a similar ~/.ssh/config as our phone, plus we can set up a systemd/user service.

At such event where we are outside and our work computer are also on, we can always connect to our home computer with extra steps from our phone: go to cloudserver, and then ssh -p 2222 user@localhost will serve the job well.

SSH tunnel between home and office

We put the following in our office’s ssh configuration:

# file ~/.ssh/config

Host remote-tun
    User                user
    HostName            domain.tld
    LocalForward        8022 localhost:2222
    RemoteForward       2223 localhost:22
    DynamicForward      8257
    ServerAliveInterval 120

Host home
    User                user
    HostName            localhost
    Port                8022
    ServerAliveInterval 120

Then we set up a systemd user service at work:

# file ~/.config/systemd/user/tunnel.service
[Unit]
Description=SSH tunnel to domain.tld

[Service]
Type=simple
Restart=always
RestartSec=10
ExecStart=/usr/bin/ssh -F %h/.ssh/config -N remote-tun

Therefore we can connect to our home computer with:

$ ssh home

Further reading

Read more from the sources below:

• ghoti’s answer on How does reverse SSH tunneling work?
• erik’s answer on How does reverse SSH tunneling work?
• OpenSSH
• SSH Tunneling and Proxying
• SSH Socks Proxy as VPN Replacement

Our browser is google-chrome, and we want it to connect to our cloud server via a proxy. Therefore we can configure google-chrome to connect to the internet via the configured proxy. The configuration should make google-chrome dependent on the proxy to be present, otherwise it should not be able to connect to the internet.

Solution

There are multitude of layers in our solution. First we need to run a command to initiate the SOCKS5 proxy connection. Then we need to configure google-chrome to make use of the SOCKS5 proxy port as its web proxy. For convenience reasons, we have to automate the setup since the first time our shell starts.

Start the connection

It can be done in as simple as:

$ ssh -TND <PORT> user@host

The choice of port is arbitrary. I selected one that is unlikely to be used by any existing services. Argument -T disables pseudo-tty allocation, while -N disables interactive prompt. Argument -D PORT is the one that actually do the creation of an encrypted SOCKS5 tunnel. It might also be useful to have -v passed for verbosity, so we can check that we are actually connected.

We also want it to be kept alive, and ideally to simplify the invocation. To keep our connection alive after running it, we can actually add ServerAliveInterval 120 to our ~/.ssh/config. It simply tells our ssh client that in case it receives no data from the server after a specified interval (here our interval is 120 seconds), it would send a message to server to get a response.

The full command we needs to reach our server to initiate our proxy is then:

$ ssh -TNvD <PORT> username@domain.tld

However to simplify that we just need to specify our user name and host name in our ~/.ssh/config file. The equivalent of -D <PORT> in the config file is the line DynamicForward <PORT>. Therefore we can compose the following config file:

# file ~/.ssh/config
Host proxy
    User                username
    HostName            domain.tld
    ServerAliveInterval 120
    DynamicForward      <PORT>

Then all we need to connect to our server is simplified to:

$ ssh -TNv proxy

Configure Google Chrome to use proxy

Apparently, it is rather trivial to force Google Chrome to honor proxy setting. Arch Linux Wiki suggests to make the following function:

function secure_chromium {
    port=<PORT>
    export SOCKS_SERVER=localhost:$port
    export SOCKS_VERSION=5
    chromium &
    exit
}

or:

function secure_chromium {
    port=<PORT>
    chromium --proxy-server="socks://localhost:$port" &
    exit
}

then source them from our ~/.bashrc.

In that manner, whenever we want to have a secure connection, we can just run secure_chromium, and enjoy our secure internet connection.

However our requirement is that we must make chrome dependent on it, that it would not be able to connect to the internet without it. The most obvious solution is to pass --proxy-server=socks://localhost:<PORT> for each run of google chrome. But it was too much work, and setting up a custom script for it is just superfluous.

The solution is actually rather simple, we just have to create the following line with that argument passed within:

# file ~/.config/chrome-flags.conf
--proxy-server=socks://localhost:<PORT>

# Note that for chromium it is ~/.config/chromium-flags

Therefore our google chrome is constrained to use the proxy server.

Automatic configuration

Currently we are using our tmux configuration with convenience scripts provided by xadfsh/xadf. We use set_tmux at the start of our session to launch a single or a series of named tmux session, and when there’s a corresponding config file for that named session, it would also be autoloaded.

Inside a named tmux session’s custom config file ssh.conf, we included the following lines:

# file ~/.tmux.conf.d/sessions/ssh.conf
new-window -t ssh:1 -n 'socks'
send-keys -t ssh:1 'clear;ssh -TNv proxy' C-m

Then we included in our autostart file:

# Launch named session ssh among other named sessions
set_tmux ... ssh ...

Note: The script set_tmux is a convenience script provided here. To use it, I recommend you to read the script carefully and understand what it is doing. See the repository xadfsh/xadf, study it, and adapt it to your case if necessary.

Therefore our connection will automatically be initiated upon entering our terminal.

However the limitation is that if the connection is severed, we would be left without a proxy. Therefore one more solution to that is to have a systemd user service:

# file ~/.config/systemd/user/proxy.service
[Unit]
Description=SSH tunnel to proxy

[Service]
Type=simple
Restart=always
RestartSec=10
ExecStart=/usr/bin/ssh -F %h/.ssh/config -N proxy

Then we can enable and start it with:

systemctl --user enable proxy
systemctl --user start proxy

If this is configured, then we no longer need our tmux based solution.

Sources

• OpenSSH: Encrypted SOCKS Tunnel
• Chromium: Making Flags Persistent
• man 5 ssh_config
• xadfsh/xadf

We have some services that we would like to be made available at office for convenience reason. The service is available from a certain port in our home computer. For security purposes we have to use ssh tunnels instead of exposing the port from our home network’s port forwarding.

At our disposal is a cloud server we control that we utilized as a bridge between our home computer and office computer. Both our home computer and work computer are connected to the cloud server. However, how do we configure the connection between our home computer and office computer?

Case Study

For the purpose of this article, we assumes that we want to expose our locally available Syncthing GUI configured in homecomputer to listen only to localhost:8384. The service would not be available from outside, as our syncthing configuration would reject any connection not coming from localhost. This is a security feature, that the service is only available locally, as it allows access to sensitive files.

1. homecomputer is our home computer
2. workcomputer is our work computer
3. cloudserver is our cloud server

Solution

With SSH tunnels, from workcomputer we can access homecomputer:8384 as if we are connecting from localhost. We can achieve this by setting up a matching pair of SSH tunnels between homecomputer and workcomputer through a single common local port on cloudserver. From homecomputer we configured a reverse SSH tunnel to cloudserver, and from workcomputer we configured a SSH tunnel to cloudserver.

As a bonus, to have a clean URL when accessing from a browser in workcomputer, we can configure a reverse proxy in a local nginx at workcomputer. A reverse proxy configuration would take a certain URL path and would configure a local port to serve the URL path. By pointing the reverse proxy configuration to the port in workcomputer we configured as a SSH tunnel to cloudserver that is forwarded to homecomputer, we can then access a service available locally in homecomputer as if we are connecting to it locally.

SSH Tunnels

Therefore we just have to configure a SSH reverse tunnel from homecomputer:

# file .ssh/config at homecomputer
# Equivalent to: ssh -R 8385 localhost:8384 username@cloudserver
Host liecorp-tunnel
    User                username
    HostName            cloudserver
    RemoteForward       8385 localhost:8384
    ServerAliveInterval 120

Then from workcomputer we configured a SSH tunnel:

# file ~/.ssh/config at workcomputer
# Equivalent to: ssh -L 8485:localhost:8385 username@cloudserver
Host liecorp-tun
    User                username
    HostName            cloudserver
    LocalForward        8485 localhost:8385
    ServerAliveInterval 120

What we did from homecomputer is basically to open a listening port cloudserver:8385 that would correspond to homecomputer:8384. Then from workcomputer we forward connection from port workcomputer:8485 to cloudserver:8385, which in turn, is forwarded to homecomputer:8384. Therefore to reach homecomputer:8384 from workcomputer, we can point our browser to http://localhost:8485 and our service at homecomputer:8384 would be available from workcomputer.

Nginx reverse proxy

To make everything tidy, we can even set a reverse proxy configuration on nginx at workcomputer:

# File /etc/nginx/sites-available/server.conf 
server {
    listen 80 default_server;
    listen [::]:80 default_server;
    server_name _;
    root    /usr/share/nginx/html;
    index   index.html

    include "/etc/nginx/default.d/*.conf";

    location / {
    try_files $uri $uri.html $uri/ /fallback/index.html;
    }

    error_page 404 /404.html;
    location = /40x.html {
    }

    error_page 500 502 503 504 /50x.html;
    location = /50x.html {
    }

    location /syncwork0/ {
        proxy_pass          http://127.0.0.1:8384/;
        include             proxy_params;
        proxy_read_timeout  600s;
        proxy_send_timeout  600s;
    }
    location /synchome/ {
        proxy_pass          http://127.0.0.1:8485/;
        include             proxy_params;
        proxy_read_timeout  600s;
        proxy_send_timeout  600s;
    }
}

Therefore we can now access homecomputer:8384 by pointing our browser to http://localhost/synchome/ instead of http://localhost:8485/.

Conclusion

To connect to a service available only to localhost at homecomputer from workcomputer, we:

1. Configure SSH Tunneling from both homecomputer and workcomputer to cloudserver, where the traffic would go through cloudserver:8385. The route cloudserver:8385 corresponds to a listening port homecomputer:8384, and forwarded traffic from workcomputer:8485. Therefore connecting to localhost:8485 from workcomputer would get us to localhost:8384 at homecomputer. To syncthing, we are accessing from localhost, and therefore is allowed to access the service.
2. Configure Nginx reverse proxy to have a clean URL without having to type ports. We set up a reverse proxy address for URL /synchome/ that would be served by http://127.0.0.1:8485/. Therefore we can access the service from http://localhost/synchome/.

This page is originally written for and is published on Static Blog Alpha at 11 August 2022. You can access the original version here.

This page is copied from my gitlab repository. You may find an updated version of the article there.

In this post, we are going to review two classes of crypto wallets. The first one we will discuss is how addresses used to be managed by a wallet software, and how such wallets are backed up. The second part will discuss on how modern wallets handle wallets, and how they are backed up. Then, the next part will discuss on ways you can secure and back up your wallets. At last, we will provide a quick review of popular wallet software available on the market.

Early Wallets

Prior to 2013,¹ wallets generate a buffer of fresh random private keys (typically 100 addresses) for receiving and change addresses for future use. Such wallets would also offer wallet backups, to backup all of those addresses. However, once the backup pool exhausted (the pre-generated 100 addresses are all used up), new addresses are generated.

As a result, the previous backup is invalidated, and new backup must be made. This, in effect, requires users to frequently back up their wallets. The result of losing the wallet without proper backups, or infrequent ones, results in the loss of funds stored in the non backed up parts of the wallet.

There are many ways to counter this issue, one of them is the use of paper wallets. Unlike its name, a paper wallet is not a wallet, but instead just a single keypair. Because of that, it promotes address reuse, false sense of security, but also requiring users to manually handle private keys. Details of its flaws could be found here.

Some other means to store your coins are the use of custodial wallets like Coinbase, web wallets (as a browser extension), cloud storage, removable media, and physical bitcoins. All of those are explored here under “Bad wallet ideas” sections.

The basic idea of backing up a traditional wallet is just by backing up every and all private keypairs. Personally, it means we are storing a lot of data that must be reproducible: not a single bit can change, or you might lose some if not all of your funds. However, the universe is very hostile to modern computers. With the presence of a Single Event Upset might actually upset your backups or devices storing your backups, no matter how hard or how well you store them.

Then, there must be a better way to secure and back up your wallets, right? Right?

The Era of Seed Phrases

According to this article, by 2019, all good wallets already adopt BIP32, a standard for hierarchial deterministic wallets. What this kind of wallet does to back up every and all possible keypairs you will ever use is by deriving all of your future keypairs from a single starting point known as a seed. That way, any modern wallet using the same seed and the same derivation path will always produce the same addresses.

The seed must have a sufficiently high entropy to be secure, and also convenient to store and back up. However, as discussed in the previous section, even by storing only the entropy, no computer is safe from a Single Event Upset. Meanwhile, storing the entropy manually by writing them down is prone to typo, material damage, hard to read handwritings, data leakage while printing, and many more attack vectors. Some ideas to store your coins are also discussed here, along with their weaknesses.

Now, how could we store the seed in a manner that is easy to use and handle, encapsulating a sufficiently high amount of entropy, and has a very good error correction? Enter the age of Seed Phrases², as a way to losslessly compress our initial entropy into a list of 12 to 24 words, and can optionally be further protected with seed extensions³. The way they accomplish it is by having a list of words, and have them represent a number (in BIP39, 0 to 2047). That way, our 12 to 24 words seed phrases store a sufficiently large number in an easily human readable series of natural language words.

The BIP39 wordlist has 2048 words, and for a phrase of 12 random words, the number of possible combinations would be 2048^12 = 2^132. Therefore, the phrase would have 132 bits of security. However, the last word in BIP39 standard is not random, but instead a checksum, hence the actual 12 words seed phrase has a security of only 128 bits, which is approximately the same strength of all Bitcoin private keys. It follows without saying that a 24-words seed phrase has a higher security compared to a 12-words seed phrase.

That way, for any good wallets that support the generation of and restoring from seed phrases, you can retrieve your funds back. That assuming all wallets use the same seed phrase standard and the same derivation paths, which is not necessarily true. Electrum, for example, uses a different seed phrase standard that tries to address the issue inherent to BIP39 standard.

Furthermore, not all wallets supports seed extension, which makes this matter more complicated. As mentioned before, a seed extension is a word, or a set of words used to extend the original entropy, and hence, a more secure way to store your coins. All and any seed extension combined with all and any seed phrases will produce valid seeds for any compliant deterministic clients. Therefore it offers possible deniality in the event where said seed phrases are compromised. For example, by having funds stored in keypairs derived from the seed phrases only, or from the seed phrases combined with fake seed extensions.

However, wallets such as Trust Wallet, Bitcoin.com Wallet, and Litewallet, do not support seed extension, and some can only accept 12-words seed phrases. Therefore, it is best for one to actually study each wallets first, before right away trust that they will support your backed up wallets.

For myself, I have found some android wallets that actually implement BIP39 properly, with standarized derivation paths. Those are Unstoppable Wallet, and Airgap. Unstoppable Wallet offers a large selection of coins (mostly ERC20, BEP20, or BEP2 tokens), while Airgap only offers a smaller selection of coins. However Airgap offers a more advanced security, and the Airgap Vault can function as a hardware wallet. Airgap is also very good on making new entropies, so far having the best options to create our own entropies. Not that Airgap is the best in making it, I just find it very interesting.

Backing Up Your Wallets

It is worth noting, that, as mentioned before, BIP39 has its own flaws: it is not true that all the user needs to restore their wallet is just their seed phrase. Some developers across the industry continue to build wallets that either:

• Don’t implement BIP standard(s).

• Implement a BIP standard, but inconsistently when compared with other wallets.

• Implement a BIP standard, but one that has not been widely adopted (and perhaps only by them).

• Don’t have clear documentation about their derivation paths, backup and recovery processes.

Wallets come and go, information gets lost, and users are left with tears. Responsible wallet developers document external recovery. Users should not have to dig through the source code to figure out the Derivation Paths or Redeem Scripts.

WalletsRecovery.org

Know Your Wallet

Therefore, before settling on any wallet, you have to understand:

1. Whether they’re a deterministic wallet or not, and if yes, of what type? Things to look out for is whether or not they have an option to recover from seed phrases. If they do, they are most likely a deterministic wallet. Most wallets that I have tested tend to derive your addresses with BIP44, BIP49, or BIP84 standards. However you still have to do your own research. This is a good starting place for you to read.

2. How they are going to derive your addresses from your seed phrases? One thing to lookout for is their derivation path. In the WalletsRecovery page, we could learn that via the Supported Paths column on their tables. You are required to study them yourselves, but as a general rule of thumbs:

   • BIP44 standard tend to use m/44'/0'/0' + Custom, with their generated addresses starts with a 1 and might looks like 1PC9aZC4hNX2rmmrt7uHTfYAS3hRbph4UN (Bitcoin donation addresses for the Free Software Foundation);

   • BIP49 standard tend to use m/49'/0'/0' + Custom, and their addresses starts with a 3, and might looks like 3MaFikrngEw8YK6T5sxVoRmmYL3awgh846 (don’t send anything to this address, I don’t know who owns it, and is generated only for demonstrative purpose!);

   • BIP84 standard tend to use m/84'/0'/0' + Custom, and their addresses starts with a bc1, and is not case sensitive. Their addresses look like bc1qm9ykrnyuknzsrrp6na6pysu7vzezf5yzlzj5hu (donation address for Xenomancy.id :) feel free to transfer some funds there lol).

3. Whether they are single address wallets or wallets that can generate new receive addresses for each transactions. Some wallets like Airgap Wallet, Trust Wallet, and many more wallets use a single address for all transactions.

   This is an expected behavior in some chains whose transaction is processed in an account model, for example in Ethereum chain (ETH), or in Binance Smart Chain (BSC). They do that because you need native funds (ETH for ERC20 tokens, BNB for BEP20 tokens) in those addresses to move tokens on that chain to another address.

   Bitcoin and coins forked from Bitcoin like Litecoin and Dogecoin uses UTXO model (UTXO stands for Unspent Transactions), where you are expected to never reuse addresses. Wallets that don’t reuse addresses include: Unstoppable Wallet and BlueWallet.

   It is worth noting that in Unstoppable Wallet, BEP2, BEP20, and ERC20 tokens (along with their native currencies ETH and BNB) uses only a single address for reasons discussed before.

To test them, you can use a randomly generated seed phrases from here, and check their generated addresses in any coins of your choosing, and compare those addresses with addresses produced by your wallet. For safety reasons, don’t actually use seed phrases generated online from that page as your actual wallet and store funds within. If, for whatever reasons, you need to use your seed phrase in that web tool, consider doing it offline by downloading the tool from source and run them in an offline device.

WARNING: YOU ARE RESPONSIBLE TO KEEP YOUR OWN SEED PHRASE SAFE AND SECURE!

To test whether or not they reuse addresses, it’s a bit trickier, and you might have to dig deeper on their documentations. I for example, learned it only by trying to actually send and receive funds from those wallet apps. That is when I learned that Trust Wallet does not generate new addresses after every transaction. The same is also true for Airgap Wallet.

However, I discovered a rather curious behavior of Airgap Wallet. Airgap Wallet did not generate new addresses after receiving a transaction, however they can detect transactions of any receive and change addresses derived from a seed phrase. How did I obtain other addresses derived from that seed phrase in Airgap Wallet? I couldn’t, but it is possible to look out other addresses from Airgap Vault, via a feature known as Address Explorer.

Some wallets didn’t directly provide information on how to export wallets externally, but we can get hints on their feature lists. For example, Unstoppable Wallet explicitly described in their “Feature List” that they supports BIP 44/49/84/69. BIP 44/49/84 refers to address standards discussed before.

Securing Your Secrets

Then, after understanding how your wallet derive your addresses, and what addresses they generate, the next step is to know how to secure your seed phrase and seed extension. In short, storing your coins boils down to backing up your seed phrase, and storing them in a safe place, or on multiple safe places for backup. According to the article, storing bitcoins consists of few independent goals. All of which can be explored below:

1. Protection against accidental loss. Modern day deterministic wallets enabling us to secure our entire wallet just by storing the seed phrase securely. Preferrably, in multiple copies stored in multiple geographic locations for redundancy. That seed phrase can be used to restore your wallet and recover all of your funds.⁴

   However, since a seed phrase can store any large amount of money⁵, it makes little sense to keep them unprotected. Therefore, you should also further secure your seed phrase with a seed extension. Note that not all wallets support storing seed extensions.

   I think, it is also a common sense not to store your seed phrases along with their seed extension. It will literally be the same as storing non-protected seed phrases. Not storing your seed extension at all is also dangerous, as losing access to that is essentially the same as losing your seed phrase.

   To conveniently store your seed phrases and seed extension, consider storing them with pencil and paper. Then that paper should be stored in the dark while avoiding extreme heat and moisture. Some people might decide to store them in a stamped or engraved metal. Whatever your favorite way to store your seed phrase, make sure to keep the paper (or metal plates) safe and secret like cash or jewelry.

2. Verification that the bitcoins are genuine, and privacy and protection against spying. Storing your seed phrases only store the seed required to derive private keys. To learn about how many coins are stored within, we need a crypto wallet. Ideally, we should have a private full node that would independently verify incoming and outgoing transactions from our addresses. However it is impractical to do it nowadays, as per February 2022, the size of Bitcoin blockchain is 324 gigabytes.

   There are lightweight wallets that do not store the entire blockchain history, and confirms transactions by communicating with external full nodes. Lightweight wallets have their weaknesses, especially as they do not validate the rules of bitcoin, while sending addresses to third parties and receive wallet balance and history, while also skipping several security steps that can make their users vulnerable.

   If you value privacy, consider setting up a full node. However, if you value speed and minimal storage use, consider using light wallets. Practically all mobile wallets are lightweight wallets, simply because it is impractical to store the entire blockchain history in a mobile device.

3. Protection against theft. As discussed before, anyone with access of your keys, has access to your coins. The same is true with seed phrases and seed extension, as they provides a mean to derive all of your keys. It is not even about a physical possession of the seed phrases and seed extension, but also extends on the security of devices you use to manage your wallets.

   Therefore, you have to ensure that the wallet software running on your device, your device’s operating system and its hardware are safe and secure. Also worth noting that generally desktop computers lack specific mobile features such as Secure Storage and Biometrics to secure your secrets. Generally, a linux-based system is less vulnerable to malwares.⁶

4. Easy access for spending or moving bitcoins. It really depends on whether or not you need to move your coins often or not. If you need to move your coins around often, consider having a separate wallet of which a reasonable amount of funds that are expected to be moved around are stored within. The rest of your funds that don’t need to be moved around that often should be stored in a more secure wallet. Consider setting up a cold storage device, or having a hardware wallet that would secure your private keys and would not expose them to your hot devices.⁷

   A compromise to all above that I find useful is to repurpose an old android device as your own airgapped hardware wallet, an approach that is done by Airgap with their dual app model.

As a summary, this page cleverly explains as follows:

bitcoin wallets should be backed up by writing down their seed phrase, this phrase must be kept safe and secret, and when sending or receiving transactions the wallet software should obtain information about the bitcoin network from your own full node.

However, it is worth noting that we should also understand the nature of our wallets of choosing (see Know Your Wallet section).

This page is originally written for and is published on Static Blog Alpha at 11 August 2022. You can access the original version here.

BIP85 opens up the possibility of backing up only one seed, and use it to derive any future seeds. The operation is also just a one-way operation, any derived seeds cannot be used to find out its parent seed. Therefore, it simplifies backup processes a lot.

Our backup could, for example, only stores our seed phrase. Then, for any new derived seeds, we can just store the derived seed’s index. A backup of our parent seed, is then effectively also a backup of all of our keys.

There has been some uses in crypto wallet technologies. One of hardware wallet vendors that implement BIP85 is coldcard, at least as far as I discovered. One software wallet that does support deriving new seeds based on BIP85 that I know of is Airgap Vault.

In some cases, maybe you don’t want to rely on coldcard firmware or Airgap Vault to derive your child seeds. Therefore, you can also derive your seeds with the use of Ian Coleman’s Mnemonic Code Converter. Again, I am not sure how many times do I need to emphasize it, but:

EXPOSING YOUR SEED PHRASE TO A NON-AIRGAPPED DEVICE MIGHT COMPROMISE YOUR SEED PHRASE!¹

Airplane mode is available to most modern devices. Also, incognito mode that would delete all history and caches after the tab is closed is not hard to find in all good modern browsers, be it in mobile or desktop.

To use the Mnemonic Code Converter offline, you can either:

• load the page first in an incognito tab, then turn on airplane mode and use the tool;² or

• download the latest version of the tool’s releases, then copy it to an offline device³, and run the tool in an incognito mode.

However, it is worth noting that, unlike that advertised in the linked coldcard documentation, it is not true that all we need to back up our wallet is just our parent seed and its BIP85 derivation index. According to the BIP85 specification, the derived entropy is different according to its application number. In case of deriving child BIP39 seeds, the derivation path format is: m/83696968'/39'/{language}'/{words}'/{index}'

We also discovered that there are at least three factors to consider: language, words, and index. If you mostly use English-language wallets, then the language index would be 0'.⁴ For words, it refers to the number of words in your seed phrase, that could either be 12-words, 18-words, or 24-words. Meanwhile, index part is trivial: it’s just the derivation index of your mnemonics.

For example, if you want to derive a 12-word seed phrase from your seed phrase with index = 0, the derivation path would be: m/83696968'/39'/0'/12'/0'. For index = 1, the derivation path would be: m/83696968'/39'/0'/12'/1'. Meanwhile, if you want to derive a 24-word seed phrase instead, they would be: m/83696968'/39'/0'/24'/0' and m/83696968'/39'/0'/24'/1' instead.

Therefore, assuming English-language seed phrases,⁵ all you need to write down to back up all of your seed phrases is the following:

1. Your parent seed phrase. It encodes your original entropy, and without it, no entropy can be derived. Additionally, you also need your seed extension if you have one.⁶

2. The size of your derived seed phrase. It could be a 12-words, 18-words, or 24-words seed phrase.

3. Your child seed phrase index.

Assuming you just back up your seed phrases, without backing point 2 or point 3, there would be almost unlimited amount of combinations that you have to look out for.⁷ In coldcard, even though the index runs just from 0000 to 9999 with marketedly, “there are only 10,000 possible choices”, is not entirely correct. Given that each derived seed can either be in a 12, 18, or 24 words format, the actual number of possible choices are 30,000.⁸ For Airgap Vault, there can only be 30 possible choices though, as the app can only derive child seed phrases from index 0 to index 9.

Given all of the above as our considerations, and that assuming only English seed phrases, also that every tool we are going to use implements BIP85 standard properly, we can conveniently record our derived seed phrases by storing these four values: parent seed phrase, parent seed extension, words, and index.