Everything is a file, know how to cope with them.

File management is about organizing, copying, moving, deleting and renaming files.

To understand file management, one must first understand the filesystem layout and how users and groups are used to set permissions on files and directories.

The Filesystem

A file system (FS) is a complex data structure that dictates the way files are stored on a physical medium. It exists in several flavours, some of them you may already know: FAT, NTFS, ext4, HFS, ISO 9660, …​ There is no ideal FS, all have pros and cons. The FS you usually use, whether you know it or not, depends on the operating system (OS) you’re on. Nowadays, Windows OSes rely on the NTFS file system, while Linux OSes mostly use ext4 and MacOS has its own APFS to store files.

From a user perspective, the FS is where files and directories are stored (by the way, a directory is nothing but a special file). This leads to a layout specific to each OS. The Linux OS inherits the Unix FS layout where all is organized under a unique root directory called / (slash).

This is not the place to list and explain the Linux FS organisation, but we must mention the most commonly found directories and their purposes.

/bin, /sbin, /usr/bin, /usr/sbin

directories where system commands reside. You can find out in which directory a command is located by using the which command.

/root

homedir of the superuser.

/etc

most of the system wide configuration files are stored there.

/home

directory containing the home directories of regular users.

/lib, /usr/lib

system wide shared libraries.

Browsing the filesystem

When you have no graphical interface to show you where you are and no mouse to click the files or directory you want to select, you have to do things the classic way.

Example 1. A happy little journey
Commands used

cat, cd, echo, ls, man, pwd

  1. Start your terminal application (whether it is WSL on Windows, Terminal on MacOS or some terminal under Linux).

    This note is for MacOS users only.

    Chances are the default shell on your operating system is Zsh. You need to switch to Bash as those two shells don’t behave exactly the same way. To do so, issue the following command in your terminal:

    $> chsh -s /bin/bash

    If this command does not work, you can change the default shell on your operating system following the instructions in Change the default shell in Terminal on Mac.

  2. Find out where you are located in the filesystem right after you log in.

  3. Make your computer display Hello on the terminal.

  4. Change directory to /tmp (this is a system wide directory to store temporary files) and list its content.

  5. You can list content in several ways. Reading the manual of the ls command, figure out how to display the list of files

    1. in long format;

    2. sorted by time, reversed;

    3. both at the same time.

  6. Travel to the / directory and get the list of all files and directories located at the root of the filesystem.

  7. Go back to your home directory (the one you where in when you first logged in). There are several ways to do this. Try to find at least two.

  8. From there, display the content of the file named /etc/services

You’ve just learned the basic commands to move around the FS (cd) and get the content of a directory (ls) or a file (cat). Another major command you’ve used is man. All commands are heavily documented through manual pages.

Local file management

Most commands you will be using are related to file management. Very often, you will have to list directory contents, change the current directory, copy files and directories, move or rename files, remove files or directories. Organizing content often requires to create directories and to get information on files.

You have certainly figured out already that most files are text files you may want to read, display on screen or edit (more on that later as there is an entire section dedicated to text editors). These files, amongst others, are really easy to compress and if you want to share a bunch of them with colleagues, you may want to create an archive to send them all at once.

These operations are commonly done using a graphical file manager but can be achieved using the CLI as well.

Some actions are easier to achieve using the CLI, like finding a file by its attributes or searching a file for specific content. You may also want to compare files line by line to find out if they differ.

Example 2. Local file management
Commands used

cat, cd, cp, file, find, ls, man, mkdir, mv, pwd, wc

  1. Inside your homedir, create a directory called LFM and make it your current directory. You can check you reached the right destination using which command?

  2. Without changing directory, find all the files in /usr/share/doc with a name starting with README and a size between 2 and 10 kilobytes.

  3. Copy one of them to your current directory (the one from gzip would be great if it appears in the list, but anyone will do). If you are on a Mac and the /usr/share/doc directory does not exist, use any README file on your system whose size is less than 10kB.

  4. Find out how many characters, words and sentences this file contains (you may want to define what a sentence is).

    • Create a file named rubish.docx by echoing some text in it (using a redirection we will explain later):

      $> cat > rubish.docx

      Type anything you want (it may make some sense to type something useful) and end your input by hitting kbd:[Ctrl+D] (this is keyboard shortcut to inputting EOF — End Of File).

  5. What size is your new file? You can find out using two different easy ways (just use simple well known commands). Give them both, explaining your choice.

  6. What kind of app can open this file? What is its type?

  7. Print the content of this file in the terminal. Does it contain exactly what you typed in earlier?

  8. Create a directory named temp inside the LFM folder and move the rubish.docx file there, changing its name to rubish.txt.

Remote file management

Working remotely in a connected world

Everything you can do locally about file management can be done remotely (on a distant computer) through a remote login.

Due to increasingly security concerns, the only relevant way to login remotely nowadays is using SSH (the secure shell). It creates an encrypted tunnel from your local computer to the remote host. Of course, to be able to log remotely, you must have a user account on the distant computer.

Using the SSH protocol, it is also easy to copy files from and to a remote computer or to synchronize directories. But, as most of the servers are HTTP based, you can as well use command line browsers to get and put files there.

Example 3. Remote file management
Commands

cd, curl, diff, echo, man, mkdir, rsync, scp, ssh, wc, wget

  1. Using ssh log into the remote computer named ens-srv-eftcl.emse.fr (use credentials from the school).

Unfortunately, this machine is only accessible from the school (wired network, eduroam WiFi network and emse-invite WiFi network).

  1. On the remote computer, create a RFM directory under your homedir and make it you current directory. Then, using command line tools only, download the file https://ci.mines-stetienne.fr/cps2/course/cls/assets/data/lorem.ipsum and name it lorem_ipsum.txt.

  2. Make a copy of this file named place holder.txt (mind the white space in the file name and deal with it), then find out how many words and sentences it contains (this time you may consider a sentence as a part of the text ending with the . character).

  3. From another terminal, copy the local LFM/README to your remote RFM directory and compare it with the same file on the distant system if it exists. If the file on one side is compressed, you must decompress it before making the actual comparison.

  4. Synchronize the temp directory and its content with the remote computer (put it in the RFM folder).

  5. Make some changes to the content of the tmp folder (add files, make changes to files already there, …​) of this hierarchy and try to synchronize again.

  6. Then, make some changes remotely and synchronize the other way (from remote to local).

If you are not comfortable yet with command line text editors, you can simply add content to a text file by echoing something at the end of it, using redirections.
For example, echo "Another line at the end." >> file.txt.