Standard Streams 2

Standard Streams 2

PIPE

In UNIX, pipe operation is to direct a standard output of a command to be the standard input of another command. The standard output of a program can be redirected to another program as a standard input, the standard error can only be redirected to a file.The "bc" program receives input from the keyboard. In processing this input, the generated errors are written to the screen via the standard error channel, and the results generated during the input process are also given to a pipeline instead of the standard output device monitor. The standard output of the "bc" output from the pipeline is given as standard input to the command "wc -l". "wc -l" processes the standard input from the pipeline and sends the errors to the screen via the standard error channel, while the normal results are sent to the same screen via the standard output channel. In this way, UNIX commands can be connected to the end-point to perform very meaningful and challenging tasks easily.

Example usage of pipe:

Example commands using pipe (|):

program1 | program2 : Standard output of the program1 is redirected to program2 using pipe.

program1 2> errors.txt | program2 : Standard output of the program1 is redirected to program2 using pipe. Also standard error of program1 is redirected to a file named errors.txt

The operators "<", ">", ">>", "2>", "2>&1" are used to direct input and output.

These operators are placed between a command set (command and parameters) and a file, and input and output are directed.

In output directories made using ">", the output file overwrites the file if it already exists. The ">>" operator can be used to prevent this and add it to the file.

If the file to which the output is directed does not exist, it is created automatically.

command > file.txt 2>&1 : both standard error and standard output are merged into the "file.txt" file.

"tee" Command

The "tee" command copies the standard input as if it were a standard output, and at the same time writes the standard input to the specified file.

In general, it is used to transfer the intermediate results of the commands connected to the end via the boring method.

The "tee" command overwrites the specified file under normal circumstances. However, if the contents of the file need to be lost, and the backend is desired, the "tee" command should be used with the "-a" option in the form "tee -a file".

To read my previous article: Standard Streams 1

Standard Streams 1

Standard Streams 1

What is standard input, output and error ?

Each UNIX program has a standard input, a standard output, and a standard error channel.

Programs can import entries from the standard input, as well as from the user or from a file.

If the output of the programs is written in such a way as to use the standard output, the user can easily transfer the output in a different environment that the user wishes, such as screen, printer, file.

Example: The "ls -l> file_list.txt" command writes the output of the running "ls" application to file file_list.txt.

Software that receives data from standard input, processes them, and directs output to standard output are also called "filters". Filter software can perform complicated operations when properly appended to each other.

Each program communicates over 3 channels.Standard INPUT channelStandard OUTPUT channelStandard ERROR channelEach communication channel has a system-assigned number. These numbers are 0 for standard input, 1 for standard output, and 2 for standard error.The program retrieves the information that the user wants from the standard input channel 0, ie under normal conditions, from the keyboard.The program generates input jobs and processed information from the user. This information is also displayed to the user from the standard output channel 1. Under normal circumstances, the program displays the processed information on the screen.The progam can make some mistakes while receiving input from the user. The standard error channel 2 is used to notify the user of unusual situations like this. Program errors are displayed to the user via the standard error channel unless otherwise specified. Errors that occur under normal conditions are displayed on the screen.Although the standard error and standard output device have the same screen, they can be separated easily because they are transmitted from different channels and they should be considered separately.

Normally, when a program is run, it writes its processed information to the screen on channel 1, which is the standard output channel.However, at the end of the program, when the characters "1>" and immediately following the file name are specified, the information written to the standard output channel is written to the specified file. Hence, the standard output channel is disconnected from the screen and the output of the program is directed to the file.Assuming the character ">" is used instead of "1>", the system assumes channel 1 is the standard output channel. In other words, if the channel number is not specified in the redirection operator, this is the standard output channel number 1.If the standard output made with the characters "1>" and ">" is directed to the file, if there is a file with the same name in the specified location, the contents are lost without any warning and the new content is written to the output file.If the output to be created is to be added behind an existing file, the "1 >>" or ">>" characters must be used to direct the file. The system understands that the standard output is appended to the end of the file, and adds the output generated by the program immediately after it does not damage the contents of the existing file.If the file does not exist, the file is automatically created by the system, directing it to the file by appending or overwriting it. 

Example Usage: 

program > file.txt
program 1> file.txt
program >> file.txt
program 1> file.txt  

Normally, when a program is run, the errors and errors that occur during data processing are written to the screen from channel 2, which is the standard error channel.However, at the end of the program, when the characters "2>" and immediately following the file name are specified, errors written to the standard error channel are written to the specified file. Therefore, the standard error channel is disconnected from the display and program errors are directed to the file.If the standard error made with the characters "2>" is redirected to the file, if there is a file with the same name in the specified location, the contents are lost without any warning and program errors are written to the file. If the error to be created is to be added behind an existing file, the "2 >>" characters must be used to redirect the file. The system understands that the standard error is to be appended to the end of the file and adds the error generated by the program immediately after it does not damage the contents of the existing file.If the file does not exist, the file is automatically created by the system, directing it to the file by appending or overwriting it.

Example Usage: 

program 2> file.txt
program 2>> file.txt

The "2> & 1" characters must be written at the end of a command in order for a program to combine standard output and standard error channels and write it on the same channel over a single channel.

It is pointless to merge standard output and standard error channels into a single channel as long as the resulting combined output is not redirected to a file.

The standard output and standard error channels combination operand "2> & 1" must be used after a standard output is directed to a file so that the standard output and standard error can be combined and directed to a file.

Example usage:

program 1> file.txt 2>&1
program 1>> file.txt 2>&1 

To read my previous article: Disk Management 2

Disk Management 2

Disk Management 2

Link Files

File linking can be created in UNIX file systems. The file is only available in one place on the file system, but can be accessed in many places only once. The link files are not functionally different from short-cuts in the MS-Windows environment. The goal is to provide users with unobtrusive access to files and directories.

The link files in the MS-Windows operating system are ordinary files and are interpreted by the shell (explorer.exe). When a different shell is used, the destination file can not be reached via the link files. Conversely, in UNIX operating systems, link files are a feature provided directly by the file system. Even if the shell changes in UNIX, the mechanism of link files can continue to work.

There are two types of link files in UNIX. Hard link and symbolic link. The file type of the symbolic link files is the character "l". The access authorities are also "rwxrwxrwx". When accessing the file with a link, the access control authority of the target file is valid, so access authorities of the link files are insignificant.

A link to the link file can also be defined. When you access a link file, all links are routed by the file system and finally the actual file is reached.

Archive Commands

tar (Tape ArchiveR) - archives files and directories. Archiving creates a single large archive file that can be saved on the tape media by adding all the files you have made.

"-c" means create and allows creation of archive file
"-t" allows the archive to be tested.
"-x" will open the archive.
"-v" means verbose, and lists the files in the archive when it is being created, tested, and opened.
"-f file" specifies that the archive in the file name is to be used. If the "-f" option is not used, the standard input device is used for opening and testing, and the standard output device is used for creating. 

Zip / unzip - Creates and opens zip files that we know quite well from the MS-Windows platform.
 
You can specify the compression level between zip [-1 ..- 9]. "-1" is the worst compression (fast), "-9" is the best compression (slow).

Compression Commands

compress - a compression tool that has been around forever in all UNIX's. It compresses the file given as parameter and saves it as ".z" extension, also deletes the original file. 

The behavior of gzip is the same as compress, but provides a much higher compression ratio than compress when the file extensions are ".gz".

The behavior of bzip2 is the same as compress and gzip but provides a much higher compression ratio than gzip while the file extensions are ".bz2". 

compress, gzip and bzip2 are not archive programs; They only compress.

GNU tar software also has the ability to compress gzip and bzip2 formats while it is being archived; The -z parameter can be used to compress gzip.

Apart from this software there are many more compression commands (7zip, lbzip2, xz, lrzip). You can choose to compress these software with different capabilities depending on the structure of the file.

To read my previous article: Disk Management 1

Disk Management 1

Disk Management 1

Disk Partition and File System

Disk partitioning allows multiple file systems to be created on a disk. Each logical disk partition hosts a file system. A file system allows files to be stored in a directory in an order. Definitions for required cataloging (required for quick access, location indicators, etc.) and access control (which file and how accessible) can also be performed on a file system basis.

Very different file systems have been developed and used for different purposes. File systems developed for multitasking systems often have different attributes than those developed for single-tasked operating systems. In a multitasking system it may be necessary to define special mechanisms at the file system level because of the possibility of processing the same file for more than one process at the same time. Similarly, file systems developed for multi-user systems are quite different from file systems designed and developed for single-user systems. The infrastructure that will allow access control rules to be defined in multi-user systems is provided by the file system.

An example of a file system is:
In Windows: NTFS (New Technology File System), FAT
In Linux: ext2, ext3, ext4...

File Systems

Many different file systems have been developed with the different needs that arise. The most common file systems for Linux are ext2, ext3 and raiserfs file systems. Besides, it can be used under Linux with up to 20 file systems including fat, vfat, ntfs, is9660 filesystems.

Ext3 and raiserfs file systems are based on transaction logging as well as advanced database management systems. Due to these attributes, these file systems have high reliability. It is not possible to lose data easily. In the event of power failures and unexpected shutdowns, the file system correctly and successfully stores information that can restore itself to its last good state.

Many current Linux distributions, including Suse Linux, use the ext3 file system by default.

File systems should be checked for consistency before use. Often, boot scripts automatically do this for file systems that are used at boot time.  

File size and file system limits can be inspected from figure below:

Disk Partitioning

One of the applications that can be used for partitioning under Linux is fdisk. Another widely-used alternative to fdisk is DiskDruid. These applications are basically used to separate multiple logical partitions of a physical disk.

The fdisk program runs briefly and gives a brief usage information. A disk device file must be supplied to fdisk. The disk device files are "/dev/hd[abcd]" for the IDE interface and /dev/sd[abcdefgh] for the SCSI interface.

Mounting File Systems

Once a partition is defined and a file system is installed through the mkfs program, this file system can be used by connecting to any directory in the UNIX directory tree structure. The so-called mounting is to use the files in the specified file system when attempting to reach a particular directory.  

The mount program is used to link file systems. From the moment a directory is linked to a file system, every user that goes through this directory will see the files provided by the new file system. If there are saved files in this directory before linking, these files become invisible / unreachable. When the connected file system leaves (unmount) the files will be accessible again. 

A file system that is actively used (for example, the file in which it is being read or written) can not be separated from the directory structure. Programs that use the file system must be terminated.

Some of the current distributions come with an integrated Linux kernel patch called 'supermount' or 'automount'. With this application, removable media can be mounted or unmounted at the time of access. For example, when a user inserts a CD media into the CD-ROM drive of the system, the drive is automatically connected, and when the eject button is pressed, the connection is resolved.

Disk Usage Information

The occupancy information of file systems and files/directories can be accessed through the df and du programs.   

Example usage of df command is given in the figure below:

df (Disk Free) - Indicates occupancy information of all connected file systems. In the example, the "-k" option is used to display the values in KB units.

du - Estimates file space usage.

To read my previous article: File Permissions and Access Rights 2

File Permissions and Access Rights 2

File Permissions and Access Rights 2

Special Access Rights Setuid and Setgid

To allow a user who is not authorized to access a file to temporarily access the file, the setuid and setgid programs are authorized to execute the programs. Setuid works with the authority of the user who owns the program for the duration of the program. A program marked with setgid works with the authority of the group that owns it for the duration of the program.

To give setuid access: "chmod u+s file_name"
To give setgid access: "chmod g+s file_name"

The "setuid" and "setgid" private study authorities apply only to compiled program files. The setuid or setgid run permission for a shell program is not different from the normal run permission.

Sticky "t" is a special case for the privileged directories. A directory with the "t" privilege can be written by all users, but can not be deleted or updated other users' writes (unless they are given write permissions on a file basis). However, a directory with an access rights of "rwxrwxrwx" is writable by everyone and they can also delete others' files.



It would not have been possible for ordinary users to change their passwords stored in /etc/passwd and /etc/shadow without the setuid scheme. Since the information stored in these files can only be accessed by the root user, the passwd program is setuid as root.

As it can be seen in the example above owner of the /etc/passwd file is the root user. However binary file located at /usr/bin/passwd has the setuid access right which means anyone can execute this file with the root authority.

For security reasons, setuid/setgid should not be done despite all these benefits, unless it is needed.

Sticky 't' Right

Sticky bit prevents other users to delete a file created by another user in the temporary file writing directories required by all user programs. In such directories, no user can delete another user's file or directory through sticky special authority, even though the user, group and others have all the permissions (read, write, execute). A temporary file created by a user in a directory such as "/tmp" without the "Sticky" special authority could easily be deleted by another user for the reason that all the permissions were granted to everyone else.

Setuid, setgid and sticky bit is set just like regular file permissions. For setuid we use 4, for setgid we use 2 and for sticky bit we use 1. For example:

All rights are given to a file including setuid, setgid and sticky bit:
chmod 7777 example_file

A file is given read permission to owner and group only and has sticky bit:
chmod 1440 example_file

Changing Owner and Group of a File/Directory

"chown" command is used for changing the owner of a file or directory and "chgrp" command is used for changing the group of a file or directory.


If the "chown" or "chgrp" command is used with the "-R" option, the entire directory and filename under the command directory will be applied if the given parameter is an directory. If the "-R" option is not given, the command is applied only to the directory/file given as parameter.

chown -R newOwner:newGroup directory_name: This command sets newOwner and newGroup as the owner and group of every file and directory under directory_name recursively.

Ordinary users can not use the "chown" command. If they change ownership of a file of their own, they lose their authority over the file and can not take ownership of the file without the help of "root". The "chown" command can not be used by ordinary users to prevent this, whereas ordinary users have limited use of the "chgrp" command.

To read my previous article: File Permissions and Access Rights 1