Standard Streams 2

Standard Streams 2

PIPE

In UNIX, pipe operation is to direct a standard output of a command to be the standard input of another command. The standard output of a program can be redirected to another program as a standard input, the standard error can only be redirected to a file.The "bc" program receives input from the keyboard. In processing this input, the generated errors are written to the screen via the standard error channel, and the results generated during the input process are also given to a pipeline instead of the standard output device monitor. The standard output of the "bc" output from the pipeline is given as standard input to the command "wc -l". "wc -l" processes the standard input from the pipeline and sends the errors to the screen via the standard error channel, while the normal results are sent to the same screen via the standard output channel. In this way, UNIX commands can be connected to the end-point to perform very meaningful and challenging tasks easily.

Example usage of pipe:

Example commands using pipe (|):

program1 | program2 : Standard output of the program1 is redirected to program2 using pipe.

program1 2> errors.txt | program2 : Standard output of the program1 is redirected to program2 using pipe. Also standard error of program1 is redirected to a file named errors.txt

The operators "<", ">", ">>", "2>", "2>&1" are used to direct input and output.

These operators are placed between a command set (command and parameters) and a file, and input and output are directed.

In output directories made using ">", the output file overwrites the file if it already exists. The ">>" operator can be used to prevent this and add it to the file.

If the file to which the output is directed does not exist, it is created automatically.

command > file.txt 2>&1 : both standard error and standard output are merged into the "file.txt" file.

"tee" Command

The "tee" command copies the standard input as if it were a standard output, and at the same time writes the standard input to the specified file.

In general, it is used to transfer the intermediate results of the commands connected to the end via the boring method.

The "tee" command overwrites the specified file under normal circumstances. However, if the contents of the file need to be lost, and the backend is desired, the "tee" command should be used with the "-a" option in the form "tee -a file".

To read my previous article: Standard Streams 1

Standard Streams 1

Standard Streams 1

What is standard input, output and error ?

Each UNIX program has a standard input, a standard output, and a standard error channel.

Programs can import entries from the standard input, as well as from the user or from a file.

If the output of the programs is written in such a way as to use the standard output, the user can easily transfer the output in a different environment that the user wishes, such as screen, printer, file.

Example: The "ls -l> file_list.txt" command writes the output of the running "ls" application to file file_list.txt.

Software that receives data from standard input, processes them, and directs output to standard output are also called "filters". Filter software can perform complicated operations when properly appended to each other.

Each program communicates over 3 channels.Standard INPUT channelStandard OUTPUT channelStandard ERROR channelEach communication channel has a system-assigned number. These numbers are 0 for standard input, 1 for standard output, and 2 for standard error.The program retrieves the information that the user wants from the standard input channel 0, ie under normal conditions, from the keyboard.The program generates input jobs and processed information from the user. This information is also displayed to the user from the standard output channel 1. Under normal circumstances, the program displays the processed information on the screen.The progam can make some mistakes while receiving input from the user. The standard error channel 2 is used to notify the user of unusual situations like this. Program errors are displayed to the user via the standard error channel unless otherwise specified. Errors that occur under normal conditions are displayed on the screen.Although the standard error and standard output device have the same screen, they can be separated easily because they are transmitted from different channels and they should be considered separately.

Normally, when a program is run, it writes its processed information to the screen on channel 1, which is the standard output channel.However, at the end of the program, when the characters "1>" and immediately following the file name are specified, the information written to the standard output channel is written to the specified file. Hence, the standard output channel is disconnected from the screen and the output of the program is directed to the file.Assuming the character ">" is used instead of "1>", the system assumes channel 1 is the standard output channel. In other words, if the channel number is not specified in the redirection operator, this is the standard output channel number 1.If the standard output made with the characters "1>" and ">" is directed to the file, if there is a file with the same name in the specified location, the contents are lost without any warning and the new content is written to the output file.If the output to be created is to be added behind an existing file, the "1 >>" or ">>" characters must be used to direct the file. The system understands that the standard output is appended to the end of the file, and adds the output generated by the program immediately after it does not damage the contents of the existing file.If the file does not exist, the file is automatically created by the system, directing it to the file by appending or overwriting it. 

Example Usage: 

program > file.txt
program 1> file.txt
program >> file.txt
program 1> file.txt  

Normally, when a program is run, the errors and errors that occur during data processing are written to the screen from channel 2, which is the standard error channel.However, at the end of the program, when the characters "2>" and immediately following the file name are specified, errors written to the standard error channel are written to the specified file. Therefore, the standard error channel is disconnected from the display and program errors are directed to the file.If the standard error made with the characters "2>" is redirected to the file, if there is a file with the same name in the specified location, the contents are lost without any warning and program errors are written to the file. If the error to be created is to be added behind an existing file, the "2 >>" characters must be used to redirect the file. The system understands that the standard error is to be appended to the end of the file and adds the error generated by the program immediately after it does not damage the contents of the existing file.If the file does not exist, the file is automatically created by the system, directing it to the file by appending or overwriting it.

Example Usage: 

program 2> file.txt
program 2>> file.txt

The "2> & 1" characters must be written at the end of a command in order for a program to combine standard output and standard error channels and write it on the same channel over a single channel.

It is pointless to merge standard output and standard error channels into a single channel as long as the resulting combined output is not redirected to a file.

The standard output and standard error channels combination operand "2> & 1" must be used after a standard output is directed to a file so that the standard output and standard error can be combined and directed to a file.

Example usage:

program 1> file.txt 2>&1
program 1>> file.txt 2>&1 

To read my previous article: Disk Management 2

Disk Management 2

Disk Management 2

Link Files

File linking can be created in UNIX file systems. The file is only available in one place on the file system, but can be accessed in many places only once. The link files are not functionally different from short-cuts in the MS-Windows environment. The goal is to provide users with unobtrusive access to files and directories.

The link files in the MS-Windows operating system are ordinary files and are interpreted by the shell (explorer.exe). When a different shell is used, the destination file can not be reached via the link files. Conversely, in UNIX operating systems, link files are a feature provided directly by the file system. Even if the shell changes in UNIX, the mechanism of link files can continue to work.

There are two types of link files in UNIX. Hard link and symbolic link. The file type of the symbolic link files is the character "l". The access authorities are also "rwxrwxrwx". When accessing the file with a link, the access control authority of the target file is valid, so access authorities of the link files are insignificant.

A link to the link file can also be defined. When you access a link file, all links are routed by the file system and finally the actual file is reached.

Archive Commands

tar (Tape ArchiveR) - archives files and directories. Archiving creates a single large archive file that can be saved on the tape media by adding all the files you have made.

"-c" means create and allows creation of archive file
"-t" allows the archive to be tested.
"-x" will open the archive.
"-v" means verbose, and lists the files in the archive when it is being created, tested, and opened.
"-f file" specifies that the archive in the file name is to be used. If the "-f" option is not used, the standard input device is used for opening and testing, and the standard output device is used for creating. 

Zip / unzip - Creates and opens zip files that we know quite well from the MS-Windows platform.
 
You can specify the compression level between zip [-1 ..- 9]. "-1" is the worst compression (fast), "-9" is the best compression (slow).

Compression Commands

compress - a compression tool that has been around forever in all UNIX's. It compresses the file given as parameter and saves it as ".z" extension, also deletes the original file. 

The behavior of gzip is the same as compress, but provides a much higher compression ratio than compress when the file extensions are ".gz".

The behavior of bzip2 is the same as compress and gzip but provides a much higher compression ratio than gzip while the file extensions are ".bz2". 

compress, gzip and bzip2 are not archive programs; They only compress.

GNU tar software also has the ability to compress gzip and bzip2 formats while it is being archived; The -z parameter can be used to compress gzip.

Apart from this software there are many more compression commands (7zip, lbzip2, xz, lrzip). You can choose to compress these software with different capabilities depending on the structure of the file.

To read my previous article: Disk Management 1

Disk Management 1

Disk Management 1

Disk Partition and File System

Disk partitioning allows multiple file systems to be created on a disk. Each logical disk partition hosts a file system. A file system allows files to be stored in a directory in an order. Definitions for required cataloging (required for quick access, location indicators, etc.) and access control (which file and how accessible) can also be performed on a file system basis.

Very different file systems have been developed and used for different purposes. File systems developed for multitasking systems often have different attributes than those developed for single-tasked operating systems. In a multitasking system it may be necessary to define special mechanisms at the file system level because of the possibility of processing the same file for more than one process at the same time. Similarly, file systems developed for multi-user systems are quite different from file systems designed and developed for single-user systems. The infrastructure that will allow access control rules to be defined in multi-user systems is provided by the file system.

An example of a file system is:
In Windows: NTFS (New Technology File System), FAT
In Linux: ext2, ext3, ext4...

File Systems

Many different file systems have been developed with the different needs that arise. The most common file systems for Linux are ext2, ext3 and raiserfs file systems. Besides, it can be used under Linux with up to 20 file systems including fat, vfat, ntfs, is9660 filesystems.

Ext3 and raiserfs file systems are based on transaction logging as well as advanced database management systems. Due to these attributes, these file systems have high reliability. It is not possible to lose data easily. In the event of power failures and unexpected shutdowns, the file system correctly and successfully stores information that can restore itself to its last good state.

Many current Linux distributions, including Suse Linux, use the ext3 file system by default.

File systems should be checked for consistency before use. Often, boot scripts automatically do this for file systems that are used at boot time.  

File size and file system limits can be inspected from figure below:

Disk Partitioning

One of the applications that can be used for partitioning under Linux is fdisk. Another widely-used alternative to fdisk is DiskDruid. These applications are basically used to separate multiple logical partitions of a physical disk.

The fdisk program runs briefly and gives a brief usage information. A disk device file must be supplied to fdisk. The disk device files are "/dev/hd[abcd]" for the IDE interface and /dev/sd[abcdefgh] for the SCSI interface.

Mounting File Systems

Once a partition is defined and a file system is installed through the mkfs program, this file system can be used by connecting to any directory in the UNIX directory tree structure. The so-called mounting is to use the files in the specified file system when attempting to reach a particular directory.  

The mount program is used to link file systems. From the moment a directory is linked to a file system, every user that goes through this directory will see the files provided by the new file system. If there are saved files in this directory before linking, these files become invisible / unreachable. When the connected file system leaves (unmount) the files will be accessible again. 

A file system that is actively used (for example, the file in which it is being read or written) can not be separated from the directory structure. Programs that use the file system must be terminated.

Some of the current distributions come with an integrated Linux kernel patch called 'supermount' or 'automount'. With this application, removable media can be mounted or unmounted at the time of access. For example, when a user inserts a CD media into the CD-ROM drive of the system, the drive is automatically connected, and when the eject button is pressed, the connection is resolved.

Disk Usage Information

The occupancy information of file systems and files/directories can be accessed through the df and du programs.   

Example usage of df command is given in the figure below:

df (Disk Free) - Indicates occupancy information of all connected file systems. In the example, the "-k" option is used to display the values in KB units.

du - Estimates file space usage.

To read my previous article: File Permissions and Access Rights 2

File Permissions and Access Rights 2

File Permissions and Access Rights 2

Special Access Rights Setuid and Setgid

To allow a user who is not authorized to access a file to temporarily access the file, the setuid and setgid programs are authorized to execute the programs. Setuid works with the authority of the user who owns the program for the duration of the program. A program marked with setgid works with the authority of the group that owns it for the duration of the program.

To give setuid access: "chmod u+s file_name"
To give setgid access: "chmod g+s file_name"

The "setuid" and "setgid" private study authorities apply only to compiled program files. The setuid or setgid run permission for a shell program is not different from the normal run permission.

Sticky "t" is a special case for the privileged directories. A directory with the "t" privilege can be written by all users, but can not be deleted or updated other users' writes (unless they are given write permissions on a file basis). However, a directory with an access rights of "rwxrwxrwx" is writable by everyone and they can also delete others' files.



It would not have been possible for ordinary users to change their passwords stored in /etc/passwd and /etc/shadow without the setuid scheme. Since the information stored in these files can only be accessed by the root user, the passwd program is setuid as root.

As it can be seen in the example above owner of the /etc/passwd file is the root user. However binary file located at /usr/bin/passwd has the setuid access right which means anyone can execute this file with the root authority.

For security reasons, setuid/setgid should not be done despite all these benefits, unless it is needed.

Sticky 't' Right

Sticky bit prevents other users to delete a file created by another user in the temporary file writing directories required by all user programs. In such directories, no user can delete another user's file or directory through sticky special authority, even though the user, group and others have all the permissions (read, write, execute). A temporary file created by a user in a directory such as "/tmp" without the "Sticky" special authority could easily be deleted by another user for the reason that all the permissions were granted to everyone else.

Setuid, setgid and sticky bit is set just like regular file permissions. For setuid we use 4, for setgid we use 2 and for sticky bit we use 1. For example:

All rights are given to a file including setuid, setgid and sticky bit:
chmod 7777 example_file

A file is given read permission to owner and group only and has sticky bit:
chmod 1440 example_file

Changing Owner and Group of a File/Directory

"chown" command is used for changing the owner of a file or directory and "chgrp" command is used for changing the group of a file or directory.


If the "chown" or "chgrp" command is used with the "-R" option, the entire directory and filename under the command directory will be applied if the given parameter is an directory. If the "-R" option is not given, the command is applied only to the directory/file given as parameter.

chown -R newOwner:newGroup directory_name: This command sets newOwner and newGroup as the owner and group of every file and directory under directory_name recursively.

Ordinary users can not use the "chown" command. If they change ownership of a file of their own, they lose their authority over the file and can not take ownership of the file without the help of "root". The "chown" command can not be used by ordinary users to prevent this, whereas ordinary users have limited use of the "chgrp" command.

To read my previous article: File Permissions and Access Rights 1

File Permissions and Access Rights 1

File Permissions and Access Rights 1

File and Directory Access Reviews

In UNIX, all kinds of system resources have the rights to read, write and execute on defined users and groups. The file attributes configuration determines who can read, write, and execute a resource.
 
By default, a user has all rights to all resources under his or her home directory.

A special UNIX user, 'root', has all rights over all system resources.

Above "ls -l" is shown where the owner of the file, the group, and the access authorities are located.

The first character on the left is the file type ("-" ordinary file, "l" link, "d" directory, etc.)

The next 3 characters are the access authority of the owner of the file
The following 3 characters are the access authority of the group of the file
The last 3 characters are the access authorities of the others
 
The * mark on the right side of the file names indicates that the file is executable by the current user. This is a feature of the ls program running under Linux.

As can be understood from the examples, there are three kinds of authority, and these authorities read, write and operate with the order.

If there is a letter in the authorization definition, the corresponding authorization is given. If there is a "-" sign, the authorization is not given.

When three of these definitions of authority come side by side, the authority of the owner, group and others of the file can be determined.

For example:"---------" means that no one has any authority"r --------" means that the owner only has the authority to read, the same group and others have no authority over the file"rw-rw-r--" means that the owner and the group have the authority to read and write, and others are only authorized to read"rwx-r-xr-x" indicates that the owner has read, write and execute authority, the group has authority to read and execute the group, and others have read and execute authority.

The writing authority defines the ability to make changes and delete at the same time. The person who is authorized to write to a file can be deleted.

Access to files and directories is regulated by the "chmod" command.

"chmod u+x hidden_file" - gives the right to run the "hidden_file" file. 

"chmod ug = rw important_file" - only the read and write authority is given to the owner and group of the "important_file" file, and the authority of the other does not change.

Multiple authorizations can be made at the same time, separated by commas.

"chmod ug+rw, ug-x, o-rwx bigfile" - the owner and group of the "bigfile" file are given read and write permissions, the owner of the file and its group are authorized to run, and no authority is given to the others.

The special powers of stick, setuid and setgid will be discussed later.

The access authority of the file can be given in octal numbers for chmod command.

You need to use 4 for "r", 2 for "w", 1 for "x" and finally 0 for "-". The figures corresponding to the authorities are written and collected in groups of 3. The 3-digit number is the number of authority.

For example:

"-rw-r-x---"        means (4+2+0) + (4+0+1) + (0+0+0) = 650
"-rwx-wx--x"     means (4+2+1) + (0+2+1) + (0+0+1) = 731
"-rwxrwxrwx"  means (4+2+1) + (4+2+1) + (4+2+1) = 777

We can change the access rights of any file by both following ways (they both mean the same):

chmod +rwx any_file.ext
chmod 777 any_file.ext

The permissions will be "-rwxrwxrwx".

To read my previous article: File and Directory Commands

File and Directory Commands

File and Directory Commands

UNIX Command Format

In UNIX, commands consist of a single word. The second word that comes next to the command is processed as a parameter. Example: "cd /home/user"

Options are given with the "-" sign. More than one option can be written to a command individually, or it can be written by a single "-" sign.
Example: "ls -l -a" and "ls -la" produce the same result.

The GNU version also supports long option input.
Example: "mail -h" and "mail --help" give the same output.

Option parameters are used immediately after the option with or without spaces. In the GNU version of the commands, the parameters of the long-style options are given after the "=" character after the option.

For example: "ls -w 160", "ls -w160" and "ls --width = 160" are the same. The parameter "-w" is "160".

The parameters of the commands are given after the options.
Example: "ls -w 160 directory1 directory2" The command parameters are "directory1" and "directory2".

To execute the commands at the same time on a single line, you can put a ";" between them.
Example: "configure; make; make install "

"&&" field means "and". If the command on the left is successful, the command on the right is executed, otherwise it is not.
Example: ls directory && cd directory

However, if the "ls" command lists the "directory" directory, it will run "cd directory". Any attempt to enter a non-existent directory will be blocked.

"||" field means "or". If the command on the left is successful, the command on the right is not executed.

Example: ls directory || mkdir directory

If the "ls" command can not list the "directory", the "directory" is defined with "mkdir directory". The directory "directory" may be listed by "ls", but it will not be attempted to be defined with mkdir because the "directory" already exists.

mkdir (MaKe DIRectory) is used to create a new directory, rmdir (ReMove DIRectory) is used to delete an empty directory.
The command 'rm -rf directory_name' can be used to delete a content-rich directory with all content.

If the UNIX commands can run in error-free format, the user will not be informed that they have successfully run and finished. Feedback is given when an error occurs, only.



On older Linux systems, when running "rm -rf /" as root, all files are deleted from the system until there are no more files left. Current linux systems have been taken care of within the rm command This should be considered as a further indication of the need to work with root identity only when necessary.

Some useful commands combined with parameters:

touch <file_name>: It creates a file named "file_name", if it already exists then it updates the date modified.

rm <file_name>: Deletes file named "file_name".

rm -f <file_name>: Deletes file named "file_name" without asking permission.

rm -i <file_name>: Deletes file named "file_name" with asking permission.

rm -r <directory_name>: Deletes the directory and all the contents of it recursively.

rm -rf <directory_name>: Deletes the directory and all the contents of it recursively without asking permission.

In UNIX, there is no such thing as a file extension like Windows. A file name can have more than one dot ("."), Which is often used to make the file name more meaningful. Parts separated by periods, do not have to carry information about the format of the file.

For this reason, a file ending in ".txt" can be either an executable file or an image file.

Copy, Move and File Commands

The cp command is used to copy files and directories. Common forms of use are:

cp <source> <target>: copies the file/directory specified as "source" to the destination specified as "target"

cp -r <source> <target>: copy the source directory and all files and directories under it to the target directory. If target does not exist, creates it.

cp -r -p <source> <target>: copy the source directory and all files and directories under it to the directory target. If target does not exist, creates it. In addition, copy files and directories with protection of their access rights.

Mv <source> <target>: Moves the file/directory specified as "source" to the location specified as "target". This command is also used to change the file/directory name (mv <old_file_name> <new_file_name>).  

An existing file can be overwritten as a result of cp and mv operations. In some cases unwanted information loss may occur.

With the file command we have information about the type of file. Example:

Pattern Matching

In UNIX, pattern matching is a very useful and powerful feature. In many places it makes operations much easier.

* - replace any character string or nothing.

? - is used in place of any character. 
[] - The characters in the range are used instead of one character.

ls page * - fetch all the directories/files starting with "page" directory/file. 
ls page?.txt - fetch all directories/files starting with "page", continuing with any character and ending with ".txt". 
ls page??.txt - fetch all directories/files starting with "page", followed by any two characters and ending with ".txt".

ls page?[0-2].txt - Fetch all directories/files starting with "page", continuing with any character, continuing with 0, 1, 2 characters and ending with ".txt". 
ls page*[13579].txt - Fetch all directories/files starting with "page", followed by a single odd digit and ending with ".txt".

Example Usage of "file" Command

find /usr/include -name "stdio.h": Search for the file named "stdio.h" in the directory "/usr/include" and all directories under it.

find / home -name "*. [Ii] [Ss] [Oo]": Search all files in .iso, .iso, .iSo ... in all directories under /home.

find / -type l: search all link files from the root

find /home/user -user root: search the files owned by the root user in all directories under the /home/user directory.

find /home -uid 500: Looks for the files owned by the user with user id 500 in all directories below the /home directory.

find /home -size +2048M: Look for files larger than 2 GB in the /home directory and all directories under it.

find /home -size + 10240k: Look for files larger than 10MB in the /home directory and all directories under it.

find / -type f -mmin -90: Search for modified files in the last 90 minutes. 
find / -type f -mmin +90: Searches for files that have been "modified" before the last 90 minutes.

To read my previous article: Change of Active Identity and PAM