Splitting Large Files using `split` command and `for` loop (for sequence)

Looping through and splitting sequentially named large files.

 

for i in {1..4}; do split -d -l 20000 Phase-2-Slot-$i.csv p2s$i\_ --additional-suffix=\.csv; done


End results

Input files:

Phase-2-Slot-1.csv
Phase-2-Slot-2.csv
Phase-2-Slot-3.csv
Phase-2-Slot-4.csv


Output files:

For Phase-2-Slot-1.csv -> output files are p2s1_00.csv, p2s1_01.csv, p2s1_02.csv so on

For Phase-2-Slot-2.csv -> output files are p2s2_00.csv, p2s1_02.csv, p2s1_02.csv so on

Take away Points:

• Escaping Underscore:
  
  The variable $i has been used in output file names and an underscores ( _ ) after it Neither the variable nor the underscore will be reflected in the output file name if the underscore is not escaped. The variable $i_ is valid and different from our $i. The work around is to escape the underscore ( _ ) after the variable $i . Note the escaped underscore in the above command  ( \_ )

• The sequential loop using for command
  
  Especially note the curly braces, (not parenthesis) and the two dots in between the integers.

If the leading integer is smaller than the trailing one, the counter will step up and vice versa.

for i in {a..b}  a < b := Step up counting
for i in {a..b}  a > b := Step down counting

Miscellaneous Linux Shell Commands

• Extract Postgres table list and re-execute sql command on all tables:

 

psql -U odoo13 odoo13 -h localhost -tqc '\dt' | awk -F "|" '{print $2}' | sed -r 's/[ ]*$//g'|xargs -i echo "select '{}', count(*) from {};"| psql -U odoo13 -h localhost
 

psql : Extract table names or a db

awk : Filtering result and extract (and print) table name

sed : Eliminate trailing space from table name

xargs and echo : generate desired sql [ Note {} ]

 

• Using shell for:

for table in `mysql table_name -Nse 'show tables;'`; do mysql table_name -e 'drop table '$table ; done

• Let's Encrypt certbot command to register domain and extend subdomain:

sudo certbot --nginx -d domain.com

sudo certbot --nginx -d domain.com,subdomain.domain.com --expand

 

•  DNS lookup for TXT and MX 

`dig -t txt ilyn.global +short`

`dig ilyn.global txt +short`

 

`dig -t mx ilyn.global +short`

`dig ilyn.global mx +short`

Removing "+short" will give full result

• Telnet a mail server and check if email user exists

 

Shell command: telnet gmail-smtp-in.l.google.com 25

Here 25 is the email server port

Email server commands after telnet is successful
 

helo <something>
mail from:<email@address>
rcpt to:<email@address> 

 The response will reveal if the user exists

Syncing two remote directories with rsync command

 rsync, a command that syncs the two remote directories.

Though straight forward by typing password, we my need to sync with key (PEM) files. We need to set remote shell and as parameter we can set the key file name.

An example:

Local to Remote:

rsync -rPav   -e 'ssh  -i ~/.ssh/id_rsa'  /local/source/directory/  user@remote_server:/remote/destination/directory/  

 

Remote to Local:

 rsync -rPav   -e 'ssh  -i ~/.ssh/id_rsa'  user@remote_server:/remote/source/directory/  /local/destination/directory/

 

Please note, the trailing slashes ( / ) in both source and destination directories.

To update (if source has newer) and delete parameters "u" and "--delete" need to use:

rsync -rPavu --delete

 

The things become even easier if we maintain a config file for ssh. For example, our .ssh/config file contains something like this:

HOST remote-machine
        IdentityFile ~/.ssh/id_rsa.pem
        Hostname our-remote-host.example.com
        User user

Then we can invoke like below:

Local to Remote:

rsync -rPav   -e 'ssh'  /local/source/directory/  remote-machine:/remote/destination/directory/  

 

Remote to Local:

 rsync -rPav   -e 'ssh'  remote-machine:/remote/source/directory/  /local/destination/directory/

JWT token in CORS session Cookie

This article is based on the experience of Nextjs frontend (a framework of react) and Nodejs backend with Express and cors middleware.

 

Without these settings in both front end and backend - session cookie will not be accepted (both by server and browser).

N.B. This is workable only in localhost or over HTTPS only. HTTP and trivial IP address other than localhost will not work.

 

 In front end, we need to set these 2 header options along with Ajax request ( native fetch or with axios ):

  credentials: 'include' 

and  

  mode: 'cors'  

 

 

In back end, we need to use cors library or middleware along with the parameter/option

  origin: <exact client url> 

and 

  credentials: true

 

The cookie setting header should include these options:

  sameSite: 'none',
  secure:      true,
  maxAge:    Cookie expiry time in milliseconds
  httpOnly:   true

 

 

We should use CSRF cooke library for further protection from CSRF attack.

Alternatively we can use Authorization header that bears the token manually saved in local storage (or browser cookie), and should take necessary protection from XSS attack.

HTTPS on localhost - for React and Laravel

Generating Self-signed Certificate: (Linux - ubuntu)

openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout ./selfsigned.key -out ./selfsigned.crt

(source)

Launching React app with the certificate/key: (Linux)

HTTPS=true SSL_CRT_FILE=./ssl/selfsigned.crt SSL_KEY_FILE=./ssl/selfsigned.key npm start

or we can set the variables in .env file

(source)

Launching Laravel on HTTPs: ( using ngrok )

after running: ( ➜ php artisan serve --host <IP_ADDRESS> )

 

 ./ngrok http <IP_ADDRESS>:<PORT>

(source) - (ngrok)

First lex program - sections of a lex code file

Lex installation in Windows ( gnu flex)