Block Busting

Hello, I'm Kundu 🍀

https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/dns-failover.html

Goal

- Customer incoming through Nginx(NOT Nginx Plus), and Nginx doesn't support failover like ha-proxy. 

   I'd like to use dns-failover for stable service.

Setup

[Route53] - [Health checks] 

1. Set your Name 
2. Select what to monitor, I select Endpoint
3. Endpoint setup
   1. Public IP address for each Nginx
   2. Host name (I add this '/etc/nginx/conf.d/443_health.conf' --> server_name health.mydomain.com)
   3. HTTPS (443)
   4. Path for Nginx (in my case: /etc/nginx/html/status/health.html) 
   5. set health.html  ( $ echo "alive" > health.html )
4. Set String matching 'Yes'
5. Search string 'alive', I already set 'STEP 3-5'
6. Set Health checker regions

Now I have 2 Health checks

[Route53] - [Hosted zones]

Add 2 Record Sets

Edit Record Set each.

• Routing Policy is up to you. (Multiplevalue Answer, 
• Health Check to Associate must same IP address (Alias Values)

Kerberos user management sample flask code.

You could re-use this code for Kerberos host management as well

@app.route('/krb/user/<username>', methods=['GET', 'DELETE', 'POST', 'PUT'])
def userAPI(username):
    result_code = "success"
    result_message = "ok"

    if request.method == 'GET':
        cmd = "/usr/bin/kadmin -p account/admin -w ADminP@ssW0rd -q \"getprinc " + username + "\""
    elif request.method == 'POST':
        cmd = "/usr/bin/kadmin -p account/admin -w ADminP@ssW0rd -q \"addprinc -policy bhero -pw " + username + "123!@# " + username + "\""
    elif request.method == 'DELETE':
        cmd = "/usr/bin/kadmin -p account/admin -w ADminP@ssW0rd -q \"delprinc -force " + username + "\""
    elif request.method == 'PUT':
        password = username + "*()890"
        cmd = "/usr/bin/kadmin -p account/admin -w ADminP@ssW0rd -q \"cpw -pw " + password + " " + username + "\""

    try:
        cmd_result = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
        message = cmd_result.stdout.read()

        fd = open(logfile, 'a')
        fd.write(str(datetime.now()) + " cmd : " + cmd + " result : " + message + " \n")

        if message.find("already exists") != -1:
            result_code = "failed"
            result_message = username + " is already exists"
        elif message.find("does not exist") != -1:
            result_code = "failed"
            result_message = username + " is not exist"
        elif request.method == 'PUT':
            if message.find("changed.") == -1:
                result_code = "failed"
                result_message = message
        elif request.method == 'GET':
            auth_time = "[never]"
            message = re.sub("Last successful authentication: .*", "Last successful authentication: " + auth_time, message)
            result_message = message
        fd.close()
    except subprocess.CalledProcessError as e:
        result_code = "failed"
        result_message = e.returncode

    message = {'result': result_code, 'message': result_message}
    fd.close()

    return json.dumps(message)


@app.route('/krb/user', methods=['GET'])
def showAllUsers():
    result_code = "success"
    result_message = "ok"

    if request.method == 'GET':
        cmd = "/usr/bin/kadmin -p account/admin -w ADminP@ssW0rd -q \"getprincs\" | grep -v \"\/\""

    try:
        cmd_result = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
        result_message = cmd_result.stdout.read()

    except subprocess.CalledProcessError as e:
        result_code = "failed"
        result_message = e.returncode

    message = {'result': result_code, 'message': result_message}

    return json.dumps(message)

macOS User Configuration

1. /etc/krb5.conf 

[libdefaults]
default_realm = ABCDEF.COM
allow_weak_crypto = false
rdns = false

[realms]
ABCDEF.COM = {
kdc = kdc.abcdef.com
kdc = kdc2.abcdef.com
admin_server = kdc.abcdef.com
kpasswd_server = kdc.abcdef.com
}

2.  /etc/ssh/ssh_config

• After MacOS update, this configuration usually reset. So after update you must check this configuration.

GSSAPIAuthentication yes    => Allow authentication protocol for ssh kerberos support
StrictHostKeyChecking no     

3. kinit at the MacOS terminal

kinit --kdc-hostname=kdc.abcdef.com,kdc2.abcdef.com sfixer@ABCDEF.COM

Install

yum --disablerepo=*  --enablerepo=base,update install -y dmidecode krb5-libs

Configuration

1. Set files if you need

• /etc/hosts
• /etc/ssh/sshd_config

2. Run ntp update

ntpdate -u pool.ntp.org

3. Registration for principal with kadmin account at the new kerberos server. And create keytab

# addpric
/usr/bin/kadmin -p account/admin -w RkaWkrdldi -q "addprinc -randkey host/dev1-api-all.abcdef.com"

# ktadd
/usr/bin/kadmin -p account/admin -w RkaWkrdldi -q ktadd -k "/home/ec2-user/seeds/keytabs/dev1-api-all.abcdef.com host/dev1-api-all.abcdef.com"

# chmod
chmod og+r /home/ec2-user/seeds/keytabs/dev1-api-all.abcdef.com

4. Add kdc hosts

cat /home/ec2-user/seeds/hosts

10.100.56.52      dev1-api-lucky21.abcdef.com       dev1-api-lucky21              
10.100.56.51      dev1-api-lucky11.abcdef.com       dev1-api-lucky11              
10.100.56.50      dev1-api-lucky01.abcdef.com       dev1-api-lucky01              
10.100.56.21      dev1-api-point11.abcdef.com     dev1-api-point11                
10.100.56.22      dev1-api-point12.abcdef.com     dev1-api-point12                 
10.100.56.20      dev1-api-point01.abcdef.com     dev1-api-point01                
10.100.56.23      dev1-api-point21.abcdef.com     dev1-api-point21                
10.100.56.24      dev1-api-point22.abcdef.com     dev1-api-point22                 
10.100.0.162      dev1-proxy-out21.abcdef.com        dev1-proxy-out21             
10.100.0.161      dev1-proxy-out11.abcdef.com        dev1-proxy-out11    

5. Copy keytab(Step. 3) file to kerberos client

/etc/krb5.keytab

Setup api server for this setting

Launce ec2 & setup nginx + gunicorn + flask 

When setup kerberos client

Just run command

• curl -s krb5-client.abcdef.com/seeds/krb-svr-config | /bin/bash

Install

1. OS (AMI Linux 2)

$ cat /etc/system-release
Amazon Linux release 2 (Karoo)

2. Installed Package

• releated krb5
• ntp

$ sudo yum list installed | grep krb
krb5-devel.x86_64                     1.15.1-19.amzn2.0.3            @amzn2-core
krb5-libs.x86_64                      1.15.1-19.amzn2.0.3            @amzn2-core
krb5-server.x86_64                    1.15.1-19.amzn2.0.3            @amzn2-core
krb5-workstation.x86_64               1.15.1-19.amzn2.0.3            @amzn2-core
pam_krb5.x86_64                       2.4.8-6.amzn2.0.2              @amzn2-core

$ sudo yum list installed | grep ntp
fontpackages-filesystem.noarch        1.44-8.amzn2                   @amzn2-core

Reference

• https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/managing_smart_cards/installing-kerberos
• https://gist.github.com/ashrithr/4767927948eca70845db

Settings

Summary

1. EC2 (2EA) - master, slave (HA)

2. DNS (Route 53, abcdef.com for sample)

- kdc.abcdef.com

- kdc2.abcdef.com

Configuration

1. /etc/krb5.conf 설정

- realm domain should upper string

$ cat /etc/krb5.conf
# Configuration snippets may be placed in this directory as well
includedir /etc/krb5.conf.d/

[logging]
 kdc = FILE:/var/log/kerberos/krb5kdc.log
 admin_server = FILE:/var/log/kerberos/kadmin.log
 default = FILE:/var/log/kerberos/krb5lib.log

[libdefaults]
 default_realm = ABCDEF.COM
 dns_lookup_realm = false
 dns_lookup_kdc = false
 rdns = false
 ticket_lifetime = 24h
 renew_lifetime = 7d
 forwardable = true

[realms]
 ABCDEF.COM = {
  kdc = kdc.abcdef.com:88
  kdc = kdc2.abcdef.com:88
  admin_server = kdc.abcdef.com:749
  default_tkt_enctypes = aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96 aes256-cts-hmac-sha384-192 aes128-cts-hmac-sha256-128 des3-cbc-sha1 arcfour-hmac-md5 camellia256-cts-cmac camellia128-cts-cmac des-cbc-crc des-cbc-md5 des-cbc-md4
  default_tgs_enctypes = aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96 aes256-cts-hmac-sha384-192 aes128-cts-hmac-sha256-128 des3-cbc-sha1 arcfour-hmac-md5 camellia256-cts-cmac camellia128-cts-cmac des-cbc-crc des-cbc-md5 des-cbc-md4
  permitted_enctypes = aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96 aes256-cts-hmac-sha384-192 aes128-cts-hmac-sha256-128 des3-cbc-sha1 arcfour-hmac-md5 camellia256-cts-cmac camellia128-cts-cmac des-cbc-crc des-cbc-md5 des-cbc-md4
 }

[domain_realm]
# .example.com = EXAMPLE.COM
# example.com = EXAMPLE.COM   

2. /var/kerberos/krb5kdc/kdc.conf 

[kdcdefaults]
 kdc_ports = 88
 kdc_tcp_ports = 88

[realms]
 ABCDEF.COM = {
  kadmind_port = 749
  max_life = 9h 0m 0s
  max_renewable_life = 7d 0h 0m 0s
  master_key_type = des3-hmac-sha1
  supported_enctypes = aes256-cts-hmac-sha1-96:normal aes128-cts-hmac-sha1-96:normal des3-cbc-sha1:normal arcfour-hmac-md5:normal
  database_name = /var/kerberos/krb5kdc/principal
  admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab
  acl_file = /var/kerberos/krb5kdc/kadm5.acl
  dict_file = /var/kerberos/krb5kdc/kadm5.dict
  key_stash_file = /var/kerberos/krb5kdc/.k5.ABCDEF.COM
 }

3. /var/kerberos/krb5kdc/kadm5.acl 

*/admin@ABCDEF.COM  *

4. Create KDC database

kdb5_util create -r ABCDEF.COM -s

5. Create KDC admin

• https://web.mit.edu/kerberos/krb5-1.5/krb5-1.5.4/doc/krb5-install/Add-Administrators-to-the-Kerberos-Database.html

# kadmin.local
kadmin.local:  addprinc account/admin@ABCDEF.COM
     
NOTICE: no policy specified for "admin/admin@ABCDEF.COM";
assigning "default".

Enter password for principal admin/admin@ATHENA.MIT.EDU:  (Enter a password.)
Re-enter password for principal admin/admin@ATHENA.MIT.EDU: (Type it again.)

Principal "admin/admin@ABCDEF.COM" created.
 
kadmin.local:

6. KDC database Backup & Restore

• https://docs.oracle.com/cd/E19683-01/817-0365/aadmin-3/index.html
• https://docs.oracle.com/cd/E19496-01/806-1971/6jb7j7amu/index.html

Create script & run crontab & propagate to slave server

#!/bin/bash

/usr/sbin/kdb5_util dump /var/kerberos/slave_datatrans


/usr/sbin/kprop -f /var/kerberos/slave_datatrans mgmt-krb-kdc02.abcdef.com > /dev/null

• FYI, domain information in /etc/hosts

% cat /etc/hosts

... 
10.100.125.156     mgmt-krb-kdc02.abcdef.com mgmt-krb-kdc02

7. Daemon start & enable

systemctl start krb5kdc.service
systemctl start kadmin.service
systemctl enable krb5kdc.service
systemctl enable kadmin.service

 I want to use Linux OpenVPN client for the tunnel.

OpenVPN Client Install

Insatll (Linux AMI 2)

# amazon-linux-extras install epel
# yum -y install openvpn easy-rsa iptables-services

Setting

# Copy autologin profile for auto login.

vi /etc/openvpn/client/vpn_cli01.conf

# Service start open VPN Client

systemctl status openvpn-client@vpn_cli01.service
systemctl start openvpn-client@vpn_cli01.service

# Kernel option

'net.ipv4.ip_forward' default = 0

I need to forward ipv4, so I change 0 -> 1

[ec2-user@ip-10-??-???-??? ~]$ cat /etc/sysctl.conf
# Kernel sysctl configuration file for Red Hat Linux
#
# For binary values, 0 is disabled, 1 is enabled.  See sysctl(8) and
# sysctl.conf(5) for more details.
# Controls IP packet forwarding
net.ipv4.ip_forward = 1
# Controls source route verification
net.ipv4.conf.default.rp_filter = 1
# Do not accept source routing
net.ipv4.conf.default.accept_source_route = 0
# Controls the System Request debugging functionality of the kernel
kernel.sysrq = 0
# Controls whether core dumps will append the PID to the core filename.
# Useful for debugging multi-threaded applications.
kernel.core_uses_pid = 1
# Controls the use of TCP syncookies
net.ipv4.tcp_syncookies = 1
# Controls the default maxmimum size of a mesage queue
kernel.msgmnb = 65536
# Controls the maximum size of a message, in bytes
kernel.msgmax = 65536
# Controls the maximum shared segment size, in bytes
kernel.shmmax = 68719476736
# Controls the maximum number of shared memory segments, in pages
kernel.shmall = 4294967296

# OpenVPN Setting on admin webpage

When I want to connect VPC1 -> VPC2 and set like this.

• I assign to user : 192.168.xxx.xxx/16
• VPC2 IPv4 CIDR : 10.50.0.0/16

# EC2 network setting

I'd like to use this EC2 instance for routing.

So, you must disable this option. (OpenVPN Server & Client both)

# Setup Routing Table

You must consider request & response both routing table.

Set OpenVPN custom account

# 1. myadmin account create                                                        
                                                                                
    % sudo useradd -s /sbin/nologin "myadmin"                                
    % sudo passwd myadmin                                                    
                                             

# 2. Configuration file path                                                              

    /usr/local/openvpn_as/etc                                                       
               
                                                                 
# 3. custom admin login setup users.0

    % sudo vim /usr/local/openvpn_as/etc/as.conf           
                                                                                
    ## allowed to access via the bootstrap auth mechanism.                       
    boot_pam_service=openvpnas                                                  
    boot_pam_users.0=myadmin                                                 
    # boot_pam_users.1=                                               
    # boot_pam_users.2=                                                         
    # boot_pam_users.3=                                                         
    # boot_pam_users.4=                                                         

  
                                                                                
# 4. Restart openvpn server                                               
    % sudo service openvpnas restart  

Setup environment: AWS

OpenVPN Server 

Select AMI for purchase license

Select Region & Continue to Launch

I choose 'Launch through EC2'. I need to set more details.

I recommend 'Launch from Website' . This option is much easy to setup.