VM Config
All Distributions
VM Specifications:
- CPU: 1vCPU (2 cores)
- RAM: 2 GB
- HD: 100 GB
When creating a Rocky 8 VM with hardware version 15, VMware uses a UFEI config with secure boot by default. Keep it simple and use use BIOS.
When creating a Ubuntu 64bit VM with hardware version 15, VMware uses a BIOS config.
VM Hardware Changes
- Use BIOS. EUFI without secure boot is silly.
- DO NOT enable CPU hot add (disables numa rules)
- DO NOT enable RAM hot add (disables numa rules)
- Remove Floppy drive from hardware
- Disable COM1
- Disable COM2
- Disable Parrallel
- Disable Floppy
Change the hostname:
Rocky
Make sure to change "SOFTWARE SELECTION" from "server" to "minimal install."
Rocky 8 - Verify and/or Update the IP
Rocky 9 - Verify and/or Update the IP
Verify DNS is setup correctly.
The output should look something like:
Verify NTP is setup correctly.
The output should look something like:
Ubuntu
Verify and/or Update the IP
No DNS changes are required. Ubuntu uses a dynamic resolv.conf file for connecting local clients to the internal DNS stub resolver of systemd-resolved.
Verify NTP is setup correctly.
The output should look something like:
User Creation
Rocky 9.1 Only
During the install, you can only setup the root account. Creating a secondary user account was not an option. This only seemed be required for Rocky 9.1. 9.2 gave the option during the install.
So we need to:
- Add the localadmin user.
- Change the password for localadmin.
- Ad the localadmin to the wheel (for ssh and sudo).
LVM Expansion
Ubuntu Only
For whatever reason, Ubuntu does not use the the entire disk during the OS installation even if it is specified to use it. So we will do this before the cleanup.
Various commands to show the size of drives.
Display the size of the volume group.
Extend the logical volume to use all of the remaining disk space.
Force the OS to see the extended space.
It is probably wise to do a good old reboot.
!!! RUN AS ROOT !!!
In case you did not get the memo...
Run all of the proceeding commands while logged in as: root.
Optional: Ubuntu Only.
In CentOS and Rocky, a root user is enabled by default. Ubuntu disables the root user by default because it does not have a password. To enable root login, change the root password from Keepass and follow the steps below. If you would like unlock enable root login:
- Change the root password: sudo passwd root
- Unlock the root account: sudo usermod -U root
- Allow root to login through SSH: sudo sed -i 's/#PermitRootLogin prohibit-password/PermitRootLogin yes/g' /etc/ssh/sshd_config
- Restart SSH: sudo systemctl restart sshd
Use the following command to login as root. You will be prompted for the password for the user that you are currenlty logged in as and NOT the root password.
Install
Rocky
Update the repo list, update the bits, install pre-reqs. With CentOS 7, install "dnf" and substitute "yum-utils" for "dnf-utils."
Ubuntu
Update the repo list, update the bits, install pre-reqs.
NTP Setup
Regardless of the operating system, the following lines are how the top of the config should appear.
Rocky
You are able to set the NTP servers during setup. All you have to do is verify the settings are correct. The location of the config file is:
Ubuntu
During the OS install, you are not prompted for the NTP servers. Chrony was installed during the install section above. You still have to change the default NTP servers. The first sed will delete any line that contains "maxsources 1." The following two lines will replace the defaults with the specified NTP servers.
All Distributions
Test the Chrony sources.
Set the timezone.
Verify time settings.
SELinux Config
This section only applies to Rocky.
Verify SELinux status:
Change option SELINUX from disabled to enforcing and reboot after activating.
AppArmor Config
This section only applies to Ubuntu.
Verify AppArmor status:
IPv6 Network Config
All Distributions
This command works in all distrobutions. Verify the current state of IPV6.
- 0 - IPv6 is currently active
- 1 - IPv6 is currently disabled
Verify current IPv6 status.
CentOS 7 and Rocky 8 both come back with null.
Ubuntu comes back with the following: inet6 fe80::250:56ff:fea9:2bae/64 scope link
In Ubuntu, if "ip a | grep inet6" is run, it is not persistant after a reboot.
Ubuntu
To disable IPv6, change yes to no.
Verify if IPv6 has been disabled:
Rather than completely disabling IPv6 all together, the IPv6 stack can be enabled, but prevented from assigning network addresses to network interfaces. Edit the following file and append the following lines.
Run the following command to apply the changes or a reboot will work too.
FirewallD Config
This section only applies to Rocky.
Setup FirewallD to start during startup and run now.
Check the service status.
List zones.
List the default zones.
Disable default firewall ports.
Apply the firewall changes.
Show the current config.
UFW Config
This section only applies to Ubuntu.
Check the status of UFW.
If UFW is disabled, enable it.
Create the default policy based on best practises.
Use the following command to allow SSH from any network.
Show the policies in a numbered order.
SSH Server Config
Make a backup of the sshd_config.
After creating a backup of this file, this step is optional and is performed if you wish to check all the options that are currently enabled in this configuration file. You can check it out by running the following command in your terminal:
Make the following changes to the sshd config:
- Protocol 2: protocl 1 is a high security risk. Since version 7.0 of OpenSSH, protocol 1 is automatically disabled during compile time. We are forceing protocol version 2 regardless.
- HostKey /etc/ssh/ssh_host_ecdsa_key: is a less secure cipher than the others. So we are not going to use it.
- LoginGraceTime: specifies how long after a connection request the server will wait before disconnecting. There is a STIG to set the interval to 30 seconds or less.
- PermitRootLogin: prevents the root account from being able to SSH into the server. There are slight differances between CentOS 7, Rocky 8, and Ubuntu 20.04. That is why there are several sed commands.
- MaxAuthTries: refers to the maximum number of authentications attempts per connection. This is not a lockout mechanism, it will not protect you from any brute force attack. When the limit is hit, the sshd stops accepting authentication attempts closing the TCP connection.
- MaxSessions: the max number of open sessions to a server at a time. Example, if this is set to 3, you could only be SSHed into the server 2 times and one SCP session.
- PermitEmptyPasswords: specifies whether the server allows logging in to accounts with a null passwords. If you intend to use the scp utility to make backups over the network, you must set this option to yes.
- X11Forwarding: this is a STIG. Enabling X11 Forwarding on the host can permit a malicious user to secretly open another X11 connection to another remote client during the session and perform unobtrusive activities such as keystroke monitoring, if the X11 services are not required for the system's intended function, they should be disabled or restricted to the user's needs.
Verify the changes:
Verify if the SSH ciphers have already been changed:
The lines below force better SSHD security:
- KexAlgorithms: controls which key exchanges are allowed.
- Ciphers: used to encrypt the data after the initial key exchange and authentication is complete.
- Message authentication codes (MAC): Encryption provides confidentiality, message authentication code provides integrity. We need both.
SSH Server Config - Key Login
This section is optional.
To disable users from logging in to this server using passwords, make these changes. This will require SSH keys to be created and used.
- PasswordAuthentication: no
- ChallengeResponseAuthentication: no (this is the default)
- PubkeyAuthentication: yes
SSH Server Config - Auto Logout
This section is optional:
Rocky 8 uses a older version of OpenSSH than Ubuntu 20.04. The SSH auto logout settings work in Rocky 8's version. This functionality is broken in Ubuntu 20.04's version. As a result, the workaround was to create a script. See the "Auto Logout Script" section for more information on how to install the workaround.
ClientAliveInterval and ClientAliveCountMax: there are two ways to configure this. There is a client alive message that can be sent in intervals. If no responce is received, after max count, the user will be logged off. The interval is in seconds. Add the m for minutes.
Example 1: ClientAliveInterval 30m * ClientAliveCountMax 2 = 60m
Example 2: ClientAliveInterval 60m * ClientAliveCountMax 0 = 60m
Verify the changes.
SSH Server - Moduli Generation
The /etc/ssh/moduli file contains prime numbers and generators used by the SSH server for the Diffie-Hellman key exchange. Your current /etc/ssh/moduli is probably not unique. Generating a new file will harden the server but generating this file will take a long time. This would have to be done on every server AFTER cloning.
The commands to create a new moduli are below. The -G and -T switches only work in CentOS 7 or Rocky 8. Ubuntu 20.04 is using a newer version of the Open SSH server that requires different commands. The commands for the new version are from a OpenBSD man page on ssh-keygen.
The TOTAL time to replace the moduli file is around 1.5 hours on a Linux VM that has 2 cores (1vCPU) and 2GB of RAM.
CentOS 7 and Rocky 8 - moduli keygen
Original stribika blog commands:
- ssh-keygen -G /etc/ssh/moduli.all -b 4096
- ssh-keygen -T /etc/ssh/moduli.safe -f /etc/ssh/moduli.all
- mv /etc/ssh/moduli.safe /etc/ssh/moduli
- rm /etc/ssh/moduli.all
CentOS 7 and Rocky 8 - moduli keygen
Original stribika blog and Ubuntu 12.04 man page combo:
- ssh-keygen -G moduli-4096.candidates -b 4096
- ssh-keygen -T moduli-4096 -f moduli-4096.candidates
- mv moduli-4096 /etc/ssh/moduli
- rm moduli-4096.candidates
Ubuntu 20.04 - moduli keygen
OpenBSD man page on ssh-keygen modified:
- ssh-keygen -M generate -O bits=4096 moduli-4096.candidates
- ssh-keygen -M screen -f moduli-4096.candidates moduli-4096
- mv moduli-4096 /etc/ssh/moduli
- rm moduli-4096.candidates
According to stribika's 2015 blog, if you choose to enable the "diffie-hellman-group-exchange-sha256" KexAlgorithms, delete the lines where the 5th column is less than 2000. The commands from stribika are here:
Original stribika 2001 bit truncation:
- awk '$5 > 2000' /etc/ssh/moduli > "${HOME}/moduli"
- wc -l "${HOME}/moduli" # make sure there is something left
- mv "${HOME}/moduli" /etc/ssh/moduli
According to Mozilla's 2017 guide, all Diffie-Hellman moduli in use should be at least 3072-bit long. The commands below are a modifed version of stribika's. Mozilla's were giving an access denied error.
Original Mozilla's 3072 bit truncation:
- awk '$5 >= 3071' /etc/ssh/moduli > /etc/ssh/moduli.tmp
- mv /etc/ssh/moduli.tmp /etc/ssh/moduli
Original stribika 2001 bit truncation updated to Mozilla's 3072-bit-long truncation:
SSH Client Config
Make a backup of the ssh_config.
After creating a backup of this file, this step is optional and is performed if you wish to check all the options that are currently enabled in this configuration file. You can check it out by running the following command in your terminal:
Make the following changes to the ssh_config:
- Protocol 2: protocol 1 is a high security risk. Since version 7.0 of OpenSSH, protocol 1 is automatically disabled during compile time. We are forceing protocol version 2 regardless.
- The remainder of the changes are recommended from stribika's blog.
Verify the changes. I did not want to use grep in the verification because I wanted to see check to make sure the spacing looked appropriate.
The following part is optional.
To disable users from logging in to other systems using passwords, make these changes. This will require SSH keys to be created and used.
- sed -i 's/# PasswordAuthentication yes/\n PasswordAuthentication no/g' /etc/ssh/ssh_config
- sed -i '/ PasswordAuthentication no/a\ ChallengeResponseAuthentication no\n' /etc/ssh/ssh_config
- sed -i '/ ChallengeResponseAuthentication no/a\ PubkeyAuthentication yes' /etc/ssh/ssh_config
Debian 10 Extras
The following pieces were notes from a Debian 10 minimal install project that was aborted.
Add "AllowUsers localadmin" to the bottom of sshd_config:
- echo "AllowUsers localadmin" | tee -a /etc/ssh/sshd_config
Run the following command to modify the "visudo" config. Do NOT edit this in vi/vim/nano etc. Append user group to the end of the file.
- /usr/sbin/visudo
- localadmin ALL=(ALL:ALL) ALL
- tail /etc/sudoers | grep localadmin
Customize the VIM config by creating the file in the respective users home dir (vi ~/.vimrc). Then paste the following.
- " highlight search
- :set hlsearch
- " when searching, ignore text case
- :set ignorecase
- " change the line number color
- :highlight LineNr ctermfg=grey
- " show line numbers all of the time
- :set number
- " enable pasteing with right mouse click
- :set mouse=r
Ubuntu - Disable Message Of The Day (MOTD)
This section is optional.
This was only tested with Ubuntu 20.04.
The default Ubuntu MOTD is tacky. We created our own.
If you like the default Ubuntu login, but want to cut down on some of the noise at login, you can disable specific items such as the dynamic MOTD news. An example of the news would be: "Super-optimized for small spaces - read how we shrank the memory footprint of MicroK8s to make it the smallest full K8s around."
Verify the motd-news setting. Disable the motd-news setting with a sed command.
- cat /etc/default/motd-news | grep -i ENABLED
- sed -i 's/ENABLED=1/ENABLED=0/g' /etc/default/motd-news
Remove the executable permission from the specific files that you want to turn off.
- chmod -x /etc/update-motd.d/10-help-text
- chmod -x /etc/update-motd.d/50-motd-news
- chmod -x /etc/update-motd.d/90-updates-available
For a list of what is disabled:
- find /etc/update-motd.d/ -name "*" -perm 644
To disable all Ubuntu MOTD content, follow these steps:
- Verify the motd-news setting.
- Disable the motd-news setting with a sed command.
- Remove the executable permission from all files.
- For a list of what is disabled:
Message Of The Day (MOTD) Setup
This section is optional.
This was not tested with CentOS 7.
This was setup and tested on Rocky 8 and Ubuntu 20.04.
Banner: a message that displays when a SSH session is started and BEFORE the login.
MOTD: a message that displays AFTER a user has logged in to a SSH session.
Make the following changes to the sshd config:
- PrintMotd: if this is enabled, the user will get a MOTD displayed AFTER login. The contents of the /etc/motd file is printed to the screen. This is NOT a bash script. It is only a flat file.
- PrintLastLog: after succesful login, a message shows the date/time and the IP address of the last login.
- Banner: this is a message that is printed to the screen BEFORE login.
If you do need to make changes to the sshd_config, the commands are here:
- sed -i 's/.*PrintMotd.*/PrintMotd no/g' /etc/ssh/sshd_config
- sed -i 's/.*Banner.*/#Banner none/g' /etc/ssh/sshd_config
The commands above here are defaults, so no changes should be required. PrintLastLog does need to be changed from the default.
If any changes were made to sshd_config, restart sshd.
MOTD Setup
Upload bash script to:
For Rocky 8, put the bash script in: /etc/profile.d/. For Ubuntu, most of the documenation said to put the bash script in: /etc/update-motd.d/. I put the bash script for BOTH distros in: /etc/profile.d/ and the scripts worked as expected. Lastly, make the MOTD bash script executable (regardless of the distro/location).
Auto Logout Script
This section is optional.
I tested this bash script for Rocky 8 and Ubuntu 20.04. I put it in: /etc/profile.d/ and the script worked as expected in both distros. Lastly, make the bash script executable (regardless of the distro/location).
This script works for both console and SSH sessions.
This will log you out of your current session. It will not log you out if you have a dialog or vi open.
Autologout.sh Setup
Upload bash script to:
Cleanup - Kernel and bits
All Distributions
Reboot the system to apply all of the changes above.
Use the following command to login as root. You will be prompted for the password for the user that you are currenlty logged in as and NOT the root password.
Stop logging. Ubuntu might bark that another service is dependant on rsyslog. Run: systemctl stop syslog.socket rsyslog.service
CentOS and Rocky
CentOS 7
Remove any old kernels and packages:
Rocky 8+
Remove any old kernels and packages:
Ubuntu
Remove any old kernels. The "purge-old-kernels" command will keep the current kernel and one previous kernel version. The "apt-get -y autoremove --purge" command does the same thing without errors. The --purge removes old kernels. Without the purge, it only cleans old packages. purge-old-kernels -y --keep 1.
Clean out old apt-get packages:
Cleanup - NICS
Remove the udev persistent device rules.
CentOS 7 and Rocky 8
Remove traces of the MAC address and UUIDs.
Rocky 9
Remove traces of the MAC address and UUIDs.
Ubuntu 20.04
Remove traces of the MAC address. UUID does not seem to exist in netplan.
Mrs. Clean - Script
The SSH server host keys must be recreated with each new VM. Otherwise, the same host keys would be on every VM and that would be a huge problem. So, we delete them. Each distribution handles this a differently.
CentOS and Rocky will recreate the SSH keys after the next reboot or the next time the SSH service restarts.
Ubuntu will NOT recreate the ssh keys. Additional commands are required.
The idea of the following script is the same as a Windows "run once" startup shortcut. After the script has run once, it will delete itself.
The /etc/machine-id file contains the unique machine ID of the local system that is set during installation. The machine ID is a single newline-terminated, hexadecimal, ID string. This was added to RHEL 7.1 to identify the machine in the network. Ubuntu used this parameter as well. This parameter must be unique. When engineers clone a VM, the machine-id is not changed, and the VM may have the same machine-id. The engineer should change the machine-id of the cloned VM for the server to be uniquely identifiable over the network.
Only /etc/machine-id needs to be regenorated as /var/lib/dbus/machine-id is a symbolic link to /etc/machine-id.
systemd-random-seed.service is a service that restores the random seed of the system at early-boot and saves it at shutdown. Saving/restoring the random seed across boots increases the amount of available entropy early at boot. On disk, the random seed is stored in: /var/lib/systemd/random-seed.
Make the script executable.
Mrs. Clean - Cron Job
To add new crontab jobs, use the command below. This will use the default editor. In most cases, the default editor is vi/vim. You cannot use vi/vim on its own. You have to use the crontab command.
By using "sudo" in the command, this will add the jobs to the root accounts cronjob. As a result, using "sudo" in the actual commands is not necessary. After the text editor is open, paste the following line.
- crontab -e
- @reboot /usr/local/bin/mrs-clean.sh
Verify the crontab job is scheduled for root.
Cleanup - Files and history
Force the logs to rotate and purge old logs.
Issue the following commands to truncate the logs.
Clean all mail.
Clean /tmp out.
Clean various goodies and bash histories.
Ready to rock?
Shutdown the system.
Perform the following tasks:
- Verify that the NIC is on the correct network.
- Change the CD drive to "client device."
- Remove any VMware snapshots.
- Convert the VM to a VMware template.
References
General Template Notes
- https://oitibs.com/linux-vm-template-creation/
- https://lonesysadmin.net/2013/03/26/preparing-linux-template-vms/
- https://gist.github.com/AfroThundr3007730/ff5229c5b1f9a018091b14ceac95aa55
- https://community.spiceworks.com/how_to/151558-create-a-rhel-centos-6-7-template-for-vmware-vsphere
- https://blog.no-panic.at/2016/08/11/a-clean-start-how-to-prepare-a-minimal-debian-template-for-lxc-containers/
VMware Customization Requirments
- https://docs.vmware.com/en/VMware-vSphere/6.7/com.vmware.vsphere.vm_admin.doc/GUID-E63B6FAA-8D35-428D-B40C-744769845906.html
- https://kb.vmware.com/s/article/56409
SSHD Config, hardening, and moduli key ssh-keygen
- https://chewett.co.uk/blog/2535/fixing-ssh-keygen-unknown-option-g-or-t-on-ubuntu-20-04/
- https://cryptsus.com/blog/how-to-secure-your-ssh-server-with-public-key-elliptic-curve-ed25519-crypto.html
- https://infosec.mozilla.org/guidelines/openssh
- https://man.openbsd.org/ssh-keygen.1
- https://manpages.ubuntu.com/manpages/xenial/man1/ssh-keygen.1.html
- https://medium.com/@jasonrigden/hardening-ssh-1bcb99cd4cef
- https://stribika.github.io/2015/01/04/secure-secure-shell.html
Increasing the drive size
- https://askubuntu.com/questions/1269493/ubuntu-server-20-04-1-lts-not-all-disk-space-was-allocated-during-installation
- https://www.linuxtechi.com/extend-lvm-partitions/
- https://kb.vmware.com/s/article/1006371
- https://blue42.net/linux/5-steps-linux-lvm-extend-volume-no-reboot/#the-command-on-centos7-xfs
- https://ma.ttias.be/increase-a-vmware-disk-size-vmdk-formatted-as-linux-lvm-without-rebooting/
random-seed Info
- https://www.redhat.com/en/blog/understanding-random-number-generators-and-their-limitations-linux
- https://unix.stackexchange.com/questions/388803/why-is-the-random-seed-only-saved-to-disk-on-boot-and-shutdown
- https://manpages.debian.org/testing/systemd/systemd-random-seed.service.8.en.html
machine-id Info
- https://www.thegeekdiary.com/centos-rhel-7-how-to-change-the-machine-id/
- https://unix.stackexchange.com/questions/402999/is-it-ok-to-change-etc-machine-id
Cronjob creation
- https://www.2daygeek.com/execute-run-linux-scripts-command-at-reboot-startup
- https://askubuntu.com/questions/408611/how-to-remove-or-delete-single-cron-job-using-linux-command
Info on creating a service
We did not create a systemd service. We ended up creating a crontab job.
- https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system_administrators_guide/chap-managing_services_with_systemd
- https://bitraboy.wordpress.com/2018/09/17/how-to-run-an-install-script-after-reboot-centos-rhel/
- https://blog.knoldus.com/creating-custom-daemons-in-linux/
- https://gist.github.com/Ham5ter/6cf693f67a1792b55f294969647a4e6a#file-regenerate_ssh_host_keys_init-sh
- https://newbedev.com/linux-schedule-command-to-run-once-after-reboot-runonce-equivalent
- https://www.redhat.com/sysadmin/replacing-rclocal-systemd