27 Apr. 2012

SLES 11 tomcat problem, RHEL subnet fail

SLES 11 is not your friend

SuSE Linux Enterprise Server 11 is Novell's (latest) enterprise Linux distribution. They tout it as a great product, but in my job as an admin, I've noticed it is worth about as much as Novell's stock. If you are looking into linux and want to give it a try, or looking for the next good solution for your server operations, I highly suggest you steer clear of SuSE. Brought to you by a company that has yet to create something they've not driven into the ground, SLES is quite possibly the worst distribution I've ever had the misfortune of using.

My face every time I need to use YaST

I encountered a somewhat strange error on two of our SLES 11 boxes (dev/prod pair). The Apache Tomcat 6 install was taking 100% of one CPU on each box. Luckily they were both dual CPU machines, so our websites still responded, but even with 0 traffic these things were cranking away. I searched and searched and eventually happened upon a fix released by Novell regarding Tomcat using large amounts of CPU. It had something to do wtih a new linux scheduler and some silly incompatability in Java with it or something of that sort. So I ran through the fixes, setting the scheduler into some backwards-compatible mode and whatever else Novell said I should do. No dice. Still had Tomcat raping exactly 1 full CPU on each box.

After a few hours of beating my head against a wall, I did what any good sysadmin would (and something I should've done first). I updated shit. All of it. Turns out, after updating the IBM Java JRE that was installed from sr7 to sr8, the incompatability that Java had was no longer an issue and my tomcat process dropped back down to .5% cpu like it should. Welp, lesson learned.


RHEL isn't perfect

I really wasn't OVERLY familiar with RedHat until my most recent job when we started moving to CentOS. Before that, I was more familiar with Debian and BSD (ArchLinux anyone? I fanboy it). As I've become more familiar with it, I've actually grown to love it quite a bit. It is a solid, logically set up, well documented, smoothe system to work on (none of which apply to SuSE). It isn't, however, without its quirks. Here's how to fix one of them.

At work we run a large VMWare environment like most large entities probably do, or at least should. Because I clone VMs and deploy new servers frequently enough, I figured I would spare myself the trouble of sifting through tons of system files buried who-knows-where just to set up a new network config (the clone uses the original's, and that's no good...). So I wrote myself a neat little script, complete with interactive questionnaire for your network settings. Real slick. After being completely impressed with myself, I cloned a new server and fired up my script. I filled in all the network information, and voila, a shiny new network config and I didn't have to do anything! Awesome!

But then I noticed it.

None of my subnets for my secondary IP addresses (on aliased interfaces) were correct! Nooooooooooooooooooooooooo!!!!! Figuring that my script probably just fubar'd something, I poked around in my network config files. Everything looked solid in my ifcfg-eth0* files. IP Address was correct, prefix was correct, gateway, etc.. EVERYTHING was flawless. But for some reason, no matter what was listed in the ifcfg-eth0* files for the prefix (I had PREFIX=25), it would pop up on the /16 subnet. WTF? I was able to shut the network down and manually bring the interfaces/IPs online properly using ip link and ip addr, but that is hardly kosher... And forget writing a script to do that automatically on boot. Isn't that what /etc/init.d/network is? So I set out doing some detective work.

Four hours later I was at my wit's end. I had tried everything I could think of. I had 2 other boxes this was working fine on. I had rebuilt the config files from scratch 10 times, copied them from working boxes and replaced IPs/subnets 10 times. Still nothing. Tired and annoyed, finally I decided to try one last thing before I ragequit. I replaced the PREFIX=25 entry in my config file with NETMASK=255.255.255.128. Now, the manuals say that NETMASK has been deprecated in favor of CIDR notation and to use PREFIX instead. But when I restarted the network with the NETMASK command in there, THEY CAME UP WITH NO PROBLEMS WHATSOEVER AND IN THE RIGHT SUBNET! WHY!?

So, after wading around through the darkest recesses of linuxdom (or maybe its in bold on the front of some manpage or doc, idk), I discovered why it was throwing my IPs on the /16 subnet. When NETMASK was deprecated in favor of PREFIX, a routine called ifcalc() is run that calculates subnets and stuff. I guess this program is from before 1993 when CIDR didn't exist because it can only compute classful-range subnets, that is only values of 0 or 255 per octet. Your subnet not a /8,/16,/24, or /32? No worries! With ifcalc(), it soon will be! As I was running on a /25 (255.255.255.128), ifcalc was thrown for a loop and decided to round down to /16 for whatever reason I still to this day do not understand.

Dear Red-hat, please do not deprecate things that are vital components of a system and you do not have a working replacement for. Thank you.

Update:

Looks like something very similar was reported as an official bug here.

3 Apr. 2012

Evading your work's site block:

Remember: This can get you fired. Also remember: Your network monitoring team can still see the huge amounts of encrypted traffic you are using to watch youtube at work and may eventually wonder what your appeal is with yourhomeserver.com and why you are using an encrypted connection to shuffle ~200kbps constantly. My job isn't to follow your workplace rules, that is your job. My job is to show you how to break them :D

Things you will need:

Once you have these in place, you need to decide which instructions you want. For stupid people or for smart* people.

*The qualification for "smart" in this context is the ability to properly pronounce the word banana.


For stupid people:

For smart people:

Congratulations! You are smart. Or at least you think you are. This is just a brief explanation of how it actually works and what is going on behind the scenes, not just step by step instructions on how to set up your computer to get to the WoW forums to flame your guildie about aggroing a boss early.

What this setup does (a not so well known ability of ssh) is sets up a "tunnel" to your ssh server that will essentially forward all network traffic you send into the tunnel to the server, and the host will act as if the traffic were local on the server. SOooooooooooooo, logically it follows that if you set up your web browser to fire off requests to this tunnel and listen for responses from that tunnel (instead of its usual operation which uses your network's gateway as the "tunnel"), you will be browsing quite literally from your home (or friend's) server.

Your computer's setup: PuTTY can be configured to listen on a port on your local system, which will then tunnel to the designated port on a remote system as a dynamic SOCKS proxy. Any port on your local system (assuming no local firewalls prohibit) works just fine. The remote host/port need to be ones that are not blocked by your internet firewall, (the likelyhood of your home server being blocked is small, but usually few ports are open). To better mask, run it over port 443, which is where https runs. This will, however, prevent your home server from running a ssl web server as well, since SSHD will be listening on the port instead of your web server. Once this SOCKS proxy is setup in PuTTY, anything aimed at the port you selected will tunnel to the sshd server remotely. So, open up your browser, select "use proxy" or similar, make sure it is using a SOCKS proxy (not http or what have you), and use the address of localhost:port where port is the port you had PuTTY listen on. I tried this on Opera, and on my particular system it would only run using the loopback address 127.0.0.1:myport instead of the hostname localhost. YMMV.

Your server's setup: If you are running Windows on your home box, install cygwin and then OpenSSH. If you are running linux, you probably already have it. Make sure in the config file for openssh server (I think defaults to /etc/ssh/sshd_config) that the line "AllowTcpForwarding" says yes. Restart sshd. That's all. And this is why you should love Linux and FOSS. Oh be sure to note or change what port sshd is running on and to forward it if you're behind a NAT router. SSHD, when receiving tunneled TCP requests, takes command and sends them out from the server as if they were its own. It forwards the responses it gets back through the tunnel.

Now that you've made it through, you can refer to the stupid section for step by step if you can't figure out PuTTY's quirks on setting this up :)

Setting up a Kerberos5 server in Arch Linux:

1. Ensure krb5 package is installed (should be)

2. Edit /etc/krb5.conf to reflect your desired realms/hostnames. NOTE: I did have issues using localhost or the hostname on my system. It wouldn't run properly. Didn't have time to troubleshoot, tried loopback addr (127.0.0.1), worked.

3. Ran /usr/sbin/kdb5_util create -s to create KDC db.

4. Created (wasn't there by default) the kadm5.acl. NOTE: It seems most distros, particularly Cent/Fedora/RedHat have all this in /usr/kerberos and /var/kerberos/krb5kdc. Arch keeps this all in /var/lib/krb5kdc (binaries in /usr/sbin). I needed to symlink the /usr/local/var/krb5kdc directory to the /var/lib/krb5kdc where Arch keeps it.

5. Once this was symlinked, I was able to launch the kadmind process, then run kadmin.local to login with the password created and then add principals at the prompt via "addprinc username". If you want accounts to have admin privileges, be sure to do "addprinc username/admin", and add the line
*/admin@YOURREALM.COM *
to the kadm5.acl file.

6. Test by running kinit to get a ticket, klist to show it, kdestroy to destroy it.