ryan chewning dot net

Roaming the Mobile Seas

I’ve gotten around a bit in the mobile network scene. I started many years ago on Cingular (pre and post ATT merger) as they were switching to TDMA, GPRS and the 3G roll-out /merger of the networks. I went to Verizon some years later because I got tired of standing in the middle of Clifton Ave to try and make a phone call. I’ve been pretty loyal to Verizon, though all the brew-ha-ha over their involvement with the NSA/CIA/Feds in the illegal surveillance of American citizens caused me to depart for friendlier waters. Which is how I found myself with T-Mobile aka, I don’t think I have service here either. After a while of dealing with that, I went back to Verizon since getting consistent service has become more important for the moment than whether or not big brother/sister is listening (they are anyway). You might say wait a minute, aren’t you forgetting a network? While I’ve never had a personal phone on the Sprint/Nextel network, my work phone is Sprint and my older work phone was Nextel. So I’ve had experience with all the networks in one form or another.  I believe that Verizon has one of the most consistent networks as far as performance and stability. Anyway, enough of my rambling about networks.

And now the Good Stuff

My new every two recently came up and ever the techy/gadget guy I decided to dump the failing crackberry platform for an Android device, specifically the HTC Thunderbolt, which is one of the first 4G phones for Verizon’s LTE network. While its rocking Froyo (2.2) out of the box but with an OTA upgrade to Gingerbread (2.3) it still has Sense 2.2. There are some things that I like about Sense 2.3 / 3.0 that I miss (my work phone is the HTC Evo which has the latest Sense). I had to root the phone, which I found to be a pretty easy process; its just a lot of hoops to jump through to get the pre reqs installed. After rooting I tried several different Roms and while they were all pretty solid, I actually like the stock Verizon Rom best. That being said, I did remove all the verizon cruft that comes locked on to the phone (I’m looking at you Verizon with your VZNav, CityID, V Cast and some other stuff that I just didn’t care for).

I find that the phone runs great with the occasional hangup when multiple accounts try to sync while I’m attempting to do something. This is especially true when I’m in a weak coverage area for LTE or am on the 3G connection.

The other issue I’ve noticed and its probably a bug in the radio stack or Android software is when I’ve been out of range of the network for a long period of time, say a couple of hours, I notice that I have to toggle the mobile internet option a few times before getting LTE service back or even 3G data. I’ve noticed that a similar thing happens when I’ve been stuck in an area that is limited to 3G/1X service as well. A few times I have had to completely restart the phone because just boggling the Mobile Internet service wasn’t enough to get the phone to “see” the LTE network or realize that data services could be turned back on.

Other than the few issues with the phone seeing the network, this phone has been blazing fast. In the Lebanon area I’ve seen speeds around 12mbps down and 3 to 6mbps up. Down around Kenwood both speeds seem to double which leads me to believe that I’m probably hitting the maximum size of the network connection to the towers in the Lebanon area.

Many may wonder how the battery life is since all LTE phones are dual radio for the time being. It probably wouldn’t have hurt HTC to stop trying to compete with the iPhone in size and sleekness and just throw on a bigger battery. The Thunderbolt has a 1400mAh battery which is a bit smaller than its EVO cousin.

The battery makes it though most of my work day and that’s with me reading emails, txting and making phone calls in areas where the reception is pretty terrible because of the building construction. I can leave work with 30% to about 10% battery life remaining. Though I usually try to throw the phone on the charger for a bit around lunch time just to make sure it has a bit more battery life left just incase I need to make use of it later in the day.

All in all I would give the Thunderbolt 4 of 5 stars because of battery life, and a few of the network issues that its had. This phone is also far superior to another smart phone I’ve had including the two different versions of Blackberry curves that I’ve owned.

Or, how I finally post something straight to the web without holding on to it for months and months in my drafts folder until I forget what it was about and delete it...

Yes, the blog is still alive. I haven’t done much with it though which is kinda sad. That being said I feel like I’ve been doing a bit recently that I’m both excited about and looking forward to having it done, documented and posted. Hopefully I’ll feel like doing all of that sooner rather than later, but after a long day at work, the last thing I really want to come home and do is write more documentation.

Since the last thing that’s posted is a bunch of MySQL stuff from when I was at UC almost two years ago I’ve been a busy guy. I left UC for a full-time gig all be it a lower position in the grand scheme of things as a computer technician for a school district in Warren County. Its not a bad gig as I’m about to reach a year there, but its not all that I’d hoped it would be as far as opportunities and my desired career path as a Linux Systems Administrator / DBA (MySQL or Oracle [someday]) / Network Administrator. Don’t get me wrong. Its important to have the folks that change out busted computers and fix the technology at the physical level, but its just not what I want to spend the next several years of my life doing.

Which leads me to what I’ve been doing besides work. As I sit at a local Starbucks (that I will soon be working at again so I can get a newer car sooner) I’m typing this on my laptop that is happy connected to the OpenVPN server that’s running at home. Its running on a PC using Ubuntu LTS server. This box is also handling the firewall, IDS, DNS, DHCP and IPv6 gateway for the home network. The sad part about all of this, the load on the box is still way down on the load even when there is torrenting or gaming afoot. In addition to the single box, I’ve got another one setup as the fail over point and two other servers inside the home that are handling other servers such as radius for WiFi,system monitoring and notification software, a la Zabbix. All of this is clearly over kill for the house, but its the process and actually touching these things in a non-mission critical environment so I can say I’ve touched this stuff and know something about how it all works together.

Dreamhost Drama

And this site is running on the Linode servers I happily set up about three months ago because Dreamhost lost its competitive edge for me in their pricing. Honestly they probably lost it some years ago, but I wasn’t comfortable enough with administering a Linux server to make the jump to Linode. For example with Dreamhost you have to have their hosting plan which can be anywhere from $10.95 a month down to $5.95, if you prepay for 10 years. I was prepaid every three years which brought my monthly cost down to $7.95 a month for their hosting plan. If I wanted decent performance though I would have to go to a VPS and that started at $15 per instance at 300MB of RAM and scaled at a rate of $1 per 20MB in addition, if you needed to have a secured site (SSL’d) you had to pay $3.99 a month per IP address. Needless to say that was a bit too pricey for me. I believe I was paying about $70 a month for hosting through Dreamhost by the time it was all said and done. Once I crunched the numbers it became pretty clear to me that a move was in order and I went to the only other hosting company that I would trust and it was Linode. Not to sound dramatic about the move, but when you start paying a flat $20 per server with 512MB of RAM, 20GB of disk space and 200Gbs of transfer and your transfer pools between all of your servers, life is a bit nicer. Yes there is the headaches of keeping _everything_ updated on the servers, but for the price, the control and flexibility you can’t beat Linode. And I should mention that the little extras like extra IP addresses are only a dollar a month and their backup solution is $10 per server per month, which can save your ass if you get hacked or have fat finger syndrome.

Other Bits

So while I’ve been doing all this goofy stuff around my house, I’ve finally started to write my first complex Bash program/script. It sounds silly in this day and age with all the smart phones and fast 3G/4G networks out there, but I’m working to make a script to email/sms me important weather related information. In another life I’m a amateur radio operator that’s pretty involved with volunteer stuff, one of them is the weather. I’m a trained weather spotter for the National Weather Service’s SkyWarn program. I actually do the HAM stuff though. I’m one of several net control operators for the Cincinnati section. The script would respond both to updated information that’s relevant to severe weather outbreaks, like the Convective Outlooks published by  Storm Prediction Center and emails sent to an address that would be periodically checked and a response generated based on the commands in the email. That’s a useful feature so I can get weather information for any other place that I might be at or traveling to. I’ll try to post it up when I have something together.

These are some tweaks that we are using in our default MySQL installations at UC. There are some application specific changes that we will need to make but I will add those later.

Note these should be favorable for using both Innodb and MyISAM database engines together. There will be application specific tweaks on another post.

vi /etc/my.cnf or use your favorite text editor.

Find the following sections and make the changes:
key_buffer_size
This should be set to about 20%-25% of available memory on the box since we will not be using MyISAM exclusively. If we are, the number goes to 30%-40%.

innodedb_buffer_pool_size
For innodb tables this should be 70% – 80% of available memory. If we have MyISAM tables as well then set this closer to to 70%.

innodb_additional_mem_pool__size
This doesn’t really need to be adjusted. This is for additional memory that would be available to the innodb engine to do misc. tasks. Use SHOW INNODB STATUS to see how much is being used. If nothing else, bump to 20M.

innodb_log_file_size
Very important, especially for write intensive loads. 256M is a good size. Be careful though as this requires a longer start up time if you have to do a recovery.

innodb_log_buffer_size
Default size is alright. But if dealing with blobs a lot then bump to 16M. This memory is flushed every second though so too much memory is a waste.

innodb_flush_log_at_trx_commit
This is very tricky and needs to be thought out carefully before being set. efault value of 1 will mean each update transaction commit (or each statement outside of transaction) will need to flush log to the disk which is rather expensive, especially if you do not have Battery backed up cache. Many applications, especially those moved from MyISAM tables are OK with value 2 which means do not flush log to the disk but only flush it to OS cache. If you’re not concern about ACID and can loose transactions for last second or two in case of full OS crash than set this value. It can dramatic effect especially on a lot of short write transactions. The log is still flushed to the disk each second so you normally would not loose more than 1-2 sec worth of updates. Value 0 is a bit faster but is a bit less secure as you can lose transactions even in case MySQL Server crashes. Value 2 only cause data loss with full OS crash.

table_cache
Default value should be fine. Might even be able to be turned down to 1024.

thread_cache
Thread creation/destruction is ususally an expensive operation. Start at 16 and watch for fast growth of Threads_Created variable. If you have this. Start pushing the number up. Ideally we don’t want to be creating threads in normal operation.

query_cache_size
If your application is read intensive and you do not have application level caches (APC for php or memcache) this can be great help. Do not set it too large as it may slow things down as its maintenance may get expensive. Values from 32M to 512M normally make sense. Check it however after a while and see if it is well used. For certain workloads cache hit ratio is lower than would justify having it enabled. Zabbix is a good example of an app that should not have this enabled.

innodb_flush_method
DO NOT USE O_DIRECT on any virtual machine that is using the SAN. There are reports that using O_DIRECT in a SAN backed MySQL server will slow a simple select by a factor of 3 times. Either fdatasync or O_DSYNC should be used. Leave at the default.

innodb_file_per_table
Insert this in the [mysqld] section so that all the innodb tables are in seperate files and less disk space is consumed. With innodb, once space is allocated in a file, the only way to get it back is to dump and reload the database.

Ah, found it finally. How to do Master-Master Replication and have Failover with Heartbeat and MON of all things. Need to go through this and tweak it for me. Notes soon but for now enjoy this!

http://kjalleda.googlepages.com/mysqlfailover

So we are in the process of migrating from RHEL 5 XEN in various patched states to XenServer w/CentOS 5.2

For some of our testing we do not have direct access to our current SAN and are still waiting for a new SAN to arrive. In the meanwhile we are using the Local Storage Repository (LSR) with a nice USB drive hanging off the front.

A little upfront about Citrix’s XenServer, it’s RHEL derived. When doing the default install it creates everything as LVMs this has its benefits and drawbacks. This makes it really easy to add storage capacity but makes it easy to corrupt data if the underlying structure is not on a hardware raid (yep, not following best practices on the dev system – but that’s why you backup and backup often)

Anyway on to the meat of the post – Quick little trick to get the new disk in to the LSR.

First, turn the new disk in to a Physical Volume (PV) that can be used by the LVM:
pvcreate /dev/sdX

You’ll then need to extend the Volume Group (VG) by adding the new PV to it:
vgextend "VG Name" "/dev/sdX or path/to/physical/device"

To get the VG name:
vgdisplay
It will list all the Volume Groups on the server.

Now the LVM has been extended to the new size but you have to tell Xen that there is an update as well to do this you’ll need to do a few things.
We have to know the UUID of the Storage Repository (SR) that we want XenServer to re-scan and update its information about. The UUID can be found from running “vgdisplay” and dropping the “VG_XenStorage-” from the “VG Name”.

xe sr-scan uuid=SRUUID

And bam! We have increased the LSR size. Just don’t forget to keep regular backups!

Just to say that I’m still alive and going. I’ve been consumed by work, so much so that I’ve not had much time to put toward the writings that I want to do here nor really had the drive at the end of the day (but I’m working 18 – 20 hour days for the next few weeks).

In the beginning there was a blog and the blog was…well a blog, about technology. Welcome to the place I showcase the work I’ve done and my experience with technology. Here I hope to post every now and again my two cents on technology. Sometimes I’ll try to write in a how-to manner things that I’ve found tricky to do because there wasn’t much help from the great search god , or I just want to keep the information in a centralized spot of sorts. I hope that you’ll poke around a little and take a look at some of the projects that I have going on. Some of them will have svn repositories available once the alpha version is ready.

Thanks and happy coding.

Ryan