JS Ext

Wednesday, May 14, 2014

Another hard disk dead

It's that time of year again.  The seasons are changing.  I have to mow my lawn again.  The temperature is getting warmer.  Hard disks are dying!  I had expected that moving my server into my basement would reduce the deaths of the hard disks.  This death might be a little unique, though.  The hard disk is brand new.  I bought it 3 months ago.  It is a Hitachi 4TB hard disk.  I never had a Hitachi before, but it was 4TB and it was on sale for the same price as Seagate and Western Digital 3TB disks.  I decided to take a chance.  At 3 months, I consider this hard disk a dud.

My server has 7 hard disks attached to it, although one of them is an SSD (the operating system).  Five are inside of the case while 2 are in an external eSATA enclosure.  The Hitachi is in the external enclosure.  I know the enclosure is fine, since there is another hard disk in it that is working fine.  This hard disk was my first chance to really expand my capacity.  Most of my hard disks are between 1.5TB and 3TB.  I previously combined two 1.5TB drives into a single 3TB device and two 2TB into a 4TB device in my RAIDZ.  The 4TB drive replaced the two 1.5TB drives, matching the capacity of the two 2TB drives.  With the 4TB dead, I'm running in DEGRADED state until a replacement can get shipped to me.  I don't need the capacity yet.  Right now, the biggest thing driving my data growth is doing backups of my Windows PC, but I barely use it anymore, so I was able to reduce the backup frequency.

Monday, May 12, 2014

iPhone 5 and Galaxy S5 - There is something about 5

I have seen lots of reports about the design chief for the Galaxy S5 stepping down.  He isn't stepping down because the S5 was a bad phone.  He is stepping down because the S5 was only an incremental upgrade over the S4.  The S5 had some great upgrades, but none were earth-shattering.  My buddy upgraded from an S4 to the S5, mostly because of the fingerprint scanner.  He was greatly disappointed, though.  From the outside, they are so similar he accidentally brought his S4 home from work, thinking it was his S5.  My wife is going to upgrade from an S2 to the S5.  She considered the S4 which would have been free, but we decided that the IP67 rating of the S5 and the Qi charging (semi-false advertising, but that is a rant for another day) was worth paying a little bit for the S5.

This whole situation reminded me of the iPhone 5 rollout last year.  The S5 is an incremental upgrade similar to how the iPhone 5 was an incremental upgrade.  I wrote about the iPhone 5 upgrade last year.  I will dive deeper into the S5 next time, but for now, I want to compare the "upgrades".

Charging

The iPhone 5 released a new charging connector that was not compatible with older iPhones.  While this new connector was better, it also meant you had to buy all new connectors.  All the old wires were now useless.

The S5 continues to use the standard Micro-USB.  All your old cables still work.  There is talk about the S5 supporting Qi wireless charging (another standard), but it turns out you have to buy an after-market replacement cover from Samsung.  Unofficial qi receivers are available, but they remove the IP67 water resistance.  If you buy the cover (which we are going to do) then you have a more convenient way to charge your phone that supports a standard.

64bit (iPhone 5S)

The iPhone 5S is 64bit.  The S5 is still 32bit.  Right now it isn't that important, but it will become a big deal in the future.  We don't know when in the future it will become important.  While 64bit does increase your performance, so does doubling your cores (S5 has 4 cores).

Size

The iPhone 5 was about 20% lighter and thinner.  That is nice.  It isn't what most people want, though.  Many people like having the bigger screens.  Rumor has it Apple has finally got the memo and the iPhone 6 might have a larger screen.  The S5 on the other hand....didn't change.  It looks identical to the S4.  At least the iPhone 5 changed something.  My buddy wouldn't have made the same mistake with an iPhone 5.

Post-Launch Marketing

As a general rule of thumb, ignore all pre-launch marketing.  That is why we are jumping right to post-launch.  After the iPhone 5 launched, Apple continued to market it as if it was a huge improvement over the iPhone 4S.  They wouldn't acknowledge that it was an incremental upgrade.  Apple portrayed the iPhone 5 as the next big thing (hehe).  Samsung on the other hand realized that the S5 was an incremental upgrade after launch.  That is why the person in charge stepped down.

This is where I think both companies got it wrong.  There shouldn't be anything wrong with an incremental upgrade.  You shouldn't market it as something great, or fire someone for mediocrity.  Most people get upgrades every two years.  They tend to skip a model.  When asking people why they were upgrading to the iPhone 5, the most common response I got was it was time to upgrade...since they were on either the iPhone 3 or the iPhone4.  Apple does have a lot of fan boys that will pay for the yearly upgrade, however.  Samsung has to compete with Apple, so it sticks with the yearly rollout as well.  These companies should acknowledge this. Besides, what else can you really put into the phones?  Apple has some catching up to do (screen size, wireless charging, IP67), but not by much.

The next big thing should come out next year.

Wednesday, May 7, 2014

Make a Backup Before Upgrading Ubuntu

I am a little obsessive about backing up my data.  I use ZFS with RAID-Z to prevent data loss due to silent data corruption.  Since ZFS can be problematic due to its licensing, I also have a USB hard disk that contains a copy of all my data.  In the event that my house gets destroyed, all my data is backed up to an EC2 micro instance.  My EC2 instance runs Owncloud with the data stored on a ZFS volume.  Since the EC2 instance is outside of my home, I try to keep it up to date.  During the last update, serious problems occurred.

After upgrading my EC2 instance to Ubuntu 14.04, my ZFS volume disappeared.  Since there was a kernel upgrade, I assumed the zfs module just needed to be recompiled.  Finding the instructions on how to do this was pretty painful, but that is a rant for another day.  The recompile of the zfs module failed, however.  The zfs module won't compile against the 3.12 version of the Linux kernel.  I would have to patch my kernel to get it to work.  I decided to try and roll back the kernel, but that just made things worse.

This is where I wished I took a backup.  Not a backup of my data.  A backup of the Ubuntu OS.  Now that 14.04 is out, I am having a hard time finding Ubuntu AMIs for older versions that work.  I fired up another EC2 instance using Gentoo.  Gentoo is the distro I use at home, and I know ZFS works well with it.  I used this instance to import the ZFS pool and copy the data to an ext4 volume.  I will create a new Ubuntu 14.04 instance that will use the ext4 volume for Owncloud.

This experience has showed that I did a few things right.  First, my data was on a different EBS volume.  I did backup the Mysql database, but not with enough frequency.  The software I installed was pretty minimal and minimally customized.  This makes it a lot easier to create a new instance and get running.

I definitely have a few things that I know I need to do in the future.  First, I need to take snapshots of the OS.  Second, I need to keep the configuration backed up as well.  That way, if I need to create a new instance, I can more easily recover.  Finally, I should back up the Mysql data to the data volume.

Monday, May 5, 2014

Static methods for all....and (another) reason to not use them

When I switched to my current team, I was surprised to find an endless supply of static methods and variables.  It seemed like the team really liked them.  I can understand why.  Most of the team were younger and weren't really used to design patterns and object orientation.  For them, the only reason to extend and object is to make it convenient to call common methods.  Being new to the team, I didn't try to force a paradigm shift.  I brought it up a handful of times, but everyone seemed to like their static methods.

Recently, however, I was tasked to fixing a recurring issue.  Inside of our Android app, some of our static variables were being reset to null.  It seemed like it happened when the app crashed, so we put in code to reinitialize the static variables in the Activity that gets called when the app crashes.  We were still getting sporadic null values.

Then a major problem occurred.  We started using a 3rd party Android library that was returning null values.  I disassembled the code and the developer had used static variables to store some values.  Sound familiar?  At first it happened when the app crashed, but then we noticed it happening when we ran out of memory.  The screenshot library has a memory leak....ugh!  We plugged the memory leak and the nulls in the 3rd party library went away.  The incident got me thinking.

I started doing some research on the topic.  I found lots of places talking about Android memory management.  There seemed to be a debate of where to put long-lived objects.  Some (including Google) recommended using a static variable to create a singleton.  Others suggested keeping a reference to it from the Application class.  I then found more details on what we think is going on.

First, a few things that you might not realize.  Java classes are loaded into memory.  If nobody is using a Java class, then that class can be unloaded from memory.  If someone decides to use that class again, then the class can be reloaded into memory.  This type of thing might happen if you run out of memory.

Apparently, one of the differences between a JVM and Dalvik is the definition of a "class being used", or when is it OK to unload a class.  It looks like the JVM looks for references to the class while Dalvik looks for references to an instance of the class.  It turns out that it is a very important distinction.  In the above examples, the entire class was static.  All the methods and variables were static.  There was no instance of the class.  There wasn't a need for one since everything was static.  This works in the JVM world, but it doesn't work in Dalvik.  When you are low on heap space, Dalvik will unload the static class, since by it's definition, nobody is using the class.  Then, someone calls a static method which reloads the class, reinitializing the static variables, usually to null.

I made the decision that we shouldn't use completely static classes any more.  While it is generally bad practice to use static only classes, I had a concrete technical reason to stop using them.  I rewrote the code to use instances of the classes that are referenced from the Application class.  Anyone that wants to call them needs a Context.  Luckily almost all of our code already had a reference to a Context.  We haven't had an issue since I did the refactoring.

Sunday, April 20, 2014

XP is close to dead (but why is it still alive?)

Now that Windows XP is officially out of support, various websites are reporting the current Windows XP share.  I have decided share my stats.  Currently, 5.7% of the Windows visitors to my blog run Windows XP.  That is not a tiny amount.  That is more than my Windows Vista readers (3.16%).  This begs the question, why is it so high?

1) Timing

Windows XP was released in 2001.  This was around the time that non-technical people were getting computers.  Services like @Home were providing broadband internet for the masses.  This means the operating system that those masses of people were running was Windows XP.  Just about every computer you purchased at the time was running Windows XP.  For many of the current internet users, Windows XP was their first operating system.

2) Backwards compatibility

This is something that affects business users more than consumers.  In the workplace, your computers often have speciality programs that MUST work.  This means upgrading has a cost that is much larger than the licensing cost for the new version of Windows.  There is massive testing effort.  Every native program must be tested by QA engineers before a roll out.  To make things worse, many programs need to be heavily modified to work in newer versions of Windows.  This is not always cheap, or even possible.

3) People dislike change

This is probably the biggest reason.  Because XP was the first version that most people used, people know how to use it.  They know the ins and outs of the OS.  They know how to update drivers, open task manager, browse the network and change the desktop wallpaper.  Users have spent users getting used to these administrative interfaces.  Microsoft has a habit of completely rewriting all of those interfaces for every major Windows upgrade.  Users don't want to relearn how to do all of those tasks.

Tuesday, April 15, 2014

The Screenshot Saga: Episode 2 - Background All the Things!

We get the fix pack, and we did a quick regression test.  Everything seemed like it still worked.  I told my project manager that it would take me some time to integrate the server flush fix.  You see, the method we call to flush to the server already worked in one way....in the foreground.  Any professional library developer would know that you need to maintain backwards compatibility.  Therefore, I assumed that they would provide a new method for the background version.  I was wrong.  They changed the existing method.  Our issue (the freeze after taking a screenshot) was gone.  I checked everything in, let Jenkins build the APK and told our tester that it was ready to test.

We are near the end of our release cycle so it took a few days for the tester to actually test the screenshot functionality.  I was in for a nasty surprise though.  The tester reported that the screenshot functionality wasn't working!  I fired up Logcat to figure out what was going wrong.  The "fix" pack did a lot more than just fix our one issue.  While it also fixed a bunch of other issues that other clients complained about, it also reduced the logging to Logcat.  After 2 days of looking into it, I finally disassmbled their Jar file.  I used the debugger to step line by line into their code.  That is when I discovered the first problem.

The very first thing the library does is send the killswitch request to see if the library should be enabled.  That is done in the foreground, delaying the start of the application.  One of this company's clients complained, so they put the killswitch request in the background.  The problem is the library initialization takes two steps:  start() and enable().  start() fires off the killswitch request and enable() uses the killswitch response to determine if the library should start up.  Since start() now runs in the backround (and returns immediately), enable() has a high likelihood of failing.  If the server is running fast that day, then enable() will work.  That is why my quick regression worked when I was first given the fix pack.  While looking at the source code for enable(), I noticed that it checks an internal boolean to see if enable() already ran successfully.  This means I could call enable() right before I take the screenshot.  It is a hack, but at least I'm not calling undocumented API calls, like I did in my previous hack.

The next issue was that not all the screenshots were being sent to the server.  This one drove me nuts.  It took me a while before I decided to look at the source code for takeScreenShot().  I was appalled.  What used to be a foreground action was now in the background!  You see, another client complained about something being slow.  This time is was the screenshot code.  The library developers did what they always do: they shove it in the background.  The problem was I call the flush code right after I take a screenshot.  Yay race condition!  There was no guarantee that the screenshot was done being taken before we invoke the manual flush.  Once again, this was something that would work part of the time, which is why it passed our quick regression.

It is obvious that the developers that own this library don't know anything about multithreading.  You can't just throw everything in the background as soon as someone complains about code running in the foreground.  It requires thought and planning.  There are ramifications.

The next thing that irritates me is the fact that some features are barely supported.  If a company provides half ass support for a feature, then say it up front.  Don't let us get invested in a feature that is just going to cause us more problems.  In this case, it was the manual flush.  My company wants to guarantee that the server gets the screenshot.  They don't want to gamble with the possibility that the user kept our app open for another 60 seconds.  If your app/library implements a feature in a shitty way, then you really don't support that feature.  Let us know that up front and move on.

Sunday, April 13, 2014

The Screenshot Saga: Episode 1 - Customer Support Attacks

I recently had the (dis)pleasure of adding a 3rd party library to my company's Android app.  While this library had a few features, we only needed two features: the ability to take screenshots (of our app) and the ability to turn that functionality in the event of a catastrophic failure.  The idea was to take pictures of certain transactions in the event that one of our customers wanted to dispute a transaction (I said $100, not $1000).  The developers had exactly no say in the picking of the library.  We were not even allowed to implement it ourselves.  We had to use this library.

The company that wrote this library owes me a nice bottle of single malt scotch.  Customer service was a nightmare.  For starters, the killswitch didn't work.  It eventually came out that this was my company's fault, but it took some....extreme....measures to figure this out.  The library fires off an HTTP GET and looks for a response.  The developer who wrote the JSP used the JSP Designer view, not the text editor view.  Therefore, the JSP didn't return the needed 0 or 1.  It returned a <html><body>1</body></html>.  A little bit of logging would have helped on that one.

The next issue was the fact that we couldn't take a screenshot of the entire scrollable area of the page.  It didn't matter what View you passed in.  The library would get the root View of the page and take a screenshot of that.  The company kept giving us some blanket statement about how only our "developers know when the entire view is visible".  I was not happy about their support staff's veiled insult towards me.  I eventually disassembled their Jar file.  I tracked down the one line where they inadvertently took the screenshot of the root view, not the passed in view.  I got so frustrated with their support staff, I actually pulled up their source code on a WebEx!!  They still didn't believe me.  That issue was fixed in a new release of the library.

The next issue was sending the screenshot to the server.  Taking the screenshot doesn't automatically send the screenshot to the server.  You have to either wait for the flush interval or use a manual flush.  My company opted to use the manual flush while most other clients used the interval.  It turns out that so few people actually use the manual flush, that nobody noticed that it runs in the "foreground".  There was a noticeable freeze in the app.  I disassembled their code and turns out that call new FlushTask().execute().get();  For those of you not familiar with Android development, what that means is the flush is run in the background, but we wait for the flush to finish....essentially negating the fact that the flush is running in the background.  My first instinct was to run it in my own AsyncTask, but for some reason, the FlushTask actually used a thread local variable, causing it not to work when invoked from a background thread.  Why would they do this!!!  Luckily FlushTask() was public so I could invoke it directly.  Technical support was no help.  First, they told me to just use my own AsyncTask.  Can you believe we pay them money for this?  Finally they tell us that another company complained about the same thing and that their next fix pack was going to fix the issue.

Our saga continues.  We got the next fix pick and it was a mess.  Tune in next time!