Difficulty in Learning to Program Securely

There are some general tips for secure programming.  Check pointers.  Validate input from users.  Use parameter binding when performing SQL.  Those are pretty obvious.  Every day, more and more alerts go out about security vulnerabilities.  In almost none of the alerts does the actual security hole get discussed.  You sometimes see example exploiting code, but you almost never see example exploited code.  How are software engineers supposed to learn from the mistakes of others?  It seems like we have to repeat the mistakes of others for us to learn those lessons.  I understand proprietary software vendors don't want to release the source code for their software and they fear that doing that would allow more people to hack their software, but how as an industry can we grow if the details are hidden from us?

Time and time again I hear about another company or another product being vulnerable to a major security flaw.  Many of those flaws sound like they are preventable.  The art of programming securely doesn't seem to exist.  Blindly following guidelines doesn't always work.  It often makes the problem worse.  Developers need to be educated about writing secure code.  I find it amazing that an industry that is so important isn't allowed to learn from its own mistakes.  Everything is kept secret.  Our industry will always be doomed to repeat itself as long as new developers are added to the workforce.

When GPS Attacks!

My mom recently purchased a house.  She was so excited about her new home that she wanted to host Christmas.  A few weeks later when Christmas came rolling around, I packed up my car and drove to her house.  I used Waze as my navigation app.  For most of the ride I followed the same path I always follow to visit my mom.  After we got closer, that is when the GPS kicked in and started taking us down roads I was less familiar with.  I blindly followed the GPS until we arrived at a house that looked surprisingly empty.  The house was the right number, but the street was the wrong name.  I fired up Google Maps and it wanted to send me to the same location.  The township and zip code were correct.  Even the neighborhood name was correct.  I called my mom and she told me the intersection she lived on.  Luckily, I was only 2 blocks away.  Two minutes later and I was at my mom's new house.

After my brother gets there, he told me that he had meant to tell me not to use the address when using a GPS.  He told me to use the cross street.  Apparently all GPSes take you to the same wrong house.  Even my mom's TomTom took her to the same wrong house.  This would have been just a funny family story if a different topic didn't come up during dinner.  The underwriter of the homeowner's insurance was cancelling my mom's policy and she doesn't know what to do.  My brother is involved with that industry (home inspections) so he wanted to read the letter.  The letter explained that they had driven by the house and saw that the roof was in bad shape.  They would not insure my mom's house until she pays for a new roof and get the house re-inspected.  My brother would never have let my mom buy a house with a bad roof.  He would be able to recognize a bad roof as well, since he worked as a roofer while going to school.  He goes out trying to figure out what is wrong.  His only guess is that the underwriter didn't like the awning over the back porch.  That is when I brought up the GPS fiasco.

This wouldn't be the first time a company caused harm due to GPS issues.  Now my mom has to call up her insurance company and try to explain that there is a high probability the underwriter went to the wrong house!  So much can be lost in translation.  This underwriter is going to cause lots of stress and may cost my mom lots of money if they don't admit their mistake (if they made one).  At this point, I don't know if the problem is the awning or the misplaced house, but I'm leaning towards a misplaced house.

Target Data Breach and the Lack of Journalism

I work pretty close to a Target.  So close that my wife sends me Target to pick up baby supplies from time to time.  Unlucky for me, my wife sent me on November 27th.  That is the first day that credit cards started getting stolen from Target.

Like all Target customers, I want to know what happened.  As an avid Krebs On Security reader, I knew I would find a great explanation on what happened there.  For technical news that makes it to the mainstream, I like to compare what the mainstream says versus what the technical sites say.  For issues like the Federal Affordable Care Act exchange, I found the mainstream to report on the opposite of what the technical sites reported on.  For the Target data breach, I found that nobody was doing any actual investigation, except for Brian Krebs.

Every article I found about the data breach seemed to have similar wording as the various blog posts on Krebs.  Some articles seemed to quote Krebs word for word but didn't actually put quotes (") around the words.  All of them refer to Krebs but most of them never actually linked to the source material.  When I finally pulled up Krebs, I was surprised how much more information was available in his posts versus what was in the mainstream media.  He goes into detail about the discovery process and other interesting details.

I find it disappointing that the media doesn't link to Krebs'es website.  The reason they call it the World Wide Web is because sites link to each other.  Online news organizations don't want you to leave their website.  They also can't rip off an entire expose by Krebs.  I am fine with them giving minimal (dumbed down) information.  When articles are posted about international relations, economics or California state law, I appreciate that the articles are dumbed down.  This allows me to quickly get a general understanding of the topic.  I wish they would post reference links so that I can easily drill down to learn more details for the topics.  Some topics, like economics, I don't fully understand but I can understand the more technical publications.  At first, I thought this desire was a result of reading too many academic papers.  Then I remembered high school English class where I was taught to cite my sources.

As for the Krebs coverage of the data breachs, the part that I am most interested in was not covered: there was no detail on how the data breach occurred.  This is not a surprise.  The breach is too new.  I'm sure one day in the future I will be reading the front page of Krebs and will see a post about how it happened.

Old Production Code Can Have Defects

I have been struggling lately with this idea that once code works in Prod, it is infallible.  I have worked on a few defects recently where we identified bugs in code that is over 4 years old.  The recommendation always included fixing the 4 year old defects, but management seems to dislike that.  In their minds, if the code worked fine for 4 years, then we must have changed something that caused the issue.  While true to a certain degree, the problem lies in the kind of defect that occurs.  I have been seeing two types of "old" defects that manifested themselves recently.

The first type of defect is the good ol race condition.  In my college multithreading class, I was taught that any possible path of execution is just that....possible.  Therefore any possible path that is wrong must be eliminated.  We had to prove that every possible execution path was correct.  Just because you work to reduce the likelihood of a particular execution path, doesn't mean it will never occur.  The reason for this is because of the decision making process that thread schedulers go through.  Changes in your operation system, your hardware, your JVM version or even the code inside of the threads being scheduled could change the order in which lines of code get executed.  This means you must eliminate race conditions because they could cause future problems!  That is now how management thinks, though.

The second type of defect is one where the calling code changed in a way that triggers the defect.  This is the most common  scenario I have seen.  Imagine a Set< String > in Java.  A Set object contains a collection of String objects.  The set is supposed to be unique.  If you try to add the same string twice, then the second Set.add() is ignored.  I saw an implementation of a Set-like container that claimed the same unique property.  The problem was the add() method didn't honor that contract.  You could add the same String as many times as you want.  In this case, it wasn't a String, though.  The developer required the api-user to enforce the unique property by checking the "key" property of all the objects in the set.  It was a very large object, non-changing object (which is why it was being cached).  An error was introduced in the call stack 10 frames up.  This error caused the code 1 frame up to check check for a null key, but it still had the full correct object that it was adding in.  This caused the code to call add() on the set 60k times.  Obviously we ran out of memory.

In both of these scenarios, fix your code.  I don't care how old it is.  A defect is a defect.

Failure to Share Media in Person

I had some family over and my brother wanted to show everyone some videos he created.  He had copies of the video on a Micro SD card.  The problem is my entertainment center in my living room doesn't really support in-person sharing.  I have two devices hooked up to my TV that could play media.  The first is a Windows PC.  This is my gaming PC.  The PC isn't in my living room, though.  It is in my crawlspace under the living room.  I ran HDMI and USB cables from the crawlspace to the living room.  Due to the length of the USB cables, they don't work well for data transfer.  The USB cables are for game controllers.  My brother tried plugging in his Samsung Note and the device wouldn't event recognize the USB connection.

My second device is my MK802 III.  This is the primary display for my TV.  I have MX Player Pro installed.  I put in the Micro SD card, but I didn't know how to "mount" the card at the time.  The only time I use the SD card slot is when I travel with the MK802.  When you boot the MK802 with an SD card in it, the card gets mounted on boot.  After everyone left (when there was less pressure to get it to work) I found the menu option to mount the SD card.

This drove my brother nuts.  All this technology and it wasn't usable for him.  We settled on plugging the SD card into my laptop, then copying the files over SMB and Wifi.  These were HD video clips, though.  Needless to say, it was going to take a while.  The clips were available on Youtube, though.  So, we fired up the Youtube app on the MK802 and immediately ran into issues.  The problem was the MK802 was connecting over Wifi.  While I do have 3 access points in my house, the laptop and MK802 were right next to the one access point.  On top of that, some family members were on their phones and tablets browsing the internet.  My Wifi bandwidth was exhausted.

We switched back to the gaming PC.  My brother fired up IE to go to Youtube and started making fun of me more.  Adobe Flash wasn't installed.  He started saying how I never use the PC and I fired back saying who uses IE?  Google Chrome was installed....and running....with Youtube already opened!  He started playing videos in Chrome and noticed the taskbar wasn't going away when he put the video in fullscreen mode.  I started to talk about how this was a bug in Windows 7 and he cut me off talking about how it doesn't happen to him.

I need to find a better way for people to come over and transfer video.  I should plug an ethernet cable into my MK802.  Since the MK802 is rooted, I should install a Samba server so that other guests can download the files while we are watching them.  I also should do some testing to make sure I can plug in a USB Mass Storage device.  It is much easier to plug in the SD card into a USB reader rather than the slot in the MK802.  The MK802 is semi-hidden while the USB Hub that connects to it is more easily accessible.

Thread Saftey: Protect your Lock

Java provides the ability to lock on yourself.  You can do this using the synchronized keyword as part of a method definition.  Many programmers don't use that form of the keyword, though.  You may have seen this code before:

public class MyObject {
 private final Object lock = new Object();
  public void someMethod() {
   synchronize ( lock ) {
   // actually do something
  }
 }
}

There are two points to having a private lock like this.  For both scenarios, imagine if you synchronized on the method instead of the lock.

1) If a developer extends MyObject, then any methods that they write that synchronize on the method will block your methods as well.  Both codes are critical sections of the same lock.  This could either be a good thing or a bad thing.  If the developer extending MyObject doesn't look at (or doesn't have access to) the source code, then it could be a really bad thing.  Deadlocks of occured doing this.  You can also run into performance issues.  If your class is senative to those types of issues, then I highly recommend making the object final, regardless of the synchronization type you use.  This prevents people from extending your class

2) A developer that is using MyObject can synchronize on an instance of the object.  If the implementation of MyObject is


In my opinion, it is better to be educated so that you can make the right decision for your scenario.  The problem with a 'private' lock is you no longer let users of your api know that a method is synchronized.  JavaDoc and various auto-complete technologies let the user know that a method is synchronized.  They use the method keyword to determine that.  Over use of the private lock pattern leads to a bunch of synchronized methods that are documented as not synchronized.

Thread Safety: Beware of Java's Hashtable, Vector and Collections.synchronize*()

I am shocked at how bad people are at multithreaded programming.  I took one class in college on the topic, and that was all I needed.  That one class gave me far more knowledge than most programmers I know.  This isn't a statement about how good I am....this is a statement about the poor state of programmers today.

One of the first mistakes I see is just throwing a Collections.synchronize*() call around a data structure or using the synchronized data structures.  While very handy, these wrappers give a false sense of security.  Consider the following code:

public class MySet {
 private final List list = Collections.synchronizeList( new ArrayList( 10 ) );
 private final Map map = new Hashtable();
 public void add( MyObject obj ) {
  if ( !map.containsKey( obj.getId() ) ) {
   list.add( obj );
   map.put( obj.getId(), obj );
  }
 }
}

This class makes use of a synchornized wrapper and a synchronized data structure (Hashtable).  Therefore, access to the internals of 'list' and 'map' are protected and thread-safe.  You know what isn't threadsafe?  MySet!  Imagine two threads running at the same time that are both calling add().  Below is a sequence of what line each thread is on.  You will see Thread B stall 3 times waiting for Thread A to finish with the easy-to-use wrappers.

A: MySet.add()
B: MySet.add()
A: if ( !map.containsKey( obj.getId() ) ) {
B: if ( !map.containsKey( obj.getId() ) ) { // B must wait for A to finish with 'map' before it starts
A: list.add( obj );
B: list.add( obj ); // B must wait for A to finish with 'list' before it starts
A: map.put( obj.getId(), obj );
B: map.put( obj.getId(), obj ); // B must wait for A to finish with 'map' before it starts

At the end of the execution, 'list' will have 2 entries in it while 'map' will only have 1.  Congradulations!  You have know written a race condition.  On top of that, your add() method now has 3 critical sections!

The correct way to write the synchronization should look closer to this: (I say closer because it doesn't HAVE to look like this)

public class MySet {
 private final List list = new new ArrayList( 10 );
 private final Map map = new HashMap( 64 );
 private final Object lock = new Object(); // I will explain this in a future post

 public void add( MyObject obj ) {
  synchronize( lock ) { // lock only once, do your business, then get out
   if ( !map.containsKey( obj.getId() ) ) {
    list.add( obj );
    map.put( obj.getId(), obj );
   }
  }
 }
}

In the updated code, we create a singe lock that protects both data structures.  Then, we use the non-thread safe versions of the data structures.  In this case, Thread B must wait for Thread A to completely finish the add() method before it can do the first check.  This code correct.  Correct in this context means there is no order that Thread's A and B can run in that would violate the internal invarience of the object.  It is impossible for 'list' and 'map' to get out of sync with each other.  It gets even better.  Since there is only one critical section, the code actually runs faster!

XSLT-based replacement for JSPs

Every time I start a new project, I like to throw in something new and unique.  I experiment with using new technology or mixing old and new technology.  Sometimes the experiment is a complete failure.  Sometimes the experiment leads to a pattern that I start using over and over again.

For this project, I decided to try using an alternative to JSPs.  I wanted to replicate some of the features that JSPs support, though.  The main three features that I wanted were support for some sort of tag library, support for reading information from some sort of Java bean and support for basic flow control like loops.  There were a few things that JSPs didn't support that I wanted as well.  I also wanted to support unit testing with mock support and support for reading all files out of the classpath as opposed to the filesystem.  I want the app to get deployed as a "bag of jars" instead of a war file that requires Tomcat to be setup.

The System

The system I came up with is merges multiple technologies together, but the main technologies are XML and XSLT.  To understand the system, lets take a single GET request and decompose the main steps that occur.

GET /html/index.html

Step 1 - Create the page-specific bean

Every page gets a bean class and bean factory.  For the index.html file, the bean class would be IndexBean.java and the factory would be IndexBeanFactory.java.  The rendering engine would use reflection to instantiate a new IndexBeanFactory class.  This factory would then create an instance of the IndexBean class using the request object.  The factory would make any midtier calls that are needed, then call setters on the bean.  The bean acts as a simple container for data.  In the JSF world, the bean mimics a request-scoped ManagedBean.

Step 2 - Serialize the page-specific bean to XML

I found a really great Java library for serialization to XML.  The library is called Simple.  It is really easy to use.  In our example, the render engine takes the IndexBean instance that was created by the factory and serializes it to XML.  Now we have an XML document that represents all the data that would be visible on the page.

Step 3 - Transform the Bean XML to HTML and Custom Tags (Intermediate XML)

This is where the "replacement" of the JSP file comes in.  In this example, the developer would write index.xsl instead of index.jsp.  This XSL file contains a mix of HTML and Custom tags.  You can use XPath and the <xsl:*> namespace tags to do logic, like loops.  As a devloper, you are transforming the page bean xml to HTML.  It isn't 100% HTML, though.  The goal is for the "syntax" of this "visual code file" to be easy to learn for JSP/JSTL/Facelets developers.  When you want an HTML tag, you just put an HTML tag.  If you have a "tag library", you can use those tags.  This requires developers to learn XSLT, but as I have pointed out previously, every developer lists XSLT regardless of their experience with it.

Step 4 - Transform the Intermediate XML to HTML

While the Intermediate XML has HTML tags, it also has Custom tags.  The final step in the process is to perform an XSL transformation to translate the Custom tags to HTML tags. This is where your Tag Library developers come in.  They maintain their own XSL file that contains all the custom tags.  You can even import multiple tag libraries together if you have multiple teams doing this.  This step is what allows UI developers to use reusable code in their pages.


In this example, you end up with an HTML file from an XSL file.  Although a bit awkward at first, it should be easy to pick up.  For developers, they are only using technology that they should already know!  Your tag library developers stay in XSL/HTML land.  No more Java code.  Now, lets talk about some of the goals again.


Tag Library

This system supports having a tag library.  Developers use the custom tags in Step 3 while the tags are implemented in Step 4.  My app makes use of JQueryMobile.  For every component that I use, I have created a custom tag for it.  This makes it fast for me to write pages, since I don't need to copy and paste the HTML for a ListView over and over again.  The tag library is even segregated, so if you had a special team that owned this file, they are not sharing the file with the developers.  These xsl files are read from the classpath, so they can even be bundled with different jar files.

Reading from Java Beans

In the JSP and Facelets world, developers use Expression Language (EL) to access various Java beans.  With this technology, we serialize a single bean (tree) into XML.  This allows us to use XPath to read from the "Java Bean".  Although we are technically reading from XML instead of the bean, it can conceptually be thought of as reading directly from the bean.  This is implemented in Step 2.

Flow Control

Flow control that simulates JSTL is implemented in Step 3.  Developers use standard XSLT tags like <xsl:forEach> to do flow control.  These contructs are not new.  Developers should be semi-familiar with it.  The learning curve should be pretty minimal.

Testing with Mock support

Every step in the pipeline can be unit tested.  These previous step in the pipeline can be mocked to perform a true unit test.  You can even integration test the entire pipeline by mocking out the midtier calls.  You can use XmlUnit to to perform an XPath assertion.  You use Test Driven Development on your UI Layer!  You can unit test your bean factory and xsl page seperately.  You can unit test without starting Tomcat!

Reading from the Classpath

Xsl files are read from the classpath, not the filesystem.  This means you put them in src/main/resources for Maven projects.  In Debug mode, you can modify the xsl files and hit refresh in the browser.  You can TDD the xsl files using JUnit.  You can split out different parts of your website into different jars (rcs repositories) so that different teams don't conflict with each other.

Android Support

All of this technology uses either simple 3rd party Java libraries or parts of the JRE.  Most of the 3rd party libraries advertise the fact that they work on Android.  While I am using a servlet engine, that servlet engine (TJWS) advertises the fact that it works on Android.  On top of that, there is nothing in the design that forces it to work with a servlet engine.  In my current implementation, the factory object gets an HttpServletRequest object passed in.  Switching that out with a generic "request" bean would completely decouple the system from a servlet engine.  Alternatively, you might be able to implement the HttpServletRequest interface in a way that wraps the native request object for your webserver.  Either way, you now have a UI technology that works on Android.

Ease of Development

Another feature I implemented that makes development easier is segragation of the pipeline.  You can request the outputs of steps 2 through 4 just by changing the url.  In the example above, we requested index.html.  This gives you the output of Step 4.  If you were to request index.xml, then you get the output of Step 2: the serialized XML of the Java Bean.  If you were to request index.phtml, then you get the Intermediate XML that contains a mix of HTML and Custom tags from Step 3.  A developer can hit Step 2 in the browser to see what the XML structure looks like so that they can write the XPath expressions correctly.  Tag developers can look at Step 3 to see the before and after of their Tag transfomations.

Internationalization

Step 4 also performs simple variable replacement out of a ResourceBundle.  This allows built in internationalization that follows Java's best practices.  Java and Android tend to use different internationalization systems, though.

Performance Considerations

One thing to keep in mind is the scalability of your system.  One thing that JSP has over this system is performance.  JSPs get translated into Java code which gets compiled to class files.  The tag libraries are implemented in Java, so they are pre-compiled into class files.  The class files are JIT-compiled into (hopefully) optimized assembly.  This means a JSP-based system will perform better than this system.  On the other hand, these performance implications are the same for Facelets, so if you are using Facelets instead of JSPs, you shouldn't see much of a performance impact due to the XSLT transformation.

Ubuntu Tablet PC (Toshiba Satellite u925t)

Ubuntu installed pretty easily on my Toshiba Satellite u925t.  I loaded Ubuntu 13.10 onto a USB stick and it installed without any issues.  The laptop wasn't as aggressive with Secure Boot as my Acer laptop was.  All the hardware that I use worked out of the box.  I tested Wifi, the touchscreen, the mouse pad and the camera.  I did not test Bluetooth, but I do see the icon on my system tray.

While the touchscreen is pretty accurate, I did run through a bunch of customizations to increase the sizes of tool bars and other things.  Having larger handles is just easier.  I use Onboard as my on screen keyboard.  I launch the keyboard on startup.  I keep it hidden, but I allow it to pop up when it detects text input.  I also keep the touchable icon on the bottom left of the screen.  I use the "Small" layout, which is the closest to what I want.  I wish the hide button was a regular key in that layout though.  I don't always want the keyboard to show up and I found it annoying to long-press the Enter key to bring up another popup to hide the keyboard.  I have not figured out a way to configure Onboard to not show the keyboard when I'm in laptop mode.

I installed Google Chrome, but I found that Mozilla Firefox is much more touch friendly.  Google Chrome strives to have the same look and feel across all platforms.  This means it ignores the size changes I made to make the toolbars and scrollbars larger.  I installed a few Firefox Addons that allowed me to scroll using a drag gesture and to long-click to open in a new tab (there is no way to right-click in tablet mode and the u925t's touchpad does not right-click very well).  As always, Flash blockers are essential to preserve our battery life when browsing the internet.  I installed Genymotion to allow me to run Android apps.  I will dive deeper into Android-on-Linux in a future post, but overall it runs ok.

I configured the power button to put the laptop into suspend mode when pressed.  This makes the laptop feel more like a tablet.  I increased the Unity Panel size to 54 pixels.  I haven't figured out a way to right click while in tablet mode, though.

Overall, I have very happy with this setup.  I spend most of my time in tablet mode.  I go to laptop mode when I blog or when I program.  I believe that this style of operating system is the future.

Feature Request: Optional Permissions in Android

Imagine yourself as an Android developer for a bank....lets call it JeffBank.  Banks usually have two types of customers: people with deposit (checking/saving/CD) accounts and people with loans.  Now, JeffBank wants to roll out a mobile app.  You are writing screens for two different lines of business.  This isn't that abnormal.  It happens all the time.  When someone logs in, you get one of three different experiences: deposit, loan and both.  You roll out the app and the negative reviews start piling in.

You start reading and you realize people are complaining about permissions.  You see angy comments questioning the need for access to the camera.  You see acusations of being involved with the NSA because you have access to read the GPS location.  One thing I learned from being an Android developer is that Android users do look at the permissions an app uses, and are very vocal when they feel you are doing something "shady".

This problem stems from the fact that your loan users are getting a deposit account app as well as a loan app.  Users tend to think of themselves, and not about the company they are doing business with.  While a bank is a little more obvious than other types of split personality companies, your loan users still don't understand why you need a camera or GPS for making a loan payment.  You can try to calm them down, but you don't always know who they are because they never actually installed the app!  You can't tell them that if you had a checking account, you could deposit a check by taking a picture of it, or that if you needed an ATM, the app would allow you to find the closest ATM.  Even if you try, would your loan customers even listen to you?

What Android needs is an optional permission system.  When a user tries to install an app, they should see a list of required and optional permissions.  For the optional permissions, the user would have the option to revoking those permissions on install or after the fact.  This allows JeffBank to make a mobile app and not piss off the loan customers while still providing features to the deposit customers.  Developers would still have the option for required permissions, like internet access.  It would be silly for a bank app to flag that as optional.

Android 4.3 introduced a hidden screen that allows you to turn off some permissions for apps, but there is no telling how the app will respond to permissions being turned off.  Current developers aren't developing for that.  If you turn off any permission and it breaks, then the developer is just going to tell the user "well, don't do that!"  Right now, all permissions are required.  By giving that power to the developers as opposed to the users, you allow the developers to do if-checks to make sure the permission has been granted.  If a deposit user has no plans on using the ATM locator feature, they can disable the GPS for the app.  The mobile app could detect that it doesn't have permission to read the GPS location and can hide the menu option, or show it, but have an error message directing them to give the app access to GPS.

Overall, this feature would create more trust between mobile users and mobile developers.  It would cut down on hate-reviews asking why JeffBank is trying to take naked pictures of them.