Scala Play 2.0 web services on Heroku

I noticed two issues with Scala Play web services on Heroku.

Firstly, for some reason my initial implementation worked when running Play locally, but did not work in production on the Heroku server. The approach was using java.net.URL as an input stream and blocking while the content was downloaded. So, don’t do that (the approach that worked is detailed below).

Secondly Heroku uses a reverse proxy between the user and the web server, so getting the user IP isn’t going to work as it will be the IP of the proxy instead. So what you need to do is look for the “X-Forwarded-For” attribute that the proxy adds to the header information for you.

Here’s what worked for me:

def index = Action { implicit request =>

   val ipAddress = request.headers.get("X-Forwarded-For") match {
      case Some(ip: String) => ip
      case None => request.remoteAddress
   }
   val ip = if( ipAddress.equals("127.0.0.1"))
      "10.0.0.1" // insert your IP here for development mode testing
      else ipAddress

   val promise: Future[play.api.libs.ws.Response] = WS.url("http://freegeoip.net/json/"+ip).get()
   Async {
      promise map { res =>
         val locationJSON:JsValue = Json.parse(res.ahcResponse.getResponseBody())
         val lat = ( locationJSON \ "latitude" ).asOpt[Double].getOrElse(0)
         val long = ( locationJSON \ "longitude" ).asOpt[Double].getOrElse(0)
         val city = ( locationJSON \ "city" ).asOpt[String].getOrElse("")
         println("City: "+city)

         Ok(views.html.location(city,ip))
      }
   }
}

What is Systems Engineering – A simple explanation

If you look in the regular places, you’ll see that Systems Engineering is described, but the words are all Gartneresque. That’s all well and good for a wiki or report, but what does Systems Engineering actually mean?

Basically, it’s a process to reduce complexity.

First of all we define everything as a System. A computer can be a system, a factory can be a system, even people following processes can act as a system. In fact the details of what’s inside the system can even be largely ignored at the start of the process. A system has inputs and outputs, but at a high level, so long as the outputs are correct relative to the inputs, the insides of your black box system could be microchips, cogs, bacteria, or something else entirely. It really doesn’t matter.

To solve the problem, the first thing you do is describe the problem as a series of inputs and outputs, which form your requirements. Whether you’re transporting cargo, processing information, or trying to achieve some effect, your system will have requirements, which define the boundaries of that system. For example a space shuttle may need to deliver one ton of cargo to low earth orbit within 24 hours of launch. As you can see, the requirements don’t detail the solution, only the boundaries of the problem.

With your requirements in hand, you now have the boundaries of the system. You have a black box, and ‘something’ will go inside that box to solve all of your problems.

So how does this reduce complexity? The black box, ie the system can be reduced into smaller and smaller sub systems. At each reduction you are architecting the system and defining the boundaries between which the engineers of each sub system will work. For example, you might break a space shuttle down into a fuel subsystem, life support subsystem, navigation subsystem, etc. This makes a lot of sense, because the chemical and mechanical engineers designing the fuel subsystem, likely don’t care about and don’t know how to engineer the software navigation subsystem, or the medical life support subsystem, so they interface through the tightly defined boundaries created when breaking the system down into its subsystems. These boundaries make the problem workable and ensure the best person for the job is designing the exact details of each subsystem. This is a good thing because too much detail in an architecture can miss the mark through naivety, you really just want to design the boundaries and the effects achieved by the super system.

Systems Engineering then focusses on Verification and Validation, where each sub system is tested against the requirements, and the overall System is approved to undertake the mission.

Without the clearly defined boundaries, a complex System can quickly become something unwieldy. Using the space shuttle again as an example, what if the fuel line requirements were not clearly defined? It’s easy to imagine a scenario where engineers (who are often not in the same building or even country) get their wires crossed and the fuel is not delivered fast enough, or at the right temperature, or even the right fuel type for the engine. Would the shuttle take off? And if it did, will it achieve the mission, or or will it fall back to Earth? Even if the fault was discovered before the shuttle is launched, the costs of re-engineering can be astronomical (no pun intended). Particularly delays, where you may have to pay a bunch of staff to do nothing for weeks, re-engineering costs can quickly add up.

Another example is anything relating to Services Oriented Architecture or SOA. In SOA, every service can be regarded as a subsystem. It can be quite flexible, as software services can be reused with no additional work, and depending on how well the system is designed it can degrade gracefully if services become unavailable or have not yet been developed.

Having well defined boundaries and a series of tests to be performed on each subsystem, depending on which requirements the subsystem had allocated to it, allows the engineers on each subsystem to ignore the other side of the boundaries, which reduces the complexity of the problem to something manageable.

Traceability:

As you move through the process, the complexity of the links between every line of every artifact increases, as if absorbing the complexity of the system. It can be difficult to manage these traces, but worth it to maintain the context of the system. Without context the purpose of a sub task is often lost (“Why are we testing this nut against ISO1404 again? Why does this wire need to transfer 10Mbits? Can I get away with using a cheaper cable?”). But with the context of traceability you can not only see exactly why each task is important and which stakeholder is asking for it, you can be sure that every subsystem is tested and every requirement is met. Being able to trace back to stakeholders is also great, as anyone can identify the stakeholders originally responsible for every component of the system being built and tested, and ask them for feedback and clarification if necessary rather than simply being lost in some enormous process.

Agility:

A common criticism of this process is that it all sounds very ‘waterfall’, something that has been replaced by the ‘agile’ software development life cycle. But three points should be made:

1. A lot of what people these days call agile is simply staged waterfall. You can still stage the requirements under this method. You can also stage the order the subsystems will be implemented, and even defer finalising or even deciding on whole parts of the design, because they’ve been neatly compartmentalised for you through Systems Engineering. You can use Systems Engineering to define the goals of the subsystems, then use the Agile method to develop any or all of them.

2. An Agile organisation is one that is not burdened by classical engineering like this. But really their flexibility does not come down to their technical people, these extremely agile startups are actually in search of a business model. And it goes without saying that you can’t architect a system to meet the user needs if you don’t know what the user needs are in the first place. Under these conditions you don’t use Systems Engineering, but the fact that the system under development needs to be constantly re-engineered is quite wasteful, a price you just have to pay for continually going back to the drawing board. For established businesses however, the project needs can usually be quite easily determined and documented, making the entire process much more efficient, which saves both time and money. Systems Engineering can be the difference between a project being feasible or not going ahead at all.

3. Sometimes the Systems Engineering process itself can be quite costly, depending a great deal on the tools being used and the level of mandatory processes and regulations in place at the organisation. The size of the system will determine whether the costs of using a rigid systems engineering process is proportional or not. For vehicles, infrastructure, any enterprise software, weapon systems, etc, it would be crazy not to, but for a very small component individually (maybe a resistor or a motor), the complexity of that job might not warrant the overhead of the full systems engineering process. Once again though it’s up to the tools and the organisation, Agile Systems Engineering is definitely possible

And at the end of the day if done right, eliminating all of that waste and having your systems delivered on time and to specification will ultimately save you a great deal of time, money and heart ache.

 

Implementing a tree hierarchy in GWT

A simple example of the Node hierarchy described in this post.

If you’re like me, you’re using GWT to push the boundaries of what is possible in an interactive web UI.

One of the data structures I’ve been working with recently that I didn’t see much documentation on is the tree hierarchy, as shown on the right.

Nodes within nodes, within nodes, within nodes, etc.

This method uses the GWT composite to give you full control of the widgets, provides data persistence and provides a simple way to integrate the gwt-dnd library.

We have two classes, one of which is abstract:

public abstract class Node extends Composite {
int nodeID = 0;
Node aboveNode = null;
Node belowNode = null;
Node parentNode = null;
NodeContainer parentContainer = null;
NodeContainer childContainer = new NodeContainer(this);
NodeData data = null;
int level = 0;
Widget widget = null;
}

public class NodeContainer extends VerticalPanel {
ArrayList<Node> nodes = new ArrayList<Node>();
Node parent = null;
}

Each node is stored within a NodeContainer, is also holds a NodeContainer to hold other nodes inside of it.

The NodeContainer needs some logic to handle the nodes, including:
- Add(Node node, Node aboveNode)
Pseudo code: Find the aboveNode and insert this node below it. Update the above, below and parent references of the three nodes so they have an up to date picture of their neighbours.

- Remove(Node)
Pseudo code: Find the node and remove it from the node container, update the above and below for the above and below nodes to point to each other, closing the hole where this node was.

With these functions in place, we can now write a movement function in the node itself.

public void move( Node moveTo, Node aboveNode )
this.getParent().remove(this);  // Remove me from my parent
moveTo.add(this, aboveNode)
this.save(); // Persist node after saving

Great, we can move the node around. For example you might hit a button or press tab to move the node to the right in the hierarchy, or shift+tab to move it backwards. To move it to the right essentially what you’re doing is making the node a child of the node above it. The code is simple:
node.moveTo( this.getAboveNode(), null );

There’s no aboveNode requirement once we’re into the destination node, so that’s null.

Persisting the data
You might have noticed in the move code, we implement a save() function. Each node has a reference to it’s ‘NodeData’. This is your data and all it needs to do is implement a simple interface.

public interface NodeData {
Long getParentID();
void setParentID( Long parentID );
Long getAboveID();
void setAboveID( Long aboveID );
Long getID();
void save();
}

So using the methods that you have written in your data object in order to fulfill the NodeData interface contract, a node is able to set the attributes corresponding to the data ii contains then hit the save() method. Obviously your save method will vary, but in general all it needs to do is send these attributes to the server and save them to the DB. My recommended option would be a simple data store put, using objectify (if you’re using AppEngine) because it’s so damn easy.

The way you style the Nodes to look the way you need them to be is by extending the Node class. It’s simply a composite so you can treat it like one. In the initialising function for your own node extending Widget, you’ll need to call initWidget() in the node class, and then add the nodeContainer to the widget you’ve initialised in the node class. From here you can decorate that initialising widget however you’d like with buttons, text boxes, etc.

I haven’t been able to fit everything into this post (for example loading the widgets, including the algorithm for sorting sibling nodes by aboveID and the recursion function needed to draw the tree), but hopefully this has given you the basis to form a solution. If you’d like the code I’m using for my own hierarchy then just leave a comment and I’ll upload it to GitHub.

Cheers,
Nick Cooper

What is agile?

There’s a lot of confusion around the concept of agile, and particularly when it should be used. Basically, agility is a property of a class of business methodologies. Being agile is great, given the choice between agile and cumbersome who wouldn’t choose agility? But the thing to realise is that you might be accepting other trade offs.

A good analogy is a military air craft. Agility is a great thing to have depending on the role. As it turns out, for a fighter jet agility is one of the most important properties you can have. Dog fighting involves a tight decision loop and the faster you can react and Read more »

Documenting a Startup Episode 3

After uploading the version 1 How Do You Feel app, and feeling pretty good about it (no pun intended) I decided to do some work (just an hour or two here and there after work) to pretty it up a bit. I can’t have something horrible floating around on the web with my name on it surely? It looks like this personal experiment (continuous integration) is actually working, as even the smallest changes are pretty rewarding to see on the live URL, making me more motivated to keep making them instead of just reading Hacker News or learning some wanky programming language I’ll probably never use until bed time.

Progress

Version 1 – The debut

The concept here is that there’s a simple question, how are you feeling today?

There are two axis you can Read more »

Documenting a Startup Episode 2

In one little action I’d completely humiliated my fiance.

After fiddling with GWT for a few hours my amazing app howdoyoufeel.org was ready to share, complete with a horrible color scheme and some very ‘special’ to say the least faces.

I post it on my facebook feed for laughs and my ever supportive fiance, without even realizing what she was linking to put it on her feed as ‘probably the best thing ever’.

The next day I’m chuckling over how crap version 1 was but how liberating to have it out in the wild free and from my own fears of not meeting the expectations of others, having thoroughly lowered them. While checking the visitor analytics, I idly comment  that there are actually quite a few users from Canada and Sri Lanka. She freezes.

Canada and Sri Lanka? Those are her home countries, and therefore her friends who clicked her link! She rushes to the website hoping that it’s the inspired genius she’d expect from her can-do-no-wrong fiance.

Sorry babe, it’s version one, the minimum viable product, as crap as it gets ; )

“I hate you”

The occupational hazards of a startup!

Can I make money online?

Talking about making money on the internet has a bit of a stigma about it these days. This is probably because of all the hyped up ‘get rich quick’ schemes that are advertised so heavily on so many less than desirable sites, however, the fact of the matter is – Yes, you can make money on the internet. It’s a multi-billion dollar industry.

Now before I continue: Don’t worry I’m not going to try and sell you anything. There’s no affiliate links, no CPA products, no ebooks, nothing. The only way we will profit off this blog is if we one day decide to advertise.

So now with that little disclaimer done, lets get to the reality of the situation. As much as would-be internet millionaires who try program after program trying to make their millions so they can live on a beach and never have to work again don’t like to believe, for all intents and purposes making money on the internet is just like running any other business except, you guessed it, on the net.

There are several ways people can do this, whether gaming Google with blackhat SEO techniques (not recommended!), eBooks (which can range from extremely informative to just plain rubbish), selling products on eBay, or mastering the art of getting cheap google AdWords traffic through split testing and optimisation; at some point along the line you’re going to realise “Hey, this is hard work!”.

You’re going to be making spreadsheets and entering all sorts of data to calculate which of your numerous split tests is performing the best and deserves your full budget. You’re going to be sending and receiving emails every day to customers, product suppliers, or content writers. You’re going to be carefully managing your finances because every time you run that $10,000 campaign you stand to lose thousands of dollars.

But the news isn’t all bad, you’ll get to work your own hours, be your own boss, and be able to tell people that you are that guy who actually did it. Also, being that you are by definition an entrepreneur you’re not constrained by the chains of a ‘salary’ and can stand to add a couple of zero’s to that bank balance and pay off the mortgage.

There are of course two distinct reasons why you’d take a normal business and put it on the internet.
1. The increased connectedness brings together people with niche interests in a way that they never before could. If there’s a cat collar enthusiast community, suddenly it might become profitable to create luxury cat collars and sell them to the community. The product has nothing to do with the internet, but it’s being on the internet that makes the entire business viable.
2. Internet ads are much more efficient than the old print and television ads. You can target very specific demographics including individual users who have visited your site before but not purchased. And once targetted you can measure the exact performance of your ads with statistics as useful as dollars spent vs dollars made (if you give the formula the value of a sale). The power of online advertising is killing traditional media and bringing normal non-internet industries online.

Your limit rests only on that one factor, how hard are you willing to work for it? This scares off 99% of the people who are typically attracted to internet riches. The ones who are attracted to it for the wrong reasons. The question is: which type of person are you?

Your turn
If you’re just starting out leave us a comment with a question about anything at all, we have heaps of experience and want to see new players succeed. Also feel free to sign up to our rss

Good luck

Did you like this article? Check out the free task management tool I’m developing at teamsgo.com. You’ll love the simplicity, and your colleagues will love how much more orgnised you are. http://www.teamsgo.com

How to open a business in eight easy steps

So you want to open a business. Great! Maybe you’re even most of the way there. But like it or not, there are eight steps you’ll need to follow. Skip any and you’ll probably find that no one will want to buy what you’re selling. Luckily, they’re pretty easy.

Remember when you open a business, if you’re not returning value for the money you receive then you’re either receiving charity from someone (which is ok for a little while) or stealing from them. The more people you help, the more value you’re creating, and the more money you deserve. The skill is turning a problem into an income.

So to get money, let’s create some value. The eight steps are: Read more »

Filesharing now an official religion

http://torrentfreak.com/file-sharing-recognized-as-official-religion-in-sweden-120104/

Here’s the official press release:

“The Church of Kopimism is recognized by the state of Sweden 

Just before Christmas, the Swedish governmental agency Kammarkollegiet registered the Church of Kopimism as a religious organisation. This means that Sweden is the first country to recognize kopimism as a religion. 

The Church of Kopimism have tried to become registered as a religious organisation by Kammarkollegiet for more than a year.

- Since Kammarkollegiet has been strict with formalities, we had to apply three times, a happy Gustav Nipe – board chairman for the organisation – says. He continues, I think it might have something to do with the governmental organisations abiding by a very copyright friendly attitude, with a twisted view on copying. Read more »

Why we could be alone in the universe

A bit of a philosophical post today.

First I’ll start with a belief of mine, it’s highly probable we’re living in a simulation.

This is because:
1. We’re reaching the stage where  we can create more and more complex simulation worlds. Our artificial intelligence agents, while still very primitive, are getting a lot better. Once we can simulate a human brain and also scan a living brain we will very likely be able to digitize people. In addition, quantum computing will allow us to simulate every atom of the universe. These simulations tend to be simplifications of our current universe, but as computing power increases the effort required to create a simulate tends towards being trivial.

2. If creating a simulation is trivial, then so long as the civilisation creating them survives, it makes sense to assume an infinite number of simulations would spring up. If infinite Read more »