The Problem

Recently I was tasked with providing fast full-text searching, across multiple tables, for a reasonably big database (~2m records).

Immediately I reached for some sort of Solr or Elastic solution. Being the lazy developer that I am, I chose the path of least resistance (most samples online on how to get started), and chose Elastic [edit: would seriously consider Angolia for my next project, if I had less than 10k records to index].

Getting Elastic set up was relatively simple. Having access to a few AWS credits, I used the AWS ElasticSearch service because I didn’t want the responsibility for maintaining the server.

Now to populate with some data.

Taking data from a database and putting it in an Elastic index must be a common scenario with lots of examples and help, right?

Well, not quite as common as I thought, and not as easy as I was hoping.

I was “hoping” for a nice simple user interface; point at my database, select what data I want, import into Elastic, done. I thought, with the relative maturity of Elastic, that it would be baked into the product, or someone would have created an add-on already. At the least, I thought, there would be a plethora of examples to show how to get data from a database to Elastic.

What I found was lacking, and its the reason why I’m making this post to help others looking to do something similar.

Logstash

The currently recommended solution for indexing database data is Logstash (up until 2015 the recommended way was through Rivers).

Logstash. Looks great from the diagram

With Logstash you can connect to your data, filter it, transform it, and then add it to Elastic. Great!, just what I was looking.

Logstash takes quite a bit of work to set up and configure though. The setup is fairly straightforward; just download and install on Windows or Linux. After installation, there is no UI with Logstash, just access to a command line command, to which you supply configurations.

Unfortunately Logstash does not have a way to connect with databases out of the box. To connect logstash to your database you will first need to jump through a couple of hoops:

1. You will need to install the JDBC plugin for Logstash, the instructions on how to do this can be found here.
2. The JDBC plugin does not include any database drivers so you will need to find the driver you need and then download. I am mostly connecting to Postgres so I need to download the Postgres JDBC driver. Make a note of where you download this driver to, you will need the path to the driver to your configuration later.

Providing you have no issues with your Java installation, you should now be ready to go.

Logstash Configurations

Most of your time will be spent fine tuning configurations. Simple configurations are straight forward i.e. connect to a single table with basic field types and import into Elastic. Anything outside these parameters can be challenging.

A basic configuration to connect to a Postgres database and import into a new Elastic index could look something like this:

# file: simple-out.conf
input {
    jdbc {
        # Postgres jdbc connection string to our database, mydb
        jdbc_connection_string = "jdbc:postgresql://localhost:5432/mydb"
        # The user we wish to execute our statement as
        jdbc_user = "postgres"
        # password
        jdbc_password = "mypassword"
        # The path to our downloaded jdbc driver
        jdbc_driver_library = "/path/to/postgresql-9.4-1201.jdbc41.jar"
        # The name of the driver class for Postgresql
        jdbc_driver_class = "org.postgresql.Driver"
        # our query
        statement = "SELECT * from contacts"
    }
}
output {
    elasticsearch {
        protocol = http
        index = "contacts"
        document_type = "contact"
        document_id = "%{uid}"
        host = "https://urltomyelasticinstance"
    }
}

The configuration file above can be run with the following Logstash command:

"\pathtologstashbindirectory\logstash.bat" -f simple-out.conf

1. Connect to the postgres database.
2. Get all the data from the ‘contacts’ table.
3. Connect to your elastic instance.
4. Create a new index called ‘contacts’
5. Create a new document type called ‘contact’
6. Automatically create mappings for the fields in the contacts table.
7. Populate the index with the data.

Even More Productivity?

After experiencing the immense productivity gains from using Ionic for mobile development, surely using Ionic 2 would be even more productive, right?!!?.

Sadly. No. Not yet, at least.

I had hoped to be writing a post telling everyone how awesome Ionic 2 is.

I had hoped to be falling over myself to tell you about how quickly it was to get started, the incredible workflow, the intuitiveness of the framework, the terseness of the code.

Instead, what follows is a moan colored by disappointment and frustration from using Ionic 2, Typescript & Angular 2.

Ionic, Angular 2 And Typescript

Ionic is a cross platform mobile app framework. It uses Html & JavaScript (Angular) to create mobile apps that can be run on Android and/or IOs.

Ionic 2 is more of the same, but with a few notable differences. You will, inevitably, be using Angular 2 and Typescript to create your apps.

Angular is a JavaScript framework which makes creating single page web apps easier than just using plain JavaScript. Angular 2 is google’s second attempt at the framework, and a significant departure from the first.

Typescript is a superscript of JavaScript. It allows you to write tidier, more elegant, maintainable, JavaScript.

Now, both Angular 2 and Typescript are great in my book, but they are ‘new’. This leads me to my first gripe about the Ionic 2 stack, the learning curve:

Gripe 1: The Learning Curve

There are several steep learning curves involved with the Ionic 2 stack.

Typescript is a big departure from vanilla JavaScript. The syntax should not be alien to anyone who currently uses ES6/ES2015 but it will still take some getting used to.

The other steep learning curve is with Angular 2. Although the syntax in Angular 2 is more elegant and easier to learn than Angular 1, they both have steep learning curves.

Also, there are also changes with Ionic 2 which are a departure from Ionic 1 which you must learn. However these changes are not as severe as the changes in the Angular 2 and using Typescript.

Gripe 2: Beta

Ionic 2 is a beta.

Angular 2 is just out of beta.

Typescript is relatively new.

If you regularly program via stack overflow ‘copy and paste’ you will be frustrated by the lack of help and the ‘out-of-date’ help that you find.

Gripe 3: The Showstopper: Transpiling & Bundling

Ok, here’s the thing that’s eating me most: transpiling & bundling.

This is not a gripe with Ionic 2 per se; it is a gripe about modern web development.

Before, with Ionic 1, using the web tools to debug your app, feedback was instant. You made a change to your code and the changes were reflected immediately on screen.

Now, with Ionic 2, when you save, all your code needs compiled from typescript to something the browser can understand. On top of that, Ionic 2 now bundles your application for use in the browser every time a change, no matter how small, is made.

The upshot, is that when you save even a small project, the wait before you get feedback is greatly increased. I have spent my time waiting 30 seconds upwards waiting for a simple change in my JavaScript to take effect on screen!.

(Maybe I am using it wrong? Please, please, correct me if I am.)

This is a BIG problem for me. The thing I liked most about Ionic 1 (and web development in general) was how immediate the feedback loop was.

Things were so much more productive in the web world compared to using compiled languages. Working with a scripting language had its issues, but it felt so productive not having to wait on the compiler to finish.

Transpilers & bundlers have added a compile step to web development which affects productivity.

Largely due to my opinion on the frustrating feedback loop with Ionic 2, I will not be using Ionic 2 for new projects yet.

Sure, there are a lot of benefits from using Typescript to create a maintainable code base and productivity gains from using using Angular 2, but these are currently outweighed by the negatives.

Ionic 1 is still more productive for me and I will be continuing to use it in the near future.

As always, this is just my opinion. If I am missing something obvious, or have got this horribly wrong, please correct me in the comments below….

Code Run Through

This post follows on from posts about hybrid mobile app development using the new WordPress REST API and Ionic: Part 1, Part 2

What follows is a run through of the import parts of the code, for the application shown in Part 2.

By now you should have a [very basic] Ionic app running in your browser. The app will allow you to:

• Log into your remote, WordPress REST API enabled, website.
• Make a post from the mobile app to your WordPress site.

Pretty simple stuff.

The code that performs the magic is pretty simple too.

*I have created both an Ionic 1 and Ionic 3 repo for the App code. However, below I describe the structure for the Ionic 1 repo only.

Ionic Project Structure (Ionic 1)

Ionic Folder Stucture

As you can see from the folder structure below there are quite a few folders in our Ionic App.

However, the important files and folders are as follows:

• www/js: This folder contains all the Javascript code for the app.
• www/js/app.js: The main entry point for the app. Things like routes are configured here.
• www/js/controllers.js: The logic behind what happens on the screens is placed inside this file e.g. what should happen when the login button is pressed.
• www/js/services.js: Reusable services to perform business logic e.g. authentication and connection to WordPress
• www/templates: This folder contains the html for screen in the app.
• ionic.config.json: This file contains configuration options for the app. In Part 2, we changed a setting in this file to point to our WordPress site.

The folder structure is much like any other Angular application, so we will head straight to the code to see what the key lines are:

App.js

In app.js we configure the Angular app before it runs.

In this file we state:

• What our routes are i.e. what are the urls for our app, what html should be loaded and what Javascript files should control the html.

// Ionic uses AngularUI Router which uses the concept of states
// Learn more here: https://github.com/angular-ui/ui-router
// Set up the various states which the app can be in.
// Each state's controller can be found in controllers.js
 $stateProvider
 .state('login', {
 url: '/login',
 templateUrl: 'templates/login.html',
 controller: 'LoginCtrl'
 })
 .state('report', {
 url: '/report',
 templateUrl: 'templates/report.html',
 controller: 'ReportCtrl'
 })
 // if none of the above states are matched, use this as the fallback
 $urlRouterProvider.otherwise('/login');

• Any injectors/interceptors we want to use. We will see later that, when we get an authentication token, we want this to be automatically passed to every subsequent HTTP call we make. An interceptor is a way of achieving this.

Controllers.js

The controllers.js file contains the logic for the Login and Report screens.

The LoginController has one function: try to login when the user clicks the Login button

$scope.login = function () {
 // contact our login service with the data from the username and password fields
 LoginService.loginUser($scope.data.username, $scope.data.password).then(function (data) {
 // if it is a success, go to the Report screen
 $state.go('report');
 }, function (data) {
 // if there is an error pop it up onscreen
 var alertPopup = $ionicPopup.alert({
 title: 'Login failed!',
 template: 'Please check your credentials!'
 });
 });
 }

The ReportController has a single function too: try to post the score and report data to WordPress (though our WordPress service).

$scope.createReport = function () {
 // show a saving... message while we contact the service
 $ionicLoading.show({
 template: 'Saving...'
 });
 // pass through the values from the score and report fields to the service
 WordPressService.createReport($scope.data.score, $scope.data.report).then(success, failure);
 }

Both controllers are very simple, it is in the Services.js where the real work is performed.

Services.js

The services.js is the work horse of the app, it contains 3x services:

• LoginService. This service is used to contact a WordPress REST API end point and request a JWT authentication token.

// the important bit, contact the end point and ask for a token
 $http.post('/server/wp-json/jwt-auth/v1/token', data).error(function (error) {
 failure(error);
 }).success(function (data) {
 // you are now logged in, save to session storage, the auth interceptor will pick up
 // and add to each request
 $window.sessionStorage.token = data.token;
 success(data);
 });

• WordPressService. The WordPress service contacts our WordPress REST Api and tries to create a post.

var data = {
 title: score,
 excerpt: report,
 content: report,
 status: 'publish'
 };
 // the important bit, make a request to the server to create a new post
 // The Authentication header will be added to the request automatically by our Interceptor service
 $http.post('/server/wp-json/wp/v2/posts', data).error(function (error) {
 deferred.reject(error);
 }).success(function (data) {
 deferred.resolve(data);
 });

• AuthInterceptor. The interceptor service checks localstorage to see if we were given a token, if we have, then it adds it to every Http request.

request: function (config) {
 config.headers = config.headers || {};
 //if there is a token, add it to the request
 if ($window.sessionStorage.token) {
 config.headers.Authorization = 'Bearer ' + $window.sessionStorage.token;
 }
 return config;
 }

Any Questions?

That’s basically all there is to it.

If you have any questions, or any amendments that I can make to the Github repo, then please comment below….

This post follows on from a previous post, espousing the virtues of the WordPress REST API for hybrid mobile app development.

At the 2016 WordCamp in Belfast, I gave a talk about using the WordPress REST API for mobile development. I gave two code walk throughs at the conference; what follows is a walk through of the football mobile app.

The football app uses Ionic to create a cross platform mobile app that:

• Connects to a remote, REST API enabled, WordPress site.
• Authenticates a username and password from the device (using JWT – JSON Web Tokens).
• Shows a ‘Result’ screen. The screen allows users to input a score and match report.
• Posts this ‘Result’ to the remote WordPress site.

N.B. All the Ionic 1 code is available on Github. There is also an Ionic 3 code repo available here.

The steps to create a shiny new mobile app, which allows soccer coaches to post match results from their phones, is as follows:

Ionic 3 to WordPress

WordPress Site

We need to first create a WordPress website which will act as our back end engine to our app.

• Create a new WordPress site (or use an existing one).
• Install and activate the following plugin:
  • JWT Authentication For WordPress REST API: https://en-gb.wordpress.org/plugins/jwt-authentication-for-wp-rest-api/
  • Make sure to set your auth key after activating the plugin, e.g.:

    define('JWT_AUTH_SECRET_KEY', 'your-top-secrect-key');

*Also on some servers you will need to enable http auth in your .htaccess file

RewriteEngine on
RewriteCond %{HTTP:Authorization} ^(.*)
RewriteRule ^(.*) - [E=HTTP_AUTHORIZATION:%1]

Mobile App

• Install Node on your machine.
• Install Ionic and Cordova using the following NPM command/cmd:

  npm install -g cordova ionic

• Download the github code (cd to the folder you want to put it in first)

  git clone https://github.com/conorw/wp-rest-api-ionic.git

  Or for Ionic 3

  git clone https://github.com/conorw/wp-rest-api-ionic2.git

• Using the command line navigate (cd) into the created directory

cd wp-rest-api-ionic

• Get node to install Install the projects dependencies

npm install

• Change the ionic.config.json file to point to the URL of your new WordPress site

  {
   "name": "teamthingmobile",
   "app_id": "",
   "proxies": [
   {
   "path": "/server",
   "proxyUrl": "https://thenameofyoursite.com"
   }
   ]
  }

• Start Ionic to show a web verison of your app

  ionic serve

  Done!

  You should now have a working mobile app, executing in your browser, which you can now use your WordPress credentials to login.

If not? Give me a shout below…….

In the next post I will highlight the important code in the sample, and explain what it means.

*For an Ionic 3 version of the code see this post.

WordPress, not just for bloggers

I love WordPress!

Unfortunately, in my everyday job as a software engineer, I don’t get to use it much.

However, with the availability of the WordPress REST API, there are more and more reasons to use it on “real” software projects.

The WordPress REST API, in very simple terms, is a way for apps and other websites to communicate with the powerful WordPress “engine”.

Pic’N’Mix WordPress

With the WordPress REST API you can use WordPress as an ‘Application Framework’. With an application framework you can ditch the heavy blog website concept and just use the great bits of WordPress that you need in your shiny, light-weight app or website. Give your new app or website a head start by harnessing dependable WordPress engine features like user authentication, content management, media management, extensions, SEO, translations etc.

A great use case for the WordPress REST API is to use it to supercharge Mobile App development. By Pic’n’Mixing the best parts of WordPress with your mobile development technology of choice, you can rapidly get an app up and running.

Pic’n’Mix WordPress

You can use WordPress as a dependable back end to your Mobile App. The mobile app stays light-weight, the only thing it needs to utilize WordPress is a Http client. If your app can make a web call, you’re all set.

But it’s not used in production apps, right?

There are some fantastic examples of apps, out in the wild, which use WordPress as a powerful back end. 3Advance were quick to recognize its utility and have created two apps already which use WordPress in the background:

Launch Alert: Meg Bitton Live iOS App gets 148 ⭐️⭐️⭐️⭐️⭐️ reviews in just 12 hours!

Launch Alert: Apps Developed to Watch and Stream Portrait Masters

3Advance are notorious for quickly turning app ideas into reality. They found the WordPress community and REST API perfect for rapid application development.

Getting Started

To get started using WordPress as your next application framework is as simple as installing a normal self-hosted WordPress site or create a site on WordPress.com.

The REST API is enabled by default on self-hosted and WordPress.com sites.

Check it out for yourself; go to your favorite WordPress site and add “/wp-json/wp/v2/posts” to the web address to see a REST API call in action e.g.

http://www.angrybirds.com/wp-json/wp/v2/posts

There are numerous examples on the internet of how you can start integrating the data returned from the REST API with your app. So instead of rehashing yet another example, here are some links below.

WordPress REST API Documentation: https://developer.wordpress.org/rest-api/

Used with a progressive web app: /wordpress-rest-api-part-2/

Used with React:

https://snipcart.com/blog/reactjs-wordpress-rest-api-example

Used with Vue.js

https://github.com/bstavroulakis/vue-wordpress-pwa

So much code, written so many times

Android, IOs, Windows Phone, Web, Responsive Web; customers don’t really understand, they want their app in ALL the places. “Surely it’s the same functionality, just make it work on a different device, how hard can it be?!?”.

Unfortunately, plenty hard.

Different programming languages, different screen sizes, different App stores and different hardware makes multi platform development a pain for mobile developers. What we end up doing is re-writing the same functionality in different bespoke apps, using different languages, for each of the hardware devices; not ideal, but sometimes unavoidable. Writing the same code, multiple times is something that many of us unfortunately have to swallow.

Server Side API

There are some things we do to try to minimize the amount of code we need to duplicate on each device.

One of the approaches, to reduce code duplication, is to create a comprehensive set of web services. Using services, we can implement most of the data retrieval/manipulation and business logic, server-side. All the devices will now be able to access and use these services; code once, use everywhere.

However, these services only get us partially on the way to eliminating code duplication.

Hybrid Apps

One increasingly popular solution, to making apps that can be written once and run anywhere, is by using ‘hybrid’ development i.e. write the app in a non-native language that can be ran on ‘all’ devices. There are a slew of hybrid approaches available for developers to use e.g. ReactNative, Telerik, ManifoldJS and Ionic.

Personally, I have not used all of them. They all look promising, and they all have their pros and cons. However, recently, I started using Ionic (as it suited my situation perfectly) and am wholly impressed by what Hybrid development offers.

Ionic To The Rescue

My situation entailed a large SPA website, written mostly in Angular, requiring a mobile app to be created that mirrored some functionality of the website.

The thought of having to duplicate the website features, using multiple languages, and then having to support all these codebases, was horrific. We had moved as much logic to the server side as possible, but there was still a great deal of functionality in the website that could not be moved and needed to form part of the mobile app; cue Ionic.

Iconic is a hybrid mobile platform which basically embeds a website inside a mobile app. The website in the mobile app can be a normal responsive html/css/js website but can also utilize various native functionality (e.g. camera, GPS, senors etc) by calling a layer, that sits on top of the native system, called Cordova.

Since Ionic allows you to create functionality in Javascript, and we had lots of Javascript (Angular) to utlilize, Ionic was a perfect fit for us.

Rapid Application Development

Never before I have been so productive developing mobile apps. The productivity gains from using Ionic have been immense:

• We have been able to share Angular components and services with our web AND mobile apps.
• If we find a component or service that cannot be easily shared, it forces us to refactor and break down the item into more reusable parts.
• Quite a bit of CSS can be re-used too (if refactored well enough) to make styling consistent, across not only mobile devices, but web too.
• We are maintaining and testing a ‘largely’ single code base using a pervasive coding language (Javascript). A problem found on one of the form factors is fixed once and deployed everywhere.
• One of the biggest productivity gains is with the workflow. Most of the development/debugging can be performed through a web browser. Using Ionic you are able to change your Html/Js/Css and automatically see live changes. No need to start an emulator or wait on the IO between desktop and device. Sometimes, you absolutely need to test something on a specific device, but I find, that for most work, the ‘webview’ suffices.

Not Quite There Yet

There are some trade offs with using hybrid development in preference to native development, not least performance. But for most ‘line of business’ applications the performance is ‘good enough’.

Ionic itself is great and getting better, but has some limitations. Cordova has its limits and ‘idiosyncrasies’ and can be a pain to work with. However, just having to maintain one single codebase between the mobile platforms (with web thrown in for good measure), coupled with an awesome workflow, on apps which are ‘good enough’ for most business needs, and gets an app in clients hands sooner, makes Cordova’s ‘idiosyncrasies’ bearable.

Alternative Viewpoints?

This form of hybrid development has worked well for me but there are so many other workflows and frameworks out there. Has anyone any experience with the other frameworks?

I have a confession, I prematurely optimize.

Looking for efficiencies in some code you have just splurged seems like too much fun. We have all done it; one moment we are motoring along, knocking features out left right and center, next minute you disappear down the rabbit hole looking for optimizations.

It feels productive though.

What if my app needs to handle 1m users? What will 0.2ms extra download time do to my SEO rankings? Will users go somewhere else if my code is not running as fast as it could?

These are all semi-legitimate concerns, but at what stage does it become essential to address them?

YAGNI (You Aren’t Gonna Need It)

The truth is, most of the times, there are plenty of more fundamental issues to address, most of them related to the question “What do users really care about?”. Optimization is only important after users decide your product is actually worth using, not before.

If anyone asks, in principle, I am totally for the essence of YAGNI (You’re Not Gonna Need It), in practice, however, my track record is mixed. Theory says we should develop minimum viable products, get these products to customers without delay, defer commitment as late as possible for irreversible decisions, reduce waste, reduce inefficiencies etc etc. But far too many times I have spent hours optimizing something which:

• Delays the time it takes to get to the customer, thus reducing time for feedback.
• Makes technology selections too early based on optimization concerns.
• Creates deep and incomprehensible inheritance models, because that’s what a book told me to do.
• Introduces unnecessary complexity to the solution which developers behind me have to take even more time to learn (decipher).

All on things that:

1. Weren’t eventually even used in production.
2. Didn’t need as optimized as much as they were. Not now. Not ever.

This is time I could have used discovering what users actually liked or disliked.

Permission Granted

Hopefully by confessing my problem to you that it is the first step in my rehabilitation. Next time you see me pre-optimizing you have my permission to slap me, Batman style, and shout “YAGNI”.