Scaling WordPress

Standard

*In this post I will attempt describe how you can prepare your WordPress site so that it can be scaled.

WordPress is generally not scalable

The usual deployment scenario is to have the database, core files, user files and web server, all on the same server.

In this scenario, the only scaling that can be performed, is to keep adding memory and power to a single server (vertical scaling).

Scaling Options

‘Scaling vertically’, which is also called ‘scaling up’, usually requires downtime while new resources are being added and encounter hardware limits. When Amazon RDS customers need to scale vertically, for example, they can switch from a smaller to a bigger machine, but Amazon’s largest RDS instance has only 68 GB of memory.

‘Scaling horizontally’ means that you add scale by adding more machines to your pool of resources. With horizontal scaling it is often easier to scale dynamically by adding more machines into the existing pool.

Horizontal vs Vertical Scaling

Horizontal Scaling For WordPress

To enable horizontal scaling for WordPress we need to first decouple the various parts from each other (web server, files and database). Once decoupled, we can place each part on separate servers and scale them as necessary.

Decouple Before Scaling

Decoupling the database is ‘easy’. You can easily move your WordPress database to its own server by taking a backup and migrating (guess what, there’s a plugin for that). Once moved, you just need to change the ‘DB_HOST’ value in your wp-config file.

Decoupling the web server from the files (WordPress core files and media library files) is more complicated. From what I can gather there are 2 options for doing this:

  1. Host only your library files on a remote cloud server. This article explains how to achieve this.
  2. Host both your core files and library files on a mounted volume linked to a remote cloud server. This article explains how to link your uploads folder to cloud storage, but you equally link your entire WordPress folder.

I prefer option 2 because:

  • Option 1 keeps your core files coupled to the web server.
  • It also only caters for new uploads, not existing files too.
  • In addition, it involves rewriting the url links for your files.
  • Option 2 keeps your url links the same so you revert back or change cloud provider easier.

However option 2 is way more complicated to set up. In my next post I will show how Docker can help with this process.

Decoupling complete, now scale!

You can now scale your WordPress site.

In the next post I will show how to use Docker, in particular Docker Cloud, to make this process painless.

Deploy a Developer

Standard

Most software projects inevitably suffer, especially in the humanitarian domain, unless the huge gap between developers and end users is recognized and addressed.

One of the best ways to do this (reduce the gap between these two diametrically opposed groups) could be by actually embedding a developer in your core team on your next mission or assignment.

Why Do Projects Fail?

Over a 15 year career as a software developer I can attest to the fact that poor requirements definition is the chief culprit for failing projects. I have developed mobile, desktop and server applications, worked for several companies ranging from large corporate companies to small start up companies, in a number of diverse sectors, and the single most challenging problem with creating great software is a reoccurring one; requirements definition. End users and developers can’t, won’t and don’t communicate effectively and efficiently leading to the poor definition and implementation of requirements.

poor_requirements_definition

As you can see from the above diagram, the main reason projects fail, by a large large margin, is poor requirements definition; finding our what users need and translating this into solutions.

What’s the Big Problem?

In my experience this big problem could largely be solved by bridging the gap between the two groups at either end of the process: those who make the software and those who the software is bring made for.

But it should be easy right? We are all humans, end users are human, software developers are human (mostly).

How could there be much of a problem of communicating ideas from one group to another?

Well, it turns out, there are two big issues: developer apathy and the requirements gathering process itself.

Issue #1 : Lack of Empathy

The first issue is that transposing the ideas from a non-technical user to a technical/techhie developer (who is more comfortable talking to code than people) is a big challenge. A challenge that the software industry has still failed to adequately address and affects software projects again and again and again.

The problem is that there is a huge gulf in experience, skill sets, even the language used for the same things, between a non-technical ‘ordinary’ user and the people who sit behind desks and write code. Software developers lack the empathy to adequately understand the domain and the users.

Ordinary end users talk a completely different domain language and can have completely different experiences than desk-ridden developers.

end_user

End users, come in many shapes and sizes. They can range from the very technical to the not quite so technical.

developer

On the other hand, Software Developers are generally desk ridden, highly technical and actually take pleasure from being able to talk a different language from mere mortals.

How many times have you talked your IT department and felt like they were taking pleasure in talking down to you using highly technical terms?

I will let you into a secret, we absolutely do that on purpose!

Do not underestimate the apathy (and even contempt) that developers have for end users. We view users as inferior beings. Even when users tell us directly that they have problems using something we have built, our first instinct is to scoff and assume it is user error and they should be smarter to use it correctly.

This is genuinely what we think you are doing when you ring us with a problem.

This is genuinely the kind of thing we think end users are doing when they ring us with a problem.

Issue #2 : Noise & Barriers

So with issue #1 the problem is that the two groups have often completely different perspectives and find it difficult communicating with one another. Issue #2 is that, in most software projects, there are numerous barriers placed between the developers and end users by the actual requirements gathering process.

The two groups don’t actually get even the chance to miscommunicate directly. There are layers upon layers to make ensure that, even if a clear message is stated by end users on their requirements, by the time this reaches the developer, it has been changed into something different (a bit like a game of Chinese whispers that a lot of us played in the playground as kids).

requirements_gathering_process

The Requirements Gathering Process

Take the above simplified example of a requirements gathering process. We just want to get information from end user to developer in a clear efficient way.

At the start we have the user telling a requirements gatherer of their issues.

The gatherer generally translates these findings using their own domain knowledge into a series of documents.

These documents are reviewed by a ‘middle manager’ who filters and translates the raw requirements, again using his/her own judgement.

The middle manager produces a set of faceless requirement documents which are passed to the developer to develop.

At each step some ‘understanding’ of what the end user wanted to express is lost or mistranslated by people and paperwork.

Then, when refinements are needed, the process starts all over again.

The problem?, too many steps and too many intermediaries.

Lets get these two groups talking to, and understanding each other, a bit more. Wouldn’t it be far more effective and efficient if these two groups were allowed to communicate directly with one another?

Of course it would, but because of Issue #1: the fact that they don’t even know how to talk to one another, it is also pretty impractical.

So how do we make it practical?

Turning Apathy Into Empathy

I am not saying it is easy to achieve.

Developers don’t necessarily want to come face to face with users.

Users may feel frustrated with the prospect of having to communicate with (sometimes) abrupt and condescending developers, but, from experience, the most productive and effective projects that I have worked on have involved the end user communicating directly with the developers from day 1.

Not only is it more productive for the two groups to communicate directly but it also increases the chances that the solution produced will be usable (i.e usable by the end user), useful (actually solves a stated problem) and used (not thrown in the bin like so many other projects by users. Users will have actively participated in the creation of the solution, and have a stake in the solution created.

So here my tips, based on experience, of how to help bridge the gap between developer and end user.

  • Get developers out of the office (Genchi genbutsu). Get developers out of their comfort zone. Consider deploying them on your next mission to get them to start to understand the domain and gain empathy with users. We need developers to attempt to understand users better, having empathy with end users not only increases the understanding of the domain but also gives developers incentive to make better choices during the development stage. Giving a human face to a requirement, as opposed to a faceless requirements document is hugely influential on the decisions that developers make. This type of understanding and empathy should not be understated in affecting the quality of solution your developers produce!

 

  • Use Visuals. Using visuals, rather than language and documents to share ideas cuts out a lot of the potential miscommunication issues and helps to speed up the software development process. Best to use visuals and prototypes, both developers and uses both speak this language!

prototype

  • Develop a common domain language. Right from the beginning set out a Common language for communication, try to find a common denominator to terms and phases used by both sides.

 

  • Regular Feedback & Pilot Exercises. Fail Fast. Fail Often. Facilitate a consistent, regular, and always open, channel of communicating. Frequency is key here. For example have a weekly show & tell from developers on progress and the direction they are currently taking, allow feedback from end users that will be allowed to have an impact of the future direction of the solution. Build it, scientifically measure the feedback, and learn from the lessons end users are teaching you.

 

  • Customer Champion. It may be impractical to keep going back to all the end users regularly, but at least make sure that you have a representative user who can be called upon regularly to give feedback on choices and solution direction.

 

  • Hire a Software Developer with domain experience. Find a ready-made developer who already understands the domain. If you are lucky enough to find a developer with domain experience, hire them, your project will thank you for it.

 

  • Consider an Innovation Hub. Bring the development team permanently closer to your organisation by housing them beside your team. See Andrej Verity’s recent blog post for some great advice on how to do this in a humanitarian context.

A Challenge to You

So the challenge I put to you is, “will you consider deploying a developer with your team on your next assignment?”

Are you willing to stand up to your own IT department or IT contractors and demand a more closer working relationship?

Are you willing to drag developers kicking and screaming away from their desks and into the field? Believe me, as a developer myself, you will face resistance with this!

That is the challenge to you, it would not be a challenge if it wasn’t challenging, if it wasn’t difficult!

Why Write This Blog?

Standard

I write code for a living, I love doing it, but I want to do it better.

I forget things, really important things.

Birthdays, meetings, anniversaries, names, outstanding bills, all fall victim to my absent-mindedness; nothing is safe. As programmers we have better things to do with our time, right? Why should we have to remember trivial things like anniversaries when there is code in front of us to crank.

Most of us coders don’t have a personal assistant to keep us on track, and most of us don’t conform to the basement-dwelling, cola drinking, RPG playing, colostomy bag wearing, nerdy stereotype (at least not all of the stereotypes). We are ordinary folk who just like as little distraction as possible while engrossed in some code.

programmers

So what to do, how can we still keep our heads in code while having as few untimely interruptions from the outside world as possible?

Over the past few years I have been gradually getting better at remembering and organizing things. By using widely available software programs and apps I have able to save face on several occasions now, but things are still not perfect.

This blog chronicles my attempts at learning, from current tools and practices, and from you, about what I am doing wrong and how we can do it better.

Left to my own devices I can be a lazy and unproductive programmer.

I am scatterbrained; forgetting meetings, birthdays and names.

I fret that I will forget important things and so try to deal with them as soon as I encounter them or just sweep them under the carpet.

Procrastination is a habit that I cling onto like a life raft when things start to get difficult.

I like to think of myself as a typical coder 🙂

productivity

BUT,

One redeeming quality that I do have is that I know that I have these flaws and recognize that they stop me from being a better programmer.

Procrastination makes easy things hard, hard things harder.

Mason Cooley

 

I have been around just long enough to know that there is hope for coders like me. I know there exist tools, practices and tips that will enable me to shake the shackles of these common programmer traits and become a more rounded, confident and productive coder. This blog is about exploring these topics and getting opinions on what works and what doesn’t.
EDIT:

LOL! in the course of writing this post I found myself spending an hour [2 hours]: looking for an avatar, browsing Facebook and looking at a whole host of memes from quickmeme.com.