Vercel: Developer-focused, powerful & cost-effective

Standard

*this post is the 3rd of a 4 part series of posts exploring Supabase, Vercel & Svelte. This post goes deeper into Vercel

Part 1: Intro
Part 2: Supabase
Part 3: Vercel
Part 4: Svelte

In a previous post, I outlined why Supabase, Vercel & Svelte make up my current ideal technology stack. This post delves deeper into Vercel and why it has helped me fall in love with full-stack development again.

Productivity as a Priority

As a “seasoned” software engineer, I’ve navigated through numerous deployment & hosting platforms.

I am always on a quest for a more streamlined, developer-friendly approach. The goal is always to developers space to focus on the things that really matter.

That search has led me, again and again, to Vercel in the past few years.

Here’s my personal take on why Vercel has become an indispensable tool in my development stack.

Discovering Vercel: Weekend project to production apps

My journey with Vercel began during a weekend project launching a web app for a local charity. The goal was to deploy an easy-to-use and maintained web application. The app should be created with minimal fuss and minimal cost to the charity.

Over the course of the weekend, I was able to create a web app & CICD pipeline at Zero ongoing cost to the charity.

Since then I’ve used Vercel on a number of other production projects.

Seamless integration with developer workflows

The biggest selling point of Vercel is its seamless integration with developer workflows.

An estimated +100m developers incorporate GitHub somewhere in their workflow, I am one of those 100m.

Vercel hooks easily into your github repo. After linking to GitHub, vercel recognises what type of project you’ve got going on, and automatically builds and deploys your web-based and/or node project on a branch push.

It may sound like a simple thing, but it saves a huge amount of time, complexity, and expense for a significant number of web & node developers out there.

This single feature, coupled with a host of other performance and developer-focused features, makes Vercel highly persuasive for web and node developers.

Develop App & API together

A big game changer for me was the ability to develop & deploy my [web] apps and APIs at the same time.

API and APP in the same repo

If you include an “API” folder in your repo, Vercel “automagically” spins up a serverless function to serve the API endpoints.

Being able to develop an App and API in the same codebase is a big win for smaller startup teams where the code is changing rapidly.

There is no disconnect between the two; if a new feature or API route is deployed, they are both deployed together and not out of sync.

Even better, if you are developing with Node for the API, and a JavaScript framework for the front end, the same developer can understand and update both aspects.

Performance as someone else’s problem

Vercel doesn’t just deploy your application; it helps optimize and scale it.

Leveraging global content delivery networks (CDNs) and smart caching strategies helps your application load fast, regardless of where your users are.

Whether your webpage or API experiences a handful of users or a sudden surge in traffic, performance remains consistent.

This peace of mind, knowing that scalability and optimization aren’t something I have to actively manage, allows me to focus on other, client-centric, tasks.

Vercel Pricing (March 2024)

The Cost

The pricing tiers underline the feeling that Vercel was built for developers, by developers.

For smaller teams, being able to quickly try out ideas at no cost, allows them to iterate at speed. Not having to raise a purchase order or ask engineering for some resources, just to create a proof of concept, is perfect for startups and small teams.

Even when you get past the proof of concept stage, the pricing is extremely persuasive for small and large teams alike.

Embracing the Future with Vercel

My experiences with Vercel have made it a staple in my development workflow.

Ease of use, performance optimizations, and scalability it offers are fantastic for the types of projects I regularly work on (small team start-ups).

For any development team, in a similar situation, looking to streamline their development & deployment processes, I can’t recommend Vercel highly enough.

*this post is the 3rd of a 4 part series of posts exploring Supabase, Vercel & Svelte.

Part 1: Intro
Part 2: Supabase
Part 3: Vercel
Part 4: Svelte

Supabase: Making development easy, scalable & affordable

Standard

*this post is the 2nd of a 4 part series of posts exploring Supabase, Vercel & Svelte. This post goes deeper into Supabase

Part 1: Intro
Part 2: Supabase
Part 3: Vercel
Part 4: Svelte

In the previous post I outlined why Supabase, Vercel & Svelte make up my current ideal technology stack. This post delves deeper into Supabase and why it has helped me fall in love with full-stack development again.

What is Supabase?

Supabase is a managed service which encompasses (but is not limited to) all of the following: authentication, database, file storage & serverless functions.

Supabase is like other “backend as a service” (BaaS) offerings like Firebase, but with a few notable differences; the project is open-source & is centred around an open-source relational database (Postgres).

What makes it so useful?

As I have lamented in the past, app development is complicated, and anything that reduces stack complexity can help focus developers on the things that really matter.

I tried Supabase for a weekend project for a local charity and achieved so much in a single weekend that I would consider myself an advocate for the product. Following that experience, I have now used Supabase successfully for two additional production projects and plan to use it in the future for similar scenarios (small team startups).

Advocating Supabase at a JavaScript meetup. Slides below…

Creating a relatively simple app over a weekend is not a huge accomplishment; there are other services and no-code platforms that can do something similar in the same timescales.

However, experience has taught me to get into the weeds with a product and then extrapolate into the future to gauge the real value of a tech stack. Low code and no-code tools are great, but at some point, in a growing project, you will hit a wall.

What makes Supabase stand out is that coupled with other developer tools like Svelte, it can be at least as productive as no-code tools without the drawbacks e.g. vendor lock-in, limited customization, up-front costs and scalability.

Embracing Open-Source and Community

My gravitation towards Supabase is also influenced by its open-source ethos which promotes transparency, collaboration, and community-driven innovation.

Being open to open source is more than just being idealistic, it’s also pragmatic.

The Supabase project is open source e.g. the code that runs its managed service can be downloaded and used on a server of your choosing.

If Supabase decides to increase the managed service cost to a level where it no longer makes sense to use it, you can manage the services yourself elsewhere.

Supabase has been completely transparent about its open-core business model from the start, hopefully, this model continues to work for them.

However, relying on open-source projects is not without potential pitfalls, especially when open-source companies’ heads get turned by greedy VCs and start over profiteering.

At one time, Elastic was my tool of choice, but the change in licence by the company has left a bad taste.

However, even though open-source licences can change, it is still better than the closed-source alternative where you are completely at the whims of the vendor from day one.

Simplifying the Complex

Creating apps is a complicated process even without having to worry about managing servers.

Delegating responsibility for managing auth, database, and storage to a managed service allows small teams to concentrate on more impactful concerns.

Not only does Supabase take these concerns away from you but it does it all in an easy-to-use dashboard.

The developer experience in general has been, dare I say it, enjoyable. Using the Supabase tools and libraries has been successful in reducing the complexity and lines of code in my apps.

The Security Model: Easy to Understand

The simplicity of the row-level security in PostgreSQL is easy to configure and understand.

It presents a straightforward yet robust framework that drastically reduces the risk of misconfiguration—making security accessible to all of the team, even for newcomers, from day one.


However, it’s not perfect.

I have had experience in the past with different approaches to securing data. My least favourite way in the past was to implement the security rules totally in code i.e. lots of if/then statements hidden away in code that only the core developing team could understand or change.

In contrast, in my opinion, the “best” way I have experienced is to use declarative authorization rules, defined in the data schema e.g. Amplify authorization rules.

In the example below, any user can read from the “Todo” table/graphql type, but only the person who created the row can update or delete their own data.

## Configure schema and auth rules
## in one place 
type Todo 
  @model 
  @auth(rules: [{ allow: public, operations: [read] }, { allow: owner }]) 
  { 
     content: String 
  }
## Implementing something similar
## using Postgres/supabase
create policy "Allow select, update and delete for users based on id" on "public"."Todo" as permissive for all to public using ((auth.uid() = id));
 
create policy "Read for all users" on "public"."Todo" as permissive for select to public using (true);

It would be great if Supabase could cater for the type of declarative security as above, if anyone knows if it can, please reach out.

Scalability and Performance: Meeting Tomorrow’s Needs Today

Premature optimization is the root of all evil, let someone else grapple with the demon

In the past, I have spent countless hours trying to eek out marginal gains in performance in case my app goes viral. Spoiler alert… it didn’t… and I’ll never get those hours back.

Let someone else (with probably more expertise) obsess about performance and scalability.

Supabase’s seamless scalability ensures that as you grow, your backend does too—smoothly and reliably. This peace of mind allows you to focus your energies on innovation and enhancing user experience, secure in the knowledge that your technological foundation is a given.

The Cost-Effectiveness of Dreaming Big

Free and Pro pricing (March 2024)

In the world of startups, where every resource counts, Supabase’s pricing model is perfect.

The free tier is generous enough to battle-test your idea. The follow-on tiers are predictable and fair.

It’s not just about infrastructure costs where Supabase shines. The comprehensive savings in developer hours it enables through its exceptional developer experience is significant.

Again, this efficiency allows you to channel resources into areas that directly amplify user value and platform growth.

A Comparison with the Giants

In my career, I have used other back-end-as-a-service offerings and Supabase compares favourably for the projects I’ve been doing lately i.e. small team startup.

I have used all of the following comparable technologies in production environments: Firebase, Retool, AWS Amplify, Budibase.

I have tried, but not implemented the following tools: Planetscale

I have not tried, but want to look at, the following: Parse, NHost, Backendless, AppWrite

My advice, if any is needed, is to look at your particular situation and try out any or all of the tools above on a pet project. The “try out” part is key, all these sites have wonderful marketing websites which promise the earth. It’s not until you get down into the weeds on developer experience and pricing that the suitability becomes clearer.

*this post is the 2nd of a 4 part series of posts exploring Supabase, Vercel & Svelte. This post goes deeper into Supabase

Part 1: Intro
Part 2: Supabase
Part 3: Vercel
Part 4: Svelte

The Power Trio: Svelte, Supabase, & Vercel – My [current] Tech Stack of Choice

Standard

*this post is the 1st of a 4 part series of posts. This post gives a quick overview of Svelte, Vercel & Supabase, the following posts will go deeper into the technologies.

Part 1: Intro
Part 2: Supabase
Part 3: Vercel
Part 4: Svelte

Choosing the right stack for your project is akin to setting the foundations of a building. It needs to be solid, reliable, and scalable.

As a software engineer for over 26 years, I’ve finally found a stack that feels like home: Svelte, Supabase, and Vercel.

This trio has not only supercharged my own productivity but has also proven indispensable in my role as CTO of a small startup. When every decision weighs heavily on our budget and future, finding a stack that offers ease of use, scalability, reliability & cost-effectiveness, has been thoroughly reassuring.

Supabase, Svelte & Vercel

Why SvelteKit, Supabase, & Vercel?

Each component of this stack brings something unique to the table.

Svelte, with its simplicity and speed, allows us to build web applications that are incredibly fast and easy to maintain. It eliminates the complexity typically associated with front-end development. It makes the developer experience delightfully smooth. This has been a game-changer for us. In a startup environment resources are limited, and we need to move quickly without sacrificing quality.

Then there’s Supabase, an open-source Firebase alternative, which has been a revelation. It offers the backend services we need – authentication, database, and storage – without the overhead of managing these systems ourselves. Its PostgreSQL foundation means we’re building on top of a powerful, open-source database. Not only that, its easy-to-use APIs save us countless hours that would otherwise be spent on backend development.

Vercel provides a seamless deployment and hosting solution that integrates perfectly with SvelteKit and GitHub. Its global CDN ensures our applications are fast, no matter where our users are. Its commitment to developer experience makes deploying our applications as simple as a git push. In the fast-paced environment of a startup, Vercel’s scalability and ease of use are invaluable.

Productivity Gains

The synergy between Svelte, Supabase, and Vercel has significantly boosted our productivity. The reduction in context switching, the streamlined development process, and the ease of deployment means we can go from idea to production incredibly fast. In a small startup, where each member often wears multiple hats, being able to focus more on solving our users’ problems and less on the intricacies of our tech stack is a massive advantage.

Scalability for Startups

For any early-stage startup, the ability to scale efficiently is critical. This stack ensures that we’re not just building for the present but are also prepared for future growth. Supabase and Vercel, in particular, offer scalable solutions that grow with us. Both ensure that we can handle increased loads without a hitch (and without surprise bills). This peace of mind allows us to focus on innovation and delivering value to our users, rather than worrying about our infrastructure.

A Personal Reflection

My personal journey through the realms of large corporations and startup agencies has taught me the importance of choosing the right tools. In the past, I’ve dealt with the complexities of custom builds and the challenges of managing primitive services on platforms like AWS, Azure & GCP. While powerful, they often require a significant investment in time and resources to manage effectively.

In my current role, where the margin for error is slim, and our budgets are tight, the simplicity, efficiency, and scalability of the Svelte, Supabase, and Vercel stack have been a blessing. It’s a setup that supports rapid growth and innovation, aligning perfectly with the transparent, agile, forward-thinking ethos of our startup.

My 26 years of experience across different spectrums of the tech industry has solidified my belief that software development is hard, the tech stack should make it easier, and abstracting the difficult parts away to scalable services should always be considered.

With this particular tech stack, I have personally found a great balance between interoperability, scalability and extensibility. A word of caution though, what works for one team does not necessarily mean its right for others. However, for our situation, it’s been a testament to how the right set of tools can not only enhance productivity but also empower a team to focus on what truly matters – creating value for the users.

*this post is the 1st of a 4 part series of posts. This post gives a quick overview of Svelte, Vercel & Supabase, the following posts will go deeper into the technologies.

Part 1: Intro
Part 2: Supabase
Part 3: Vercel
Part 4: Svelte

NIDC 2023 – Startup Priorities

Standard

Recently I had the absolute pleasure of talking at the 2023 Northern Ireland Developers Conference.
My talk was titled “The Art of Bootstrapping: Focus on the things that REALLY matter” and took a look at what the priorities of a startup should be.

TL;DR: According to the data, startups spend too much time & resources on relatively unimportant things, compared to a lack of focus on product market fit. To get an idea of what “product-market fit” equates to in practice for startups, see the following advice from Andrew Gadzeki (founder of Acquire.com)

The NIDC is my favorite conference to go to for several reasons, but the main reason is that it is a conference by developers, for developers. Everyone is there to share or to learn. In the spirit of sharing, I wanted to share the slides from the talk in case they can help anyone else.

High-Speed Web: Start with the End in Mind

Website Speed
Standard

The speed of your website is important to users. But in this age of pay-per-compute, reducing processing all along the chain is important to keep owner costs down too.

Fortunately, in today’s modern web, both of these seemingly competing requirements actually complement each other; reducing processing costs can be achieved by improving site speed.

To explore how to improve site speed and reduce overall processing, lets start with the end in mind and work backwards.

Starting with the customer, what consititutes a ‘good’ user experience, in terms of speed?

Ultimate User Experience?

Imagine you are a first time visitor to your site. The ideal situation would be if an immediate view of your site was instantly available to the device you’re using. No javascript processing, no querying for data, no image optimizations, just the browser showing HTML and [inline] CSS (or even one big snapshot image of your content).

Bear with me, I know some sites need to show dynamic data and some do not, but remember, at this point, you’re just an ordinary user. You dont care about what goes on in the browser and on the server. Neither do I care what static vs dynamic is, or what pain it takes to achieve results. You just want a great experience.

As a user, I want to see the content immediately. By the time I’ve made sense of the initial page (0-3.8secs), I want to interact with it.

If the data a customer is viewing is updated server-side while the page is open, these updates should be pushed to me automagically. Getting new data should not rely on me to fetch the data e.g. by hitting some kind of refresh button to call back to the server.

If the customer leaves the site and comes back to it, by the time I have indicated that I wish to view the page (preemptive rendering?), it should already be fully loaded and no stale data exists on the screen. If any updates have been made to the page since I last saw it, the changes, and only the changes, should have been pushed to my device, using as little bandwidth as possible.

All sounds great? But are these demands even possible using the latest tools/technologies for web and server?

Server Side Rendering

Arguably one of the most important things is to show the content of your site in the quickest way possible.

Generally, the steps that a browser takes to display a web page are as follows:

  1. A request to the server is made for a webpage.
  2. The server decodes this request and the page, and its resources (files), are downloaded.
  3. The web browser uses the page resources to build the page.
  4. The page then is rendered (displayed) to the user.

The bottle-neck in this process are steps 2 + 3 i.e. downloading the resources and ‘building’ the page as the resources become available. Rendering a ‘built’ page, stage 4, is what a browser is really good at.

Can we improve, or even skip, steps 2 + 3 to give users a better experience?

If the customer’s browser isnt going to do the hard work to pull all the resources together to build the page, who’s going to do it then?

Combining and building the page can be perfomed on the server instead. The subsequent ‘built’ page can be the resource that is served to the customer.

Server side rendering is an ‘old’ method of serving web pages. Before client-side Javascript was as powerful as it is today, SRR was a common way to serve dynamic web pages.

What’s changed is that now you can use Javascript on both the client and server. You can now replace PHP, Java, ASP etc on the server with NodeJs. This certainly helps with code reuse between client and server, but are we actually any further forward?

The principles are still the same. Client browser makes call to server, server dynamically creates a webpage containing an initial view of the page, server then delivers the page to client.

Server side rendering certainly solves the processing problem for the customer but we’ve just pushed the problem server-side. Not only have we increased the amount of processing that we, the site owner, has to pay for. We are also not much further in improving the overall speed of the site. Certainly, the customer may see a more complete view of your site sooner. Also, the server may have better access to some of the resources. But the overall amount of processing to build the page stays relatively the same.

If we, as a site owner, are now paying for the processing to ‘build’ an inital view of the site how can we make this process as efficient as possible?

Static First

The vast majority of content on the web changes infrequently. Even sites with ‘dynamic’ content, usually only a small amount of the total content of a page is truly dynamic.

If the same content is going to be viewed by more than one visitor, why generate it more than once? It might not be a big issue if you only have 2 visitors. But even with 11 visitors, you still might be doing 10x more processing than was needed.

If as much of your content can be precompiled i.e. the data has already been fetched, the styles applied and html generated preemptively, it can be delivered to the user quicker. If fact, if the content is already compiled, we can take the server out of the chain completely for this interaction, and allow the browser to access the content directly from a location close to the customer.

The ‘static first’ strategy is to compile a static view of the page first, serve it from a CDN, delay enabing javascript until the page has loaded, then hydrate any stale dynamic data.

By adopting static first you can potentially reduce your hosting costs to cents per month, as opposed to dollars, AND provide a blisteringly fast experience for your customers

But what about pages that are never viewed? To statically generate an entire site, you need to generate and manage updates for all potential routes in your website. You might be generating hundreds or thousands of web pages that real users may never visit. However, although ‘real’ users may not visit these pages, it is likely, and welcome, that at least 1x crawler bot will want to access these pages (unless it is content not for the eyes of search engines).

Caching Vs Static Site

So if having assets ready and available on the network, close to the user, is preferable. Is this not just server caching?

Yes and no. There are a number of different caching options available for your site. You can cache the individual items that are referenced by your page e.g. images, css files, database queries etc, or you can cache the page itself, or both. A static first strategy will try to cut the cord with the server. The strategy does not require database query caching and processes as much into a cachable unit i.e. page caching. Caching is generally performed dynamically i.e. the caching happens when one or more users access a particular page. Static site generation is performed pre-emptively i.e. before any users access a particular page. Caching and static site generation both have the aim of making reused assets as available and as close to a user’s device as possible; the difference is if this is done entirely pre-emptively or dynamically.

Static First, Only Update Dynamic Data

However, frequent updates may be unavoidable depending on the type of site. For dynamic sites, it is not feasible to continually pre-compile all views for all users, especially when the data is changing frequently.

But remember again, your user does not care. Mostly, they dont understand the difference between static and dynamic sites; they want to see the important content, fast.

You can aim to statically compile as much of the page as possible beforehand, but the ‘dynamic’ parts will involve some sort of processing. As a first time user, I may accept having the page load, seeing a placeholder where the dynamic data should be, and then ‘hydrating’ the data on page load. On the other hand, the user may, in this instance, prefer a slightly slower page load, if the page loads with the data already fully ‘hydrated’. The choice probably depends on what makes your customer happiest.

Subsequent Visits & Data caching

Up until now we’ve generally been concentrating on the scenario when customers first visit your site. When a customer visits your site again, the situation is slightly different. A returning customer will already have, on their device, many of the resources for the page. If nothing has changed, they may even already have everything to show the page again.

As a returning user, it makes little sense for me to have to contact the server again, have a new view generated by the server and download a new page. If only a small subsection of the page I already have has changed, this is unnecessary processing.

The ideal situation is if the server actually pushes my browser updates. When this happens my browser doesnt have to continually ask if new data is available. An even better scenario is if the server has already pushed me the data before I open the page again.

Even if you dont consider websockets and/or service workers you still have the opportunity to cache api data on the server. If a piece of data has not changed since the last time your browser (or any other browser) asked the server for it, it introduces unnessecary processing. Not for the feint hearted, but api caching can be achieved using the ETag header of a HTTP call.

Final Note. Software Development is hard.

There are only two hard things in Computer Science: cache invalidation and naming things.

— Phil Karlton

There are lots of difficult things about software development, but cache invalidation is the devil.

To reduce processing, all the methods above require caching in some shape or form. Static website generation is just page caching by another name. Putting things in cache is easy, knowing when to update them is incredibly difficult.

If your website page changes url, should you rebuild your entire site in case other pages reference the changed page? Does this re-introduce more processing than its worth?

If you statically compile your stylesheets inline into the page, what if a stylesheet changes? Does every page need compiled again, even if it doesnt make use of the changed style?

If a property in a row of data in your database changes, how do you invalidate your api cache? What if the data is referenced by another item in the cache?

If you are a masochist and like solving these type of problems, have at it. For the rest of us mere mortals, look to use a tool or service that helps you manage these problems.

Here is a non-exhaustive list of some newer tools and services that help in this space:

Hugo – Static website builder

Shifter – Compile your WordPress site to static

Vercel – JAM Stack + React Server Side Rendering + Hosting

Netlify – JAM Stack + Hosting

Webiny – Almost serverless Static website builder + API Generation

Svelte – Javascript framework that precompiles to plain JS so you dont need to deploy and load a framework with your webapp.

FOSDEM 2020 & Web Fonts Performance

Standard

FOSDEM is a free and opensource conference held in Brussels every year. There is no admission charge to the 2 day event and the topics under discussion are incredibly diverse. If you are interested in anything and everything opensource, FOSDEM has you covered.

This year was my 2nd year at the conference and, yet again, it did not disappoint. Most of my time was spent in the Web performance & Javascript developer rooms which were packed out. I also managed to get to some ethics talks and to the Quantum Computing room (but quickly scarpered out of there when things got complicated).

I learned quite a bit there, but one quick win that I want to share with you was on web font performance. During the Sia Karamalegos talk on web fonts she shared some quick tips on getting your fonts to load quicker. The full talk is embedded below, but the tips are as follows:

  • Preconnect External Fonts. If you are loading external fonts e.g. Google Fonts, a quick performance tip was to use the “preconnect” tag. Using this tag will establish a handshake with the external server before the resource is loaded. With the connection warmed up, it will take less time to download the resource when the time comes. *A small caveat to this, only preload things definately used on the page, otherwise it is doing extra work that might not even be needed.*
/* the usual way of loading google fonts */
<link href="https://fonts.googleapis.com/css?family=Muli:400"
      rel="stylesheet">
Google fonts load waterfall showing wasted time from loading from extra connection time
Wasted Time Loading Google Fonts
/* A better way of loading */
<link rel="preconnect" href="https://fonts.gstatic.com/" crossorigin>
<link href="https://fonts.googleapis.com/css?family=Muli:400"
      rel="stylesheet">
Google fonts load waterfall showing preconnect
Performance Improvement
  • Preload Self-hosted Fonts. If you are loading self-hosted fonts, the tip was to use the “preload” tag to load the fonts quicker. The “preload” tag will fetch a resource (without actually executing it). This means that the files will be downloaded right away and will not have to wait on other resources to be loaded.
/* loading self-hosted fonts */
<link as="font" type="font/woff2"
  href="./fonts/muli-v12-latin-regular.woff2" crossorigin>

<link as="font" type="font/woff2"
  href="./fonts/muli-v12-latin-700.woff2" crossorigin>
Google fonts load waterfall showing local font waiting to load until after CSS
Wasted Time Loading Self-Hosted Fonts
/* a better way */
<link rel="preload" as="font" type="font/woff2"
  href="./fonts/muli-v12-latin-regular.woff2" crossorigin>

<link rel="preload" as="font" type="font/woff2"
  href="./fonts/muli-v12-latin-700.woff2" crossorigin>
Self-hosted waterfall showing preload

Resource Hints

To explain the difference in the “rel” tags, here is a quick cheet sheet from @addyosmani.

Full Video

Please check out the video below for the full talk which also includes tips on FOIT , variable fonts, and tooling.

Full Talk On Font Performance

Firestore Saving User Data

Standard

Cloud Firestore + Authentication FTW

The last post on this site was about how Firebase’s Cloud Firestore was a great option for saving data, especially in new or prototype applications.

In a previous post I also showed how you can use Firebase’s handy authentication mechanism to easily register and authenticate users to your site.

This post will attempt to show how the Firestore and authentication mechanisms can be combined to store user preference information.

Authentication Recap

In the auth post we saw how easy it was to setup and that, when a user authenticates, they have a unique id when you can then use.


firebase.auth().onAuthStateChanged(function(user) {
if (user) {
// User is signed in. uid is now available to use
var uid = user.uid;
}
});

The code above shows how we can watch for when a user logs in. After they have logged in we can then use the uid as a unique identifier for each logged in user. The next step is to start creating some data in our Firestore that we can link to this authenticated user each time they log in.

Save User Related Data

Saving user related data e.g. profile information, with Cloud Firestore is easy. Once you have an authenticated user’s unique identifier you can start to save documents in your Firestore with the UID as an identifier. Each time the user logs in you just look up the document in your Firestore to retrieve the related information about this user/login.


firebase.auth().onAuthStateChanged(function(user) {

var dbUser = db.collection('users’)

.doc(user.uid).set(

{

email: user.email,

someotherproperty: ‘some user preference’

});

});

In the code above we wait for the user to log in, retrieve their UID and create a document in the Firestore using the uid as the index. From this point we can add as many properties to this document that we need, and retrieve the information anytime the user logs in.

Simple stuff! Give Firebase a go, you won’t regret it.

Firebase Cloud Firestore

Standard

NoSQL, huh, whats is it good for?

I am not a huge fan of NoSQL. If you can keep your data structured, you should keep your data structured. Also, I have never, to date, worked on a project where a relational database was not scale-able enough to cater for the amount of data I needed to throw at it. NoSQL databases, like Firestore, have their uses, but for most production projects I’ve worked on so far, a relational database was either the ‘best’ choice or ‘good enough’.

However, sometimes, the structure of the data is unknown or likely to change. One such occasion is when you are starting out on a project. For prototype web applications I have found that NoSQL (Mongo, Firebase etc) can really speed up the initial stages of the project. Firebase’s new Cloud Firestore offering is a really useful NoSQL database for prototyping projects.

Real-time Database Vs Cloud Firestore

Firebase used to call its database offering ‘Realtime Database’ it now has a new offering called ‘Cloud Firestore’. Both are currently supported and available through the console, but it does appear that Firebase are trying to coax users to use the newer, but still in beta, Firestore.

The main difference between the two seems to be the data model. One of the frustrating things about the Realtime database was that you needed to de-normalize your data in order to use the queries effectively. If your data was hierarchical in nature you found yourself having to jump through hoops in order to use the realtime database effectively e.g. because you are always returned the entire sub-tree for a query, you needed to create separate trees for your data to avoid bloat in your result sets.

Cloud Firestore FTW

Firestore is not just any old NoSQL database, it has the following compelling features:

  • Easy to setup. The Firebase docs do a great job of helping you get started in whatever language you use (web, IOs, Android, Node, Go, Python, Java).
  • Easy to use. I have tried to think of a way that the Google team could have made it even easier to use the SDK… but failed.
  • Realtime. Real-time means real-time, try it out, create a simple project, hook your UI up to some data, watch the data change as you change the data in your console.
  • Scalable. So they say, sadly I’ve not had to use the scalability super powers yet.

Recommendation

Even though its still in beta, I would thoroughly recommend choosing Firestore over the realtime database for new projects. For existing projects, which currently use the realtime database, the choice is not as simple. Migrating to the Firestore is not a simple task due to the vastly different datamodels. Take a look at the pros and cons to determine if it is indeed worth the migration effort.

 

In the next blog post I’ll show how you can get started with Firestore and build upon the Authentication post to tie authentication events to saved user data.

 

Firebase Authentication

Standard

Tools For Your Tool-Belt

As a freelance developer its good to have a number of different tools in your tool-belt for when the situation demands. Firebase, and in particular, Firebase Authentication, has proven really useful over the past couple of years.

JWT

Most projects that I work on nowadays require some sort of user authentication. In the past I have used JWT or basic auth as my go-to solutions for authentication. However, even those these methods are fairly straightforward, I still had the nagging feeling that “This is a common problem, there must be a way to make this even easier.”.

JWT is great and easy to get started with but it is not always plain sailing; basic auth is well….. basic. JWT is very similar to Firebase Authentication in that you can use a 3rd party to authenticate users (using the Auth0 service) and not have to set up your own authentication server and process. However, manually persisting tokens on the client side and handling the clean up on expiry, logout etc, still felt verbose to a lazy dev like myself.

Firebase Auth

Firebase Authentication is easy to set up, the wide range of SDKs are easy to use, is free to start using, and “supports authentication using passwords, phone numbers, popular federated identity providers like Google, Facebook and Twitter, and more”.

Most of my experience using Firebase authentication has been in web apps or hybrid mobile app development; below are some really quick steps on how to get started for web:

  1. Sign up for a Firebase account and create a new project.
  2. Choose “Add firebase to your web app” and follow the instructions i.e. include the JS library and add the configuration details to your app.
  3. Add a screen which asks the user for an email address and password. Then pass these details to the “createUserWithEmailAndPassword” SDK function to create a new user.
  4. Add another screen which asks for an email address and password, then pass these details to the “signInWithEmailAndPassword” function. When the function returns you should now have an authenticated user! I could go into more detail on the other methods you can use e.g. “signOut”, but most of them really are self-explanatory and the docs do a great job of showing you how.

Add a login screen with text boxes and a login/register button, then hook the button click to the Firebase SDK methods

Facebook, Google, Twitter, GitHub

To add Facebook, Google, Twitter or Github authentication just:

  1. Enable the required type of authentication through the “Authentication” tab on your Firebase console (following any specific instructions for each provider).
  2. Add a button to your login page for each provider you wish to use.
  3. Hook up the button click to use the appropriate SDK method on the client e.g. “signInWithPopup(new firebase.auth.GoogleAuthProvider())”
  4. After the user has authenticated themselves through the popup window, your call should return and you should now have a [Google] authenticated user.
  5. *A word of caution for hybrid mobile developers* Popups will not work on your hybrid mobile app, you will need to use native Facebook/Google libraries to achieve the same thing e.g. using facebook sign-in for Ionic.

Mobile, federated, logins are waaaaaay more difficult!

Populating Elastic With Your Database

okay-we-get-it-elastic-search
Standard

The Problem

Recently I was tasked with providing fast full-text searching, across multiple tables, for a reasonably big database (~2m records).

Immediately I reached for some sort of Solr or Elastic solution. Being the lazy developer that I am, I chose the path of least resistance (most samples online on how to get started), and chose Elastic [edit: would seriously consider Angolia for my next project, if I had less than 10k records to index].

Getting Elastic set up was relatively simple. Having access to a few AWS credits, I used the AWS ElasticSearch service because I didn’t want the responsibility for maintaining the server.

Now to populate with some data.

Taking data from a database and putting it in an Elastic index must be a common scenario with lots of examples and help, right?

Well, not quite as common as I thought, and not as easy as I was hoping.

I was “hoping” for a nice simple user interface; point at my database, select what data I want, import into Elastic, done. I thought, with the relative maturity of Elastic, that it would be baked into the product, or someone would have created an add-on already. At the least, I thought, there would be a plethora of examples to show how to get data from a database to Elastic.

What I found was lacking, and its the reason why I’m making this post to help others looking to do something similar.

Logstash

The currently recommended solution for indexing database data is Logstash (up until 2015 the recommended way was through Rivers).

logstash

Logstash. Looks great from the diagram

With Logstash you can connect to your data, filter it, transform it, and then add it to Elastic. Great!, just what I was looking.

Logstash takes quite a bit of work to set up and configure though. The setup is fairly straightforward; just download and install on Windows or Linux. After installation, there is no UI with Logstash, just access to a command line command, to which you supply configurations.

Unfortunately Logstash does not have a way to connect with databases out of the box. To connect logstash to your database you will first need to jump through a couple of hoops:

  1. You will need to install the JDBC plugin for Logstash, the instructions on how to do this can be found here.
  2. The JDBC plugin does not include any database drivers so you will need to find the driver you need and then download. I am mostly connecting to Postgres so I need to download the Postgres JDBC driver. Make a note of where you download this driver to, you will need the path to the driver to your configuration later.

Providing you have no issues with your Java installation, you should now be ready to go.

Logstash Configurations

Most of your time will be spent fine tuning configurations. Simple configurations are straight forward i.e. connect to a single table with basic field types and import into Elastic. Anything outside these parameters can be challenging.

A basic configuration to connect to a Postgres database and import into a new Elastic index could look something like this:


# file: simple-out.conf
input {
    jdbc {
        # Postgres jdbc connection string to our database, mydb
        jdbc_connection_string = "jdbc:postgresql://localhost:5432/mydb"
        # The user we wish to execute our statement as
        jdbc_user = "postgres"
        # password
        jdbc_password = "mypassword"
        # The path to our downloaded jdbc driver
        jdbc_driver_library = "/path/to/postgresql-9.4-1201.jdbc41.jar"
        # The name of the driver class for Postgresql
        jdbc_driver_class = "org.postgresql.Driver"
        # our query
        statement = "SELECT * from contacts"
    }
}
output {
    elasticsearch {
        protocol = http
        index = "contacts"
        document_type = "contact"
        document_id = "%{uid}"
        host = "https://urltomyelasticinstance"
    }
}

The configuration file above can be run with the following Logstash command:

"\pathtologstashbindirectory\logstash.bat" -f simple-out.conf
The configuration, when run, will:
  1. Connect to the postgres database.
  2. Get all the data from the ‘contacts’ table.
  3. Connect to your elastic instance.
  4. Create a new index called ‘contacts’
  5. Create a new document type called ‘contact’
  6. Automatically create mappings for the fields in the contacts table.
  7. Populate the index with the data.
Easy, right?
The challenges come when your scenario deviates from this very simple example.
In the next post I’ll detail some of the issues that you might come across when deviating from this simple example.