Shopify Search Engine

Project Specification

  • Fast. Shopify's default search was not meeting performance expectations.
  • Synonyms of searched words.
  • Redirection of certain terms to various site pages.
  • Custom fields used in searching and more page types returned than Shopify's default.
  • Custom weighting of fields and "Boosting" of certain result sets based on criteria.

Application Design

Click headers to expand/collapse

General Application Design Principles

Microservices...to a point. Never a monolith, but let scale justify a new application service. I don't necessarily agree with the argument that monoliths should be the initial design for startup apps, but understand the reasons and err in that direction.
Data is generally grouped for the entire app in a single instance. Data is not combined with other apps, but neither is it split out to a single concern. My reasons for this are the minimum financial and management costs associated with provisioning a database exceed the benefits from reliability gains.
Use technologies that minimize ongoing maintenance of the application. This generally means heavy use of serverless technologies. The reasoning behind this is that maintenance typically exceeds development in terms of cost and time, so the additional costs for not managing infrastructure are justfied at small to medium scale.
Some cloud-specific technologies are used, but generally cloud-agnostic technologies are preferred. This generally means heavy use of containers.
Continuous Integration & Deployment is used in all cases. In nearly all cases, the flow is from GitHub to Azure DevOps/VSTS to the deployment target.

Data

Persistence

Azure Storage Blobs / Tables
Technology:

Azure Storage Blobs / Tables

Function:

House the data to be indexed as well as other app data

Search Engine

Azure Search
Technology:

Azure Search

Function:

Properly index the site data for rapid searching

Cache

RedisKubernetes / DockerAzure DevOps
Technology:

Redis

Deployment:

Kubernetes / Docker

Deployment:

Azure DevOps

Function:

A cache to house frequently requested data to improve performance

Back-End

API

ASP.NET Core / C#Kubernetes / DockerAzure DevOps
Technology:

ASP.NET Core / C#

Deployment:

Kubernetes / Docker

Deployment:

Azure DevOps

Function:

Provide data to the Shopify Admin UI, allowing modification for logged in store owners

Change Detector

Azure Functions / C#Kubernetes / DockerAzure DevOps
Technology:

Azure Functions / C#

Deployment:

Kubernetes / Docker

Deployment:

Azure DevOps

Function:

Through a combination of webhooks and polling, detect changes in the front end and hand them off to a queue

Indexer

Azure Functions / C#Kubernetes / DockerAzure DevOps
Technology:

Azure Functions / C#

Deployment:

Kubernetes / Docker

Deployment:

Azure DevOps

Function:

As a queue fills, make changes to the index according to the messages

Front-End

Shopify Admin UI

React / Typescript JSKubernetes / DockerAzure DevOps
Technology:

React / Typescript JS

Deployment:

Kubernetes / Docker

Deployment:

Azure DevOps

Function:

Provide an Admin UI to store owners to make changes to how the search engine works

Shopify Application Proxy

React / Typescript JSASP.NET Core / C#Kubernetes / DockerAzure DevOps