Introduction

What is the common essential feature among apps we use in our daily lives, such as Coupang, Baemin, and Yanolja? It's search functionality. In Coupang, we search for products; in Baemin, we search for food, and in Yanolja, we search for travel options. We also use search on platforms like Google and Naver to find restaurants or shop. Search has become an indispensable part of our daily lives, and it's a must-have feature in all modern applications.

However, developing search functionality is by no means an easy task, as backend developers and product managers are well aware. Creating a proper search functionality involves significant challenges. Users expect Google-level search quality even within applications, and as that expectation rises, the technical requirements become more complex.

So, what are the technical requirements that make developing search functionality difficult?

There are two primary methods for developing search functionality: using the search capabilities (specifically, query capabilities) provided by a database and adding a separate search engine to synchronize with the database.

Using Database Query Capabilities

The advantage of the first approach is that it allows for relatively simple development in the early stages of a service. It leverages the query and indexing capabilities provided by the database, eliminating the need for additional solutions. Developers can work with the database's query language, resulting in a smooth learning curve.

For example, in a hotel reservation app, you can use a simple SQL query to find hotels in the Pangyo area and provide them to the user. In the early stages of a service when user requirements are not complex, database search capabilities are sufficient.

However, as search requirements become more demanding, significant limitations and performance issues arise. There are functional limitations such as handling typos, synonyms, and scoring calculations. For instance, if a stock market app doesn't handle synonyms, users must enter "NAVER" instead of "네이버" to find relevant information. Any slight typo might result in "no results found."

Moreover, as search criteria become more complex, the complexity of queries increases. What if a user wants to search for a 5-star hotel with an antique atmosphere that allows pets and provides breakfast in Pangyo? As users require more specific search results, query length grows, and you may need to create indexes for numerous conditions. Building these indexes consumes resources, and not using indexing leads to performance issues as the database scans all records to find matching results.

In conclusion, development time increases significantly due to testing, debugging, and other factors. Even when the functionality is complete, it can negatively impact service speed and more.

As your service expands, you'll ultimately require a separate search engine.

Connecting a Search Engine to the Database

Databases and search engines are fundamentally different software designed for distinct purposes.

Databases primarily focus on data integrity, consistency, and durability in data processing. They emphasize balanced performance for read and write operations, concurrency, availability, and recoverability in disaster scenarios. Querying data is relatively less critical, except for specialized databases designed for handling massive data extraction (OLAP databases).

In contrast, search engines are designed to find the most relevant results for user intentions within stored data. They assume features like typo correction, synonym handling, searching for related but not identical results, and ranking search results. Therefore, the database's capabilities are inherently limited.

Due to this trade-off relationship between the two software types, it's advisable to store and process data in databases while handling search functionality separately in search engines. However, this choice leads to a complex development challenge, and hiring skilled backend developers capable of developing and maintaining search functionality becomes a crucial task.

First and foremost, adding a search engine means dealing with two different systems. You'll need to learn and use the search engine's language, resulting in a learning curve. Every time you deploy your service, you must test both systems simultaneously. API changes will occur as well. The backend's dependencies will deepen, and product development speed will gradually slow down. There will be a dramatic drop in development productivity.

Above all, the most challenging issue is synchronizing the database and search engine. To search for data stored in the database through the search engine, both systems must have identical data. Storing the same data in duplicate already raises inefficiencies.

Moreover, you'll need to establish mechanisms to synchronize new events like user registrations and adding products. These mechanisms might involve creating data pipelines based on messaging systems like Apache Kafka or using separate package connectors. Building these solutions takes time and adds costs.

Once the synchronization process is deployed, you'll need to monitor and manage it. You have to track whether synchronization occurs at the same speed as transactions in the database, ensure there are no security issues, and troubleshoot in real-time. If an issue arises, you'll have to determine whether it's a database or search engine problem. Naturally, you'll need to purchase solutions like Datadog. If, by any chance, the search engine's indexing speed falls significantly behind the database's, you'll have to re-synchronize from scratch.

This complexity affects application performance, deteriorates user experiences, decreases retention, lowers conversion rates, and ultimately leads to customer churn.

Is there an easier way to develop search functionality?

Using only database query capabilities is insufficient for implementing proper search functionality. Integrating a search engine with the database requires specialized search development skills and increases development complexity. So, is it truly impossible to enhance the user experience with high-quality search during the early stages of a service?