For a while now, I’ve been building findatechjob.dev to help developers find their dream jobs without all the spam and distractions of the big jobs boards.

At 8am every morning, I’d be filled with dread as I checked my website to see whether the daily jobs load had succeeded.

Most of the time it worked. When it didn’t, I had to manually restart the VM. Shocking.

What Went Wrong

To get fresh jobs data into findatechjob, I ran a Cloud Run Job each morning. It hit various job APIs, transformed the results into a standard format, and dumped them into a Meilisearch server running on a VM in GCP.

This worked, for the most part. It cost $8/month, search latency was good, and it was easy to set up.

The problem was the machine. I was running Meilisearch on an e2-micro instance, 2 shared vCPUs and 1GB of memory.

This was enough at first, until it wasn’t. Over time, the number of jobs I was gathering increased, so Meilisearch had to store more data in memory. Jobs are small, so that wasn’t the problem.

The problem was indexing. Each time I loaded a new dataset into Meilisearch, it had to re-index everything. This caused a memory spike on every load. Sometimes the VM survived. Other times it didn’t.

The Core Insight

Eventually the sad mornings wore me down, and I had plans to add far more jobs to the site. Something had to give.

I came to a realisation. Serving search requests is an inherently different task to building an index, yet I was expecting my tiny VM to handle both with the same resources.

So why not separate them?

The Solution

My solution was simple. I started running two instances of Meilisearch, one to index, one to serve.

Indexing

First, I replaced the VM with Meilisearch running as a second container in my Cloud Run Job, and loaded jobs data there instead.

This ephemeral Meilisearch was used only to build an index. The index was then compressed and uploaded to GCS.

I recommend reading this blog post, which outlines a similar architecture in far more detail than mine.

One issue I found with that approach is that it uses a Meilisearch dump, but those require re-indexing on load. This is why I uploaded the whole compressed data.ms directory instead. It comes out to about the same size.

Serving

With the index uploaded to GCS, I deployed a second Meilisearch instance as a Cloud Run Service. I built my own container image, wrapping the official Meilisearch image, that pulls the index from GCS on start-up.

To make this efficient and reduce cold-starts, I iterated on a few things:

  • Compressed the data.ms directory before uploading. This reduced the download size by 5x. Using zstandard gives fast decompression times too.
  • Replaced the gcloud CLI with curl to download the index. Initialising gcloud was taking 5-10s on every start-up!
  • Reduced the size of each job stored in Meilisearch. It now stores and searches only relevant metadata. Full details live in Firestore. An additional 30x reduction in index size!

Results

This solution has been working well for me.

Some stats to satisfy data-driven minds:

  • Meilisearch Resource Allocation: 1 vCPU, 1 GB memory. Actual usage barely scratches the surface. My site doesn’t get high traffic.
  • Compressed Index Size: ~5MB, with ~12,000 unique jobs.
  • Cold Start: varies between 500ms and 2000ms, depending on the mood of the Google Cloud Gods. Acceptable for me, might not be for you.
  • Cost Savings: the $8/month VM is gone. I’m now paying nothing, because my usage fits in the Cloud Run free tier.

When This Works/Doesn’t

This solution will work great for you if:

  • Your data does not constantly change.
  • You care about performance but don’t want to pay through the roof for it.
  • You don’t have vast quantities of data, or care much about cold start time.

It may not work if you need live updates to your data. This is great for me because the data loads once a day and doesn’t change, but if your data continuously changes, you’ll need Meilisearch serving requests and indexing new data all the time.

Closing Thoughts

Separation of concerns isn’t just software-design dogma. It fixed my mornings.

I haven’t had a sad morning in 2 months, and saved 100% on costs, which looks better than $8/month.

I’ll follow up with another post once the site gets more traffic, to see how this holds up.

Check out findatechjob.dev if you haven’t already!