Today, I set out to write some integration tests. Since I'm using Postgres, one hurdle people often run into is how to get a smooth, fast experience while running tests against a test database.

A lot of people like testcontainers or running a long-lived Postgres container in Docker. I personally don't like the overhead of this extra machinery – it seems simple, but in my experience it ends up being fragile somehow, and needs tending and troubleshooting.

So, I want to try embedded-postgres, which is as close to our (sqlite) dream as we get in the Postgres world. (side note, I still often dream of a more official, lightweight, embedded Postgres solution). This project works by downloading Postgres binaries (from Maven, apparently) and giving you some Go functions to start/stop a Postgres server.

The nice thing about this approach is you can (ideally) open some Go code for the first time in VSCode, find a test, click the green "run" arrow, and it should just work – no docs to read, no make/just/bash scripts to run, no docker to install/manage, etc.

And it did work for me on the first try, but it was slow. If I was iterating on a single test, running that test frequently after making changes, I was seeing ~20 seconds per test run. That felt unbearably slow.

At first, I thought it was Postgres setup time. I attempted to tackle that by setting a persistent data directory, so that Postgres wouldn't spend time in initdb.

testDB = embeddedpostgres.NewDatabase(embeddedpostgres.DefaultConfig().
			Port(port).
			DataPath("./data/pg-embedded").
			Database("test_db"))

I was surprised when that wasn't the answer – it helped a little, but I was still seeing ~18 seconds per run.

After digging around (and with plenty of help from Claude, of course), I found that most of the time was spent extracting the compressed Postgres binaries. Apparently embedded-postgres caches the downloaded files, but it will extract the archive on every run (github issue is here).

You can change this behavior by setting a BinariesPath config:

testDB = embeddedpostgres.NewDatabase(embeddedpostgres.DefaultConfig().
			Port(port).
			DataPath("./data/pg-embedded").
			BinariesPath("./data/pg-embedded-bin").
			Database("test_db"))

With that, my test time dropped to ~1 second. The first test pays the cost of setting up the database, but after that I reuse the connection, so tests drop to ~0.1 seconds each. I'm running the migrations once on startup.

I added some extra configuration to remove logs and other overhead from Postgres:

		testDB = embeddedpostgres.NewDatabase(embeddedpostgres.DefaultConfig().
			Port(port).
			StartParameters(map[string]string{
				"fsync":              "off",
				"synchronous_commit": "off",
				"full_page_writes":   "off",
				"checkpoint_timeout": "86400", // once per day, effectively never
				"log_checkpoints":    "off",
				"log_connections":    "off",
				"log_disconnections": "off",
				"autovacuum":         "off",
				"log_min_messages":   "PANIC", // suppress almost all log output
			}).
			DataPath("./data/pg-embedded").
			BinariesPath("./data/pg-embedded-bin").
			Database("test_db"))

I don't have this in CI yet – some extra gymnastics will be required to keep the cached+extracted binaries available in CI, as usual.

Personally, I love high-level testing against the actual API and database – it gives me confidence that the feature actually works when all the pieces are put together, and I find it useful to be able to reproduce a reported issue by writing a test that matches the high-level steps to reproduce from the report.