story about simplicity and building complex systems with simple tools.
Story of my first big database project
bill of materials into a label for vitamin supplement company
Story from days at gaia trying to rebuild mySQL and getting it to restart after a crash.
Story of fixing MySQL master-master replication, snapshotting, advancing the binlog skipping over conflicts AGHHHH! Redis replication allows you to just point at the source and it does the right thing.
Tell story of project I RoR project consulted on.
Unit tests took 5 minutes to spin up and run
Couldn’t easily run one test
Required setup/teardown of MySQL which was slow
Schema was brittle, required lots of setup
I wish we could say we started out this way.
The first big code push we wrote code frantically and had no unit tests.
I remember just trying to eyeball api requests and responses from our mobile client to determine if we got the logic right. Took hours. We thought we were saving time by not writing tests but actually we wasted so much more. Don’t write tests just to write tests. Use it to do all the things that you would have to check by hand each time otherwise.
One-man-band kind of ops team.
Until last november I was the only person running ops.
We were able to keep that kind of lean operational budget because the architecture was so simple and because it didn’t cause problems.
on dev they can map to same redis db, and fan it out on production as you scale.
easy to replicate a database to another then swap one keyspace over and drop the unused keyspaces on each new database afterward.
Story about an old keyspace no longer being used: 10% of redis memory
Another cleanup case: realized we didn’t need a certain data structure after it was 3 months old
30% savings
Being strategic about what data you expire can make a big difference
Don’t store data you don’t need
Do log it or store in another mechanism where you can rehydrate it back in if you need
We use our analytics service to capture all event data that is important, and we can crawl that in redshift to build reports that can be scripted to re-hydrate and backfill data when needed