Creating Scalable Systems

-
Some food for thought. Consider the following:
  1. Prefer BASE over ACID transactions
  2. Prefer asynchronous over synchronous transactions
  3. Keeping state is expensive
  4. Considering database sharding (highscalability, codefutures, Pros and Cons) by data, by transaction or by customer but avoid premature optimisation
  5. Design the system for automated rollback
  6. Create isolative structures; share nothing; such that nothing crosses the swimlanes
  7. Design systems for failure
  8. Create idempotent services where possible
Database sharding requires changes in mindset:
  1. Tables may need to be denormalised to optimise sharding (as well as to workaround cross-shard joins/ queries)
  2. Scale-out instead of scale-up
  3. Do away with replication where possible
Different sharding schemes are:
  1. Vertical partitioning – sometimes known as functional or feature partitioning where data relating to certain entities are grouped together. Different functions or features are put onto different shards.
  2. Range-based partitioning – data for a certain function/ feature/ entity is sharded using ranges (such range may be based on year, location, etc.)
  3. Hash-based partitioning – data for a certain function/ feature/ entity is sharded using a hash function (modulo operation)
Database sharding presents a number of issues:
  1. Data needs to be rebalanced from time-to-time
  2. Joining data from multiple shards (cross-shard join) is expensive
  3. Referential integrity is now an issue since referential data may now be in a different database
  4. Sharding is relatively new; no body of knowledge and lack of support

Comments

Anonymous said…
Hi

Great post. Some remarks on Sharding - it is possible to do cross-shard joins, so no need for schema changes. Just use an off-the-shelf tool for that - like ScaleBase (disclosure - I work there). They give you transparent database sharding.
16/09/2011, 05:59
Post a Comment

Popular posts from this blog

Understanding ITIL Service Management the UML way…

-
Image
I originally had a great deal of difficult reconciling the differences amongst ITIL Change Management, Incident Management, and Problem Management. Having researched on the subjects and understanding them better now, I’ve tried to put the concepts together using a UML class diagram to capture my understanding and to keep it imprinted. Similarly, I wanted to differentiate the various categories of changes (Standard, Emergency, Normal) using an easy to understand & self-explanatory manner – a UML Activity Diagram!
Read more

How to depict (Professional-Looking) Logical Network Diagrams in Astah

-
Image
I have always had to create (professional-looking) UML diagrams using Visio-like icons. You’ve got to admit that showing a bunch of boring UML boxes hardly interests anyone less technical (like anyone whose designation has the words “manager” or “director”). This is actually possible in Astah! I’ve created a template for future projects that allows me to produce logical network diagrams that look like the following: To create this diagram, you need to do the following: Assign various icons to stereotypes. I created the following for my use: Create a new UML Deployment Diagram Every element in the diagram is created as a Node The nodes are subsequently assigned the appropriate <<stereotype>>. E.g. Internet is <<internet>>, WebZone FW is <<firewall>>, Web LB is <<loadBalancer>> Ensure the “Icon Notation” is set to “Customized Icon” for each node Instead of the boring boxes, the corresponding icons automatically gets displayed!
Read more