Technical articles need to read.
Question need to solve it.
Also Include a new python API service?
header; payload; signature
Why Kafka is so quick?
sequential I/O <=> random I/O
100MB/s <=> 100KB/s
(append only log)
column-based db design vs row-based db design
Evolution of Yelp’s Elasticsearch-based Ranking Platform
B-trees and B+trees
comparsion between B-trees and B+trees
1. B+ trees, search keys can be repeated but this is not the case for B-trees
2. B+ trees allow satellite data to be stored in leaf nodess only, whereas B-trees store data in both leaf and internel nodes.
3. Deleting data from a B+ tree is easier and less time consuming because we only need to remove data from leaf nodes.
4. Leaf nodes in a B+ tree are linked together making range search operations efficient and quick.
master => slave
Replication is asynchronous.
Changes on the master aren't logged until commit.
Then slave downloads binary log.
Slave lag happens
Even when using semi-synchronous replication.
Must always query from the master.
Reading and writing data in the same transaction - or even the same statement.
INSERT INTO foo SELECT ...
bids in an auction
Slave must include recent changes made by any session.
Lag must be less than a specific number of seconds.
Data must include recent changes made by the same session.
Query may read freely from any slave without checking lag.
Reading data that doesn't change.
Reading old data is okay.
count of "likes"
How zero copy works.
1. From disk data is copy in kernal read buffer. (Performed by the direct memory access DMA engine) reads file contents from the disk and stores them into a kernal address space buffer.
2. From kernal read buffer it copy to application read buffer
3. From application buffer it will copy to kernal socket buffer and put the data into a kernel address space buffer again.
4. From socket buffer it will copy to network buffer.
1. causes the file contents to be copied into a read buffer by the DMA(Direct memory access) engine.
2. Then the data is copied by the kernel into the kernel buffer assoicated with the output socket.
3. From socket buffer it will copy to network buffer.
1.The transferTo() method causes the file contents to be copied into a kernel buffer by the DMA engine.
2. No data is copied into the socket buffer. Instead, only descriptors with information about the data are appended to the socket buffer. The DMA engine passes data directly from the kernel buffer to the Network buffer, thus elimating the remaining final CPU copy.
Every read receives the most recent write or an error.
Every request receives a (non-error) response, without the guarantee that it contains the most recent write.
The system continues to operate despite an arbitrary number of messages being dropped by the network between nodes.
HTTP2 VS HTTP1.1 difference
Json Web Token
ACID Properties in DBMS
A transaction is a single logical unit of work that accesses and possibly modifies the contents of a database. Transactions access data using read and write operations. In order to maintain consistency in a database, before and after the transaction, certain propertoes are followed.
Atomicity, Consistency, Isolation, Durability
1.The entire transaction takes place at once or doesn't happen at all.
2. The database must be consistent before and after the transaction
3. Isolation Multiple transactions occur independently without interference.
4. Durability: the changes of a successful transaction occurs even if the system failure occurs.
RDB Redis Database
AOF Append Only File
RDB is verey compact single-file point-in-time representation of your Redis data. RDS files are perfect for backups.
RDB is very good for disaster recovery, being a single compact file that can be transfeerred to far data centers.
RDB maximizes Redis performances since the only work the Redis parent process need to do in order to persist is forking a child that will do all the rest.
RDS is not good if you need to minimize the chance of data loss in case Redis stops working
Using AOF Redis is much more durable
The AOF log is an append-only log
Redis is able to automatically rewire the AOF in background when it gets too big.
AOF files arew usually bigger than the equaivalent RDB files for the same dataset.
AOF can be slower than RDB depending on the exact fsync policy.
interface Definition Language
language-neural, platform-neutral extensible merchanism for serializing structured data.
Token based authentication with Google
Reset at the start of each sampling period.
it will alow regular traffic spikes to go through the rate limter.
huge processing and memory requirements.
leaky bucket (used by NGINX)
It works by incrementing a counter when each request comes in.
Sliding windows to the rescue
It smoothes the traffic spike issue
Very easy to understand and configure
0.003% of requests have been wrongly allowed or rate limited
《Twitter snowflake is really cool》
Sign bit 1 bit
Timestamp 41 bit
Machine ID 10 bit
Sequence number 12 bit
Advantage & Disadvantages
· It's 64 bit long, it is half the size of UUIDs.
· Scalable (It can accommodate 1024 machines)
· Highly available (Each machine can generate 4096 unique IDs each millisecond)
· Some of the UUID version do not include a timestamp. Have a sortable advantage.
· Preditable for future IDs.
· The maximum timestamp that can be represented in 41 bits is ~69 years.