Redis in Action by Unknown -- Read -- Imperial Library of Trantor

Index

Redis in Action

Josiah L. Carlson Copyright

Dedication

Brief Table of Contents Table of Contents Foreword Preface Acknowledgments About this Book

Roadmap Code conventions and downloads Author Online About the author

About the Cover Illustration Part 1. Getting started Chapter 1. Getting to know Redis

Installing redis and python Using redis from other languages 1.1. What is Redis?

1.1.1. Redis compared to other databases and software

Table 1.1. Features and functionality of some databases and cache servers

1.1.2. Other features 1.1.3. Why Redis?

1.2. What Redis data structures look like

Table 1.2. The five structures available in Redis Command listing Reminder About Installing Redis and Python Redis with Other Languages 1.2.1. Strings in Redis

Figure 1.1. An example of a STRING, world, stored under a key, hello Table 1.3. Commands used on STRING values Listing 1.1. An example showing the SET, GET, and DEL commands in Redis Using redis-cli

1.2.2. Lists in Redis

Figure 1.2. An example of a LIST with three items under the key, list-key. Note that item can be in the list more than once. Table 1.4. Commands used on LIST values Listing 1.2. The RPUSH, LRANGE, LINDEX, and LPOP commands in Redis

1.2.3. Sets in Redis

Figure 1.3. An example of a SET with three items under the key, set-key Table 1.5. Commands used on SET values Listing 1.3. The SADD, SMEMBERS, SISMEMBER, and SREM commands in Redis

1.2.4. Hashes in Redis

Figure 1.4. An example of a HASH with two keys/values under the key hash-key Table 1.6. Commands used on HASH values Listing 1.4. The HSET, HGET, HGETALL, and HDEL commands in Redis

1.2.5. Sorted sets in Redis

Figure 1.5. An example of a ZSET with two members/scores under the key zset-key Table 1.7. Commands used on ZSET values Listing 1.5. The ZADD, ZRANGE, ZRANGEBYSCORE, and ZREM commands in Redis

1.3. Hello Redis

Figure 1.6. Reddit, a site that offers the ability to vote on articles Figure 1.7. Stack Overflow, a site that offers the ability to vote on questions 1.3.1. Voting on articles

Figure 1.8. An example article stored as a HASH for our article voting system Using the Colon Character as a Separator Figure 1.9. Two sorted sets representing time-ordered and score-ordered article indexes Figure 1.10. Some users who have voted for article 100408 Figure 1.11. What happens to our structures when user 115423 votes for article 100408 Listing 1.6. The article_vote() function Redis Transactions

1.3.2. Posting and fetching articles

Listing 1.7. The post_article() function Listing 1.8. The get_articles() function Default arguments and keyword arguments

1.3.3. Grouping articles

Listing 1.9. The add_remove_groups() function Figure 1.12. The newly created ZSET, score:programming, is an intersection of the SET and ZSET. Intersection will only keep members from SETs/ZSETs when the members exist in all of the input SETs/ ZSETs. When intersecting SETs and ZSETs, SETs act as though they have a score of 1, so when intersecting with an aggregate of MAX, we’re only using the scores from the score: input ZSET, because they’re all greater than 1. Listing 1.10. The get_group_articles() function

1.4. Getting help 1.5. Summary

Chapter 2. Anatomy of a Redis web application

2.1. Login and cookie caching

Table 2.1. Pros and cons of signed cookies and token cookies Listing 2.1. The check_token() function Listing 2.2. The update_token() function Listing 2.3. The clean_sessions() function Where to run cleanup functions Python syntax for passing and receiving a variable number of arguments Expiring data in Redis

2.2. Shopping carts in Redis

Listing 2.4. The add_to_cart() function Listing 2.5. The clean_full_sessions() function

2.3. Web page caching

Listing 2.6. The cache_request() function

2.4. Database row caching

Figure 2.1. A cached database row for an item to be sold online Using JSON instead of other formats Nested structures Listing 2.7. The schedule_row_cache() function Listing 2.8. The cache_rows() daemon function

2.5. Web page analytics

Listing 2.9. The updated update_token() function Listing 2.10. The rescale_viewed() daemon function Listing 2.11. The can_cache() function

2.6. Summary

Part 2. Core concepts Chapter 3. Commands in Redis

Additional documentation for commands not covered Redis 2.4 and 2.6 3.1. Strings

Table 3.1. Increment and decrement commands in Redis Listing 3.1. A sample interaction showing INCR and DECR operations in Redis Table 3.2. Substring manipulation commands available to Redis GETRANGE and SUBSTR Listing 3.2. A sample interaction showing substring and bit operations in Redis

3.2. Lists

Table 3.3. Some commonly used LIST commands Listing 3.3. A sample interaction showing LIST push and pop commands in Redis Table 3.4. Some LIST commands for blocking LIST pops and moving items between LISTs Listing 3.4. Blocking LIST pop and movement commands in Redis

3.3. Sets

Table 3.5. Some commonly used SET commands Listing 3.5. A sample interaction showing some common SET commands in Redis Table 3.6. Operations for combining and manipulating SETs in Redis Listing 3.6. A sample interaction showing SET difference, intersection, and union in Redis

3.4. Hashes

Table 3.7. Operations for adding and removing items from HASHes Listing 3.7. A sample interaction showing some common HASH commands in Redis Table 3.8. More bulk operations and STRING-like calls over HASHes Listing 3.8. A sample interaction showing some more advanced features of Redis HASHes

3.5. Sorted sets

Table 3.9. Some common ZSET commands Listing 3.9. A sample interaction showing some common ZSET commands in Redis Table 3.10. Commands for fetching and deleting ranges of data from ZSETs and offering SET-like intersections Listing 3.10. A sample interaction showing ZINTERSTORE and ZUNIONSTORE Figure 3.1. What happens when calling conn.zinterstore('zset-i', ['zset-1', 'zset-2']); elements that exist in both zset-1 and zset-2 are added together to get zset-i Figure 3.2. What happens when calling conn.zunionstore('zset-u', ['zset-1', 'zset-2'], aggregate='min'); elements that exist in either zset-1 or zset-2 are combined with the minimum function to get zset-u Figure 3.3. What happens when calling conn.zunionstore('zset-u2', ['zset-1', 'zset-2', 'set-1']); elements that exist in any of zset-1, zset-2, or set-1 are combined via addition to get zset-u2

3.6. Publish/subscribe

Table 3.11. Commands for handling pub/sub in Redis Listing 3.11. Using PUBLISH and SUBSCRIBE in Redis

3.7. Other commands

3.7.1. Sorting

Table 3.12. The SORT command definition Listing 3.12. A sample interaction showing some uses of SORT

3.7.2. Basic Redis transactions

What is a basic transaction in Redis? Listing 3.13. What can happen without transactions during parallel execution Listing 3.14. What can happen with transactions during parallel execution

3.7.3. Expiring keys

Table 3.13. Commands for handling expiration in Redis Listing 3.15. A sample interaction showing the use of expiration-related commands in Redis

3.8. Summary

Chapter 4. Keeping data safe and ensuring performance

4.1. Persistence options

Listing 4.1. Options for persistence configuration available in Redis 4.1.1. Persisting to disk with snapshots

Development Aggregating logs Listing 4.2. The process_logs() function that keeps progress information in Redis Big data

4.1.2. Append-only file persistence

Table 4.1. Sync options to use with appendfsync File syncing Warning: SSDs and appendfsync always

4.1.3. Rewriting/compacting append-only files

4.2. Replication

Example performance for SUNIONSTORE 4.2.1. Configuring Redis for replication 4.2.2. Redis replication startup process

Table 4.2. What happens when a slave connects to a master During sync, the slave flushes all of its data Warning: Redis doesn’t support master-master replication Table 4.3. When a slave connects to an existing master, sometimes it can reuse an existing dump file.

4.2.3. Master/slave chains

Figure 4.1. An example Redis master/slave replica tree with nine lowest-level slaves and three intermediate replication helper servers

4.2.4. Verifying disk writes

Listing 4.3. The wait_for_sync() function Other information from the INFO command

4.3. Handling system failures

4.3.1. Verifying snapshots and append-only files

Checksums and hashes

4.3.2. Replacing a failed master

Listing 4.4. An example sequence of commands for replacing a failed master node Redis Sentinel

4.4. Redis transactions

Delayed execution with MULTI/EXEC can improve performance 4.4.1. Defining users and their inventory

Figure 4.2. Example user inventory and user information. Frank has 43 e-dollars and an item that he’s considering selling from his inventory. Figure 4.3. Our basic marketplace that includes an ItemA being sold by user 4 for 35 e-dollars

4.4.2. Listing items in the marketplace

What is DISCARD? Listing 4.5. The list_item() function Figure 4.4. list_item(conn, "ItemM", 17, 97)

4.4.3. Purchasing items

Listing 4.6. The purchase_item() function Figure 4.5. Before the item can be purchased, we must watch the market and the buyer’s information to verify that the item is still available, and that the buyer has enough money. Figure 4.6. In order to complete the item purchase, we must actually transfer money from the buyer to the seller, and we must remove the item from the market while adding it to the buyer’s inventory. Why doesn’t Redis implement typical locking?

4.5. Non-transactional pipelines

Listing 4.7. The update_token() function from section 2.5 Listing 4.8. The update_token_pipeline() function Listing 4.9. The benchmark_update_token() function Table 4.4. Performance of pipelined and nonpipelined connections over different types of connections. For high-speed connections, we’ll tend to run at the limit of what a single processor can perform for encoding/decoding commands in Redis. For slower connections, we’ll run at the limit of bandwidth and/or latency.

4.6. Performance considerations

Listing 4.10. Running redis-benchmark on an Intel Core-2 Duo 2.4 GHz desktop Table 4.5. A table of general performance comparisons against a single redis-benchmark client and what may be causing potential slowdowns

4.7. Summary

Chapter 5. Using Redis for application support

5.1. Logging to Redis

Replacing syslog 5.1.1. Recent logs

Listing 5.1. The log_recent() function

5.1.2. Common logs

Listing 5.2. The log_common() function

5.2. Counters and statistics

5.2.1. Storing counters in Redis

Updating a counter Figure 5.1. A HASH that shows the number of web page hits over 5-second time slices around 7:40 a.m. on May 7, 2012 Figure 5.2. A ZSET that shows some known counters Listing 5.3. The update_counter() function Listing 5.4. The get_counter() function Cleaning out old counters Why not use EXPIRE? Listing 5.5. The clean_counters() function

5.2.2. Storing statistics in Redis

Figure 5.3. Example access time stats for the profile page. Remember that ZSETs are sorted by score, which is why our order seems strange compared to our description. Listing 5.6. The update_stats() function Listing 5.7. The get_stats() function

5.2.3. Simplifying our statistics recording and discovery

Listing 5.8. The access_time() context manager Gathering statistics and counters in the real world

5.3. IP-to-city and -country lookup

5.3.1. Loading the location tables

Listing 5.9. The ip_to_score() function Listing 5.10. The import_ips_to_redis() function Listing 5.11. The import_cities_to_redis() function

5.3.2. Looking up cities

Listing 5.12. The find_city_by_ip() function

5.4. Service discovery and configuration

5.4.1. Using Redis to store configuration information

Listing 5.13. The is_under_maintenance() function

5.4.2. One Redis server per application component

Listing 5.14. The set_config() function Listing 5.15. The get_config() function

5.4.3. Automatic Redis connection management

Decorators Listing 5.16. The redis_connection() function/decorator Combining *args and **kwargs Listing 5.17. The decorated log_recent() function Decorators

5.5. Summary

Chapter 6. Application components in Redis

6.1. Autocomplete

6.1.1. Autocomplete for recent contacts

Figure 6.1. A recent contacts autocomplete showing users with names starting with je Listing 6.1. The add_update_contact() function Listing 6.2. The fetch_autocomplete_list() function

6.1.2. Address book autocomplete

Listing 6.3. The find_prefix_range() function Character sets and internationalization Listing 6.4. The autocomplete_on_prefix() function Listing 6.5. The join_guild() and leave_guild() functions

6.2. Distributed locking

6.2.1. Why locks are important

Figure 6.2. The structure of our marketplace from section 4.6. There are four items in the market on the left—ItemA, ItemC, ItemE, and ItemG—with prices 35, 48, 60, and 73, and seller IDs of 4, 7, 2, and 3, respectively. In the middle we have two users, Frank and Bill, and their current funds, with their inventories on the right. Listing 6.6. The list_item() function from section 4.4.2 Listing 6.7. The purchase_item() function from section 4.4.3 Table 6.1. Performance of a heavily loaded marketplace over 60 seconds

6.2.2. Simple locks 6.2.3. Building a lock in Redis

Listing 6.8. The acquire_lock() function Listing 6.9. The purchase_item_with_lock() function Listing 6.10. The release_lock() function Table 6.2. Performance of locking over 60 seconds

6.2.4. Fine-grained locking

Table 6.3. Performance of fine-grained locking over 60 seconds Figure 6.3. Items purchased completed in 60 seconds. This graph has an overall V shape because the system is overloaded, so when we have five listing processes to only one buying process (shown as 5L/1B in the middle samples), the ratio of listed items to bought items is roughly the same ratio, 5 to 1. Figure 6.4. The number of retries when trying to purchase an item in 60 seconds. There are no retries for either types of locks, so we can’t see the line for “with lock” because it’s hidden behind the line for fine-grained locks. Figure 6.5. Average latency for a purchase; times are in milliseconds. The maximum latency for either kind of lock is under 14ms, which is why both locking methods are difficult to see and hugging the bottom—our overloaded system without a lock has an average latency of nearly 500ms.

6.2.5. Locks with timeouts

Listing 6.11. The acquire_lock_with_timeout() function Note

6.3. Counting semaphores

6.3.1. Building a basic counting semaphore

Figure 6.6. Basic semaphore ZSET Listing 6.12. The acquire_semaphore() function Listing 6.13. The release_semaphore() function

6.3.2. Fair semaphores

Figure 6.7. Fair semaphore owner ZSET Listing 6.14. The acquire_fair_semaphore() function Fair semaphores on 32-bit platforms Figure 6.8. Call sequence for acquire_fair_semaphore() Listing 6.15. The release_fair_semaphore() function

6.3.3. Refreshing semaphores

Listing 6.16. The refresh_fair_semaphore() function

6.3.4. Preventing race conditions

Listing 6.17. The acquire_semaphore_with_lock() function

6.4. Task queues

6.4.1. First-in, first-out queues

Figure 6.9. A first-in, first-out queue using a LIST Listing 6.18. The send_sold_email_via_queue() function Listing 6.19. The process_sold_email_queue() function Multiple executable tasks Listing 6.20. The worker_watch_queue() function Task priorities Listing 6.21. The worker_watch_queues() function

6.4.2. Delayed tasks

Listing 6.22. The execute_later() function Figure 6.10. A delayed task queue using a ZSET Listing 6.23. The poll_queue() function Respecting priorities

6.5. Pull messaging

6.5.1. Single-recipient publish/subscribe replacement

Figure 6.11. jack451 has some messages from Jill and his mother waiting for him.

6.5.2. Multiple-recipient publish/subscribe replacement

Figure 6.12. Some example chat and user data. The chat ZSETs show users and the maximum IDs of messages in that chat that they’ve seen. The seen ZSETs list chat IDs per user, again with the maximum message ID in the given chat that they’ve seen. Creating a chat session Listing 6.24. The create_chat() function Generator expressions and dictionary construction Sending messages Listing 6.25. The send_message() function Fetching messages Listing 6.26. The fetch_pending_messages() function Joining and leaving the chat Listing 6.27. The join_chat() function Listing 6.28. The leave_chat() function

6.6. Distributing files with Redis

6.6.1. Aggregating users by location

Aggregating data locally Listing 6.29. A locally aggregating callback for a daily country-level aggregate

6.6.2. Sending files

Listing 6.30. The copy_logs_to_redis() function

6.6.3. Receiving files

Listing 6.31. The process_logs_from_redis() function

6.6.4. Processing files

Listing 6.32. The readlines() function Generators with yield Listing 6.33. The readblocks() generator Listing 6.34. The readblocks_gz() generator

6.7. Summary

Chapter 7. Search-based applications

7.1. Searching in Redis

7.1.1. Basic search theory

Figure 7.1. The inverted index for docA and docB Basic indexing Figure 7.2. The process of tokenizing text into words, then removing stop words, as run on a paragraph from an early version of this section Listing 7.1. Functions to tokenize and index a document Removing a document from the index Basic searching Listing 7.2. SET intersection, union, and difference operation helpers Figure 7.3. The SET intersection, union, and difference calls as if they were operating on Venn diagrams Parsing and executing a search Listing 7.3. A function for parsing a search query Listing 7.4. A function to parse a query and search documents

7.1.2. Sorting search results

Figure 7.4. An example document stored in a HASH Listing 7.5. A function to parse and search, sorting the results

7.2. Sorted indexes

7.2.1. Sorting search results with ZSETs

Listing 7.6. An updated function to search and sort based on votes and updated times Listing 7.7. Some helper functions for performing ZSET intersections and unions

7.2.2. Non-numeric sorting with ZSETs

Listing 7.8. A function to turn a string into a numeric score

7.3. Ad targeting

7.3.1. What’s an ad server?

Ads with budgets

7.3.2. Indexing ads

Calculating the value of an ad Making values consistent Calculating the estimated CPM of a CPC ad Calculating the estimated CPM of a CPA ad Listing 7.9. Helper functions for turning information about CPC and CPA ads into eCPM Inserting an ad into the index Listing 7.10. A method for indexing an ad that’s targeted on location and ad content

7.3.3. Targeting ads

Listing 7.11. Ad targeting by location and page content bonuses Listing 7.12. A helper function for targeting ads based on location Calculating targeting bonuses Being mathematically rigorous Listing 7.13. Calculating the eCPM of ads including content match bonuses Figure 7.5. The data that’s examined during a union-then-intersect calculation of ad-targeting bonuses includes all ads in the relevant word bonus ZSETs, even ads that don’t meet the location matching requirements. Figure 7.6. The data that’s examined during an intersect-then-union calculation of ad-targeting bonuses only includes those ads that previously matched, which significantly cuts down on the amount of data that Redis will process.

7.3.4. Learning from user behavior

Recording views Listing 7.14. A method for recording the result after we’ve targeted an ad Recording clicks and actions Listing 7.15. A method for recording clicks on an ad Updating eCPMs Listing 7.16. A method for updating eCPMs and per-word eCPM bonuses for ads

7.4. Job search

7.4.1. Approaching the problem one job at a time

Listing 7.17. A potential solution for finding jobs when a candidate meets all requirements

7.4.2. Approaching the problem like search

Listing 7.18. A function for indexing jobs based on the required skills Listing 7.19. Find all jobs that a candidate is qualified for

7.5. Summary

Chapter 8. Building a simple social network

8.1. Users and statuses

8.1.1. User information

Figure 8.1. Example user information stored in a HASH Listing 8.1. How to create a new user profile HASH Sensitive user information

8.1.2. Status messages

Figure 8.2. Example status message stored in a HASH Listing 8.2. How to create a status message HASH

8.2. Home timeline

Figure 8.3. When someone visits their home timeline on a site like Twitter, they see the most recently posted messages that people they follow have written. This information is stored as a ZSET of status ID/timestamp pairs. Timestamp information provides the sort order, and the status ID shows us what information to pull in a second step. Listing 8.3. A function to fetch a page of recent status messages from a timeline

8.3. Followers/following lists

Figure 8.4. To know who’s following a user, we store user ID/timestamp pairs in a ZSET. The user IDs are the people who’re following that user, and the timestamp is when they started following the user. Similarly, the users that a user is following are stored as a ZSET of user ID/timestamp pairs of the user ID of the person being followed, and the timestamp of when the user followed them. Listing 8.4. Update the following user’s home timeline Converting a list of tuples into a dictionary Listing 8.5. A function to stop following a user

8.4. Posting or deleting a status update

Listing 8.6. Update a user’s profile timeline Listing 8.7. Update a user’s followers’ home timelines Listing 8.8. A function to delete a previously posted status message

8.5. Streaming API

8.5.1. Data to be streamed 8.5.2. Serving the data

A streaming HTTP server Listing 8.9. Server and request handler for our streaming HTTP server Listing 8.10. The code to actually start and run the streaming HTTP server Identifying the client Listing 8.11. An example function to parse and store the client identifier Handling HTTP streaming Listing 8.12. A function that will verify the request and stream data to the client

8.5.3. Filtering streamed messages

Updating status message posting and deletion Listing 8.13. Updated create_status() from listing 8.2 to support streaming filters Listing 8.14. Updated delete_status() from listing 8.8 to support streaming filters Receiving streamed messages for filtering Listing 8.15. A function to receive and process streamed messages Filtering messages Listing 8.16. A factory function to dispatch to the actual filter creation Listing 8.17. The function to handle firehose, gardenhose, and spritzer Listing 8.18. A filter that matches groups of words that are posted in status messages Listing 8.19. Messages posted by or mentioning any one of a list of users Listing 8.20. Messages within boxes defined by ranges of latitudes and longitudes

8.6. Summary

Part 3. Next steps Chapter 9. Reducing memory use

9.1. Short structures

9.1.1. The ziplist representation

Figure 9.1. How long LISTs are stored in Redis Using the ziplist encoding Listing 9.1. Configuration options for the ziplist representation of different structures Listing 9.2. How to determine whether a structure is stored as a ziplist

9.1.2. The intset encoding for SETs

Listing 9.3. Configuring the maximum size of the intset encoding for SETs Listing 9.4. When an intset grows to be too large, it’s represented as a hash table.

9.1.3. Performance issues for long ziplists and intsets

Listing 9.5. Our code to benchmark varying sizes of ziplist-encoded LISTs Listing 9.6. As ziplist-encoded LISTs grow, we can see performance drop Keeping key names short

9.2. Sharded structures

Sharding LISTs Sharding ZSETs 9.2.1. HASHes

Listing 9.7. A function to calculate a shard key from a base key and a secondary entry key Being consistent about total_elements and shard_size Listing 9.8. Sharded HSET and HGET functions Listing 9.9. Sharded IP lookup functions Storing STRINGs in HASHes

9.2.2. SETs

Listing 9.10. A sharded SADD function we’ll use as part of a unique visitor counter Listing 9.11. A function to keep track of the unique visitor count on a daily basis Listing 9.12. Calculate today’s expected unique visitor count based on yesterday’s count Other methods to calculate unique visitor counts

9.3. Packing bits and bytes

9.3.1. What location information should we store?

Listing 9.13. Base location tables we can expand as necessary Listing 9.14. ISO3 country codes

9.3.2. Storing packed data

Listing 9.15. A function for storing location data in sharded STRINGs

9.3.3. Calculating aggregates over sharded STRINGs

Listing 9.16. A function to aggregate location information for everyone Listing 9.17. Convert location codes back to country/state information Listing 9.18. A function to aggregate location information over provided user IDs

9.4. Summary

Chapter 10. Scaling Redis

10.1. Scaling reads

Remember: Write to the master Figure 10.1. An example Redis master/slave replica tree with nine lowest-level slaves and three intermediate replication helper servers Encryption and compression overhead Using compression with OpenVPN

10.2. Scaling writes and memory capacity

Scaling write volume Presharding for growth 10.2.1. Handling shard configuration

Listing 10.1. A function to get a Redis connection based on a named configuration Listing 10.2. Fetch a connection based on shard information

10.2.2. Creating a server-sharded connection decorator

Listing 10.3. A shard-aware connection decorator Listing 10.4. A machine and key-sharded count_visit() function Multiple Redis servers on a single machine Alternate methods of handling unique visit counts over time

10.3. Scaling complex queries

10.3.1. Scaling search query volume 10.3.2. Scaling search index size

Sharding SORT-based search Listing 10.5. SORT-based search that fetches the values that were sorted Listing 10.6. A function to perform queries against all shards Listing 10.7. A function to merge sharded search results Sharding ZSET-based search Listing 10.8. ZSET-based search that returns scores for each result Listing 10.9. Sharded search query over ZSETs that returns paginated results

10.3.3. Scaling a social network

Scaling posted message database size Sharding timelines Listing 10.10. An example of how we want our API for accessing shards to work Listing 10.11. A class that implements sharded connection resolution based on key Scaling follower and following lists with sharding Listing 10.12. Access follower/following ZSET shards Listing 10.13. Sharded connection resolution based on ID pairs Listing 10.14. A function that implements a sharded ZRANGEBYSCORE Using this method for sharding profile timelines Listing 10.15. Updated syndicate status function

10.4. Summary

Chapter 11. Scripting Redis with Lua

11.1. Adding functionality without writing C

11.1.1. Loading Lua scripts into Redis

Listing 11.1. A function that loads scripts to be called later Keys and arguments to Lua scripts Returning non-string and non-integer values from Lua Table 11.1. Values returned from Lua and what they’re translated into

11.1.2. Creating a new status message

Lua scripts—as atomic as single commands or MULTI/EXEC Lua scripts—can’t be interrupted if they have modified structures Listing 11.2. Our function from listing 8.2 that creates a status message HASH Listing 11.3. Creating a status message with Lua Writing keys that aren’t a part of the KEYS argument to the script Script loaders and helpers

11.2. Rewriting locks and semaphores with Lua

11.2.1. Why locks in Lua? 11.2.2. Rewriting our lock

Listing 11.4. Our final acquire_lock_with_timeout() function from section 6.2.5 Listing 11.5. A rewritten acquire_lock_with_timeout() that uses Lua Listing 11.6. A rewritten release_lock() that uses Lua Table 11.2. Performance of our original lock against a Lua-based lock over 10 seconds

11.2.3. Counting semaphores in Lua

Listing 11.7. The acquire_semaphore() function from section 6.3.2 Listing 11.8. The acquire_semaphore() function rewritten with Lua Listing 11.9. A refresh_semaphore() function written with Lua

11.3. Doing away with WATCH/MULTI/EXEC

11.3.1. Revisiting group autocomplete

Listing 11.10. Our autocomplete code from section 6.1.2 Listing 11.11. Autocomplete on prefix using Redis scripting Table 11.3. Performance of our original autocomplete versus our Lua-based autocomplete over 10 seconds

11.3.2. Improving the marketplace, again

Listing 11.12. The purchase item with lock function from section 6.2 Listing 11.13. The purchase item function rewritten with Lua Table 11.4. Performance of Lua compared with no locking, coarse-grained locks, and fine-grained locks over 60 seconds

11.4. Sharding LISTs with Lua

11.4.1. Structuring a sharded LIST

Figure 11.1. First and last shard IDs for sharded LISTs Figure 11.2. LIST shards with data

11.4.2. Pushing items onto the sharded LIST

Listing 11.14. Functions for pushing items onto a sharded LIST Limitations to this sharded LIST

11.4.3. Popping items from the sharded LIST

Listing 11.15. The Lua script for pushing items onto a sharded LIST

11.4.4. Performing blocking pops from the sharded LIST

Why not use a MULTI/EXEC transaction? Listing 11.16. Our code to perform a blocking pop from a sharded LIST

11.5. Summary

Appendix A. Quick and dirty setup

A.1. Installation on Debian or Ubuntu Linux

Listing A.1. Installing build tools on Debian Linux Listing A.2. Installing Redis on Linux Listing A.3. Installing the Redis client libraries for Python on Linux

A.2. Installing on OS X

Listing A.4. Installing Redis on OS X Listing A.5. Installing the Redis client library for Python on OS X

A.3. Installing on Windows

A.3.1. Drawbacks of Redis on Windows

Compiling Redis in Windows yourself

A.3.2. Installing Redis on Windows

Figure A.1. Redis running in Windows

A.3.3. Installing Python on Windows

Listing A.6. Installing the Redis client library for Python on Windows

A.4. Hello Redis

Listing A.7. Testing Redis from Python Running Python in other ways Redis on OS X and Windows Configuring Redis Is hiredis available on non-Linux platforms?

Appendix B. Other resources and references

B.1. Forums for help B.2. Introductory topics B.3. Queues and other libraries B.4. Data visualization and recording B.5. Data sources B.6. Redis experiences and articles

Index

A B C D E F G H I J K L M N O P Q R S T U V W X

List of Figures List of Tables List of Listings

← Prev
Back
Next →

← Prev
Back
Next →