Sidekiq threads are assumed to be connected to a single redis. So when connecting the application to multiple redis, the number of sidekiqs needed corresponds to the number of redis. This article guides to install multiple sidekiq on the same rails application in the most concise way
config/initializers/sidekiq.rb
1 2 3 4 5 6 7 8 | Sidekiq.configure_server do |config| config.redis = { url: "redis://#{ENV.fetch('REDIS_SERVERNAME')}:6379/0", namespace: ENV['REDIS_NAMESPACE'] } end Sidekiq.configure_client do |config| config.redis = { url: "redis://#{ENV.fetch('REDIS_SERVERNAME_1')}:6379/0", namespace: ENV['REDIS_NAMESPACE_1'] } end |
First you need to understand what is Sidekiq server and sidekiq client?
- client: is where jobs are submitted to and queued (also known as Rails applications)
- server: is where jobs are retrieved (dequeue) for processing (Sidekiq process)
Sidekiq config always includes Sidekiq server and Sidekiq client. Usually the sidekiq_servers are located on different instances, so it is necessary to set the correct value of REDIS_SERVERNAME at these instances for example:
- sidekiq_1:
1 2 3 | export REDIS_SERVERNAME=redis_1_server_name export REDIS_NAMESPACE=redis_1_name_space |
- sidekiq_2:
1 2 3 | export REDIS_SERVERNAME=redis_2_server_name export REDIS_NAMESPACE=redis_2_name_space |
- rails:
1 2 3 4 5 | export REDIS_SERVERNAME_1=redis_1_server_name export REDIS_SERVERNAME_2=redis_2_server_name export REDIS_NAMESPACE_1=redis_1_name_space export REDIS_NAMESPACE_2=redis_2_name_space |
config/sidekiq.yml
This is where the list of queues used in sidekiq is defined. Each sidekiq process needs to define a separate sidekiq.yml file. eg: sidekiq.yml and sidekiq_1.yml
1 2 3 4 5 6 7 8 9 10 11 12 13 14 | :verbose: false :concurrency: 10 :timeout: 10 :queues: - sidekiq_queue_1 - sidekiq_queue_2 - sidekiq_queue_3 development: :verbose: true staging: :concurrency: 5 production: :concurrency: 5 |
Definition of worker for sidekiq_1
The rails application will call this worker’s method to send jobs to sidekiq.
1 2 3 4 5 6 7 8 | class Worker1 include Sidekiq::Worker sidekiq_options queue: :sidekiq_1_queue_1, retry: 5 def perform(*args) # các xử lý của sidekiq process được định nghĩa tại đây end |
Definition of worker for sidekiq_2
At this point, some of you may wonder why in step 1 we defined 2 variables in the rails instance but sidekiq_client only used 1 value. The reason is that we can choose a sidekiq as the default for rails applications. Usually we will choose which sidekiq has more queue types, and in sidekiq_options we just need to name the queue. For sidekiq_2, we need to define which sidekiq this worker uses via the ConnectionPool class.
1 2 3 4 5 6 7 8 9 10 11 | class Worker2 include Sidekiq::Worker sidekiq_options pool: ConnectionPool.new { redis_2 = Redis.new(url: "redis://#{ENV.fetch('REDIS_SERVERNAME_2')}:6379/0") Redis::Namespace.new(ENV['REDIS_NAMESPACE_2'], redis: redis_2) },queue: :sidekiq_2_queue_1, retry: 5 def perform(*args) # các xử lý của sidekiq process được định nghĩa tại đây end |
Conclusion
With that, we can configure our rails application to use multiple sidekiq processes! Thank you to everyone who visited this post! Looking forward to hearing from everyone.