All of our failure mitigation so far has one limiting factor: the time needed to respond to a web request. As the application currently stands, we’re making our Stripe API call inside our user’s HTTP request, which is slow. If the Stripe API gave us a temporary error, our best response would be to retry the request in a few moments, but we can’t necessarily do that in the space of a typical web request.
Moving our payment processing to a background job, however, allows us to get some response back to the user more quickly, as well as gives us more leeway for robust error handling. Using a background job also makes it easier to integrate other tasks, such as sending a receipt back to the user or integrating with reporting tools.
We’re going to use Rails ActiveJob, which was added in Rails 4.2, to handle the logistics of background processing. (A blog post by David Copeland[37] was helpful in putting together this section.) ActiveJob is a Rails front end that can be used via an adapter to feed into several background job tools, including Delayed Job, Resque, Sidekiq, and Sucker Punch. When we write the code to conform to ActiveJob’s interface, the background job gem provides an adapter. If we decide to move to a more powerful tool, we can with minimal code change because both tools should work with the ActiveJob front end.
We’re going to use Delayed Job[38] for this because it’s easy to set up. Let’s add it to the Gemfile:
| | gem "delayed_job_active_record" |
And then run two commands from the command line:
| | % rails generate delayed_job:active_record |
| | % rake db:migrate |
The generator creates a database migration and a file bin/delayed_job that we can use to run background jobs. Delayed Job has a very simple execution model. It stores background jobs as basically serialized Ruby bindings inside the same database we use for the rest of our application. When we run the worker, it looks at the database, picks up jobs, and executes them. It may not be the best choice for a very large application because it depends on the same database as the application, which means if the database is being overwhelmed, so are the background jobs.
We tell Rails we’re using Delayed Job as our ActiveJob back end by adding a line to the application.rb file:
| | config.active_job.queue_adapter = :delayed_job |
And we’re ready to get started.
Let’s create our first job:
| | rails generate job purchases_cart |
This creates two files: app/jobs/purchases_cart_job.rb, which is our job, and spec/jobs/purchases_cart_job_spec.rb, which is our related test file.
You might be asking why we need a new purchases_cart_job when we already have a perfectly good isolated PurchasesCart workflow object. This is an exceptionally good question, and the answer, for me, has to do with how ActiveJob works.
An ActiveJob class has the following characteristics:
There are two limitations to ActiveJob that make just moving our existing code inside ActiveJob seem less than ideal to me.
First, when you invoke ActiveJob, the perform method is just executed. You don’t get a fresh instance of the ActiveJob class, and therefore, creating a complex set of states with instance variables seems out of the question.
Second, while the perform method can take an arbitrary number of parameters, those parameters all need to be serializable by ActiveJob, which means they need to be instances of BigNum, FalseClass, Fixnum, Float, NilClass, String, or TrueClass. Or they can be an instance of a class that includes the Rails GlobalId mixin, which ActiveRecord does, and which can be included by any class that implements def self.find(id). You can also include an array or hash made up of the same set (hash keys need to be a string or a symbol).
The lack of instance variables makes just copying our code over problematic. And because our StripeToken class does not implement the GlobalId mixin (and would be unable to do so without calling back to the Stripe API for its find method), just creating our PurchasesCart workflow in the controller and passing it to the job doesn’t work either. Instead, we need to pass the request parameters to the ActiveJob and do all the creation of our business logic objects there.
On the plus side, this simplifies our controller (there are some associated view changes that you can see in the code repo):
| | class PaymentsController < ApplicationController |
| | |
| | def show |
| | @reference = params[:id] |
| | @payment = Payment.find_by(reference: @reference) |
| | end |
| | |
| | def create |
| | workflow = run_workflow(params[:payment_type]) |
| | if workflow.success |
| | redirect_to workflow.redirect_on_success_url || |
| | payment_path(id: @reference || workflow.payment.reference) |
| | else |
| | redirect_to shopping_cart_path |
| | end |
| | end |
| | |
| | private def run_workflow(payment_type) |
| | case payment_type |
| | when "paypal" then paypal_workflow |
| | else |
| | stripe_workflow |
| | end |
| | end |
| | |
| | private def paypal_workflow |
| | workflow = PurchasesCartViaPayPal.new( |
| | user: current_user, |
| | purchase_amount_cents: params[:purchase_amount_cents], |
| | expected_ticket_ids: params[:ticket_ids]) |
| | workflow.run |
| | workflow |
| | end |
| | |
| | private def stripe_workflow |
| | @reference = Payment.generate_reference |
| | PurchasesCartJob.perform_later( |
| | user: current_user, |
| | params: card_params, |
| | purchase_amount_cents: params[:purchase_amount_cents], |
| | expected_ticket_ids: params[:ticket_ids], |
| | payment_reference: @reference) |
| | end |
| | |
| | private def card_params |
| | params.permit( |
| | :credit_card_number, :expiration_month, |
| | :expiration_year, :cvc, |
| | :stripe_token).to_h.symbolize_keys |
| | end |
| | |
| | end |
Basically, we just pass all the incoming parameters, along with the current user, over to the job method, which we invoke using perform_later. This causes ActiveJob to enqueue the job into Delayed Job (or whatever our back end happens to be). Our parameters—a User, a parameter hash, and the payment reference—are all serializable by ActiveJob, so we’re good.
We have two other related changes in this version of the controller. We generate the Payment reference before the job and pass it to the job (before, we generated it as part of the workflow when the Payment instance was created), and the PurchasesCartJob, as we will see in a moment, hardwires that it is successful, so it will always redirect based on the new payment reference. We don’t change our response based on the success or failure of the job because we can’t. The job is headed off into the background queue, so we won’t know whether or not it succeeds by the time we need to respond to the request.
We could just send the user to a generic URL that says something like “Thanks for ordering, you’ll be getting a receipt shortly,” but because we already have the capability to generate a unique ID before the purchase is actually created, we can do something better. We generate the reference before the job starts, passing the reference through to eventually be saved with the purchase. This gives the customer a permanent URL that will eventually have his or her order information. And it also gives us the opportunity to do something fancier on the client side should we choose, such as poll or open up a web socket so that the order information can be displayed as soon as it hits the database.
The PurchasesCartJob itself is just the logic to create the PurchaseJob workflow. Almost all of this is taken directly from the original PurchaseController:
| | class PurchasesCartJob < ApplicationJob |
| | |
| | queue_as :default |
| | |
| | def perform(user:, purchase_amount_cents:, expected_ticket_ids:, |
| | payment_reference:, params:) |
| | token = StripeToken.new(**card_params(params)) |
| | user.tickets_in_cart.each do |ticket| |
| | ticket.update(payment_reference: payment_reference) |
| | end |
| | purchases_cart_workflow = StripePurchasesCart.new( |
| | user: user, stripe_token: token, |
| | purchase_amount_cents: purchase_amount_cents, |
| | expected_ticket_ids: expected_ticket_ids, |
| | payment_reference: payment_reference) |
| | purchases_cart_workflow.run |
| | end |
| | |
| | def success |
| | true |
| | end |
| | |
| | def redirect_on_success_url |
| | nil |
| | end |
| | |
| | private def card_params(params) |
| | params.slice( |
| | :credit_card_number, :expiration_month, |
| | :expiration_year, :cvc, |
| | :stripe_token).symbolize_keys |
| | end |
| | |
| | end |
The only substantive changes involve the purchase reference. We’re taking the purchase reference and passing it along to the workflow, as you might expect. We’re also attaching the purchase reference to the tickets still in the shopping cart, which is new (there’s an associated database migration to add the payment_reference field to the ticket).
We need to add the payment reference to the ticket so that we can track the ticket through the payment process, and doing so is particularly important now that we are processing the transaction in the background. We no longer really know how much time has passed between the customer submitting the form and the card being processed. In particular, the customer may have legitimately added more items to a new shopping cart in the interim. Under the previous code, doing so would prevent the first purchase from happening because the pre-flight check would see a discrepancy between the total cost presented to the user on the form, and the total cost of the combined purchases that the workflow would find when querying the database. By tagging inventory that is already in the process of being charged, we can prevent this kind of double charging.
Our PurchasesCart workflow has to change a little bit, owing to the nature of how background jobs work. If our workflow, now running in the background, throws an exception and does not handle that exception, then the job is sent back to the queue. Typically that means the job will be reexecuted, although the specific behavior is dependent on the back end. By default, Delayed Job will reexecute failures 25 times, with an exponentially increasing wait time in between.
That’s great for us because many of the exceptions that we don’t handle are going to be due to things like API errors on the Stripe side that might be temporary and would clear if the job is run later. However, our PurchasesCart code has been operating under the implicit assumption that it’s never been executed before, and as currently written, rerunning the job with the same incoming data will cause some problems. Specifically, the workflow will create a duplicate Payment record pointing back to the same tickets, and may also try to charge the Stripe API a second time. Stripe’s idempotency protection probably prevents it from actually charging, but it seems worth trying to avoid a second call on our end, to be safe.
We need to look at all of our interactions with our database and third-party libraries to make sure they still behave reasonably if they have already been executed before.
In our case, we need to
Check for validity between the incoming form ticket purchase and the calculated price. If the purchase has already been created, we probably don’t need to recheck this. But we need to protect against the possibility that the user has added later tickets to the cart since the job started.
Purchase the tickets before the charge by changing their status. Given our data model, running that again is fine; we’d just be setting the status to a value it already has. Other data models might be less forgiving here.
We don’t want to create another purchase or purchase line items if a purchase has already been created, and we don’t want to send the charge information to Stripe if we’ve already done so.
Looking at this from the other direction, we might want to create an exception in certain cases to force a new attempt in the future. Specifically, a database failure on save has a chance of being due to a temporary condition, so triggering an exception here is still useful.
With that, the code doesn’t change a whole lot. Our initializer changes slightly to allow the purchase_reference to be passed in:
| | class PurchasesCart |
| | |
| | attr_accessor :user, :purchase_amount_cents, |
| | :purchase_amount, :success, |
| | :payment, :expected_ticket_ids, |
| | :payment_reference |
| | |
| | def initialize(user: nil, purchase_amount_cents: nil, |
| | expected_ticket_ids: "", payment_reference: nil) |
| | @user = user |
| | @purchase_amount = Money.new(purchase_amount_cents) |
| | @success = false |
| | @continue = true |
| | @expected_ticket_ids = expected_ticket_ids.split(" ").map(&:to_i).sort |
| | @payment_reference = payment_reference || Payment.generate_reference |
| | end |
| | |
| | def run |
| | Payment.transaction do |
| | pre_purchase |
| | purchase |
| | post_purchase |
| | @success = @continue |
| | end |
| | end |
We can then use that reference to see if there’s an existing purchase. If there is, we don’t need to do any pre-purchase, so we just kick out of the method with a guard clause. Also, we now use the purchase reference to filter the list of tickets in the cart to prevent against a second purchase being in flight from the same user. We also no longer catch errors in the run method, since we now want the error to not be caught so that the job can be rerun (with a slight change to the database failure spec).
| | def tickets |
| | @tickets ||= @user.tickets_in_cart.select do |ticket| |
| | ticket.payment_reference == payment_reference |
| | end |
| | end |
| | |
| | def existing_payment |
| | Payment.find_by(reference: payment_reference) |
| | end |
| | |
| | def pre_purchase |
| | return true if existing_payment |
| | unless pre_purchase_valid? |
| | @continue = false |
| | return |
| | end |
| | update_tickets |
| | create_payment |
| | @continue = true |
| | end |
Completely skipping out the pre-charge is only one choice. In some cases, for example, it might make sense to redo the validation step even if the purchase has already changed if there’s reason to believe that an outside factor might have affected validity.
If we decide to pass through the method even if there is an existing purchase, we need to change the create_payment method to be aware of the existing purchase:
| | def create_payment |
| | self.payment = existing_payment || Payment.new |
| | payment.update!(payment_attributes) |
| | payment.create_line_items(tickets) |
| | end |
All we’re doing here is trying to reset the existing purchase to the state it would be in if it had been newly created, most likely changing the status. This might be a little simplistic as our business logic gets more involved. As we currently have this structured, a failed save will raise an exception that will cause the job to be rerun.
Finally, we don’t need to charge if we’ve already got a successful response, as evidenced by the response_id being set in the purchase—this change goes in the Stripe workflow subclass:
| | def purchase |
| | return unless @continue |
| | return if payment.response_id.present? |
| | @stripe_charge = StripeCharge.new(token: stripe_token, payment: payment) |
| | @stripe_charge.charge |
| | payment.update!(@stripe_charge.payment_attributes) |
| | reverse_purchase if payment.failed? |
| | end |
Now if we run our credit card form in development, we can see that a new row is created in the delayed_jobs table, and when we run the delayed_job task, it gets evaluated, making the API call and charging the user.
We’re doing a lot of checking of existing state in the workflow as it now stands, and it’s starting to feel a little fragile to me. If you are following along with the code and the tests in the sample code, you can see that the Stripe tests take a lot of setup, and there’s a clear argument that they are too dependent on the internals of Payment. There are a few ways to mitigate that problem.
One way to reduce the amount of internal state is to split the Stripe background job into separate “prepare” and “purchase” jobs, where the purchase job is only created if the prepare job passes. The split already somewhat exists in our code, in that the Stripe job already has pre-purchase and purchase phases. At the end of this, we’ll wind up with an abstract payment prepare with concrete versions for Stripe and PayPal, and Stripe and PayPal will each have their own separate purchase job.
Creating multiple jobs clears a lot of the error checking and status checking out of the code (particularly in the later jobs), because we can now assert that previous jobs have been successful; otherwise the current job wouldn’t have been created at all. So, many of the @continue markers we’ve been using in the code will go away.
So, we have the PurchasesController call a PreparesCartForStripeJob, which in turn invokes a PreparesCartForStripe workflow. The job is unchanged except for the name of the workflow being invoked. The new workflow has the logic that we were calling pre_purchase, but with a few small changes:
| | class PreparesCart |
| | |
| | attr_accessor :user, :purchase_amount_cents, |
| | :purchase_amount, :success, |
| | :payment, :expected_ticket_ids, |
| | :payment_reference |
| | |
| | def initialize(user: nil, purchase_amount_cents: nil, |
| | expected_ticket_ids: "", payment_reference: nil) |
| | @user = user |
| | @purchase_amount = Money.new(purchase_amount_cents) |
| | @success = false |
| | @continue = true |
| | @expected_ticket_ids = expected_ticket_ids.split(" ").map(&:to_i).sort |
| | @payment_reference = payment_reference || Payment.generate_reference |
| | end |
| | |
| | def pre_purchase_valid? |
| | purchase_amount == tickets.map(&:price).sum && |
| | expected_ticket_ids == tickets.map(&:id).sort |
| | end |
| | |
| | def tickets |
| | @tickets ||= @user.tickets_in_cart.select do |ticket| |
| | ticket.payment_reference == payment_reference |
| | end |
| | end |
| | |
| | def existing_payment |
| | Payment.find_by(reference: payment_reference) |
| | end |
| | |
| | def run |
| | Payment.transaction do |
| | return if existing_payment |
| | return unless pre_purchase_valid? |
| | update_tickets |
| | create_payment |
| | success? ? on_success : on_failure |
| | end |
| | end |
| | |
| | def redirect_on_success_url |
| | nil |
| | end |
| | |
| | def create_payment |
| | self.payment = existing_payment || Payment.new |
| | payment.update!(payment_attributes) |
| | payment.create_line_items(tickets) |
| | @success = payment.valid? |
| | end |
| | |
| | def payment_attributes |
| | {user_id: user.id, price_cents: purchase_amount.cents, |
| | status: "created", reference: Payment.generate_reference} |
| | end |
| | |
| | def success? |
| | success |
| | end |
| | |
| | def unpurchase_tickets |
| | tickets.each(&:waiting!) |
| | end |
| | |
| | end |
And we have some changes to the Stripe-specific part:
| | class PreparesCartForStripe < PreparesCart |
| | |
| | attr_accessor :stripe_token, :stripe_charge |
| | |
| | def initialize(user:, stripe_token:, purchase_amount_cents:, |
| | expected_ticket_ids:, payment_reference: nil) |
| | super(user: user, purchase_amount_cents: purchase_amount_cents, |
| | expected_ticket_ids: expected_ticket_ids, |
| | payment_reference: payment_reference) |
| | @stripe_token = stripe_token |
| | end |
| | |
| | def update_tickets |
| | tickets.each(&:purchased!) |
| | end |
| | |
| | def on_success |
| | ExecutesStripePurchaseJob.perform_later(payment, stripe_token.id) |
| | end |
| | |
| | def unpurchase_tickets |
| | tickets.each(&:waiting!) |
| | end |
| | |
| | def purchase_attributes |
| | super.merge(payment_method: "stripe") |
| | end |
| | |
| | end |
The initializer is the same. We still need all the same inputs to create the purchase. The main difference is that as soon as this workflow creates the purchase, it’s done.
Our run method now covers what was done in the original pre_purchase method. We have one genuinely exceptional condition: the purchase object might already exist. My idea here, which may be oversimplified, is that if the purchase already exists, then this job is being called incorrectly and it should just end. You probably want to include a notification to the administrators, or log the failure to a file of failed purchase attempts or send the failure to a notification aggregator like Rollbar. Similarly, we end the workflow if our validity check of the expected charge and items fails. (And similarly, we probably want to tell the user in that case.)
We then create the payment instance and save it, and throw an exception if the save fails. If it succeeds, we wind up in the on_success method, defined concretely in the PreparesCartForStripe class, and we add the actual charge job to the queue. Right now, failure via a database exception causes the setup job to be automatically re-queued; we might also want some other logic to happen at some point.
The change in the run method of the PreparesCart class requires a slight change in the PayPal version as well, changing the name of the purchase method to on_success to make sure it’s called.
The new ActiveJob is pretty simple:
| | class ExecutesStripePurchaseJob < ActiveJob::Base |
| | |
| | queue_as :default |
| | |
| | def perform(payment, stripe_token) |
| | charge_action = ExecutesStripePurchase.new(payment, stripe_token) |
| | charge_action.run |
| | end |
| | |
| | end |
And the workflow takes the purchase and performs a charge:
| | class ExecutesStripePurchase |
| | |
| | attr_accessor :payment, :stripe_token, :stripe_charge |
| | |
| | def initialize(payment, stripe_token) |
| | @payment = payment |
| | @stripe_token = StripeToken.new(stripe_token: stripe_token) |
| | end |
| | |
| | def run |
| | Payment.transaction do |
| | result = charge |
| | on_failure unless result |
| | end |
| | end |
| | |
| | def charge |
| | return :present if payment.response_id.present? |
| | @stripe_charge = StripeCharge.new(token: stripe_token, payment: payment) |
| | @stripe_charge.charge |
| | payment.update!(@stripe_charge.payment_attributes) |
| | payment.succeeded? |
| | end |
| | |
| | def unpurchase_tickets |
| | payment.tickets.each(&:waiting!) |
| | end |
| | |
| | def on_failure |
| | unpurchase_tickets |
| | end |
| | |
| | end |
Our exceptional condition is if the payment.response_id already exists, meaning the charge has already been made. Again, we choose to just drop out of the charge method in that case, and again, there might be additional notification we want to do. We return a token value to prevent the on_failure clause from being invoked if the payment already exists—we’ll see a better way of handling failures in a moment.
Otherwise, we try the charge, save the new data if the charge succeeds, and reverse the purchase if it fails, just as before. We might now want to notify the users on success, which we can do with a basic Rails mailer, and we’d expect to have a system in place to notify administrators if something has gone wrong, which we will cover in the next section.