From Accessibility to Zope

The PCI DSS is clear on how to handle CV2 (also known many other acronyms: CVV, CVV2, CVN, CCV, CVC). You may not store this number “subsequent to authorization”, not even encrypted. This means the number is highly sensitive. It is treated as the one true anti-fraud measure for Cardholder Not Present transactions. Of course, this is slightly odd - it’s written right there on the card for any old waitress to see - but the Payment Card Industry makes the rules and this is the rule they’ve set.

The user friendly model for e-Commerce payment is this:

Confirm these purchases -> Enter credit card details -> Please confirm that all of the above is correct -> Thanks, here’s your order confirmation.

The final confirmation is vital for making sure customers are informed about exactly what they’re agreeing to. Customers, en masse, are stupid, and they often get to this stage before realising they’ve messed up something. The difficulty then is that the request where the web application received the credit card details is different to the one where the customer authorises it to go ahead. This means that the credit card details must persist for at least one request. In the unhappy world of statelessness that is HTTP, that translates as ‘indefinitely’.

To investigate how other software tackles this I’ve come across a startling lack of awareness in open-source shops.

First I looked at Satchmo. Satchmo stores CV2 unencrypted in the database. I couldn’t see any code that deletes it after authorisation. It also stores the card number (PAN) encrypted symmetrically with a key from the settings file. This is incredibly naïve! A compromised server would compromise the card information for the entire order history.

Then I looked at osCommerce. Unbelieveably, osCommerce appears to return the CV2 to the customer… in the address bar! Where it will be stored in the browser’s unencrypted history for maybe 90 days.

These are the technical options as I’ve thought of so far:

1. Send a PreAuth request to the gateway when you have the details and a PostAuth when they confirm. I had thought this was the unequivocally correct way - now I’m not sure. A PreAuth request isn’t an uncommitted payment authorization: funds are reserved on the card when the PreAuth is requested. Moreover, if there is an error on the confirm page that affects the amount to be billed, you need to re-request the card details. In essence, a PreAuth is a more binding transaction than the customer feels they have entered into at this stage.
2. Encrypt the card details with symmetric encryption and send them back to the browser in a hidden field. This is quite elegant, in that it remove the specific problem the PCI DSS is trying to tackle: that a compromised server potentially compromises all credit card details held. It’s still encrypted storage, though in this form it may well fall under encrypted transfer. It’s permissable to transfer CV2s as necessary, provided there’s strong encryption.
3. Encrypt the card details with symmetric encryption as above but keep the encrypted blob and give the user the key. The cryptographic strength of this protocol is only as good as the previous if the keys are cryptographically random, but it is what the PCI DSS mandates you don’t do, assuming the strictest intepretation of the standard.
4. Only request the CV2 on the confirm page. This could seem quite natural, if expressed as “Please enter your CV2 number for this card to confirm the transaction”. It doesn’t really help secure the other card details.
5. Don’t show a confirm page, or at least combine the confirm page with the form to submit credit card details, and add a big red button saying “Place this order”.

Given these the only option that I am personally happy with is 2 so that is what I intend to implement. I don’t like the PCI DSS, incidentally. I don’t like global financial companies mandating what the little people must do to protect their profits. I don’t like the way it’s written - describing what you mustn’t do without offering it’s view of approved methodologies. I think it’s paranoid about network security, it overstates the benefit of firewalls, and it’s not comprehensive enough about application security. But I like security and I’m tolerant of this PCI DSS for that reason.

I’d very much like to hear if anyone has a better solution or a different opinion.

Since I’ve been using the Django framework for web development I’ve been pointed in the direction of the Satchmo framework a few times. Satchmo is intended to be the framework for e-Commerce development built on top of Django. I’ve looked at Satchmo more than once trying to weigh up whether it would be a useful thing to use for projects, because an e-Commerce framework certainly should be useful. Satchmo doesn’t satisfy this expectation.

I have written an e-Commerce shop in Django. It took three days, much of which was spent writing the backend - basket, order processing and so on. I am absolutely certain that I couldn’t have done that with Satchmo. The first reason is that Satchmo isn’t orthogonal; it’s impossible to use Satchmo’s basket/ordering system without using Satchmo’s products front end. Front ends are very easy to write in Django and I needed something custom for this shop. Custom front ends are very hard to write in Satchmo - they involve writing custom extensions to the product model. The second reason is that Satchmo doesn’t use Enterprise Design Patterns like Money.

Satchmo also provides code that is very much outside its remit as a framework, like this strange configuration property which embeds Google Analytics. Google Analytics code is trivial to paste into the templates should I want to do so. Why would I need this built into the platform?

Satchmo doesn’t satisfy as a framework; it’s more of an off-the-shelf application regardless of what the authors claim. Now I have Django web shop code, I won’t be needing Satchmo.

I have a design pattern I use when designing an e-Commerce system. I call it the polymorphic basket and as the name suggests, it is a design pattern covering the basket. However the basket is just a special case of an order (one that is stored, typically, in a session rather than in the database), and this pattern also covers orders stored in the database.

The problem the pattern seeks to address is maintaining and pricing a list of items. The naïve solution is to record a reference to the SKU, and a number representing the quantity. This solution does not generalise well. In many shops, there is a mixture of products conforming to different conceptual models. While some products can be fully represented by an SKU code, some need bespoke customisation. It is generally an easy task to create mixed catalogues and customisation pages for these products. Data storage for these products is perhaps simplest using a Concrete Table Inheritance pattern. Even if a given merchant is only selling within one model, they may one day want to supplement their product line with perhaps just a few products sold under a different model.

The pattern is to maintain a OrderItemList of polymorphic objects conforming to a OrderItem interface. On adding an item from the catalogue, the details are copied into an OrderItem of an appropriate type. The OrderItem must encapsulate a copy of catalogue data, not references (in case that data changes or is deleted). There are no situations I have come across where we need to query the database based on the contents of the OrderItemList, so persisting the OrderItems is an ideal case for using serialisation (the Serialized LOB pattern).

The nuances of the OrderItem interface come down to experience. The OrderItemList must also be able to correctly identify and handle duplication of OrderItems - if a compatible item is added, do they stack, remain as duplicates, or refuse to be added? If items are stackable, can shoppers change the quantity they wish to purchase? Can the maximum quantity purchasable vary? An OrderItem must be queried for a price, but how is this price affected by discounts and voucher codes? How does each item affect postage and packing options and costs?

In practice even this system is not sufficient because orders are not necessarily a flat list. In some cases, OrderItems must contain child OrderItems. These are things like add-on packs and upgrades which are conceptually self-contained, but can only be ordered alongside a parent item. Child items are priced seperately but grouped with the parent item for the purposes of removing the item from the basket or changing the quantity.

I include the following example list of items (derived from experience) which a flexible e-Commerce ordering system should be able to handle within a single basket:

• DVDs - for each product, one SKU and one price (many similar examples).
• Clothing - for each product, SKUs corresponding to both colour and size. Some sizes may have different prices (many similar examples).
• Groceries - for each product, SKUs corresponding to different pack sizes at different prices. (many similar examples).
• Computers - each SKU may be upgraded with a custom combination of add-ons, at extra cost. Some of these may be available as standalone products, other times not (likewise all configurable but mass-produced goods).
• Rope - pricing is based on the length of rope to be cut from the drum at a different rate per SKU. Users might choose length instead of quantity (likewise textiles).
• Kitchens/Worktops - each SKU corresponds to a finish, but pricing is complicated, based on how many boards need to be cut to satisfy a layout, given tolerances for carpentry and mitres, and the labour cost of performing that carpentry (likewise anything bespoke).
• Antiques - each antique can only be purchased by a single buyer and must then be removed unless/until the sale falls through (likewise anything second-hand).
• Samples - given away free, but limited to one of each SKU per customer. Because they are free the normal delivery charges may not apply, and the checkout might have to be cut short because payment information is not necessary. It may not even be worth combining these into the standard order process, although it might save the merchant some overhead if the customer simply wants a few samples to be chucked in when their real order is dispatched.

In a previous post I discussed the method I used to integrate Paypal’s Encrypted Web Payments in generic SSL terms I hoped would make it easy to implement from scratch in any language. I’ve had a request from Ross Poulton to share the Python code that makes it work using the M2Crypto wrapper. So, here it is:

from M2Crypto import BIO, SMIME, X509
from django.conf import settings

class PaypalOrder(dict):
        """Acts as a dictionary which can be encrypted to Paypal's EWP service"""
        def __init__(self):
                dict.__init__(self)
                self['cert_id']=settings.MY_CERT_ID

        def setNotifyURL(self, notify_url):
                self['notify_url']=notify_url

        # snip more wrapper functions

        def plaintext(self):
                """The plaintext for the cryptography operation."""
                s=''
                for k in self:
                        s+=u'%s=%s\n'%(k,self[k])
                return s.encode('utf-8')

        __str__=plaintext

        def encrypt(self):
                """Return the contents of this order, encrypted to Paypal's
                certificate and signed using the private key
                configured in the Django settings."""

                # Instantiate an SMIME object.
                s = SMIME.SMIME()

                # Load signer's key and cert. Sign the buffer.
                s.load_key_bio(BIO.openfile(settings.MY_KEYPAIR), BIO.openfile(settings.MY_CERT))

                p7 = s.sign(BIO.MemoryBuffer(self.plaintext()), flags=SMIME.PKCS7_BINARY)

                # Load target cert to encrypt the signed message to.
                x509 = X509.load_cert_bio(BIO.openfile(settings.PAYPAL_CERT))
                sk = X509.X509_Stack()
                sk.push(x509)
                s.set_x509_stack(sk)

                # Set cipher: 3-key triple-DES in CBC mode.
                s.set_cipher(SMIME.Cipher('des_ede3_cbc'))

                # Create a temporary buffer.
                tmp = BIO.MemoryBuffer()

                # Write the signed message into the temporary buffer.
                p7.write_der(tmp)

                # Encrypt the temporary buffer.
                p7 = s.encrypt(tmp, flags=SMIME.PKCS7_BINARY)

                # Output p7 in mail-friendly format.
                out = BIO.MemoryBuffer()
                p7.write(out)

                return out.read()

The settings required are as follows:

MY_KEYPAIR='keys/keypair.pem'    #path to keypair in PEM format
MY_CERT='keys/merchant.crt'    #path to merchant certificate
MY_CERT_ID='ASDF12345'    # code which Paypal assign to the certificate when you upload it
PAYPAL_CERT='keys/paypal.crt'    #path to Paypal's own certificate 

I’ve spent this afternoon fighting with Paypal’s developer sandbox to make encrypted web payments work. This is the system of whereby the details of an order are transferred as a form field encrypted with public key cryptography, but I wouldn’t expect anyone to know that because Paypal has a wide glossary of internal terminology that is almost impenetrable to the novice.

I have never done a particularly large amount with Paypal. It’s come up occasionally but I’ve always done the hastiest job possible, perhaps pasting some code from PaypalTech.com or something. This time I’ve been working in Python with Django so I’ve had to develop everything from scratch with the M2Crypto OpenSSL wrapper. Paypal’s developer documentation lacks sufficient detail for a clean-room implementation (thought there are numerous examples to be found, none exactly corresponded to M2Crypto/Python). Furthermore, it does not give useful error messages, which can make it extremely troublesome to integrate with.

This is how I made EWP work.

1. Paypal expects data to arrive as a set of key-value pairs, which you should already have/know about. The documentation for these is extensive.
2. Make the payment system work unencrypted, using input fields with the corresponding names and values to the key-value pairs. This ensures that the information PayPal needs to receive is correct before you start faffing with encrypting that data. If you do not have a PayPal account, you can sign up for a “sandbox” account, then register as many scratch accounts as you like. Note that PayPal does not email you the confirmation emails for sandbox account; they appear in the sandbox interface under the “Emails” tab.
3. Generate an SSL keypair in PEM format with openssl genrsa . If you view this file, it is titled as “PRIVATE KEY”, but it does contain both private and public keys. The exact commandline arguments for this command are documented in the PayPal Website Payments Standard Integration Guide.
4. Generate a self-signed SSL certificate from your keypair with openssl req. An SSL certificate contains your public key, some details about the owner, and one or more cryptographic signatures. Again, this is well documented.
5. Exchange certficates with PayPal by logging in and visiting “Encrypted Web Payments” under “Profile”. You save PayPal’s certificate to a file, and upload your own. Paypal assigns a certificate ID to your certificate, which you must now add to your key-value pairs under the key cert_id . It displays this certificate ID in the table and will also email you a copy. Recall however, that PayPal’s sandbox development server does not actually email you; “emails” are stored and are available on the web interface under the “Emails” tab.
6. Generate the plaintext for the signing by encoding the key-value pairs as key=value , separated only by linefeed (\n, ASCII 0×0a) characters. CR-LF does not work.
7. Sign the plaintext using S/MIME. This requires both your private key (for the cryptography) and your certificate (to identify whose signature it is). Use these options:
   • Use binary input mode, which prevents OpenSSL munging its input.
   • Encode the data in opaque form. This implies that the text to be signed is encoded along with the signature, as opposed to detached, which doesn’t encode the plaintext.
   • Output the resulting PKCS7 structure in DER (not PEM, nor S/MIME) format. This is a binary format.
8. Encrypt the resulting DER using PayPal’s certificate (ie. public key):
   • Again, use binary input mode
   • Use the 3-key triple-DES, CBC mode block cipher. OpenSSL calls this des-ede3-cbc or just des3.
   • Output the resulting PKCS7 structure in PEM format, which is a base64-encoded format.
9. Insert the PEM blob into a form field named encrypted. There must also be a hidden value form field, named cmd, with value _s-xclick.

Why are payment gateways such a pain to integrate with? There are only two real models:

1. Merchant site directs the user to the payment gateway site with details of what they are paying for. Confirmation of the transaction is POSTed back.
2. Merchant site collect the data on a secure server, and requests payment via RPC.

So why does every payment gateway have a myriad slight variations? This means developers have to write adapters for each payment gateway, running the risk of introducing various security vulnerabilities in each. But not just that, error handling is at best muddy. I’ve used systems with non-existent error handling. I’ve even written systems without error handling (the trick is in the wording; you have to avoid saying “Thank you for your payment” and instead say “You will receive confirmation of your payment by email.” This architecture is not my design, I hasten to add).

Strangely, these variations on a theme seem to have a single meme for the documentation: to split it between at least two PDFs. These PDFs are usually along the lines of “Integration Notes” and “Advanced Integration Notes”, although if both of the above models are supported there could be one or two more. I haven’t the faintest idea why these people think that PDF documentation is better than HTML.Which is what the rest of the world uses.

The payment gateways should put their heads together and come up with a standard. Two protocols. So that with a single library, you would be able to use whatever payment gateway you want, without having to maintain dozens of adapters. This could also take the time to remove paranoid “security” checks, like verifying HTTP Referer headers (which is nonsense), and they could codify how to make Payment Gateway pages look less rubbish (because they always do).

I’m increasingly amazed by the number of banks and other secure services that seem to spread their online services over dozens of differerent domains. Simple put, a domain is one unit of trust, for a variety of reasons, and this is even assumed for security reasons in many applications (cookies and XSS sandboxing spring to mind). It’s cheaper, easier, more secure, and visibly more secure to use subdomains than purchase a separate domain to redirect users to for secure services.

Some of the culprits I’ve come across:

• NatWest (at natwest.com) use nwolb.com for online banking.
• RBS (which owns Natwest) also owns Streamline Direct, a payment gateway. RBS’ merchants’ customers get redirected onto Streamline Direct (at streamline-esolutions.com) to enter credit card details. Most won’t have ever heard of them. But if you did Google for them you’d find them at streamline-direct.co.uk and/or streamline.com.
• Paying for domains online yesterday (at streamline), I was redirected to securesuite.com, ostensibly some Mastercard security thing, and asked to enter my credit card details a second time.
• Barclays’ (at barclays.co.uk) runs their payment gateway out of epdq.co.uk
• Play.com hands over to playsecureserver1.com to take card details.

And just to contrast the way it’s supposed to work, let’s think of a few examples of big sites with secure services:

• Amazon (www.amazon.co.uk) uses https://www.amazon.co.uk.
• If you pay Google for advertising (adwords.google.co.uk), you’ll pay at https://adwords.google.co.uk.
• What domain does Paypal (www.paypal.com) use for secure services? https://www.paypal.com/.

It is relatively trivial for a hacker to obtain an SSL cert for an arbitrary domain, but extremely hard to obtain an SSL cert for someone else’s domain and then insert his machine into their DNS. Either way, he still has to compromise a web server somewhere to get his machine inserted into the chain, but web servers do get compromised, and he would have to find it beneficial to redirect to a third-party machine rather than set up some credit-card interception on the compromised host, but that’s not that hard to imagine either - maybe he can’t obtain the requisite privileges, or perhaps it’s less traceable to redirect to a different (perhaps also compromised) server.

Maybe I’m just paranoid, but more important than technical security measures are social measures: How can the public be expected to avoid phishing attacks when legitimate services are being given untrusted domains?

Mauve Internet has had two new enquiries about e-Commerce sites this week, which is good. First in a while.

I suspect that there is typically a slump in the summer as smaller business owners plan more for their weekends than the future of their business. As summer has now passed, people start looking ahead more.

This does however mean that I will have to pimp my shop codebase. It really needs tidying up - lots of things that I wouldn’t do the way they are done now that I’ve had some experience of maintaining the codebase.

I have a ton of integration to do. There are two branches to the codebase:

• One (let’s call it ’stable’) has seen bugfixes and customer-driven improvements, but has been branched a dozen times and is a huge mess.
• One has had some refactoring and more developer-driven improvements, but currently crashes due to character set issues.

After that is done, the administration interface needs to have some serious work done. Most importantly, the ImageChooser service needs to be pretty much redone. It all needs a bit of AJAX on top to make administration a more smooth experience, and I need to hook up TinyMCE to bolt in a minimal CMS.

The difficulty, if I do this work, is that I may still have to work with the aforementioned ’stable’ version even though I will have a much improved next-generation version available. Perhaps I can cut a deal on that.

I’m also considering supporting osCommerce, because it would be cheaper in terms of codebase maintenance, but I wouldn’t be able to make the same guarantees I can about implementation of bespoke features and use of future-proof technologies. This would be available as an alternative to my shop software.

What I most want to do is rewrite everything in Python. Python is much faster to develop with than PHP, and leads to much tidier and more legible code.

Well what do you know… what with all my recent work with client-side scripting and then today’s look at Mauvesoft Gallery and scanning for photos, I was just thinking about an AJAX-powered web-based Picasa clone when I stumbled across Google’s Picasa Web Albums, which I hadn’t seen before.

I’ve uploaded a few photos to it, and it’s OK I suppose, but it’s not as developed as some of the other Google webapps: it’s just a basic web gallery. It’s better than Gallery (Capital G) because Gallery is a mess these days. Gallery 2.0 is hugely bloated, and for all that bloat it isn’t very much more powerful and certainly not as easy to use as Picasa Web albums.

Still, I think Mauvesoft Gallery is a fairly good web gallery core, and it’s simple and hackable enough that I could probably rig up some AJAX over the top quite easily. It would certainly make for an easy-to-use administration interface, but I was thinking more of the on-the-fly searching, and Picasa’s wonderful ‘Timeline’ feature. I wonder whether the publishing to the world of photos is the main aim when people use web galleries, or is it that they just want access to all their photos wherever they are? (Or a third option: my brother, on holiday, uploads photos from Internet Cafes so that he can delete them from his camera to free up space.)

A fork of Mauvesoft Gallery is used to power the ImageChooser component in my e-Commerce software, and that is desperate for some attention. The UI is appalling at the moment. Perhaps I’ll download Dojo and see if that can speed up the development.