Magento

Transactional Email Deliverability of your Magento Store

The internet started with email and email continues to be a very important means of communication for a Magento site. Emails that are sent directly in relation to an activity on the website such as a registration or purchase – are called transactional emails. Transactional emails occupy a different place in the email marketing category and are governed by less strict rules worldwide.

Importance of Transactional Email for Magento stores

Deliverability of transactional emails is a key to customer satisfaction and loyalty. If a customer requesting a password reset does not get an email in time in the the inbox would result in possibly loosing the customer.

Why is Transaction Email Deliverability a problem?

If email is fundamental to internet why is email deliverability a issue?
In order to protect email infrastructure from spammers, many services created spam lists – IP addresses that have previously been used to spam and are blacklisted. There is no single authority with such lists, leading to the deliverability problem. The IP you get assigned by your cloud provider may not be the clean in all the lists and it is too difficult to find and much more difficult to get cleared. Transactional email providers come to the rescue – their business is to increase deliverbility.

What can be classified by a transactional email?

Newsletters, even opted in, do not classify as transactional email. If you do not send newsletters through Magento, all emails that go out will be transactional.
However, you maybe crossing the line if you send out upsell / crosssell in your email order confirmation for example.

Third party providers

There are many providers and it is a very competitive market a search on google for transactional email will get you many results and comparisons.
Here are a few recently updated comparisons

How to get started with transactional email for Magento?

  • Check if you have an existing subscription to a transactional email service – indirectly. For example, if you are hosted on softlayer, you may get sendgrid credits. If you use Mailchimp to send newsletters, you may have mandrill credits.
  • Signup for the service – most of them have a free tier
  • We think having a Magento plugin is not a requirement if you are self hosted on a VPC or better. Read on, we think using the SMTP service is better option than a plugin or code integration.

Before you install the Magento plugin, read this!

  1. Plugins add a drag to the system – like it or not, each plugin you add, contributes to a slowdown of Magento due to the architecture. Many plugin authors are guilty of passing in additional features into the plugin.
  2. Plugins for transactional emails are “inline” i.e. the email is sent while the purchaser is waiting for a confirmation. That is a dependency on an external system. Occasionally the service may have slowed down and that delay will be added to the wait for the customer.
  3. Local email systems are automatically configured to retry in case of upstream infrastructure failure. If configured at the system level, the email is sent only to the local system, form where it goes into a queue which the systems email service will relay. If for some reason the remote email service is not responding, the queue will remain active and a retry will be attempted after sometime.
  4. Do not select the service based on the availability of a Magento plugin – that is the least important part of the evaluation

How to setup

All providers use TLS for SMTP communication on port 587. It will be required to open port 587 in the firewall to ensure emails be sent out.

Note : Some cloud services notably Google Cloud Platform does not allow communication on ports 25 or 587. For such services you need to use a transactional email service provider that allows SMTP communication over a non standard port.

Use the guide below to get your username and password and then use the steps to setup postfix

For Mandrill

Username : mandrill username
Password : Get Key (Dashboard->Get API Keys->NewAPI Key)
Domain : smtp.mandrillapp.com

For Amazon SES

Username & Password : https://docs.aws.amazon.com/ses/latest/DeveloperGuide/smtp-credentials.html
Domain (as per region, this is for US West) : [email-smtp.us-west-2.amazonaws.com]:
Domain verification : http://docs.aws.amazon.com/ses/latest/DeveloperGuide/verify-domain-procedure.html

For Sendgrid :

Get this certificate and store in /etc/postfix/ssl
wget https://certs.godaddy.com/repository/gd_bundle-g2-g1.crt
username : sendgrid account username
password : sendgrid account password

postfix setup

    1. Ensure SASL authentication package like cyrus is installed.
    2. Ensure you have a FQDN (Fully Qualified Domain Name). The command hostname -f should report a host.domain type of name. It is preferred you use the domain you are sending from
    3. Ensure postfix is installed (and sendmail is not)
    4. Edit /etc/postfix/sasl_passwd and enter SMTP_DOMAIN, username and password as per the transactional email platform.
    5. chmod 600 /etc/postfix/sasl_passwd
    6. psotmap /etc/postfix/sasl_passswd
    7. edit /etc/postfix/main.cf and add the following to the bottom of the file
# enable SASL authentication
smtp_sasl_auth_enable = yes
# tell Postfix where the credentials are stored
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_sasl_security_options = noanonymous
# use STARTTLS for encryption
relayhost =<refer platform info>
## For mandrill
smtp_use_tls = no
## For sendgrid
smtpd_tls_security_level = may
smtp_tls_CAfile = /etc/postfix/ssl/gd_bundle-g2-g1.crt
## For Anazon SES
smtp_use_tls = yes
smtp_tls_security_level = encrypt
smtp_tls_note_starttls_offer = yes
  1. restart the postfix service
  2. test by sending an email and watching the result in /var/log/maillog
minify css js offline

Minify css and js for Magento as a build process

How to improve page load speed without server overhead so you can serve more pages.

Need for a build?

Magento being written in php, an interpretive language, the need to build is not essential for deployment. Moreover, since many small store owners are not technical or do not have a full time technical team, solutions that just work inline are preferred. For example, using plugins for css and js minify, or transfer to CDN as and when needed inline, or even use Google’s excellent pagespeed plugin.
Unfortunately, each one of these inline steps though improve page load speeds, result in a ever-so-slight slow down of the server each time. On a high traffic site, this results in inconsistent performance and user experience. We even zip the static content in .gz files so the web server (nginx in our case) does not have to spend a few milliseconds each time – assuming ofcourse you do not have a CDN that can zip.

Grunt, the task builder

We have used Grunt ( http://gruntjs.com/) as a task builder. Grunt is a popular javascript task builder written in nodejs. We use grunt to do many release oriented activities – packaging a release, installing a release, minify css, js, etc. In this article – a first of a series we plan – we will go through the process of installation of grunt and offer a solution to minify js and css flles as well as optimize images in the skin directory.

Installing grunt

  • Install nodejs and npm
    curl -sL https://raw.githubusercontent.com/nodesource/distributions/master/rpm/setup_4.x | sudo bash -
    sudo yum install nodejs npm
  • Install grunt
    sudo npm install -g grunt-cli
  • Download our Gruntfile.js and related code
    mkdir /scripts
    cd /scripts
    git clone https://github.com/luroconnect/gruntformagento.git
    cp –r gruntformagento/src/* .

Run grunt to minify css and js (and more)


cd /scripts
grunt optimize
Typical output :
Running "copy:skin" (copy) task
Created 229 directories, copied 1769 files

Running "copy:js" (copy) task
Created 197 directories, copied 893 files

Running "uglify:skin" (uglify) task
>> 30 files created.

Running "uglify:js" (uglify) task
>> 301 files created.

Running "cssmin:skin" (cssmin) task
>> 149 files created. 2.34 MB ? 1.77 MB

Running "imagemin:skin" (imagemin) task
Minified 1412 images (saved 400.26 kB)

Running "compress:skinjs" (compress) task
>> Compressed 32 files.

Running "compress:skincss" (compress) task
>> Compressed 155 files.

Running "compress:js" (compress) task
>> Compressed 332 files.

Done, without errors.

What is done by optimize :

  • Create 2 directories skin.min and js.min initially with identical content as skin and js respectively
  • Run the minifyfor css and js on the skin.min and js.min directories. .min.js files are not minified.
  • Run image optimizer on skin (png,jpeg)
  • Generate .gz gzipped files – for static delivery of gzip. See note below on nginx configuration.

Update Magento Web URLs

Update the Magento unsecure and secure skin and js URLs to point to skin.min and js.min respectively where minified content is kept.

Update nginx configuration

nginx configuration to load .gz static content if it exists

#/* static content can have expiry set to long */
location ~* \.(jpg|jpeg|gif|png|css|js|ico|swf|woff|woff2|svg|TTF)$ {
gzip_static on;
#access_log off;
log_not_found off;
expires 360d;
}

Gzip_static on tells nginx to serve the .gz file of a static file it exists rather than nginx compressing it.

Run optimizer on images in media/wysiwyg

grunt media
copy optimized images from media.min/wysiwyg to media/wysiwyg manually

Conclusion

We firmly believe in creating a documented release process. And Grunt with our Gruntfile.js goes a long way in making this a reality. In this article we have introduced the minfication, image optimization and gzip compression of static files. Try it and let us know if you have any suggestions.

This script can be run directly on the live server, but make sure you do it at a low traffic time.

How to evaluate a hosting service for Magento?

Introduction

With so many choices in hosting service it is difficult to decide what to use for hosting of a production Magento server. The first question most commonly asked is – should one use a physical (or bare metal) server or a virtual machine (or cloud server)? Many people think the obvious answer is Virtual Machines. After all this is the way the world is thinking and all cannot be wrong. But, let us take a closer look for Magento hosting. Magento is typified by 2 factors – high CPU utilization for php interpretation and mysql performance limited by both CPU and disk writes typically for operations such as reindexing and high order volumes.
 
In this article we talk about these factors and how to help evaluate your preferred platform.

CPU speed

Most cloud service providers like AWS, Azure or Softlayer do not define what you get when you ask for say 2GHz CPU – do you get 100% of the power of the CPU or is the CPU shared?
Hypervisor technology used to create Virtual Machines, easily allows one to overcommit CPU. Overcommit means that the number of CPUs on a physical hardware can be lesser than the number of CPUs in all the VMs running on that server. Emperical studies such as here(https://www.datadoghq.com/the-top-5-ways-to-improve-your-aws-ec2-performance/) have proven that overcommit does happen.
 
This means that test results and live site performance are subject to current usage of your neighbours.
In addition, VMs need hypervisors to run beneath the VM – this adds to overhead as well as latency.
 
In a study by Forrester a case was made for using bare metal infrastructure, now that some leading cloud providers have made it easy to spin a new bare metal server.
 
On a physical or bare metal server, processor power, disk space, memory and other system resources are not shared with noisy neighbours so there is high correlation between test results and live performance.

Disk speed

High speed disks vs network access disks. A SSD or a 15K SAS for example can give the boost you need when upload products and reindex or you get many orders such as during a holiday season.

  • Unthrottled performance. When you buy guaranteed IOPs for example, what happens when you exceed the limit? What if you have a burst need that exceeds the provisioned IOPs?
  • Use a locally attached SSD where available vs a network storage. Local storage will be faster than network storage by orders of magnitude.
  • Consider RAID configurations for better performance

Here is some raw disk performance metrics1 we got when testing some popular hosting providers

Provider	Softlayer  Softlayer  Softlayer	Azure	Azure   AWS  Ukfast    DO
Type		Physical   Physical   VM	VM-D2V2	VM-D2V2 EC2  Physical  VM
Disk		Magnetic   SSD	      Local	Default	Blob    EBS  SSD       Default
unbufferred 	144	   451	      200	20.6	65      57   130       150
bufferred 	155	   602	      217	1000	65      263  1500      300

Notes

  • All data in MB/sec reported by the linux dd command/li>
  • Unbuffered dd if=/dev/zero of=/tmp/test bs=256M count=4 oflag=dsync
  • bufferred dd if=/dev/zero of=/tmp/test bs=256M count=4
  • Digital Ocean performance varied in a wide range 55 MB/s to 150MB/s

This simple test shows relative disk performance on various platforms without a RAID configuration.
(Refer Roman’s wiki)

Which hosting service is preferred?

  • Physical (or bare metal) servers give the best “performance” of Magento production hosting. The key reason is that they scale in a predictable way when traffic peaks.
  • You can scale horizontally with cloud servers – many hosting providers now give the option to mix VMs and bare metal on the same subnet. >We prefer such vendors.
  • Not all VMs and servers are made equal – test before you commit.

Nginx as a load balancer for Magento

Introduction

During seasonal peaks or as traffic grows, there will be a need to add multiple app servers to your Magento store. A load balancer for Magento becomes essential. We have found that using nginx as a load balancer gives acceptable performance. We have not found many instances where we would recommend a hardware load balancer. Recent tests by nginx confirms this.

We would recommend a different load balancer only for additional features such as autoscaling.

Nginx as a load balancer offers many advantages including

  • uneven upsream servers as nginx can assign weights to each load balancer
  • self healing – takes a upstream server out of a cluster if it stops responding
  • path based load balancing
  • combination of path based and weight based load balancing
  • php upstream servers
  • SSL/TLS termination

This article assumes the process of adding a new app server to a Magento cluster is well understood. Here the focus is on the nginx configuration.

Load Balancer for Magento : Basic architecture

(more…)

Nginx or Apache : Best server for Magento

Introduction

Apache server has been for years been the default http server linux hosts use. However, recently there have been many newer “lighter” http servers. This blog article focuses on Magento hosting. Magento is a php based web eCommerce framework. Nginx requires php-fpm to process php requests. So, this comparison is really apache vs nginx + php-fpm. Apache offers MPM (Multi-Processing Module) configurations pre-fork, worker and event. In this discussion we will use the “event” MPM.
This discussion is very popular. Examples include this. We focus on Magento here.

Key Differences between apache and nginx

There are some differences architecturally that make nginx look slightly better for Magento hosting.
(more…)

Secure access to a Magento server

Today the biggest threat to your Magento production server are external threats – of being hacked. While you may not be a high value target, hackers run crawlers on the internet to discover servers with weak security and attack. In this article we discuss secure access to a Magento server. An OS level attack if successful can only be fully repelled by re-imaging the server. But preventing a OS level attack is easier than you think – if you follow some simple guidelines.

A Magento production server should have restricted access for all. Insecure, password based access should be disabled. If more than one server is used in a constellation, ssh access to the setup should be restricted to only one server.
(more…)

Will HTTP/2 help my Magento Store?

Introduction

HTTP/2 is the new http standard. Most browsers, including Chrome, Opera, Firefox, Internet Explorer 11, Safari, Amazon Silk and Microsoft Edge support HTTP/2. Nginx and other web servers too support HTTP/2. Magento 1.x and Magento 2 work very well with HTTP/2. In this article we see the benefit of HTTP/2 and give some configuration recommendations for Magento store owners or administrators.

What are key differences of HTTP/2 ?

At a high level, HTTP/2:

  • is binary, instead of textual
  • is fully multiplexed, instead of ordered and blocking
  • can therefore use one connection for parallelism
  • uses header compression to reduce overhead
  • allows servers to “push” responses proactively into client caches (more…)