Author - Pradip Shah

Case Study : How bad code can hurt performance and even break

Many websites, notably eCommerce and WordPress, live with bad code. They end up spending more effort in improving performance by adding more resources. It is like adding more horsepower to an inefficient car engine – will consume more gas and in the long run fixing the engine would be a better thing.

Not all such attempts will succeed, however. Bad code takes many forms. There are times when your application is dependent on an external service. The problem comes when this service is executed inline when your visitor is waiting for your server to respond. The two popular dependencies are sending email (smtp) and curl calls for external http access. (curl is a popular library to access remote servers programmatically).

In this article we will analyse a bad code example we found in production. A customer in India (we manage the Magento site hosting) raised a ticket that their international shipping costs stopped working. Since we take care of their release process, we investigated to see if this was an inadvertent code change that made it to production. We could find no change. Infact, there was no release made from the day the problem started. So, with permission from the customer, we investigated further.

The bad code

They had overridden the default table shipping with custom logic. The primary reason was that they wanted to give shipping costs in USD for US customers. Magento has concepts of base currency and display currency. All calculations are done in base currency (in this case INR) while the charge was in display currency (USD).

 ...
    $to_Currency = urlencode($to_Currency);
    $url = "http://www.google.com/finance/converter?a=$amount&from=$from_Currency&to=$to_Currency";
    $ch = curl_init();
    $timeout = 0;
    curl_setopt ($ch, CURLOPT_URL, $url);
    curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);
 ...

I wondered why it took over 4 years since the bad code was written to break. The reason it finally broke was that google took the URL out of service. The code below would return 0 as it did not find the appropriate tag in the output. So, all shipping was free.

Analyziing the bad code

  • Instead of using Magento’s functions for currency conversion, the developer thought of
    • Accessing the google finance page
    • Scraping the HTML output
  • Might be in violation of Google’s terms of service (programmatic access)
  • The customer has a one step checkout plugin – this code gets called each step, when anything changes. This is slowing down the checkout page.
  • There is no retry or reporting of failure
  • The conversion rate used for shipping may be different for the products, as the site conversion rates are updated daily.

Conclusion

Using curl in code that is accessed as the visitor waits is a bad idea. Apart from slowing down user interaction, the probability of an error increases and then the error has to be handled appropriately.

A non technical guide to scaling Magento (or any other website)

Performance & Scaling of a Magento web site are often confused. As a store owner who may not be technical a close analogy with real life will help in talking to your hosting providers and other experts.

It is no coincidence that hits to a website is called as traffic. We take this analogy further, to explain what factors matter to performance and scaling a website.

Website performance is like a car – higher performance cars drive faster and can cover a distance in a shorter time. Similarly, a higher performant website will serve a page fast. This is often measured as page load speed. A critical component of page load that the server is responsible for, is server response time. Like measuring performance of a car, measuring the page load speed is done in test mode with little or no traffic. Sometimes the performance is measured at a random time without looking at other traffic to the site. That is like test driving a car through traffic.

Scaling is like building highways and roads for the cars to move on. Highways are resources – CPU, memory, network that the hits to a website will utilize. The task of a Magento scaling expert is to architect a system – servers and sizes, services to run on each server, connectivity of the servers and access from internet, etc.

Hits to a website is like traffic of random cars on the highway. Each vehicle seems to have a mind of its own, joining and leaving the highway. Each visitor to the website will take their own journey visiting different pages.

Some Observations

Observation 1 : Like a car cannot drive at its highest speed possible at all times due to traffic, a website too cannot perform at its best best all the time. Understanding the factors that make the website perform at its optimal level all the time would be the task of both the developer and the server architect.

Observation  2: Like in traffic we have vehicles of different performance, in a website all URLs do not perform equally. A category page may not perform the same way as product detail page for example.

Observation 3: Better throughput will be achieved with the same resources, if the vehicle performance is improved – some bottlenecks can be avoided if the vehicles moved faster. Similarly, a better performing website is likely to scale better.

Observation  4: Like in traffic in order to scale one has to find the bottleneck in the highway that is causing the current slowdown, fix it and then look for the next bottleneck. This is a change in the hosting infrastructure and architecture, different from the website performance.

Observation  5: A traffic designers job is to ensure maximum number of vehicles can pass the highway at the best speed for each vehicle. A hosting designers job is to ensure maximum traffic is handled in a way that each hit is best served.

What lessons can we learn from traffic management

Lesson 1: To better manage traffic highway system has to be designed that is scalable. Mostly by bottleneck analysis we can derive what needs to be done. For example, is database a bottleneck, is file system access a bottleneck, etc.

Lesson 2: When traffic increases, possibly beyond the capacity of the highway, traffic management has to account for one more variable – starvation. The amount of time a vehicle has to wait at a metered light to enter the highway. The longer the wait, more frustration from the drivers who will find a better route to their destination.

Lesson : On a highway lanes are drawn. A better hosting will make lanes. The way most hosting providers take traffic is analogous to not having lanes with the hope that the maximun throughput will be achieved by letting hits contend for resources. The operating system stands to decide what process gets to use resources.

What are the recommended steps to achieve scaling?

As a first step to server side scaling, we move the database layer out to another instance or server. The main reason is that it is better to allocate resources in a single server when the workload is similar.

In our multi part series we take you through achieving scaling. The series is aimed at a store owner who need not be technical but is ultimately responsible to take a decision on the store. Until now you had to depend on an expert. However, there are no clear answers and the expert is making judgement calls based on most likely their prior experience. As a matter of fact no 2 webstores so results of efforts vary. This series will make you better informed.

We start by looking at a popular form of scaling – using FPC or Full Page Cache and other types of caches.

In order to help with scale, another important aspect is code quality specifically related to scaling. Scaling is difficult to achieve reliably if there is any externally dependent blocking service executed as part of the hit. Examples include sending email directly to a recipient or a external service, sending information from the server to an external service. All such processing should be done with some form of a queue handled either by an different process such as a cron job. Until Magento 1.9.2.4, the default email sending was inline for example, slowing the order success page being shown.

Autoscaling adds and removes servers (and hence resources) – something traffic managers cannot do with highways. This gives website scaling an advantage to be more elastic.

Magento Caches – Scaling Magento Part 2

Introduction

While Part 1 looked at FPC specifically, in this article we review all other caches needed for a Magento store. Some of these are commonly found to be referred in best practices, but a deeper understanding will help put them in perspective of Magento.

In general, for any cache the factors to consider are

  • Effectiveness – measured in terms of impact on page speed. Caches should be highly effective.
  • Invalidation flexibility – is an inbuilt automatic mechanism available? Is the mechanism too aggressive? A High score means the mechanism is very flexible.
  • Performance of miss – if a miss were to occur what would be the performance in terms of page speed. A low score represents bad performance.
  • Cost of refill –how much would it cost to refill the cache completely to be back in its most effective state?
  • Hit ratio achievable – In a real world environment if hit ratio were to be measured periodically what can be expected. A high would be over 99% hits over a reasonable period such as a day.
  • Memory required : How much memory is needed on the server side to cache the content.
Cache Type Effectiveness Invaliation Flexibility Peformance of miss cost to refill Hit ratio achievable Memory required
Browser High High Low Moderate High N/A
CDN High Low Moderate Moderate High N/A
FPC High Low Low High Moderate High
HTML Block Cache Moderate High Moderate Moderate High Low
Mysql cache High Low Low High High Moderate

Browser cache

  • Easy to setup – at the hosting level. Static resources like css, js and images should have caching enabled.
  • There is no need for invalidation as the browser checks for each cached resource if a newer version is available.
  • The ability of browsers to find changed resources means cacheable resources should have a very long expiry – say over a year.
  • Browser’s request for detecting change has a performance impact based on number of resources.
  • A key question to be answered is how merging css and js files impact browser caching performance. If merging is enabled, each page type (home, category, product, CMS) are likely to have different css and js files. Not merging allows the browser to cache individual files and reuse them across pages. However, in HTTP/1 browsers limit the number of connections to each domain. So, the advice is to not merge css and js files if using HTTP/2 or splitting domains for skin and js if using HTTP/1.

CDN

  • CDN networks cache at edge servers closer to users – reducing round trip latency from browser to server.
  • CDN also offloads the server from serving static cacheable resources, improving network performance of the server as well as freeing up server CPU to serve dynamic content.
  • CDNs may also take the load of SSL validation however, caution is needed here as the traffic between CDN and server may be unsecure making the site vulnerable to man-in-the-middle type of attacks.
  • CDNs are notorious for invalidations – some charge for APIs, others take a few minutes before the invalidations are effective across all edge points. When evaluating a CDN, this is a key factor that is not evaluated.
  • While a single miss is easily retrieved from the server or backup store, a full invalidation requires multiple GBs to be transferred to make the cache effective again
  • CDNs when full give great performance benefit on page load times

FPC

We have reviewed aspects of FPC in part 1 here.

Recap

  • FPC caches full HTML pages – except for variable content
  • Excellent for caching dynamic content
  • Requires very high amount of RAM on the server side
  • Depending on the quality of code, FPC invalidation or miss can have impact on resource utilization
  • Best implemented with autoscale – so servers are added automatically when cache is invalidated

HTML Block Cache

  • Also caches HTML but cached at the block level. Magento uses HTML blocks for building a page.
  • Since blocks may be shared across pages, these blocks do not have a high impact on invalidation as they will be regenerated once and used multiple times
  • Can dramatically improve performance if used consistently and correctly.
  • Needs developer help as many blocks are not cached by default. To cache a block, one needs a unique key that correctly identifies its variation. Check this technical blog info.
  • Invalidation can be either via a key or time (TTL). If using a key, developer needs to write appropriate event callbacks to detect change.
  • Examples of major speed up include home page blocks where latest products are shown. Depending on frequency of store updated, a 10 minute to 1 hour TTL on the block will result in dramatic improvement of home page speed.

mysql cache

Mysql can store result queries in memory. The amount of memory is specified in my.cnf as a combination of query_cache_limit (max memory for a single query result) and query_cache_size (max memory all cached queries).

  • mysql automatically invalidates a cached query if any table that was used in the query changes.
  • to access the cache mysql uses a lock on a table thereby reducing the effectiveness of the cache
  • Many mysql articles recommend smaller cache values due to the lock problem but it is best to test the size of the cache for your situation. It is best to monitor “low mem prunes” and “table locks wait”. The first gives the number of times a query was cached by removing another and the latter gives the number of times table lock was not immediately available. Both should be as low as possible.

There are other caches too

  • Php opcache
    php code is read and converted to “opcodes” which are then interpreted. An opcode cache stores the opcodes reducing one step each time. Opecode cache should have sufficient RAM and keys (equal to number of php files).
    Using opcache.php the status should be checked regularly.
    It is best to leave invalidation automatic so when the underlying php file changes the opcode cache is invalidated.
  • Operating system cache
    linux has an excellent file system cache – whenever additional RAM is available, linux will store opened files in RAM. This is important if for example you do not have a CDN. Files will actually be retrieved most likely from RAM rather than disk.
  • Magento caches should always be enabled on a production server.

Conclusion

Setting up caches is crucial for achieving performance in Magento. But with so many caches and so much advice there is overload and scatter of information. In the first 2 parts of our Magento scaling series, we have tried to pull out the important points.

Scaling Magento Series

Scaling is not just about hosting or auto scaling – many factors contribute to scalability of your Magento website. The most important and often ignored factor to scaling is the quality of code. Well written code will scale better. The next most important factor is perhaps caching. There are many types of caches that developers, managers and store owners need to understand. Understanding the benefits and compromises of a caching mechanism is important to understand scaling. We will explore the benefits as well as compromises and discipline required for various caches. We also look at hosting and autoscale mechanisms.

In this series we explore many factors that affect scaling. This has been written for store owners with some articles having technical callouts.

Introduction : A non technical guide to scaling Magento

Part 1 : Full Page Cache (FPC)

Part 2 : Other Magento Caches

Part 3 : Code quality

Part 4 : Optimizing Checkout Flow

Part 5 : Hardware

Part 6 : Hosting

Part 7 : Auto scaling

Full Page Cache (FPC): Scaling Magento Part 1

Introduction

The most important and often ignored factor to scaling is the quality of code. Well written code will scale better. The next most important factor is perhaps caching. There are many types of caches that developers, managers and store owners need to understand. Full Page Cache (FPC) is seen by store owners as a magic solution to speed issues. Understanding the benefits and compromises of a caching mechanism is important to understand scaling.

FPC Options

Magento Enterprise 1.x and Magento Community & Enterprise 2.x comes with a FPC module inbuilt. There are many plugins available for Magento Community 1.x. Some hosting providers will help setup a Varnish based FPC with appropriate hole punching. The discussion below applies to all these mechanisms as well as to Magento 1.x and Magento 2.

What is FPC?

FPC will cache most of a full page – except variable content such as a login status or items in cart or stock status of a product or sometimes even price of a product. When a hit is encountered, FPC will save CPU by returning the cached content vs re-generating the content. When a accessed page is not cached, the normal procedure to access the page will be followed and the newly generated page will be cached. While when it works well FPC is great, one needs to understand the problems a FPC might create. FPC may store the cache in files, but more likely for maximum benefit, it will be stored in memory. FPC affects resource utilization – memory and CPU. In a way it trades memory for CPU time.

What FPC is not!

Let us understand FPC better

  • Memory needed to store the entire site in FPC
    Let us say each page is 100KB and you have 10000 pages to cache. That would take about 1GB of RAM. The problem is when the number of pages or page size starts rising above this, the RAM requirement goes up. So, if you now had 20000 pages (result of each option in layered navigation for example), you would need 2GB or if each page was 120KB the 20000 pages would need  4GB. Pages are not just products – they are category pages as well. If layered navigation is added the pages multiply fast as each combination is unique and needs to be stored independently. If you start exceeding the RAM available, you need to decide what to do when you hit the memory limit.
  • Cache warming.
    Cache warming is the process of automatically adding pages to the cache before a real visitor hit comes to the cached page. When a cache is cleared, you may need to warm the cache to make FPC effective early. Cache warming uses a crawler to artificially visit pages of a site. A typical crawler will recursively crawl the site starting from the home page. This sounds logical but here are some things to think through

    • If possible find the most likely pages you need to be in the cache and warm the cache with only those pages. This will give the maximum benefit.
    • If you cannot fit all the pages in memory, the use of crawling to warm the caches becomes a problem – they will recycle pages out of memory at random, not based on the end user popularity of the pages.
    • When the cache is being warmed your resource requirement in terms of CPU will rise as both the crawler and real traffic are being served.
    • If possible crawl the site in parallel – the earlier the pages get cached the more likely a visitor to a page will already be in the cache (scoring a hit).

Performance degradation on FPC full invalidation

The above figure shows the bad response immediately when a FPC that had built to 1.5GB was cleared completely. The top image is from redis usage graph from munin and the one below is AWS cloudwatch latency averaged per minute. The latency was moderated as AWS Autoscale added more instances.

  • Invalidating the cache :
    Figuring out when cached pages are invalidated (purged / cleared) is a challenge. A more aggressive approach to invalidating (i.e. clear as little as possible) will lead to stale pages with incorrect infomration. A less aggressive approach will lead to slowdown. A FPC has to be cleared when either a code update or a data update happens.
  • FPC and robotic crawlers (BOTS)
    Even if you do not use a crawler for warming, robotic crawlers on the internet (such as google’s indexer Googlebot) will start filling the FPC cache with pages they happen to crawl. It is our advice that a site with FPC should have robots.txt and a front end processor (nginx, WAF) restricting BOTs.
  • CPU and time needed to re-generate a page
    A FPC can fully invalidate (clear) due to a (p)html or css file changing or partially due to a data change such as a product update. A miss from FPC results in the page being regenerated. The CPU requirement for a miss is much higher than a hit. If a crawler is used to warm the cache or if traffic is high, CPU requirement can be quite high as the FPC fills up. Yet, the visitor experience is not good during this period. Using autoscale, this performance degradation can be contained to some extent as additional instances are launched to handle the high CPU requirement.
  • Discipline when using FPC – know when invalidation happens
    It is important to add discipline for code update as it has the worst effect on user experience.

    • Code update should be done at low traffic times.
    • Category changes should be carefully planned at low traffic times.
    • Product additions should not have a major impact on FPC performance due to invalidations.

Our recommendation for FPC

  • Do not use a random crawler to warm the FPC cache. Use a page popularity based crawler to warm the cache if necessary.
  • Avoid using a crawler during high traffic – the crawler will compete for system resources with live traffic
  • If possible update code during low traffic times as it causes FPC to invalidate
  • If your site is horizontally scaled, pre-launch instances to your load balancer before invalidating FPC, either explicitly or indirectly, so the latency of starting an instance does not further worsen the user experience

Magento 1.x FPC Plugins

  1. Free Lesti FPC : https://github.com/GordonLesti/Lesti_Fpc. Use this guide to install
  2. Magento connect search results for FPC

Should FPC be a part of scaling strategy?

FPC is concerned with speed. Scale is concerned with the process that helps the site add resources when needed. FPC helps in scalability by reducing the use of resources per hit to the website under certain conditions. It changes the dynamics of when and how many resources will be needed.

FPC has to be considered to be part of scaling strategy, one of many parts that our scaling Magento series will delve into.

Securing your Magento site

Executive Summary

Owning a Magento website means you have a resource you own and control on the internet. Keeping it secure in all aspects is a responsibility. You need to ensure

  • your business data is secure.
  • your customer data and privacy is secured.
  • your customer’s computers are protected when accessing your website.

Your privacy policy indicates your desire to keep your customers’ data secure. You would intentionally not violate the policy but a cyber attack on your infrastructure can cause a violation.

It is not required to be a famous to be in the eyes of the attacker – most attackers today prowl the internet looking for vulnerable sites and automatically get to work. New types of attacks are being designed and hence keeping a site protected is a continuous battle.

As a person ultimately responsible for your website, you need to periodically review your security processes and see they are enhanced for new vulnerabilities.

This article has an overview of the aspects of security you need to worry about. In addition it goes in depth as well to make it actionable.

Aspects of security infrastructure for a Magento Website

  • Protection of the web edge infrastructure. Ensure you keep the bad guys out and the good guys in. Using a Web Application Firewall (WAF) with appropriate configuration will help.
  • Server protection. Ensure access to the server restricted, preferrably without a password, limit port access and protect files and folders with permissions that are just needed. Also ensure the server has the latest security patches installed. Need to know access should enforced. Code updates should be automated. A staging site should be used for updates from vendors. Plugins should be acquired from known and reputable vendors.
  • Code protection. Ensure your code is patched with the latest Magento (or any platform and plugins). In addition train developers to use safe practices to not leave holes.
  • User data protection in transit. Ensure https access to the entire site so all access is secured against in transit hacking as users use your website. Ensure all admin access is restricted either with dual password or two factor authentication or IP restricted.
  • Review all access periodically Change passwords and admin URLs regularly.
  • Run vulnerability scanners Vulnerability scanners are available for testing many aspects of security. Blackbox scanners such as Trustwave, Secure Works, help in checking if they can find external vulnerabilities. Static analyzers such as Fortify scan the code to find code level vulnerabilities.

Protecting Web Edge Infrastructure

The key role of this protection is to keep the bad guys out and allow the traffic you want in.

As with any protection of this nature, detecting bad is a key. OWASP is an online community which has come up with a core rule set (CRS). A Magento site needs a WAF protection either on the nginx server level or with an external service such as Webscale Networks.
The OWASP ModSecurity Core Rule Set (CRS) provide protections against the following category of attacks.

  • SQL Injection (SQLi)
  • Cross Site Scripting (XSS)
  • Local File Inclusion (LFI)
  • Remote File Inclusion (RFI)
  • Remote Code Execution (RCE)
  • PHP Code Injection
  • HTTP Protocol Violations
  • Shellshock
  • Session Fixation
  • Scanner Detection
  • Metadata/Error Leakages
  • Project Honey Pot Blacklist
  • GeoIP Country Blocking
System Admin Note

mod_security is a open source project available for apache and nginx. Installation is not difficult. However, you need to tune the rules to your environment. Some rule tuning may require developer or marketing input.

Server Protection

A hacked or broken in server – popularly shown in movies – gives access to your server to the hacker.  Web prowlers are continuously trying to see what ports are open in the server and what access they can get. Being a very traditional form of hacking, it is highly automated. By the very same count, protecting a linux server is not very difficult either. Follow some guidelines and review process.

  • Keep only known ports open and limit which IPs can access these ports
  • Disable any form of password access
  • Keep server patched with latest security patches
  • Run only services that you need. For example never run ftp as it gives password based access to the server.
  • Follow very strict rules on file / folder permissions as well as linux groups and users
  • Periodically scan servers for viruses signatures
  • Periodically review access keys
  • Automate routine processes and restrict with keys including code update
  • If a db server is a separate server further restrict access to this server . If using a autoscale architecture, further restrict access to app servers as well.
  • Allow no exceptions – even for a critical fix

The following system admin note has technical details for linux system administrators to execute the strategies above

System Admin Note

Depending on the configuration we recommend the following incoming ports to be open :

On the only app server without load balancer
Port of external interface Protocol / usage
80 http, main site open to the world if site not fully secure.
443 https, main site open to the world
22 ssh, possibly open only to specific IP addresses
On the db server
Interface / port Protocol / usage
External / 22 Only if you must to restricted IP addresses
Internal / 6603 Restricted to internal IPs that run app servers
Internal / 22 Ssh access from the other servers
Constellation of app servers, a db server and a nginx load balancer
Host / Interface / port Protocol / usage
Load balancer / external / 80, 443 http and https open to the world
Load balancer / external / 22 ssh, possibly open only to specific IP addresses
Load balancer / internal /2049 Nfs server
Db server / internal / 6603 Mysql, for app server access
App server / internal /1110 Nfs, for file access across all app servers

Note : If using a load balancer device or service, port 22 access should be given to only the main app server, possibly the nfs host.

System Admin Note : ssh configuration for secure access

ssh needs to be configured to a more secure mode. (Refer linux man page)

  • Change default port to non standard port (say) 2020. This will thwart an obvious attempt to breakin. However, an attacker can find the port using a port scan.
  • Disable root ssh access. This is crucial. This means that someone can never login to the server directly as root.
  • Disable password access. Weak passwords are the most common way a breakin is successful. It is common that a server being attacked will have multiple password generators trying to access the server.
  • Enable only key access. With this setting only a private key holder who has a public key stored in the .ssh/authorized_keys file will be allowed access.

/etc/ssh/sshd_config :

...
# set port to non standard 2020
Port 2020
...

# Authentication:
#LoginGraceTime 2m
#PermitRootLogin yes
PermitRootLogin no

# To disable tunneled clear text passwords, change to no here!
#PasswordAuthentication yes
#PermitEmptyPasswords no
PasswordAuthentication no

# GSSAPI options
#GSSAPIAuthentication no
GSSAPIAuthentication yes

Note : First enable key access, exchange keys, ensure ssh to keys works before disabling password

Note : When changing ssh configuration ensure a ssh session to the root server is separately running. This will prevent from being locked out of the server when changing ssh configuration.

System Admin Note : What users are needed

We recommend the following users

  • Login user
    • This user has sudo access (without password)
    • Ssh key exchanged
    • Admin logs in as this user
  • Site user (production)
    • Website will be hosted under this users (/home/production/www NOT /var/www/html)
    • This user may have ssh access to deploy code
    • This user does not have sudo access
    • If using multiple app servers, this users id will be shared across all app servers
  • Web server user (apache)
    • Nginx and php processes will be run as this user
System Admin Note : File and Directory Permission settings
  • All folders / directories under /home/production/www should have 770 – including magento’s media and var folders except app
  • All folders in /home/production/www/app should have 750
  • All files under /home/production/www (except app) should have 660
  • All files under /home/production/www/app should have 640
  • production user should be in apache group

Protect your code

  • Ensure Magento code is patched to the latest. Subscribe to Magento updates and setup process to check for patch releases on a monthly basis.
  • When the plugin vendor. Many simple looking community plugins can be weak on security and leave the site prone to attacks.
  • Train developers to use safe practices.
  • Keep the code repository (svn / git) access secure and passwordless for developer access.
  • Ensure external respository provider web access is secure and with 2-factor authentication.
  • Ensure backup servers are protected with access to write only given to automatic backup process

User Data Protection in Transit

Customer data in transit is the protection of data sent and received from your website to their browser. Internet connections to websites are not direct – the data hops from server to server in between. A server compromised on the way can tap into your data without anyone knowing it.

  • Using https for the complete site is an obvious requirement. Using higher security options in https is even better.
  • Emails sent should not contain sensitive data. A recent Magento patch fixed such an issue – passwords were earlier sent in emails.

Review all access periodically

Setup a security task force that reviews and reports on security once a month.
The charter of the task force would be evaluate the following

  • Who has access the the server and do they yet require this access. The access may be via keys and when a employee or contractor leaves, the key should be removed
  • Who has access to development environment. Have required access been withdrawn appropriately.
  • Who has access to version control systems. Have required access been withdrawn appropriately.
  • Admin access Are all admin users yet needing access?
  • Were admin passwords reset at agreed frequency?
  • Magento Patch status
  • OS patch and update status
  • External security scan (if arranged) status
  • Action taken report for the last period on any security issues
  • Backup and restore process review to ensure data and code is appropriately backed up

Transactional Email Deliverability of your Magento Store

The internet started with email and email continues to be a very important means of communication for a Magento site. Emails that are sent directly in relation to an activity on the website such as a registration or purchase – are called transactional emails. Transactional emails occupy a different place in the email marketing category and are governed by less strict rules worldwide.

Importance of Transactional Email for Magento stores

Deliverability of transactional emails is a key to customer satisfaction and loyalty. If a customer requesting a password reset does not get an email in time in the the inbox would result in possibly loosing the customer.

Why is Transaction Email Deliverability a problem?

If email is fundamental to internet why is email deliverability a issue?
In order to protect email infrastructure from spammers, many services created spam lists – IP addresses that have previously been used to spam and are blacklisted. There is no single authority with such lists, leading to the deliverability problem. The IP you get assigned by your cloud provider may not be the clean in all the lists and it is too difficult to find and much more difficult to get cleared. Transactional email providers come to the rescue – their business is to increase deliverbility.

What can be classified by a transactional email?

Newsletters, even opted in, do not classify as transactional email. If you do not send newsletters through Magento, all emails that go out will be transactional.
However, you maybe crossing the line if you send out upsell / crosssell in your email order confirmation for example.

Third party providers

There are many providers and it is a very competitive market a search on google for transactional email will get you many results and comparisons.
Here are a few recently updated comparisons

How to get started with transactional email for Magento?

  • Check if you have an existing subscription to a transactional email service – indirectly. For example, if you are hosted on softlayer, you may get sendgrid credits. If you use Mailchimp to send newsletters, you may have mandrill credits.
  • Signup for the service – most of them have a free tier
  • We think having a Magento plugin is not a requirement if you are self hosted on a VPC or better. Read on, we think using the SMTP service is better option than a plugin or code integration.

Before you install the Magento plugin, read this!

  1. Plugins add a drag to the system – like it or not, each plugin you add, contributes to a slowdown of Magento due to the architecture. Many plugin authors are guilty of passing in additional features into the plugin.
  2. Plugins for transactional emails are “inline” i.e. the email is sent while the purchaser is waiting for a confirmation. That is a dependency on an external system. Occasionally the service may have slowed down and that delay will be added to the wait for the customer.
  3. Local email systems are automatically configured to retry in case of upstream infrastructure failure. If configured at the system level, the email is sent only to the local system, form where it goes into a queue which the systems email service will relay. If for some reason the remote email service is not responding, the queue will remain active and a retry will be attempted after sometime.
  4. Do not select the service based on the availability of a Magento plugin – that is the least important part of the evaluation

How to setup

All providers use TLS for SMTP communication on port 587. It will be required to open port 587 in the firewall to ensure emails be sent out.

Note : Some cloud services notably Google Cloud Platform does not allow communication on ports 25 or 587. For such services you need to use a transactional email service provider that allows SMTP communication over a non standard port.

Use the guide below to get your username and password and then use the steps to setup postfix

For Mandrill

Username : mandrill username
Password : Get Key (Dashboard->Get API Keys->NewAPI Key)
Domain : smtp.mandrillapp.com

For Amazon SES

Username & Password : https://docs.aws.amazon.com/ses/latest/DeveloperGuide/smtp-credentials.html
Domain (as per region, this is for US West) : [email-smtp.us-west-2.amazonaws.com]:
Domain verification : http://docs.aws.amazon.com/ses/latest/DeveloperGuide/verify-domain-procedure.html

For Sendgrid :

Get this certificate and store in /etc/postfix/ssl
wget https://certs.godaddy.com/repository/gd_bundle-g2-g1.crt
username : sendgrid account username
password : sendgrid account password

postfix setup

    1. Ensure SASL authentication package like cyrus is installed.
    2. Ensure you have a FQDN (Fully Qualified Domain Name). The command hostname -f should report a host.domain type of name. It is preferred you use the domain you are sending from
    3. Ensure postfix is installed (and sendmail is not)
    4. Edit /etc/postfix/sasl_passwd and enter SMTP_DOMAIN, username and password as per the transactional email platform.
    5. chmod 600 /etc/postfix/sasl_passwd
    6. psotmap /etc/postfix/sasl_passswd
    7. edit /etc/postfix/main.cf and add the following to the bottom of the file
# enable SASL authentication
smtp_sasl_auth_enable = yes
# tell Postfix where the credentials are stored
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_sasl_security_options = noanonymous
# use STARTTLS for encryption
relayhost =<refer platform info>
## For mandrill
smtp_use_tls = no
## For sendgrid
smtpd_tls_security_level = may
smtp_tls_CAfile = /etc/postfix/ssl/gd_bundle-g2-g1.crt
## For Anazon SES
smtp_use_tls = yes
smtp_tls_security_level = encrypt
smtp_tls_note_starttls_offer = yes
  1. restart the postfix service
  2. test by sending an email and watching the result in /var/log/maillog

5 Questions to ask before selecting a Magento Managed Hosting Provider

5 Questions to ask your Magento Managed Service Provider

  1. Does the managed provider help you with setup of a CDN?
    Content Delivery Networks or CDN is the best way to get a boost of your page load time. A CDN takes away the load on your server by caching commonly used files that do not change frequently. Such files include css, js and image files. For a Magento store, images constitute a large amount of time and bandwidth of a page.
    Large CDNs have an additional advantage of having multiple end points or servers near your customers. This reduces latency and images tend to appear immediately.
    The good news is that CDNs are now a commodity – they are aggressively priced with some large CDNs having unlimited free plans.
    The only catch is that you need to “invalidate” or “purge” cached content from the CDN if you were to change the same file on the live server. Until automated, this procedure can be a burden.
    Your Magento Managed Hosting Provider should provide you with practical and complete options for a real CDNs, including help in configuring your Magento site to use your selection
  2. Does the provider have a disaster recovery plan – not just a backup?
    Backup is as good as the restore process.
    Yes! Just having backup is a essential but not sufficient.
    If your site were to be hit by a disaster, how would you be able to bring up your site on another server? Would all services start as desired?
    Funnily we have seen many magento servers do not come back to full service on a simple reboot, far less a disaster.
    Moreover, backups should be taken “off-site” – not within the same zone or provider you currently have.
    And your Managed Magento Hosting provider should give you a written disaster recovery plan based on different types of scenarios and that you can put into motion if the time arises.
  3. Does your provider regularly evaluate and tune for optimal cache performance?
    Magento site performance is quite often driven by the condition of its various caches. Caches store results in RAM vs either on disk or having to compute the results. As a quick look at the Cache Management page on Magento’s admin panel shows there are many application level caches. In addition, mysql stores query results in a in-memory cache, however requires the cache to be sized and enabled in a configuration file. The php code can be stored in a opcode cache in the php server. This also requires memory allocation and enablement.
    Since no two magento stores are the same, there is no standard configuration. It requires observation and tuning.
    Your Magento Managed Hosting provider should give you information and tune the cache, possibly even advising you with server sizing in the process.
  4. What scalability plan has the provider made for you?
    As hits increase, it may be required for your site to scale. But, how will you even know the time has come?
    In todays world, a slowdown of your Magento site is equivalent of downtime. Consider this – each drop of a second in your load time results in drop of visitors.
    If traffic spurted due to a online or offline ad, how would you know the server capacity needed to handle the spurt? Do you need autoscaling and loadbalancing technologies?
    Your Magento Managed Service provider should be on your side in advising
  5. Do you know the deliverability of your transactional emails?
    An eCommerce site sends many emails – new user registration, password change, order confirmation, abandoned cart. These emails need high rate of deliverability. If emails are directly delivered from the host, there are 2 problems – It is not known which emails are delivered and which are not. amd secondly mail servers are constantly upgrading their needs of the senders ability to send, resulting in increased non deliverabilityMany services are available to handle such emails – Amazon SES, Mandrill, Sendgrid to name a few. Each give a varying level of granularity of reports and pricing plans to support your volume.Most of these services have Magento Plugins, some premium. However, a email plugin will attempt to send the email “inline” – for example the order email is sent while the customer is waiting for the success page. A better solution is to send the email to the local system queue, and let the system process the queue by sending the email to the transaction email provider.

    Your Managed Magento Service provider should be able to configure your server’s email system to use your chosen transaction email provider?

 

Think before you execute

How system admins can use planning and checklists to improve quality

Introduction

Sounds like a right statement to make an obvious but in this article I will look at how it applies practically to a system admin

It starts at the beginning

When a task (in the form of a incident request or a service request) comes to a system admin, this is when the principle starts – how deep do you think. My observation is that a natural thing to do is to take a superficial view of the task and then either a google search is performed or a quick think is done to find commands will resolve the task.

The fundamental issue with this approach is that it is solution centric not problem centric. i.e. not enough time is spent in understanding the problem and coming up with a solution. Another problem with the approach is that there is a rush to execute as being in front of the system and executing is what will make things work.

Thinking before executing addresses both of these

  • Has the problem well understood?
    If the problem is not well understood, the advice on google would not be understood and only coincidentally will you solve the problem. Quite likely the problem will resurface. If on the other hand the problem is well understood, the solution quite often will be obvious or the advice on the internet will make sense.
  • Once the problem is understood, a detailed plan is needed.
    Most tickets require multiple steps to solve. If each step is not well documented, you are forced to plan while executing – in our experience very difficult.
  • Our approach is to come up with a detailed plan before executing. It is a modified and more comprehensive checklist. We have SOP (Standard Operating Procedures) for common tasks, but this works even if we do not have a SOP.
    We use a spread sheet.

    • At the top we have information we will need before we start executing. This includes information such as IP (internal or external) of the systems we wiil use, passwords if any that we will need, etc.
      Here is a sample of a sheet we use for rehosting.
      info_sheet
    • Below that is a table that is both a planning as well as a execution tool. The table has the following headers :

      • Function – overall heading for a set of commands
      • Step details / reference – SOP reference and commands to be executed
      • Command – the actual command that will be executed. Note the commands are created before they are executed.
      • Status – marked upon execution

    audit_trail

    Conclusion

    Checklists help being purposeful in execution and the result is a audit trail that can be used to debug issues later on.

minify css js offline

Minify css and js for Magento as a build process

How to improve page load speed without server overhead so you can serve more pages.

Need for a build?

Magento being written in php, an interpretive language, the need to build is not essential for deployment. Moreover, since many small store owners are not technical or do not have a full time technical team, solutions that just work inline are preferred. For example, using plugins for css and js minify, or transfer to CDN as and when needed inline, or even use Google’s excellent pagespeed plugin.
Unfortunately, each one of these inline steps though improve page load speeds, result in a ever-so-slight slow down of the server each time. On a high traffic site, this results in inconsistent performance and user experience. We even zip the static content in .gz files so the web server (nginx in our case) does not have to spend a few milliseconds each time – assuming ofcourse you do not have a CDN that can zip.

Grunt, the task builder

We have used Grunt ( http://gruntjs.com/) as a task builder. Grunt is a popular javascript task builder written in nodejs. We use grunt to do many release oriented activities – packaging a release, installing a release, minify css, js, etc. In this article – a first of a series we plan – we will go through the process of installation of grunt and offer a solution to minify js and css flles as well as optimize images in the skin directory.

Installing grunt

  • Install nodejs and npm
    curl -sL https://raw.githubusercontent.com/nodesource/distributions/master/rpm/setup_4.x | sudo bash -
    sudo yum install nodejs npm
  • Install grunt
    sudo npm install -g grunt-cli
  • Download our Gruntfile.js and related code
    mkdir /scripts
    cd /scripts
    git clone https://github.com/luroconnect/gruntformagento.git
    cp –r gruntformagento/src/* .

Run grunt to minify css and js (and more)


cd /scripts
grunt optimize
Typical output :
Running "copy:skin" (copy) task
Created 229 directories, copied 1769 files

Running "copy:js" (copy) task
Created 197 directories, copied 893 files

Running "uglify:skin" (uglify) task
>> 30 files created.

Running "uglify:js" (uglify) task
>> 301 files created.

Running "cssmin:skin" (cssmin) task
>> 149 files created. 2.34 MB ? 1.77 MB

Running "imagemin:skin" (imagemin) task
Minified 1412 images (saved 400.26 kB)

Running "compress:skinjs" (compress) task
>> Compressed 32 files.

Running "compress:skincss" (compress) task
>> Compressed 155 files.

Running "compress:js" (compress) task
>> Compressed 332 files.

Done, without errors.

What is done by optimize :

  • Create 2 directories skin.min and js.min initially with identical content as skin and js respectively
  • Run the minifyfor css and js on the skin.min and js.min directories. .min.js files are not minified.
  • Run image optimizer on skin (png,jpeg)
  • Generate .gz gzipped files – for static delivery of gzip. See note below on nginx configuration.

Update Magento Web URLs

Update the Magento unsecure and secure skin and js URLs to point to skin.min and js.min respectively where minified content is kept.

Update nginx configuration

nginx configuration to load .gz static content if it exists

#/* static content can have expiry set to long */
location ~* \.(jpg|jpeg|gif|png|css|js|ico|swf|woff|woff2|svg|TTF)$ {
gzip_static on;
#access_log off;
log_not_found off;
expires 360d;
}

Gzip_static on tells nginx to serve the .gz file of a static file it exists rather than nginx compressing it.

Run optimizer on images in media/wysiwyg

grunt media
copy optimized images from media.min/wysiwyg to media/wysiwyg manually

Conclusion

We firmly believe in creating a documented release process. And Grunt with our Gruntfile.js goes a long way in making this a reality. In this article we have introduced the minfication, image optimization and gzip compression of static files. Try it and let us know if you have any suggestions.

This script can be run directly on the live server, but make sure you do it at a low traffic time.

luroConnect Newsletter Subscription.

Thankyou for your interest in our newsletter- a weekly tip
to help you with Magento hosting.
Please give us your name and email so we can add you to our mailing list.

Your Name (required)

Your Email (required)