Author - Pradip Shah

How we hosted a big sale on Magento without a hitch

During an online sale that includes promotion to a list, brands worry about downtime due to “unexpected high load”. We believe that while every system has a rated limit, there can be flexibility added to your infrastructure. With some planning, including autoscale and tuning, a successful sale event can be a very likely outcome. See how we handled one such sale without a hitch.

In our experience promotions via SMS generates very high peaks. In this article we will show our recent experience with a sale day that generated a peak of 30 hits per second to Magento.

The sale

On Saturday 21st March, 2020 one of our customers had a sale event. The merchant – a fashion retail brand – is an omni channel retailer in India with about 145 stores across 45 cities. There have been previous sale events, but this one was different as most stores were closed due to the corona virus measures in place. This meant that more customers were likely to shop online than visit the store.

The sale was promoted via sms and email. The SMS list was about 1 million. Promotional messages were sent in batches of 200k every half hour. Marketing was to start at 10 AM local time.

Monitoring the website

We monitor the nginx log file, analyze and display information on our dashboard. A key component of our dashboard is the plot of number of hits / minute (we count only php hits, static hits like css, js and images are excluded). We also calculate avg response time per minute as well as standard deviation per minute. Both of these have a unit of seconds. We also have alerts setup for many parameters including slowdown and a 5xx error response.

The graph below is for a 24 hour period, including the sale time, which lasted about 5 hours with the effect keeping the site more busy than usual afterwards. Note these are server logs graphs – they look quite different from google analytics!

The traffic & response

The graph gives as an idea of the traffic to the website. The turquoise line is hits per minute served by php.

  • Traffic peaked a about 2000 hits per minute – about 30 hits per second.
  • 3476 items were added to cart during the sale.
  • Average server response was mostly below 1 second.
  • There were 3 errors – all of them related to checkout function (“This payment method is not available”)

Mysql query cache and response time

The store is a Magento 1 website. Just as the traffic / hits to the website started peaking, we got a slow alert – represented by the large black line close to the first traffic peak around 10.45 am in the previous figure. Our prior experience with the website showed that the mysql query cache plays a role in the site performance. Before the sale started, mysql query cache was turned off. On seeing the slowdown alert, we turned on the mysql query cache and saw an immediate improvement in performance. Both avg response per minute and standard deviation per minute improved – for some time.

However, standard deviation – higher number indicates some visitors faced slower response to the website than others – deteriorated with time. After the sale (10pm local time), mysql query cache was turned off – resulting in improved standard deviation, but slightly worse avg response time.

AWS autoscale

The following graph plots AWS autoscale instances over the same 24 hour period. A time based minimum 5 instances was planned during the sale from 9.15 AM to 7.30PM. At the peak 7 instances were required – our autoscale policy adds 2 instances and removes 1 at a time.

luroConnect Architecture for AWS and controls

Our standard scalable architecture uses nginx as a load balancer. The communication between the load balancer to php servers is using fcgi – individual php servers do not have nginx. A NFS server is used to share folders that need sharing – media and var folders in Magento 1. Magento cache and sessions are served from redis. Code is also shared using nfs.

A custom lambda function communicates between autoscale lifecycle hooks and the nginx load balancer adding and remove instances as the AWS autoscale policy decides.

Our cache controls including the ability to turn on and off mysql cache, resize memory allocated to redis cache, resize memory allocated to php-fpm opcache.

Interested in knowing about the advanced architecture of luroConnect?

Fill the form below and we will contact you.


My company owns the Magento site
Yes, I am a developer
I represent a Magento Agency

Magento emails & SMTP

Emails are a very critical portion of an eCommerce website. There are 2 types of emails that are used – transactional and promotional.

Transactional emails are emails the eCommerce system sends for regular customer activity – such as order confirmation email. It may also include abandoned cart email, wish list reminders, password change, etc. If a website visitor action causes an email to go out, it is a transactional email.

In this article we will discuss the best way to configure transactional email sending in Magento. This article applies to both Magento 1 and Magento 2. The focus is on what happens on production systems.

The SMTP Plugin

Traditionally Magento agencies and developers install a “SMTP Plugin”. A simple search in Magento marketplace for SMTP reveals many options. Some are general plugins that allow you to connect to any SMTP provider, others are from vendors such as SendGrid.

Advantages of such plugins are

  • Easy setup
  • Developer testing is easy
  • Websites on shared hosting have to use such a plugin

However, for growing websites, this is a particular problem

  • Outgoing ports from the application server may not be open
  • Many SMTP providers have restrictions such as rate limit on email sending or number of connections. Some SMTP plugins may not handle this condition well – just reporting the error vs trying to send again.
  • They may not be able to alert when email sending is down
  • When upstream providers’ SMTP service is down or slow, cron will slow down as each email times out on sending. (Magento sends emails in cron).

Using postfix

By default, Magento sends emails to the local system where php is running. Linux systems have many options for receiving such requests and sending out the emails. A popular software is postfix. While postfix can directly send emails to the recipients’ email systems, we are interested in the “relay host” mode of postfix. In this mode, postfix acts to relay emails sent to it from Magento to the upstream provider – such as SendGrid.

Advantages of postfix

  • Postfix has an inbuilt queue. When the upstream server’s rate limit is reached, a “defer” status is returned. Postfix understands this and it will queue deferred emails and keep retrying. Emails are not lost.
  • Magento cron will deliver to local postfix in a matter of milliseconds, freeing up cron to perform other activities.
  • Connections can be pooled so if many emails are in queue they will be pooled together and sent using a single connection. Some SMTP servers, notably gmail, have a rate limit on the number of connections.
  • Log files that can be checked by system admins. Log files can also be sent to a log collection tools. Advanced commands help checking the queue, even trimming it using conditions such as email subject.
  • Flexible configuration allows many features – on a staging server for example, only whitelisted recipients can receive emails, preventing accidental trigger of email to customers. Or emails to known spammers can be filtered by email id or domain.

Installation and configuration

Installing postfix is easy – the package is part of any linux system. For RHEL based system like CentOS :

sudo yum -y install postfix
sudo yum -y install cyrus-sasl-plain cyrus-sasl-md5

Configuring postfix to act as a relayhost.

Click on the tab below to see configuration information for the specific SMTP service being used. Commands below should be run as root (or sudo). Ownership of files created, especially the password files is assumed to be root.

  1. Sign in to SendGrid and go to Settings > API Keys.
  2. Create an API key.
  3. Select the permissions for the key. At a minimum, the key must have Mail sendpermissions to send email.
  4. Click Save to create the key. SendGrid generates a new key. This is the only copy of the key, so make sure that you copy the key to clipboard and use if for the next step.
  5. Create the hashmap from the key
    echo [smtp.sendgrid.net]:2525 apikey:{{{PASTE}}} >> /etc/postfix/sasl_passwd
    postmap /etc/postfix/sasl_passwd
    rm /etc/postfix/sasl_passwd
    chmod 600 /etc/postfix/sasl_passwd.db
  6. Add to postfix configuration file main.cf
    cat >> /etc/postfix/main.cf <<EOF
    smtpd_tls_security_level = may
    # enable SASL authentication
    smtp_sasl_auth_enable = yes
    # tell Postfix where the credentials are stored
    smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
    smtp_sasl_security_options = noanonymous
    header_size_limit = 4096000
    relayhost = [smtp.sendgrid.net]:587
    smtp_tls_CAfile =/etc/pki/tls/certs/ca-bundle.crt
    EOF
  7. service postfix restart
  1. Signup for maligun account.
  2. Get credentials for maligun. Mailgun has an option for SMTP username and password under the Domains section of the mailgun portal.
  3. Create the hashmap from the key
    echo [smtp.mailgun.org]:2525 {{user}}:{{password}} >> /etc/postfix/sasl_passwd
    postmap /etc/postfix/sasl_passwd
    rm /etc/postfix/sasl_passwd
    chmod 600 /etc/postfix/sasl_passwd.db
  4. Add to postfix configuration file main.cf
    cat >> /etc/postfix/main.cf <<EOF
    relayhost = [smtp.mailgun.org]:2525
    smtp_tls_security_level = encrypt
    smtp_sasl_auth_enable = yes
    smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
    smtp_sasl_security_options = noanonymous
    smtp_tls_CAfile =/etc/pki/tls/certs/ca-bundle.crt
    EOF
  5. service postfix restart

Amazon SES is the most cost effective SMTP provider to our knowledge. However, it is also the most difficult to setup as it requires you to verify sender emails and sandbox mode (which we will not consider here).

  1. Login to your aws account
  2. Select “Simple Email Service” from the services
  3. Select Domains from the left menu
  4. Click “verify a new domain”
  5. Enter domain and click on verify
    SES will generate the TXT records you need to add to your DNS

    SES will also generate the DKIM CNAME record to be added to your DNS
  6. Ensure domain verification green status
  7. In the SES main screen, select “Email Addresses” from the left menu
  8. Enter the sender email address. Sender email address are set in Magento. Depending on your Magento configuration different emails may need to be configured. Each email needs a mailbox as a confirmation email is sent to it.
  9. Refer https://docs.aws.amazon.com/ses/latest/DeveloperGuide/smtp-credentials.html to create SMTP credentials. Note username and password.
  10. Depending on the region your SMTP endpoint (relayhost setting in the main.cf below) will be different. Refer https://docs.aws.amazon.com/ses/latest/DeveloperGuide/smtp-connect.html
    Note : The region of SES is unimportant in that there are no restrictions from where the email is sent. Your Magento instance can be anywhere – not even on AWS.
  11. Save password
    echo [email-smtp.us-west-2.amazonaws.com]:587 USERNAME:PASSWORD >> /etc/postfix/sasl_passwd
    postmap /etc/postfix/sasl_passwd
    rm /etc/postfix/sasl_passwd
    chmod 600 /etc/postfix/sasl_passwd.db
  12. Update main.cf
    cat >> /etc/postfix/main.cf <<EOF
    relayhost = [email-smtp.us-west-2.amazonaws.com]:587
    smtp_sasl_auth_enable = yes
    smtp_sasl_security_options = noanonymous
    smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
    smtp_use_tls = yes
    smtp_tls_security_level = encrypt
    smtp_tls_note_starttls_offer = yes
    smtp_tls_CAfile =/etc/pki/tls/certs/ca-bundle.crt
    EOF

Advanced configurations

Postfix relay gateway

In a more mature environment, each php server may not be able to send on SMTP ports. Even in the presense of a NAT gateway, we would recommend a main postfix gateway to ease logging and prevent loss of emails when an instance is lost in autoscaling. With postfix we can create a relay – php servers are configured to send email to a designated email gateway server which in turn relays to the SMTP upstream server.

Sending promotional emails

Promotional emails should ideally not originate from your website – instead they should be from your marketing tools. However, Magento does support sending promotional emails. To improve deliverability of crucial transactional emails, we recommend using a different sender email domain for promotional emails. If this is configured in Magento, postfix can handle sending promotional emails from a upstream service different from the transactional service. Assuming username / passwords have been created in the SMTP service for the promotional domain, the following configuration allows selecting the relayhost to use based on the sending domain.

cat << /etc/postfix/main.cf >>EOF
smtp_sender_dependent_authentication = yes
sender_dependent_relayhost_maps = hash:/etc/postfix/relayhost_map
EOF

cat << /etc/postfix/relayhost_map >>EOF
user@promodomain  [smtp.xxx.com]:{{port}}
EOF

Add the new domains passwords as the first line in /etc/postfix/sasl_passwd

user@promodomain  {{user}}:{{password}}

luroConnect and Magento emails

The configurations described above are abridged versions of what is used in luroConnect. We use the advanced configuration with a designated server allowed to communicate on port 587 to the outside. We also use secure keys to ensure passwords are not saved in chef cookbooks / configurations. Instead they are stored in a secure key store.

Interested in knowing about the advanced architecture of luroConnect?

Fill the form below and we will contact you.


My company owns the Magento site
Yes, I am a developer
I represent a Magento Agency

Reactions to this post on linkedin

Magento 1 end of life

June 2020 is a few weeks away and many stores are not ready to move away from Magento 1. But wait, you don’t have to update – now or in the long run.

COVID-19 Update : We find many customers have had to delay their Magento 2 launch in these uncertain times. We also know many of them did not have plans to keep Magento 1 uptodate. In fact, we know agencies that have stopped support for Magento 1.

Starting at a low cost of $200 per month with no long term contract. It includes reviewing your current hosting for security, moving your website to the latest Magento 1.9 and latest php supported as well as adding additional security measures to your website. It also includes help signing you up for Mage One or Open Mage projects for support beyond Magento 1 EOL, if required.

Signup now (no credit card required) and we will be in touch with you.

What does end-of-life for Magento 1 mean?

Magento 1 End-of-Life does not mean your website will stop working. It means Adobe will stop giving fixes for Magento 1, even security patches. As php version in use goes out-of-life, no upgrades will be given by Adobe.

However, being an open source platform, your Magento 1 website will not stop working. The code and license do not restrict you from running the website.

Stay on Magento 1 for short or even long term

That is a valid option and many customers are choosing this. Makes sense if

  • You have a lot of investment in the customizations which may be difficult to replicate anywhere
  • You have a stable money generating store and any change looks like a risk
  • Are in the process of migration, but the migration may take some time

What are the options to stay on Magento 1?

  • Use paid support plan from Mage one (https://mage-one.com).
  • Use open source Magento 1 fork (https://github.com/OpenMage/magento-lts) with support from the community.

What are the risks?

  • Support from either of the above reduces over time as many websites move out of Magento 1
  • Developer support may reduce as most developers move to Magento 2
  • Plugin vendors have already stopped support or are stopping support.

luroConnect support stays for Magento 1

If you are with our managed service, we will continue to support you. The biggest risk is a security risk of vulnerabilities yet to be found. Here is our plan

  1. File system security to prevent 0-day or new unknown vulnerabilities. Our rules include not allowing execution of js or php from folders open for upload. Not allowing upload to folders where code lives. This rule will prevent many malicious code to fail as they depend the ability to upload malicious code and execute.
  2. WAF – Web Application Firewall – with strict Magento 1 rules. This prevents SQL Injection and cross site scripting related attacks from being allowed.
  3. Virtual patching – block URLs that are known to have vulnerabilities. For example, we do not allow saving of the “miscellaneous” header and footer section from being written from the admin login.
  4. Admin login protection via dual password. The first is a basic http challenge. This prevents password guess of the admin URL as 2 passwords have to be guessed.
  5. Password guess prevention by restricting how many failed attempts are allowed in a day from the same IP – implemented at the application server level without changing Magento code.
  6. Staging environment to test patches from open mage or mage one or any other source you may have. Also support php version upgrade first on staging before upgrading production.
  7. Protect source code by using secure deploy process
  8. Secure backup With a proven restore strategy

We can analyze your site for free

Schedule a call

Not happy with your website performance and want an expert to look at it?

  • We will analyze your site using public information.
  • We will ask you to give us a 1 day web server log file.
  • We will try to identify what steps if any you should take to improve your sites performance goals.

Hosting help moving to Magento 2

When moving to Magento 2, to reduce the downtime during the move, luroConnect has plans for you.

  • Staging server support plans.
  • Magento 2 transition plan with minimum downtime. Our care even includes URL rewrite rules to ensure SEO value is not lost during transition.

Meet Magento India 2020 – panel discussion

I was on a panel at Meet Magento India 2020 #MM20IN moderated by Brent Peterson of Wagento.

It was an honor to be part of a elite panel – Miguel Ignacio Balparda of Nexess / Liquid Web, Shehzad Karkhanawala of Webscale Networks, Arun of ServerGuy. The discussion was lively and well received.

While the time was too short to address all the questions, Brent was kind enough to circulate the questions ahead of time.

Here would be my responses if there was enough time!

  1. What are some of the latest trends you are seeing in the hosting world
    Integrations into application – Magento Cloud as well as pwa vendors are giving hosting included packages.
    Plugins or apps are going SaaS.
    Underlying tech with Kubernetes is changing the hosting world.
    Magento hosting has to look like SaaS to the merchant – with the flexibility.
  2. Hosting as we knew it five years ago has now become a commodity. There are some interesting approaches to hosting in the market today. In your opinion, what defines and differentiates a great hosting provider for e-commerce?
    If you look at progression of economic value – commodities, goods, services, experience. A quick example being you could use ingredients to make a cake (commodities), you could buy a mix to bake a cake (goods), you could buy a cake from a cake shop (services) or you could have your birthday party organized of which cake is a part (experience). Hosting has reached the experience level with SaaS. We have to make that happen with Magento hosting.
  3. You all provide managed hosting services. What is different about managed as compared to regular hosting?
    Managing is offering hosting at a higher level of abstraction.
    Traditionally companies like Rackspace provided a server or a managed server and charged something like $100 per server as managed or support fee. In reality you paid for a system admin.
    When you give a full application level support, you are abstracting away everything.
    While many managed hosting services talk about 15 minute response time to tickets, we measure ourselves differently – we want no tickets. If a customer has to call and tell us their site is slow or god forbid down, we have lost.
    A managed hosting provider sits on the merchant’s side between the developer and the hosting service.On one hand, we ensure the hosting bill is optimized.
    On the other, we ensure we host a 12-factor app – keeping the developers honest about the quality of their delivery. We partner with many developers.
  4. What is the relationship between the customers and the hosting company in a managed service agreement?
    I will give you an analogy – a few years ago I built a house in Bangalore. I hired an architect and a contractor. I made the contractor report to the architect and paid the architect an additional 6.5% as fee to manage the contractor who managed the workers and material. 
    We are like the architect in this example – On the side of the merchant, making things happen on the hosting.
  5. What’s the difference between public and private cloud?
    everything and nothing!
    Private cloud has power, uptime, connectivity and flexibility issues.
    If the private data center addresses some of these issues, there is no difference from managed hosting perspective.
  6. All around the globe, there are big events like Cyber Monday and Black Friday. How do you help your customers prepare for this type of event?  Auto-scaling
    Autoscaling is an obvious, but there is more to the preparation.
    Autoscaling has a latency in scaling.
    We like to understand marketing strategy – if they are sending SMS – which generates a far higher peak than almost any other marketing activity – what time and volume – so instances can be launched We encourage staggering ofcourse but it is difficult to make marketing understand that. We would prep autoscale to have more instances.
    Reduce as many moving parts as possible.
    We ensure code freeze is well before the event.
    Avoid : Product uploads, inventory sync if possible
    Indexing state
    Cache state – ensure all caches are pre-loaded / primed
  7. What are the biggest differences between a hosting environment for Magento 1 and Magento 2?
    • Varnish.
    • core performance (or performance on a miss) is tough to get in M2.
    • Rabbitmq – but we have not seen mature use of this technology as yet.
    • Build & Release process.
  8. What’s your plan for the Magento 1 EOL in June?
    We are ready to support the hosting. Security is a key concern in the longer run, but we have to live with that – react as and when we know what happens. We have merchants that have not changed their code for years. Yet on old M1, php 5.6, etc. which we will not support.
    Others have invested a lot in M1 to move with feature parity. Features on the admin panel that their packaging staff depend on or rules to decide how to ship based on warehouses.
    (During the panel discussion participants talked about hardened php. That applies to php 5.x which means the merchants are running obsolete versions for a while. I think they should at least upgrade to latest Magento and php 7.2 )
  9. Some merchants are just not ready to move to M2 due to cost and timeline pressures.  Is there anything developers and their hosting provider partners can do to help clients continue to operate after June 2020?
    Absolutely. The numbers are so large that I feel a community effort to continue, if it happens, will be much appreciated. I think the hosting community is ready to support – many of us yet support php 5.6 for example as merchants have not recognized the need to move or have unsupported plugins that have not upgraded.
  10. Security has become the most critical piece of an e-commerce merchant’s infrastructure strategy. Cyber-attacks are growing in number, intensity, and sophistication. What can a hosting provider do to protect storefronts from threats?The biggest threat is from 0-day threats and not patching ontime.
    0-day threats are hitherto undocumented or unaddressed threat.
    In medicine, when a virus hits, there are 2 strategies – handling the virus directly coming up with vaccines, etc. The other is health related advice – such as wash your hands. Crucial things that sound simple, but are instrumental in preventing someone from being infected across multiple types of viruses.I call these fundamental truths. They address a class of problems.I was an engineer when first viruses came to infect 286 computers. My employer then decided to develop expertise in virus combat. We came up with a simple set of rules that prevented many forms of viruses then. How you booted mattered then for example. What you ran on your computer mattered. I wrote some code that my customer sold to Compaq and ended up protecting millions of computers.We try to make such fundamental truths in hosting. Some are taken into standards like cross site scripting. Brent – when you became a Magento Master a few years ago, you mentioned 777 permissions as a challenge. Magento came up with what they called 2 user hosting – a web server user and a command line user. Others we have developed.We try to take a class of attacks and eliminate them.Ofcourse when a known threat exists, we try to ensure that threat is addressed directly.
  11. What’s your take on the current state of Magento Cloud?
    We have typically  3rd party opinion but from what limited 1st hand experience we have, we find that to be having many restrictions. Autoscale, tuning caches like adding more RAM to varnish, are difficult. But, Magento Cloud is there and will improve. Adobe got SaaS wrong the first time, but persevered and fixed it to be one of the most profitable companies of our time.
    I would really hope Magento stops seeing Open Source as competition. The two depend on each other.

Meet Magento India 2020

luroConnect is a bronze sponsor at Meet Magento India 2020

Look out for our invitations and announcements closer to the date of the event.

In the meanwhile, you can setup a time for a demo.

We can analyze your site for free

Schedule a call

Not happy with your website performance and want an expert to look at it?

  • We will analyze your site using public information.
  • We will ask you to give us a 1 day web server log file.
  • We will try to identify what steps if any you should take to improve your sites performance goals.

Free as in freedom, not free beer!

Introduction

The open source community has had this saying for long, though there are many, including myself, who do not understand what this difference means.

With recent changes to open source license agreements, this difference has come to the fore. For example, changes in the open source licensing of redis and mongodb has restricted how AWS and other cloud providers can conduct their business. Directly relevant to eCommerce merchants is the effect to open source Magento since Adobe’s acquisition of Magento and the path that has been followed

Magento Open Source vs Enterprise

Adobe’s business reason to have Magento is (in my opinion) to complete their offering. Adobe has seen a huge, public and successful transition from their traditional business of one-time purchase of packaged software to a subscription and even a SaaS model. With the acquisition of Magento and the Commerce Cloud licensing model, Adobe clearly thinks Magento should be packaged with hosting – hence the word Cloud in their offering and their pricing of the Enterprise license that includes cloud hosting. This transition is seen in SAP’s hybris commerce offering that includes hosting to make SAP Commerce Cloud. Unlike SAP hybris though, Magento is open source.

If you accept the Adobe Magento Commerce Cloud offering, you submit to the fixed set of features that are offered by their cloud or subscribe to a SaaS service for integration – though sometimes even that requires qualification or may not be possible.

For example, if you want PWA, you are limited and have to wait for the PWA Studio. If you want an improved search interface, you are limited to their choice. Similarly, for CDN, image optimization and Web Application Firewall, you are limited to Fastly, Adobe’s choice in the matter.

Or, perhaps, you use a plugin that is connected to a SaaS service.

When a Magento website is self-hosted though, the choice was to install a plugin or enhance the code that may require a service which has to be hosted. In the examples above, you may want to use vue-storefront or use one of many systems for search or use ImageMagick as an image optimization solution.

From Free Beer to Freedom!

The earlier licensing model of Magento pushed the decision to a “board level” – companies like ours always take the supported version. Mostly the open source version of Magento was attractive to those who were attracted by the “free beer”.

However, now the decision is that of freedom – since the paid version comes with restrictions.

If you feel guilty of being a taker of open source, you can sponsor community commits back to the open source Magento. The community participation is not negligible. Matt Asay of Adobe suggests it may be as high as 50% in this article.

(Indeed there are community participants who think Adobe is gaining from open source contributions, but that is a different blog article).

Shameless plug!

Full stack managed hosting support from luroConnect, gives you the benefit of supported opensource and the flexibility to build your own solution around it. The entire stack is based open source – from linux to Magento – including nginx, ModSecurity, redis, elasticsearch, sphinx and ofcourse mysql. (We relunctantly also allow ioncube encrypted plugins as well). Coupled with a release process from your git. Hosted on any cloud or open hosting providers – in the age of Uber you don’t need to own your data centers. Our multi-layered security approach and proactive monitoring comes standard. With additional features like a disaster recovery plan, image optimization, peep-hole maintenance and a dashboard to monitor and control key tasks such as code deployment or indexing, we bring peace-of-mind to Magento hosting. Check out our pricing and you can connect with us.

Magento Open Source vs Commerce Cloud

Magento Commerce Cloud does offer additional features. See alongside (zoom for a larger view) for a comparison taken from magento.com. Some key features like WYSIWYG editor will never be released in open source.

However, not everyone needs all of the features and some of these features are available from other plugin vendors or custom development from the many certified and non-certified agencies.

Infact, even if you are a Commerce Cloud customer, you will need customizations and potentially more plugins and even 3rdparty SaaS services to have a fully working store.

Conclusion

Magento Open Source is now a very viable option for all stores – brands and high volume stores included. With many options to customize and integrate, you have the freedom to make your own best of breed solution and not be restricted by the Adobe environment. With Managed hosting service you can get optimized and scalable websites.

Deploying a Magento PWA project

Why PWA might be the future of headless eCommerce

Progressive Web Apps (PWAs) are designed to address the mobile revenue gap indicated below.

In most markets, online retail has a higher proportional audience reach among mobile users than desktop. However, mobile sales numbers are much lower. There can be many reasons for this, some in the realm of technology.

At one time it was thought mobile engagement was best achieved using an app. This was based on data for mobile users in general, but was possibly skewed towards gaming and social media. An app has advantages of being able to deliver notifications, use mobile features the way a typically website cannot, be installed as an icon resulting in easier access.

On the flip side, there is data suggesting mobile users were reluctant to install apps due to issues like memory limitations. Other disadvantages include a need for an OK from Apple to be on the app store, lack of easy-to-test infrastructure, the rather slow process of distributing updates – for example some users may not have auto update on.

Using service worker technology, widely supported by browsers today, the PWA (or progressive web application) will give some of the benefits of an app without the cost of consuming memory, requiring an OK from Apple and being as easy to deploy new updates as it is to update a website. PWAs are also as easy to test as a website. A PWA will install on a mobile as an icon – much like an app, service workers allow push notifications as well local storage, allowing for some offline capabilities.

From a technology perspective PWAs pose a completely different problem – the relative newness of technology means developers are limited, as are systems to reliably host & deploy. The cost of development will come down as more developers and websites adopt the technology.

Hosting related issues with PWA for Magento

From a hosting and code deployment perspective. Vue-storefront for example, replicates the entire catalog in elasticsearch and uses 2 nodejs processes to run the frontend of the store. PWA Studio is expected to be Magento (read php) native, yet the reference implementation of its Upward Specifcation Is in nodejs. Both developer and production environments pose challenges.

Developer Environment

It is no more a localhost WAMP stack that you can deploy and get a development environment for PWA setup. A single project will require setup of various components (vue-storefront, vue-storefront-api, Magento, graphql, elasticsearch, redis, rabbitmq, etc).

The developer environment will affect the learning curve of the many new developers starting with these new technologies as well as the productivity of experienced developers.

Here are some challenges

  1. Launch a development environment for a new project
  2. Setup a developer environment for an existing project
  3. An ability to change and test any component easily – for example if a js file was modified that affects the UI, what component(s) need be redeployed? Can this process be automated?

It is too early for us to start work on solutions for developers – since we do not do project work ourselves. However, we are working with our partners and we have an eye on releasing a “developer stack”. Contact us if you would be interested.

Production Environment

We recently took a PWA website live. Developed by our partner Codilar, it was a first for us. Some of the challenges faced and lessons learnt are summarized below.

  1. Setting up a production environment.
    Since there were so many components, not natively supported by Magento, many configuration files had to be manually modified.

    1. Vue-storefront (the UI end that replaces varnish in a classic Magento 2) needs to communicate to Redis for Full Page Cache and the vue-storefont-api
    2. Vue-storefront-api communicates with elasticsearch and Magento 2 backend via a rest API. Ideally vue-storefront should replicate the entire catalog through a indexing process into elasticsearch, but that is not fully operational yet.
    3. Magento 2 has its own redis cache and redis session. Magento 2 FPC is not used. Magento 2.3 uses RabbitMQ in addition to connection to its database.
      Here is our architecture for the deployment. We used Virtual machines as shown. We did not use a containerised architecture. The reasons will possibly be a different blog post.
  1. Starting nodejs processes automatically. Vue-storefront uses pm2 for process management. However, developer information and documentation is written using yarn to run the pm2 processes with log files being stored in ~/.pm2. In order for better control from a system administration perspective, we installed pm2 at the global level, generated systemd files (using pm2 startup) and modified them to suit the environment. We can now use “service vue-storefront start/stop/restart”.
  2. Monitoring all the components.
    Log files for each component are taken to a central log processing server using CNCF project fluentd.
    A key challenge is observability of failures. A Magento 2 API failure is not obvious. An error return code from vue-storefront needs to be traced to vue-storefront-api to Magento. Correlating the actual hit that caused a non-fatal Magento error is another challenge.
  3. How to deploy new code with minimum or even 0 downtime
    For Magento 2, until a database change is required (via a bin/magento setup:upgrade and/or indexing), we have a process to make a deployable package, giving an opportunity to deploy with 0-downtime. Check out our bitbucket pipeline presentation.
  4. How can one deploy a vue-storefront based PWA?
    The project we migrated ran on 2 git repos – one for Magento, the other for vue. Upon deploy we need to find the files that changed since the last release and decide if the change is in Magento, vue-storefront or vue-storefront-api and decide the build steps appropriately. Presently since the repos are different, we have 2 separate builds running on the production servers. A pipeline based deploy is our next step.

Note: We think a monorepo for both Magento and vue is essential in the long run due to possibility of versioning incompatibilities.

Conclusion

This is yet early work-in-progress and we hope to update our process and keep updating this article as we go.

PWA in Magento – how do your images look?

Progressive Web Apps (PWA) in Magento opens a new opportunity to improve image quality on eCommerce websites. Use of good quality images has been known for a long time to be the a key factor in digital experience and hence conversions. In recent years many factors have changed that requires a relook at how images are served. These factors include :

  • Shoppers increasingly use mobile devices for eCommerce
  • The vast improvement in the ability of mobile devices to show high quality images
  • Improvement in the mobile internet connectivity speeds with 4G and above
  • Improvement in compression formats with introduction of webp
  • Advancement in responsive UI technology ranging from HTML picture tag and srcset attribute to PWAs that increase the application awareness of the ability of the mobile device

Through PWAs in particular, the app knows enough of the device to decide what pixel ratio as well as width of the image to get, thereby improving displayed image quality to the best possible.

PWA in Magento is now live. To see a curated list of modern eCommerce sites see https://headless.page/.

Analyzing www.hartsofstur.com for image quality

I picked a magento site and analysed for image quality. The site is the beautifully made. The site looks like a standard magento website – except that on click, the page does not seem to refresh (though the address bar changes URL) – only the content does. That is nicely done.

This site is a PWA created using Deity Falcon – a headless open source library built with ReactJS, NodeJS and GraphQL.

The Category Page

When you navigate to a category listing page from the menu, the category view uses API to get details including images.

The API URL for a category looks like this https://www.hartsofstur.com/api/catalog/products/57. (A real URL has layered navigation and other filters appended).

Such a URL returns category information as json. Let us review a snippet below :

See the “thumnail_url” (3 lines below the highlight) in the json is a standard Magento image URL from the cache. The image comes from this API for each element. The thumbnail URL is not device aware as no hint of the device was passed to the category API.

The website seems to return the same image URL (image size 600×600) for any sized device as shown below.

600×600 image is resized to 165×165 in html

600×600 image is resized to 250×250 in html

Device Pixel Ratios


Using higher Device Pixel Ratios, todays mobile phones give a very high quality image display experience to users. HTML tags have been enhanced to allow their usage. However, many eCommerce sites do not use such features.

No CDN

The other notable hosting of this website is that it does not use a CDN. Use of CDN is every more important for PWAs.

Developing PWA for Magento?

Talk to us about how our hosting stack which includes a image generator, can improve the quality of images you display.

Here is how our stack compares with default Magento for image generation.

  Magento luroConnect
When is an image generated? Pre 2.3, synchronous with frontend

2.3, synchronous with product save

Asynchronous – queued on product save
Webp support No support Supported – needs img tag change
High resolution 2x, 3x, Retina display Possible with img tag change – but makes synchronous problem worse Possible with img tag change – asynchronously generated
Other image optimizations Not possible In the roadmap
such as progressive jpeg

During the demo we will show you how the luroConnect stack can help your website.

  • Improve the quality of images displayed on your website.
  • Block any malacious hits to the site
  • Divert admin traffic to another server
  • Show what a Disaster Recovery Plan can do
  • See site performance on a dashboard
  • Review your current server size and suggest improvements to gain performance and/or reduce costs of hosting (a $100 value)

404 when moving to Magento 2

When migrating platforms for example from WooCommerce to Magento or from Magento 1 to Magento 2, it is imperative that we move all URLs  to ensure proper SEO authority is retained after the move as well as real users get a smooth Customer Experience (CX).

If it is not possible to maintain the same URLs, ensure redirects are made. This is especially true of a move from WooCommerce and also true when as part of the move the store is reorganized. When migrating data, an automated custom URL redirect process is recommended.

In spite of the best planning and intentions it is likely some URLs may be missed and redirects may not be setup.

Failing to do so has 2 negative effects

  • BOTS, especially Google may visit the new site with the older URL and receive a 404. This can reduce your site rating on Google.
  • Users that may have bookmarked or use browser history may get a 404 and get a bad CX resulting in visitors bouncing and reducing brand value.

Google Analytics does not show 404s. Google search console may show if the count is high. Neither of these is a reliable source to know what was missed.

Only the server knows for sure it severed a 404 and would have logged it in the apache or nginx access and/or error log files. On a new migration we recommend automating a 404 report from the log file, atleast once a day.

If using our luroConnect / Edge product, the dashboard can be used to setup a real-time alert for any 404 returned. This can then be actioned by developers or the agency to include a redirect.

Watch our webinar on performance and scaling in Magento

Its free!

Using analogy to vehicular traffic we explain performance and scaling in Magento.
Key takeaways

  • Know how to compare hosting options
  • Importance of good code
  • How to scale
  • Tuning Magento

We can analyze your site for free

Schedule a call

Not happy with your website performance and want an expert to look at it?

  • We will analyze your site using public information.
  • We will ask you to give us a 1 day web server log file.
  • We will try to identify what steps if any you should take to improve your sites performance goals.

Book Review : The Choice Factory by Richard Shotton

Book Review : The Choice Factory: 25 behavioural biases that influence what we buy
Author : Richard Shotton
Review of podcast interview on Everyone Hates Marketers podcast

Building a brand is tough – what’s worse is that destroying a brand is easy. Standing on the top of the roof and shouting (ok, nationwide TV ads on a major sporting event) does not build a brand with the message you want.

People have biases and they react to the message. Effectively brand is what someone perceives.

The Choice Factory: 25 Behavioral Biases That Influence What We Buy by Richard Shotton is a great book. Richard was recently interviewed on the “Everyone hates Marketers” podcast. Here is a summary of the podcast where they discussed 3 of the 25 biases. The podcast was very interesting overall, but here is the brief as a review of the biases.

Bias 1 : The pratfall effect

A pratfall is defined by Merriam Webster dictionary as “a humiliating mishap or blunder”. It’s the idea that if you admit a weakness or you exhibit a flaw you become more appealing.

Examples : VW. They went out and said, “Ugly is skin deep.” So, admit they’re ugly.
Listerine, the taste you hate twice a day. They admitted they taste awful.
Guinness, good things come to those who wait. They admitted they were slow.

It can be applied to reviews : Northwestern University did a study in 2015 … “As the product review gets better, likelihood to purchase increases until it hits a tipping point. So ,somewhere between 4.2 and 4.4 out of five. And then after that point, if the review gets any better then likelihood to purchase declines.”

Perfection was not trusted. It was seen as too good to be true. Consumers didn’t trust perfection. Most brands do not use this effect and hence people don’t trust brands.

Bias 2 : Confirmation bias.

It’s the idea that people are very good at maintaining their existing point of view. For example, if you dislike a brand and you hear a message from them. Your brain can generate counter argument after counter argument which maintains it’s existing point of view. It just doesn’t agree with the new information.

Research has shown that distraction is a good tactic to move away from confirmation bias.

Example :
“In the late-80s British Airways were struggling with perceptions of quality.

What they didn’t just do is talk about how they had amazing stewards and stewardesses, great big seats. What they did was always accompany their ads with this wonderfully evocative piece of classical music. Now there is no logical argument about quality. Therefore the brain doesn’t come up with a list of counter-arguments.“

Bias 3 : Habit

“what are the moments when people’s habits become destabilized? And therefore as an advertiser you can persuade them. ‘Cause there’s an argument that’s there’s a huge amount of complexity in life.”

People buy brands that they always do. To make them change is not persuasion as they are not even considering the brand.

A tactic discussed is “people’s habits become destabilized just after they undergo a life event. So what I mean by that — a life event is getting married, divorced, retired, moving house.”

During a research conducted by the author “what we found was that on pretty much every life event and every product we looked at people between two and three times more likely to try a new brand in that short window after they had undergone a life event.”

So, as a brand think of the moments or events in your customers’ lives that you can introduce your brand or “putting a disproportionate influence on people when they are entering your category” or personalization.

However, he cautions about over personalization “If it tries to target different messages to different people. To begin with, that will look amazing. It will have a great effect but sooner or later we ill overhear those messages and therefore we’ll know that it stands for nothing or that it can stand for many different things. So it’s value is a signal to other people about what we believe in no longer stands.”


If you liked the review or book please post your comments . If you want to see more reviews, let us know on the chat.

luroConnect – Alluring connection to your customers – improves online perception of brands by delivering an Awesome Digital Experience. Our manged hosting service helps with performance, scale and security of your web properties (blog, website, app backend) backed by a 24×7 support.

We can analyze your site for free

Schedule a call

Not happy with your website performance and want an expert to look at it?

  • We will analyze your site using public information.
  • We will ask you to give us a 1 day web server log file.
  • We will try to identify what steps if any you should take to improve your sites performance goals.