Uncategorized

Improving Headless load times – Server Side Rendering Update

PWA / Headless with Magento.

PWA and Headless started with a lot of hype and promises. Many websites have benefited from successful implementations that proves many promises were fulfilled.

However, one of the biggest drawbacks of a headless / PWA implementation of a Magento website is the page load times - especially as reported by tools and metrics not tuned for the benefits.

A key benefit headless offers is improved navigation experience, but that is not measured, even by some Real User Monitoring (RUM) tools.

Headless makes sense - almost imperative - in many cases such as a large changing catalog (we have a customer with 700k products with 1k new product uploads and additional 1k product updates daily).

While navigating, a graphql is responsible for updating the page with content and it returns only the data asked for, which can be cached across all users. For uncached catalog graphqls, the Elastic Search is most crucial part of the system for performance, but a lot of php formatting such a the menu, that went along with it is now eliminated.

This has to be a better system. So, where does it go wrong?

Issues with headless that causes speed problems

  • Large js (the application) that needs to be loaded before any content can be painted
  • Too many graphql fragments. A framework has to decide what graphql is too small that the number of graphql calls will increase. PWAStudio is a prime negative example.
  • Session locks in Magento graphqls
  • Use of POST graphqls which are not easily cacheble in varnish. Traditionally graphqls are served in POST to avoid hitting url size limits. URL are limited by the browser with 2048 considered a limit at the lower end. But, most Magento frontends generate URLs much smaller than the 2k limit and can easily use GET for queries and

These reasons and SEO requires a good implementation of SSR - Server Side Rendering

What is Server Side Rendering (SSR)

  • A headless implementation is written with a browser in mind - one that can load javascript and render the content. This is Client Side Rendering.
  • A Server Side Rendering will render the page on the server side - much like a regular Magento theme might do - and return HTML

SSR has 2 flavors

  • SSR that serves HTML without the javascript. This HTML will have no interactivity. Usable only for SEO purposes.
  • SSR that "hydrates" the HTML on the browser - by adding the events to the html and also perhaps bootstraps the application itself so it is interactive

React 18 and SSR

Headless technologies written with React have an opportunity to leapfrog the site performance by upgrading to React 18 and using the SSR features

  1. Suspense component. This allows specifying a part of the UI to be in loading state. Through this react knows the page is yet being loaded.
  2. renderToPipeableStream API. Allows HTML to be sent as a nodejs stream.
  3. hydrateRoot API. Allows the react application to be attached to the rendered HTML.

These 2 features basically allow the server side to keep rendering the application as the HTML is being returned, allowing for asynchronous rendering of components. The 3rd feature is useful when the destination is a browser - the rendered HTML is now made useful.

Current state of some popular Magento PWA technologies

  1. PWAStudio : Currently is on react 16 and does not include any SSR component.
  2. ScandiPWA : Version 6.2 and above are on react 18. Out of the box ScandiPWA does not support SSR, but it should not be very difficult to add the ssr route.
  3. VSF1 : Is based on vue and supports an inbuilt SSR
  4. VSF2 : Has an inbuilt SSR

Would you like to switch to a modern hosting platform?

Schedule a call of a free evaluation!

With features like ~0 downtime code deploy and autoscale to reduce your hosting costs, luroConnect offers you unparalleled hosting environment for Magento.

Schedule a call and we will show you how we can

  • Improve your hosting, possibly with autoscale
  • Have a managed dev, staging and production environment
  • Server performance measured every minute with alerts for a slowdown
  • A multi point health check every day
  • Optimized hosting costs

Interview with Carmen Bremen, Mage-One

Carmen Bremen is an accomplished Magento developer and community member for many years. She is a certified developer and she has been awarded Magento Master She is one of the founding members of Mage-One - a paid long-term-support (LTS) for Magento 1. We chatted with her over email recently.

luroConnect : Tell us about yourself (how many years in Magento, feathers in your cap)
Carmen : My name is Carmen and I met Magento in 2010. At first I thought: I'll never learn this, took the challenge and didn't want to give up Magento afterwards. Now Adobe unfortunately announced the end of Magento 1 and maybe that's why I became part of Mage One. I just don't want to give up Magento 1 so fast :-)
luroConnect : You must have researched and talked to merchants about M1 end-of-life. Can you summarize your finding?
Carmen : Many merchants would like to continue using their Magento 1 shop a little longer. Some have not yet decided which system they want to switch to. Others think that the investment in the shop has to pay off first and others are so used to Magento 1 that they just want to continue using it.
luroConnect : What type of patches does one expect from mage-one?
Carmen : We will offer security and compatibility patches. We will not develop new features or remove old Magento bugs.
luroConnect : Php 7.2 EOL is nearing – will you deliver patches for php version upgrades?
Carmen : Of course. The first patch we release will be the one for PHP 7.3.
luroConnect : What type of tests do you run before you release your patches?
Carmen : We use Cypress for our tests and have been working on numerous different tests for weeks.
luroConnect : How many years of mage-one support can we expect?
Carmen : We intend to offer our service for at least 5 years.
luroConnect : I know this is not legal advice I am asking, but for merchants who process payments by Paypal or Visa, a big question is what would happen after June 2020? Will they lose PCI compliance? (Refer resources if you can). 
Will mage-one satisfy the 6.2 PCI rule.
Carmen : From our perspective 100%. We offer a certificate in our customer account that can be presented to payment providers. Nevertheless, the final decision is up to the payment provider itself. However, we already have a statement from PayPal, for example, that you can continue using PayPal with Mage One.

luroConnect Support for Magento 1 past EOL

We have a 4-point plan to support you. Signup Now and we will contact you. No credit card required.

From EUR 29

Mage-One Patches for post EOL Magento 1 support. Will satisfy PCI vendor requirement.

mage-one logo

From EUR 99

Sansec eComscan examines a store for malware, vulnerabilities and unauthorized accounts. Written by well known security expert Willien De Groot, it scans files, databases and 3rd party components of Magento.

external

From USD 50*

Inbuilt into our Nginx, with M1 rules, protects from OWASP Top 10, with the ability of virtual patching.

From USD 50*

Staging environment to ensure patches are tested before taken live.

Signup now! And we will be in touch with you. No credit card required.

How we hosted a big sale on Magento without a hitch

During an online sale that includes promotion to a list, brands worry about downtime due to “unexpected high load”. We believe that while every system has a rated limit, there can be flexibility added to your infrastructure. With some planning, including autoscale and tuning, a successful sale event can be a very likely outcome. See how we handled one such sale without a hitch.

In our experience promotions via SMS generates very high peaks. In this article we will show our recent experience with a sale day that generated a peak of 30 hits per second to Magento.

external

The sale

On Saturday 21st March, 2020 one of our customers had a sale event. The merchant – a fashion retail brand - is an omni channel retailer in India with about 145 stores across 45 cities. There have been previous sale events, but this one was different as most stores were closed due to the corona virus measures in place. This meant that more customers were likely to shop online than visit the store.

The sale was promoted via sms and email. The SMS list was about 1 million. Promotional messages were sent in batches of 200k every half hour. Marketing was to start at 10 AM local time.

Monitoring the website

We monitor the nginx log file, analyze and display information on our dashboard. A key component of our dashboard is the plot of number of hits / minute (we count only php hits, static hits like css, js and images are excluded). We also calculate avg response time per minute as well as standard deviation per minute. Both of these have a unit of seconds. We also have alerts setup for many parameters including slowdown and a 5xx error response.

The graph below is for a 24 hour period, including the sale time, which lasted about 5 hours with the effect keeping the site more busy than usual afterwards. Note these are server logs graphs – they look quite different from google analytics!

The traffic & response

The graph gives as an idea of the traffic to the website. The turquoise line is hits per minute served by php.

  • Traffic peaked a about 2000 hits per minute – about 30 hits per second.
  • 3476 items were added to cart during the sale.
  • Average server response was mostly below 1 second.
  • There were 3 errors – all of them related to checkout function (“This payment method is not available”)

Mysql query cache and response time

The store is a Magento 1 website. Just as the traffic / hits to the website started peaking, we got a slow alert – represented by the large black line close to the first traffic peak around 10.45 am in the previous figure. Our prior experience with the website showed that the mysql query cache plays a role in the site performance. Before the sale started, mysql query cache was turned off. On seeing the slowdown alert, we turned on the mysql query cache and saw an immediate improvement in performance. Both avg response per minute and standard deviation per minute improved – for some time.

However, standard deviation – higher number indicates some visitors faced slower response to the website than others – deteriorated with time. After the sale (10pm local time), mysql query cache was turned off – resulting in improved standard deviation, but slightly worse avg response time.

AWS autoscale

The following graph plots AWS autoscale instances over the same 24 hour period. A time based minimum 5 instances was planned during the sale from 9.15 AM to 7.30PM. At the peak 7 instances were required – our autoscale policy adds 2 instances and removes 1 at a time.

luroConnect Architecture for AWS and controls

Our standard scalable architecture uses nginx as a load balancer. The communication between the load balancer to php servers is using fcgi – individual php servers do not have nginx. A NFS server is used to share folders that need sharing – media and var folders in Magento 1. Magento cache and sessions are served from redis. Code is also shared using nfs.

A custom lambda function communicates between autoscale lifecycle hooks and the nginx load balancer adding and remove instances as the AWS autoscale policy decides.

Our cache controls including the ability to turn on and off mysql cache, resize memory allocated to redis cache, resize memory allocated to php-fpm opcache.

Interested in knowing about the advanced architecture of luroConnect?

Fill the form below and we will contact you.

    My company owns the Magento site Yes, I am a developer I represent a Magento Agency

    Meet Magento India 2020

    luroConnect is a bronze sponsor at Meet Magento India 2020

    Look out for our invitations and announcements closer to the date of the event.

    In the meanwhile, you can setup a time for a demo.

    We can analyze your site for free

    Schedule a call

    Not happy with your website performance and want an expert to look at it?

    • We will analyze your site using public information.
    • We will ask you to give us a 1 day web server log file.
    • We will try to identify what steps if any you should take to improve your sites performance goals.

    Book Review : The Choice Factory by Richard Shotton

    Book Review : The Choice Factory: 25 behavioural biases that influence what we buy
    Author : Richard Shotton
    Review of podcast interview on Everyone Hates Marketers podcast

    Building a brand is tough – what’s worse is that destroying a brand is easy. Standing on the top of the roof and shouting (ok, nationwide TV ads on a major sporting event) does not build a brand with the message you want.

    People have biases and they react to the message. Effectively brand is what someone perceives.

    The Choice Factory: 25 Behavioral Biases That Influence What We Buy by Richard Shotton is a great book. Richard was recently interviewed on the “Everyone hates Marketers” podcast. Here is a summary of the podcast where they discussed 3 of the 25 biases. The podcast was very interesting overall, but here is the brief as a review of the biases.

    Bias 1 : The pratfall effect

    A pratfall is defined by Merriam Webster dictionary as “a humiliating mishap or blunder”. It’s the idea that if you admit a weakness or you exhibit a flaw you become more appealing.

    Examples : VW. They went out and said, “Ugly is skin deep.” So, admit they’re ugly.
    Listerine, the taste you hate twice a day. They admitted they taste awful.
    Guinness, good things come to those who wait. They admitted they were slow.

    It can be applied to reviews : Northwestern University did a study in 2015 … “As the product review gets better, likelihood to purchase increases until it hits a tipping point. So ,somewhere between 4.2 and 4.4 out of five. And then after that point, if the review gets any better then likelihood to purchase declines.”

    Perfection was not trusted. It was seen as too good to be true. Consumers didn’t trust perfection. Most brands do not use this effect and hence people don’t trust brands.

    Bias 2 : Confirmation bias.

    It’s the idea that people are very good at maintaining their existing point of view. For example, if you dislike a brand and you hear a message from them. Your brain can generate counter argument after counter argument which maintains it’s existing point of view. It just doesn’t agree with the new information.

    Research has shown that distraction is a good tactic to move away from confirmation bias.

    Example :
    “In the late-80s British Airways were struggling with perceptions of quality.

    What they didn’t just do is talk about how they had amazing stewards and stewardesses, great big seats. What they did was always accompany their ads with this wonderfully evocative piece of classical music. Now there is no logical argument about quality. Therefore the brain doesn’t come up with a list of counter-arguments.“

    Bias 3 : Habit

    “what are the moments when people’s habits become destabilized? And therefore as an advertiser you can persuade them. ‘Cause there’s an argument that’s there’s a huge amount of complexity in life.”

    People buy brands that they always do. To make them change is not persuasion as they are not even considering the brand.

    A tactic discussed is “people’s habits become destabilized just after they undergo a life event. So what I mean by that — a life event is getting married, divorced, retired, moving house.”

    During a research conducted by the author “what we found was that on pretty much every life event and every product we looked at people between two and three times more likely to try a new brand in that short window after they had undergone a life event.”

    So, as a brand think of the moments or events in your customers’ lives that you can introduce your brand or “putting a disproportionate influence on people when they are entering your category” or personalization.

    However, he cautions about over personalization “If it tries to target different messages to different people. To begin with, that will look amazing. It will have a great effect but sooner or later we ill overhear those messages and therefore we’ll know that it stands for nothing or that it can stand for many different things. So it’s value is a signal to other people about what we believe in no longer stands.”


    If you liked the review or book please post your comments . If you want to see more reviews, let us know on the chat.

    luroConnect – Alluring connection to your customers - improves online perception of brands by delivering an Awesome Digital Experience. Our manged hosting service helps with performance, scale and security of your web properties (blog, website, app backend) backed by a 24x7 support.

    We can analyze your site for free

    Schedule a call

    Not happy with your website performance and want an expert to look at it?

    • We will analyze your site using public information.
    • We will ask you to give us a 1 day web server log file.
    • We will try to identify what steps if any you should take to improve your sites performance goals.

    5 Questions to ask before selecting a Magento Managed Hosting Provider

    5 Questions to ask your Magento Managed Service Provider

    1. Does the managed provider help you with setup of a CDN?
      Content Delivery Networks or CDN is the best way to get a boost of your page load time. A CDN takes away the load on your server by caching commonly used files that do not change frequently. Such files include css, js and image files. For a Magento store, images constitute a large amount of time and bandwidth of a page.
      Large CDNs have an additional advantage of having multiple end points or servers near your customers. This reduces latency and images tend to appear immediately.
      The good news is that CDNs are now a commodity – they are aggressively priced with some large CDNs having unlimited free plans.
      The only catch is that you need to “invalidate” or “purge” cached content from the CDN if you were to change the same file on the live server. Until automated, this procedure can be a burden.
      Your Magento Managed Hosting Provider should provide you with practical and complete options for a real CDNs, including help in configuring your Magento site to use your selection
    2. Does the provider have a disaster recovery plan – not just a backup?
      Backup is as good as the restore process.
      Yes! Just having backup is a essential but not sufficient.
      If your site were to be hit by a disaster, how would you be able to bring up your site on another server? Would all services start as desired?
      Funnily we have seen many magento servers do not come back to full service on a simple reboot, far less a disaster.
      Moreover, backups should be taken “off-site” – not within the same zone or provider you currently have.
      And your Managed Magento Hosting provider should give you a written disaster recovery plan based on different types of scenarios and that you can put into motion if the time arises.
    3. Does your provider regularly evaluate and tune for optimal cache performance?
      Magento site performance is quite often driven by the condition of its various caches. Caches store results in RAM vs either on disk or having to compute the results. As a quick look at the Cache Management page on Magento’s admin panel shows there are many application level caches. In addition, mysql stores query results in a in-memory cache, however requires the cache to be sized and enabled in a configuration file. The php code can be stored in a opcode cache in the php server. This also requires memory allocation and enablement.
      Since no two magento stores are the same, there is no standard configuration. It requires observation and tuning.
      Your Magento Managed Hosting provider should give you information and tune the cache, possibly even advising you with server sizing in the process.
    4. What scalability plan has the provider made for you?
      As hits increase, it may be required for your site to scale. But, how will you even know the time has come?
      In todays world, a slowdown of your Magento site is equivalent of downtime. Consider this – each drop of a second in your load time results in drop of visitors.
      If traffic spurted due to a online or offline ad, how would you know the server capacity needed to handle the spurt? Do you need autoscaling and loadbalancing technologies?
      Your Magento Managed Service provider should be on your side in advising
    5. Do you know the deliverability of your transactional emails?
      An eCommerce site sends many emails – new user registration, password change, order confirmation, abandoned cart. These emails need high rate of deliverability. If emails are directly delivered from the host, there are 2 problems - It is not known which emails are delivered and which are not. amd secondly mail servers are constantly upgrading their needs of the senders ability to send, resulting in increased non deliverabilityMany services are available to handle such emails – Amazon SES, Mandrill, Sendgrid to name a few. Each give a varying level of granularity of reports and pricing plans to support your volume.Most of these services have Magento Plugins, some premium. However, a email plugin will attempt to send the email “inline” – for example the order email is sent while the customer is waiting for the success page. A better solution is to send the email to the local system queue, and let the system process the queue by sending the email to the transaction email provider.

      Your Managed Magento Service provider should be able to configure your server’s email system to use your chosen transaction email provider?