performance

Meet Magento India 2023 – Speaker Talk – Improve PWA Performance

On Feb 3, 2023, our CEO Pradip Shah presented a talk titled “Improving Headless / PWA Performance” at Meet Magento India in Mumbai, India.

The premise of the talk was the observation that the Headless / PWA eCommerce world promised lightning fast websites but many have ended up with slow performance, especially as measured by Google’s metrics.

This talk discussed issues and solutions that have been implemented in real eCommerce websites to increase page speed.

There are 2 versions of this for you to download

  1. The pdf version (5MB)
  2. The powerpoint version (23MB) – it has a few videos, and hence recommended to download.

Summary

Section 1 : Legacy HTML page technology is not stagnant – with Hyva and on page optimization from technologies like Nitrogen, page speeed scores of Magento traditional HTML page has improved. However, many traditional websites suffer from slower response speeds with uncached pages. Example : www.kalkifashion.com is a Hyva theme with Nigrogen front end caching and optimizaion.

Section 2 : PWA is not magic – the promise of “lightning fast” website needs work. Basic page speed principles apply and so do some new ones.

Section 3 : It is possible to make websites using Magento REST and Graphql. Each one has different challenges. REST by default does not cache and Graphql uses sessions.

Section 4 : When using graphql, avoid POST graphql for queries – use them only for mutations.

Section 5 : Study your application and you can add caching to Graphql’s that Magento may mark as not caceheable.

Section 6 : Custom graphql queries result in database queries that may need optimization. A step by step method to get the underlying mysql query generated is a useful process. We used that to achieve great results. www.mysoresareeudyog.com has a custom default filter for a listing page – latest.

Section 7 : Magento 2.4.5 allows method to prevent session locking using authentication. But by default, authenticated graphql calls are not cached – an approch similar to one presented earlier will be required to ensure optimal caching is achieved.

Section 8 : Go beyond. Add logic in frontend and cache GraphQL results in the browser. This is the ultimate power of headless / PWA. It may not help with page scores that are measured without browser cache, but user experience will improve.

Section 9 : Bulid can improve page loads. Appropriate sized chunks and using hashes instead of version. Another improvement build can give is use of cookie free domains. A static domain for example for js and css bundles allows the main domain to not go through a caching layer, improving page load speed.

Would you like to switch to a modern hosting platform?

Schedule a call of a free evaluation!

With features like ~0 downtime code deploy and autoscale to reduce your hosting costs, luroConnect offers you unparalleled hosting environment for Magento.

Schedule a call and we will show you how we can

  • Improve your hosting, possibly with autoscale
  • Have a managed dev, staging and production environment
  • Server performance measured every minute with alerts for a slowdown
  • A multi point health check every day
  • Optimized hosting costs
Click here to schedule the call
March 3, 2023 By Magento, page speed, performance, PWA, Site Speed 0 Comments Read more...

Improve page speed with Hyvä Themes

Modern Page Speed Metrics

Modern Page Speed Metrics as defined by Google are called “Core Web Vitals“.
This measure focusses on 3 aspects of user experience.

Largest Contentful Paint or LCP that measures loading performance.
Content Layout Shift or CLS that measures visual stability.
First Input Delay or FID that measures interactivity.

This obsoletes previous measures such as First Contentful Paint (FCP) or even Page load time reported by many measuring tools.

Core Web Vitals

Site speed has evolved and needs an evolution in the technology stack

The change in metrics means the technology stack has to evolve.
Many storefronts have adopted a more agile approach moving to a headless architecture.

Deploying headless adds complexity – for Magento stores, you now are dealing with two or more builds – Magento and headless. (Some headless stacks may need more builds). As a result, a headless stack is not for all stores.

There is nothing wrong with a monolithic stack – however, the Magento theme uses rather old technology that has aged.

The frontend technology world has accepted Adam Wathan‘s tailwind css as a great solution for both easing frontend development and improve page speed.

Much like jquery was the javascript library of choice for older UI, alpine.js is the library of choice for user interfaces developed using tailwaindcss.

What is Hyvä?

Hyvä Themes is a Magento theme based on tailwaindcss and alipe.js. Much like the Adam Wathan with tailwindcss, Willem Wigman the pioneer of Hyvä, believes in simplicity and componentization – making it ready for the future.

Hyvä Themes is not based on Magento’s default Luma. It is completely rewritten with modern a UI framework. The results are astonishing UI with a better page score.

Moreover, moving to Hyvä Themes is a re-theming project. Most of your existing functionality will work as is – until a module was written to be dependent on the older theme. A large website was moved by our partners Aureate Labs in under 3 months.

luroConnect partners with Hyvä

luroConnect will support Hyvä build as part of our standard Magento build process.
The luroConnect build is generally done in a dockerized container without access to the database. This ensures our build does not consume resources of the live site while running composer, setup:di:compile and static:content:deploy commands. It also makes our build easy to deploy and rollback.

luroConnect is a Managed Hosting Platform for Magento – giving a development, staging and live environment on your cloud account. With infrastructure-as-code principle used in its stack and a CI/CD inbuilt into the platform, luroConnect gives unprecedented confidence of the hosting environment on any cloud – and now use our kubernetes stack option.

Limited Time Offer for Hyvä customers

Signup with us between Jan 1, 2023 and March 31, 2023, with a go-live date before May 1, 2023 with Hyvä and get
first 3 months free of luroConnect fees (US $900 – US $3000 value depending on the plan chosen), if customer signs up for a 1 yr contract with luroConnect.

Signup with the consultation link below and see how kalkifashion.com with Hyvä is hosted on luroConnect.

Would you like to switch to a modern hosting platform?

Schedule a call of a free evaluation!

With features like ~0 downtime code deploy and autoscale to reduce your hosting costs, luroConnect offers you unparalleled hosting environment for Magento. Now with out-of-the-box support for Hyva.

Schedule a call and we will show you how we can

  • Improve your hosting, possibly with autoscale
  • Have a managed dev, staging and production environment
  • Server performance measured every minute with alerts for a slowdown
  • A multi point health check every day
  • Optimized hosting costs
Click here to schedule the call
October 6, 2022 By Core Web Vitals, page speed, performance, Site Speed 0 Comments Read more...

Magento and graphql : A slowdown saga

PWA, Magento and graphql : A slowdown saga

Note : I write this article as Magento 2.4.5 is released and fixes some major performance issues that have resulted in the dream of headless and PWA go sour for many early adopters of technologies like PWA Studio and Vue Storefront 2.

PWAs and headless came with the promise of fast navigation and have had a rough ride at best, especially with Magento backends. Early PWAs like Vue Storefront 1 were implemented using Magento Rest API with a api middleware in nodejs that used elastissearch to store cached version of magento catalog for faster access. Magento implemented Graphql to avoid the need for the rest api. Use of direct rest api is ill-advised without a middleware as it exposes a Magento admin token. A middleware would sanitize the request possibly by checking a cookie and also limit the range of queries run. Graphql prevented the use of admin tokens and instead relied on frontend cookies for authentication.Magento’s REST API also results in large datasets. Graphql has a powerful, standardized query syntax to theoretically present a path breaking solution to build Magento headless around. However, early implementations did not fully cover the entire magento data model – Magento even advised the use of REST api when graphql was not sufficient. And when real world websites with large builds and catalogs were deployed, performance issues emerged.

Graphql are ajax calls in Magento that use sessions. In a typical hit processed by Magento, a session is opened, the hit is processed and then the session is closed. (A hit here is either a frontend hit or an ajax call).

Sessions and race conditions

Graphql are ajax calls in Magento that use sessions. In a typical hit processed by Magento, a session is opened, the hit is processed and then the session is closed. A hit is either a frontend hit or an ajax call.

Magento can store information in sessions – cart data for example. In order to prevent 2 hits from writing contents to the session simultaneously – called a race condition, sessions are locked for the duration of the processing a hit. (Take for example a checkout flow and let us assume that the address and payment info is written to session. You would want to ensure both get written. If both “raced”, only one piece of information would be written). Magento identifies an internal session by setting a cookie in the visitors browser. Each hit carries the cookie allowing magento to identify the session. Since a session is unqiue for a visitor, so Magento can process process only one hit per visitor at any given time.

With ajax in general and graphql in particular, many simultaneous hits may go to the server. Magento, through the session locking mechanism, queues them up. The hits are not guaranteed to be processed in the order they were received or any order in particular.

A typical session content in redis looks like

hgetall sess_eba681783f7000bc668cc45005aeaca4
1) "lock"
2) "0"
3) "writes"
4) "5"
5) "data"
6) "_session_validator_data|a:4:{s:11:\"remote_addr\";s:14:\"116.58.200.181\";s:8:\"http_via\";s:0:\"\";s:20:\"http_x_forwarded_for\";s:0:\"\";s:15:\"http_user_agent\";s:117:\"Mozilla/5.0 (Linux; Android 10; RMX2030) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Mobile Safari/537.36\";}_session_hosts|a:1:{s:10:\"host.com\";b:1;}default|a:0:{}customer_base|a:2:{s:11:\"customer_id\";N;s:17:\"customer_group_id\";N;}checkout|a:0:{}"
7) "wait"
8) "0"
9) "pid"
10) "10.0.2.197.host.com|3735"
11) "req"
12) "POST host.com/graphql"
Most of these entries are used internally by the redis session module. Entry 6 is the main session, stored in php serialized format.

If each hit takes only a few milliseconds and the number of ajax calls are not high, the overall response time may not perceptibly slow the site.

But, if even one ajax hit takes time, the page load can slow down. If a customer opens multiple tabs or refreshes the page, the slowdown may be more severe, sometimes leading to errors.

Here is an example. We are analyzing server logs based on IP. Lines 1 and 2 indicate the same link in google was clicked twice – possibly in 2 tabs on the browser. As the page loads we see graphql’s being loaded. The last 2 items are an indication of the problem.

The 2nd last hit is the slow hit – it took 10 seconds to return a result as the item was not in cache. The problem in the last hit was that it was waiting for the 2nd last hit to be processed, although in all probability it was in cache. If the number of items in this queue grows above 6 – for example if a visitor opens tabs and accesses urls not in varnish, a 500 error may be returned, possibly resulting in the UI showing a block not fully loaded at all. A refresh typically seems to solve the problem.

Server can run out of php resources

When too many graphql calls wait, server side can run short of php resources. This time you can see 504 and 502 errors. Adding more php resources for the same server risks running out of RAM. Adding more servers does help with the problem, but that is a waste of resources.

Graphql and caching issues

Graphql in Magento needs many caches to perform well. However, upon cache clear the “galloping horses” problem can bring down a busy website.

Upon cache clear the first access to graphql results in the schema being read. The performance for this action has improved substantially since Magento 2.3, but we yet highlight it as a concern. A cache warming of this is preferred.

Queries like filter in a category page can be slow and hence caching and warming some queries is always a good idea.

Enabling varnish cache is also important – varnish does not use sessions.

Recommended way to use Magento graphql

Graphql needs authorization – it needs to know what customer is asking for the information. It needs it so that the results include say for example the correct tiered pricing for this customer. While sessions include this information, they also store state and hence need the protection of a lock.

Magento supports JWT customer tokens for graphql.
https://devdocs.magento.com/guides/v2.4/graphql/mutations/generate-customer-token.html

Use the generateCustomerToken mutation to create a new customer token.
Example :

mutation {
  generateCustomerToken(
    email: "bobloblaw@example.com"
    password: "b0bl0bl@w"
  )
}

Response

{
  "data": {
    "generateCustomerToken": {
      "token": "ar4116zozoagxty1xjn4lj13kim36r6x"
    }
  }
}

Magento suggests NOT enabling cookie and JWT token authentication at the same time to reduce the chances of encountering problems caused by the differences between the two authorization methods. However, disabling cookie was added a feature only in Magento 2.4.5

However, Magento 2.4.5 varnish sample configuration disables varnish caching for logged in user. This can reduce performance.

Conclusion

Magento performance continues to be a troublesome aspect of the eCommerce platform. At luroConnect we look at performance in great depths.

Would you like to switch to a modern hosting platform?

Schedule a call of a free evaluation!

With features like ~0 downtime code deploy and many scaling options to reduce your hosting costs, luroConnect offers you unparalleled hosting environment for eCommerce – specifically Magento and various PWA / Headless technologies.

Schedule a call and we will show you how we can

  • Improve your hosting, possibly with autoscale
  • Have a managed dev, staging and production environment
  • Server performance measured every minute with alerts for a slowdown
  • A multi point health check every day
  • Optimize hosting costs
Click here to schedule the call
September 12, 2022 By Magento, performance, PWA, Site Speed, varnish 0 Comments Read more...

Varnish, Magento and frequent product updates

In response to Erwin Hofman’s M2 performance post, a reader commented

Varnish and Magento 2

Use of varnish with Magento has been a popular speed increase. So popular, that Magento decided to incorporate in M2.

This has spread endless debates though

  • Would it not have been better to improve the core code quality instead?
  • How often and which pages do we put in cache?
  • How to organize user personalization content? (Magento has answers in terms of ajax and ESI, but I am afraid the documentation is so spread around, many developers and plugin vendors are not sure).

The question raised by the reader is another point of debate – how can a Magento store, with frequent product updates be made ready for using varnish.

Let us see how Magento works with Varnish – or what it means to say M2 supports varnish.

(Refer this article for a detailed analysis.)

Magento cache has always (yup M1 also) supported tags in caches. A typical cache is a key->value pair, typically implemented as a hash table to lookup the key and return the value. In case of a full page cache, the key is made up of the the URL and parts of the HTTP header that forms the request. When the key is found in the hash table lookup operation, it is a hit and the content is returned. When the key is not found, the request is forwarded to magento to process it. When Magento processes it, the key and the content is stored. The content is the entire HTTP response, including HTTP response headers and HTTP body.

When Magento processes a request, it returns in a HTTP response header, a “hint” header called “X-Magento-Tags”. This header has information about what type of content the page holds.

For example, a category page will have tags that represent the category and each product in that category. A product page would have a tag that represents that product.

This allows Magento to selectively clear the Varnish FPC, say when a product’s content is modified, by sending a special header to varnish to PURGE pages that match the product’s tag.

This is a communication between Magento and Varnish.

(Magento is aware of the location of varnish by the “http_cache_hosts” setting, Varnish is aware of Magento backend IP in its acl_purge list).

Indexing

In order to understand how Magento triggers cache clear events, we need to understand indexing.

Magento has many indexes – a mechanism by which changes made to admin are reflected on the frontend (the end user UI). Varnish only needs to be cleared when Magento moves products to frontend.

If index modes are set to update on save, each product save will result in corresponding pages to be removed from varnish. When making large changes such as with a csv upload, this will result in many pages being cleared multiple times. Note that a product save results in both the product as well as all the category pages of the category / categories the product belongs to.

If indexing mode is set to on schedule, a cron task that processes the “index” group is responsible for running indexing. If acceptable to the business, indexing can be set to run every few hours, reducing the rate at which varnish clears.

If you want a no hassles managed hosting, get in touch with luroConnect. We even give you a free assessment of your website.

We can analyze your site for free

Schedule a call

Not happy with your website performance and want an expert to look at it?

  • We will analyze your site using public information.
  • We will ask you to give us a 1 day web server log file.
  • We will try to identify what steps if any you should take to improve your sites performance goals.
Click here to schedule the call
January 27, 2021 By page speed, performance, Site Speed, varnish 0 Comments Read more...

Case Study : How a simple code change had a major impact on website performance

During black Friday sale, one of our customers’ websites was too slow and unusable for long periods. We were embarrassed, but all our analysis pointed to code. As part of our active support, we help with identifying code bottlenecks. We use many techniques including profiling, but a simple technique for a really slow website is analyzing the php slow log.

Observations

We track slowest URLs and chart them on our dashboard. A typical days chart looked like the one below. Most of the slow URLs were category listing page urls.

What is the php slow log?

The PHP slow request log is where PHP records information about any request that takes more than “x” seconds to execute. The number of seconds is configurable in the php-fpm.d/www.conf file.

The PHP slow request log records stack traces (also known as tracebacks) of each slowly executing script at the moment the request passed “x” seconds in execution. For example the following trace was generated for the website we were optimizing. The first line gives the date and time, the php-fpm pool name and the pid of the fpm process. The second line names the script – for Magento it will be the index.php always. The following lines have the memory address, the function name, the file name and the line number. A stack trace always lists the last call first.

[03-Dec-2020 22:54:57]  [pool www] pid 19267
script_filename = /www/usa/index.php
[0x00007ffa5ec21d30] execute() /home2/customer/77faf58/vendor/magento/zendframework1/library/Zend/Db/Statement/Pdo.php:228
[0x00007ffa5ec21c80] _execute() /home2/customer/77faf58/vendor/magento/framework/DB/Statement/Pdo/Mysql.php:93
[0x00007ffa5ec21be0] _execute() /home2/customer/77faf58/vendor/magento/zendframework1/library/Zend/Db/Statement.php:303
[0x00007ffa5ec21b30] execute() /home2/customer/77faf58/vendor/magento/zendframework1/library/Zend/Db/Adapter/Abstract.php:480
[0x00007ffa5ec21aa0] query() /home2/customer/77faf58/vendor/magento/zendframework1/library/Zend/Db/Adapter/Pdo/Abstract.php:238
[0x00007ffa5ec219d0] query() /home2/customer/77faf58/vendor/magento/framework/DB/Adapter/Pdo/Mysql.php:541
[0x00007ffa5ec218a0] _query() /home2/customer/77faf58/vendor/magento/framework/DB/Adapter/Pdo/Mysql.php:615
[0x00007ffa5ec21810] query() /home2/customer/77faf58/vendor/magento/zendframework1/library/Zend/Db/Adapter/Abstract.php:737
[0x00007ffa5ec21760] fetchAll() /home2/customer/77faf58/vendor/magento/framework/Data/Collection/Db/FetchStrategy/Query.php:21
[0x00007ffa5ec216e0] fetchAll() /home2/customer/77faf58/vendor/magento/framework/DB/Query.php:182
[0x00007ffa5ec21660] fetchAll() /home2/customer/77faf58/vendor/magento/framework/Data/AbstractSearchResult.php:233
[0x00007ffa5ec215c0] load() /home2/customer/77faf58/vendor/magento/framework/Data/AbstractSearchResult.php:107
[0x00007ffa5ec21550] getItems() /home2/customer/77faf58/vendor/magento/module-catalog-inventory/Model/StockRegistryProvider.php:143
[0x00007ffa5ec21440] getStockItem() /home2/customer/77faf58/vendor/magento/module-catalog-inventory/Model/StockRegistry.php:88
[0x00007ffa5ec213c0] getStockItem() /home2/customer/77faf58/vendor/magento/module-catalog-inventory/Model/Plugin/AfterProductLoad.php:40
[0x00007ffa5ec212b0] afterLoad() /home2/customer/77faf58/vendor/magento/framework/Interception/Interceptor.php:146
[0x00007ffa5ec21150] Magento\Framework\Interception\{closure}() /home2/customer/77faf58/vendor/magento/framework/Interception/Interceptor.php:153
[0x00007ffa5ec21070] ___callPlugins() /home2/customer/77faf58/generated/code/Magento/Catalog/Model/Product/Interceptor.php:65
[0x00007ffa5ec20fe0] load() /home2/customer/77faf58/app/code/Plazathemes/Bestsellerproduct/Block/Bestsellerproduct.php:112
[0x00007ffa5ec20ef0] getBestsellerProduct() /home2/customer/77faf58/app/code/Mageplaza/CatalogPermissions/Observer/ProductLoadAfter.php:79

How do you trace from a hit to a slow log entry

From nginx we can trace to the slow log using a match in the php-fpm access log. We format nginx and php access log in a way that helps tracing.

The nginx log entry that we traced using the format

$remote_addr|$remote_user|$time_local|$request_time|$upstream_response_time|$request|$status|$body_bytes_sent|$http_referer|$http_user_agent|$host|$upstream_addr

127.0.0.1|-|03/Dec/2020:22:55:01 -0600|8.966|8.967|GET /usa/health-and-rejuvenation/cereals-health-mix HTTP/1.1|200|20458|https://www.ishalife.com/usa/yoga-gear/bedding|Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Safari/537.36|www.ishalife.com|10.120.0.7:9000

php access log format used
access.format = %R - %u %t %m %r%Q%q %s %f %{mili}d %{kilo}M %{total}C%% %p

php access log entry to the traced url (matched by time and response time)

10.120.0.2 - 03/Dec/2020:22:54:52 -0600 GET /usa/index.php 200 /home2/customer/www/usa/index.php 8962.264 4096 48.20% 19267

The %p records the process id of php-fpm that processed this hit. Based on the time and process id, the slow log entry can be accurately traced.

Analysis

The stack trace clearly indicated that an observer was called when the event “ProductLoadAfter” was called. For this website, it slowed down the category display as the call was made for each product. A peep into the code in productloadafter observer was loading “Best Seller Products”. So, for each product in the category, it would load the (almost) identical set of best seller products. The observer object was not even used.

Let us look at the code

public function execute(Observer $observer)
    {
        /** @var AbstractCollection $productCollection */
        
           $productCollection = $this->productBest->getBestsellerProduct();
       
           foreach ($productCollection as $key => $product) {
                /** @var product $product */

                $productObject = $this->productRepo->getById($product->getId());
                if ($this->helperData->validateConditionDefault($productObject, ObjectTypeAction::TYPE_PRODUCT)
                    && !$this->helperData->checkIPPassAllRestrict()
                ) {
                    /** @var DataObject $productObject */
                    if ($productObject->getData('mpcp_usecf_hideaction') === '0'
                        && $this->helperData->getProductHideCatAndWidget()) {
                        $productCollection->removeItemByKey($key);
                    }
                    // do not use config hide action
                    if (!$productObject->getData('mpcp_usecf_hideaction')
                        && $productObject->getData('mpcp_hideaction') === '1') {
                        $productCollection->removeItemByKey($key);
                    }
                }
            }
    }

The solution

Remove the observer as apart from badly written, the result was never used on the website. Code analysis revealed 2 unused plugins.

2 unused plugins and 1 observer call was removed, resulting some category pages loading 10x faster!

Before and After stats

We monitor hits in real time and chart hits / minute and avg response time per minute. In the graphs below, the left axis is hits per minute (plotted in turquoise) and the right axis is average response time in seconds per minute (plotted in black).

BEFORE

AFTER

CPU usage of php servers

BEFORE

AFTER

Database CPU usage across multiple days

Conclusion

The importance on good code and possibly the negative effects of unused functionality was highlighted in this case study.

As users of plugins we need to understand the motivation of plugin vendors to pack in substantial functionality leads to undesired side effects, especially on functionality you don’t want to use. Budget to clean up vendor code before using.

December 29, 2020 By case study, performance, Site Speed 0 Comments Read more...

Supercharge your Magento with a Varnish cluster

What is a varnish cluster?

A varnish cluster is a set of varnish nodes, each in a different geographical location, in front of the same Magento backend.

As shown a Magento hosted in the US East region can serve varnish nodes from across the world. Access to the website from each region is directed towards the nearest varnish, benefiting from lower latency and faster page loads.

If you serve customers in different regions – internationally or across the USA, your store can benefit from a varnish cluster.

Varnish and Magento performance

Magento 2 was architected to work with Varnish for improved performance. A typical webpage – category listing or product detail page – when returned from a cached varnish page (called a HIT in varnish) typically has a TTFB of a few milliseconds. An uncached page (also called a MISS in varnish) results in TTFB of a few seconds. Optimizing a MISS is very crucial, but we will not cover that in this article. We believe varnish is one portion of optimizing your website.

What is Network Latency

Latency is the time it takes for a network packet to go from your computer to the server with a request and come back with a response – assuming the response from the server was available immediately. When a page scores a HIT in varnish, the response is almost immediate from the server. Any  Time To First Byte (TTFB) recorded on the browser can be attributed to network latency.

If you market your website to many regions around the world, and you host varnish in a single location, your visitors may be faced with higher latency. When latency of access is in a few hundred milliseconds, it becomes the bottleneck and needs optimization.

The below information from https://wondernetwork.com/pings/ gives an idea of expected latencies. To read the table use the row header as source and column header as destination. So, a ping latency from Los Angeles to Mumbai would be 267 msec.

What is the cause of latency?

Core latency is a function of distance – even light will take ~40msec to travel 13,000 km – the distance from New York to Mumbai. A network packet travels through wires at that speed, and network wires are much longer than a straight line. Moreover, a response packet has to travel all the way back.

Latency is also caused as traffic goes through network equipment. The number of switches / servers a packet has to go through depends on the network and service providers – yours as well as at the server you are connecting to.

A varnish cluster architecture

Using a GeoIP based DNS service such as AWS Route53 a users request for a domain is redirected to one of several IPs. The Magento backend is in the “default” region – where maximum traffic is expected. A varnish in the regions desired – shown US East and Europe below.

Geo-IP based DNS

A geoip based dns router such as AWS route53 can help direct traffic to the nearest varnish node based on the guessing the country the IP requesting name resolution is coming from. So users on a browser say in Australia would be directed to be served from the varnish node in Australia and one from the west states of California would be directed to the varnish there.

Since IP to region or country can never be so accurate, it should be possible for all varnish nodes to serve customers from any region. Specifically, a language or currency switcher should be available on the website.

Magento 2 supports multiple varnish nodes out of the box

An important feature in using a cache in production is its need to automatically and quickly clear contents on demand from the user and application. For example, when a product goes out of stock, the corresponding page should display the out of stock label. Magento uses a tag-based system to flush appropriate content from the cache. Magento allows setting up multiple varnish hosts and a tag-based cache clear is sent to all the hosts.

bin/magento setup:config:set --http-cache-hosts=<varnish internal ip>:6081,<varnish internal ip 2>:6081

Refer : https://devdocs.magento.com/guides/v2.4/config-guide/varnish/use-multiple-varnish-cache.html

Challenges in a varnish cluster

A varnish cluster is more complex to manage

Ensure the varnish vcl files, front end security configuration (WAF, rate limiting, etc) is managed and kept in sync on all edge nodes.

Managing includes monitoring to ensure none of the servers go down. Now, your site can be down in a specific region for example. Typical monitoring tools such as Pingdom would not work. A purposeful monitoring solution is needed.

A varnish cluster costs for additional servers

Since these would be frontend servers, the amount of RAM and their network speed requirement would depend on what traffic they get.

Number of varnish nodes

Increasing the number of nodes in a varnish cluster does not always help in improving site speed. That is because each varnish node has a different hit ratio. A lower hit ratio leads to more users getting the latency and performance penalties combined – due to a varnish MISS. Traffic pattern and latency have to be taken into account to decide on how many nodes to use in a cluster.

Difficult to warm the cache

Given the distributed nature of the cache, warming each cache independently takes more resources on the server side as well as some changes to the way a cache warmer works.

luroConnect : A modern cloud hosting platform

Schedule a call of a free evaluation and demo!

  • Is horizontal scaling manually or with autoscale right for you?
  • Evaluate if a varnish cluster will help your website performance
  • Show managed dev, staging and production environments
  • How we measure application performance every minute with alerts for a slowdown
  • Can your hosting costs be optimized?
  • How improved hosting can lead to better ROI!

On Magento Cloud? We have special offers if you switch your enterprise license to luroConnect managed AWS cloud.

Click here to schedule the call

Magento Cloud and Fastly

It is important to discuss the Magento 2 cloud decision to use Fastly as a frontend.

 

Magento 2 cloud pro version architecture is given below. (Reference : https://devdocs.magento.com/cloud/architecture/pro-architecture.html)

The architecture uses Fastly as a Full Page Cache. Varnish is not installed on the Instances in AWS.

Fastly is a CDN that uses varnish. With the Magento 2 Cloud integration, a custom plugin is used on Magento along with a custom vcl file that runs on Fastly.

Fastly has many “POP” locations. As per fastly documentation, there are 20 POP locations in North America, https://www.fastly.com/network-map

mostly in the USA. Each has its own varnish cache. When a page is not in a POP, it fetches content from the origin. A single page may have to be rendered 20 times for each POP location in North America.

Drawbacks of this architecture from a varnish cluster perspective

  • More POPs do not lead to better experience as a higher percentage of MISS on varnish results in a worse experience for more users.
  • As POPs increase the load on Magento infrastructure increases.
  • It is not possible to use a cache warmer to warm the Fastly cache.

What next?

An ideal situation would be a layered varnish configuration – each “satellite” varnish node serving a local user, caching a subset of the “main” varnish node, reducing the penalty of a varnish MISS.

Share your thoughts here or on social media.

August 25, 2020 By caches, hosting, Magento, Nginx, performance, Site Speed, varnish 0 Comments Read more...

Choosing Hosting platform for Magento : Digital Ocean

One in a while you may want to check on the technology and options available for Magento hosting – technologies change, hosting costs reduce, life becomes easier for you and the merchant. In this multi-part series we will explore cloud platforms and their suitability to hosting a production Magento website.

In this article we will explore how seriously you should consider Digital Ocean as a platform for hosting.

Digital Ocean (DO) offers VMs it calls “droplets”. With an excellent blog and a friendly attitude towards developers, DO is a serious hosting contender. Many developers naturally recommend using DO to merchants for hosting Magento. How real is it? Let me explore a few pros and cons.

DO Choice of vCPU and memory

DO offers “shared CPU” and “dedicated CPU”. For eCommerce hosting we always prefer dedicated CPU. Shared CPUs work differently on all cloud providers and quite often when you need it the most, you do not get enough. Each vCPU is a Intel hyper-thread. As of this writing (July 2020), we see use of Intel Xeon Gold 6140 @ 2.30 GHz with DDR4 memory at 2.6 GHz. Our simple memory speed test showed about 1.8GHz effective throughput to memory.

DO disk : Love / hate relationship!

I love the disk speed – you get good SSD performance with no throttling. Even for attached storage. Unlike other cloud platforms, you don’t have to juggle with figuring out how many IOPS you need, once you understand for that platform how IOPs translates to speed.

To test the disk I use a simple but effective method to test the disk speed. I create a file with /dev/zero using dd of 1GB. dd gives me the write speed. Here is the command I use :

dd if=/dev/zero of=$tempFile bs=1M count=1024 conv=sync oflag=direct
tempFile has the path to a temp file in the mounted disk I am testing

For both direct disk as well as mounted block device we see 200-350 Mbps disk speeds across all droplets our customers use. This is the best we have seen in cloud platforms. Physical hardware can give upto 450Mbps speeds.

So, why the hate?

We have seen disk crashes – thankfully on staging servers. So, when it comes to production servers, we always recommend customers to have a DR plan to minimize loss of data when this happens. Our suspicion is that the storage is not in a managed RAID, hence a disk crash is a droplet specific event.

Network

File transfer speed test

: A large file transfer using scp on the internal network is done in about 130 MB/sec – about a 1 Gb/sec speed.

NFS

NFS of block storage performs poorly. We use nfscache so most reads are served from cache. However, if there is a need for a large number of reads or writes, the performance can drop dramatically.

Examples of NFS hurting performance

Magento 1 : As images are created in the frontend, there is a check to see if an image file exists before an image url is included in a page. This invariably results in slowdown.

Magento 1 and 2 : large log files. Magento stores log files in a shared folder. Multiple lines per hit of logs will result in slowdown.

Experience :  We were in the process of taking a Magento 2 website live that had about 2500 errors written to a log file. The issue was related to M1 migration resulting in some attributes not defined in M2. App servers saw 30% CPU in I/O wait and the site came to a crawl. Php access logs showed under 10% CPU utilization.

Non availability of autoscale

DO does not come with an autoscale option (you could get autoscale in their Kubernetes which we are not considering here). This means you may have to keep capacity for a holiday sale for example.

Managing server costs

Server costs even when you shut them down

DO charges for servers that are shut down. If you do not want to be charged fully for a server, storing an image is an option.

Recommendation

We host customers on Digital Ocean if they do not have a need for scaling and are willing to have a DR plan. This generally increases the cost of the overall solution.
Also stores that do not need scaling such as :

  • PWA – Vue-storefront based stores which use nodejs
  • A high hit ratio varnish FPC magento store (depends on many factors, but essentially requirement for scaling multiple app servers is less likely)

 

July 31, 2020 By hosting, Magento, performance, PWA, scale, Site Speed 0 Comments Read more...

Magento emails & SMTP

Emails are a very critical portion of an eCommerce website. There are 2 types of emails that are used – transactional and promotional.

Transactional emails are emails the eCommerce system sends for regular customer activity – such as order confirmation email. It may also include abandoned cart email, wish list reminders, password change, etc. If a website visitor action causes an email to go out, it is a transactional email.

In this article we will discuss the best way to configure transactional email sending in Magento. This article applies to both Magento 1 and Magento 2. The focus is on what happens on production systems.

The SMTP Plugin

Traditionally Magento agencies and developers install a “SMTP Plugin”. A simple search in Magento marketplace for SMTP reveals many options. Some are general plugins that allow you to connect to any SMTP provider, others are from vendors such as SendGrid.

Advantages of such plugins are

  • Easy setup
  • Developer testing is easy
  • Websites on shared hosting have to use such a plugin

However, for growing websites, this is a particular problem

  • Outgoing ports from the application server may not be open
  • Many SMTP providers have restrictions such as rate limit on email sending or number of connections. Some SMTP plugins may not handle this condition well – just reporting the error vs trying to send again.
  • They may not be able to alert when email sending is down
  • When upstream providers’ SMTP service is down or slow, cron will slow down as each email times out on sending. (Magento sends emails in cron).

Using postfix

By default, Magento sends emails to the local system where php is running. Linux systems have many options for receiving such requests and sending out the emails. A popular software is postfix. While postfix can directly send emails to the recipients’ email systems, we are interested in the “relay host” mode of postfix. In this mode, postfix acts to relay emails sent to it from Magento to the upstream provider – such as SendGrid.

Advantages of postfix

  • Postfix has an inbuilt queue. When the upstream server’s rate limit is reached, a “defer” status is returned. Postfix understands this and it will queue deferred emails and keep retrying. Emails are not lost.
  • Magento cron will deliver to local postfix in a matter of milliseconds, freeing up cron to perform other activities.
  • Connections can be pooled so if many emails are in queue they will be pooled together and sent using a single connection. Some SMTP servers, notably gmail, have a rate limit on the number of connections.
  • Log files that can be checked by system admins. Log files can also be sent to a log collection tools. Advanced commands help checking the queue, even trimming it using conditions such as email subject.
  • Flexible configuration allows many features – on a staging server for example, only whitelisted recipients can receive emails, preventing accidental trigger of email to customers. Or emails to known spammers can be filtered by email id or domain.

Installation and configuration

Installing postfix is easy – the package is part of any linux system. For RHEL based system like CentOS :

sudo yum -y install postfix
sudo yum -y install cyrus-sasl-plain cyrus-sasl-md5

Configuring postfix to act as a relayhost.

Click on the tab below to see configuration information for the specific SMTP service being used. Commands below should be run as root (or sudo). Ownership of files created, especially the password files is assumed to be root.

Sendgrid

  1. Sign in to SendGrid and go to Settings > API Keys.
  2. Create an API key.
  3. Select the permissions for the key. At a minimum, the key must have Mail sendpermissions to send email.
  4. Click Save to create the key. SendGrid generates a new key. This is the only copy of the key, so make sure that you copy the key to clipboard and use if for the next step.
  5. Create the hashmap from the key 
    echo [smtp.sendgrid.net]:2525 apikey:{{{PASTE}}} >> /etc/postfix/sasl_passwd
postmap /etc/postfix/sasl_passwd
rm /etc/postfix/sasl_passwd
chmod 600 /etc/postfix/sasl_passwd.db
  6. Add to postfix configuration file main.cf 
    cat >> /etc/postfix/main.cf <<EOF
smtpd_tls_security_level = may
# enable SASL authentication
smtp_sasl_auth_enable = yes
# tell Postfix where the credentials are stored
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_sasl_security_options = noanonymous
header_size_limit = 4096000
relayhost = [smtp.sendgrid.net]:587
smtp_tls_CAfile =/etc/pki/tls/certs/ca-bundle.crt
EOF
  7. service postfix restart

Mailgun

  1. Signup for maligun account.
  2. Get credentials for maligun. Mailgun has an option for SMTP username and password under the Domains section of the mailgun portal.
  3. Create the hashmap from the key 
    echo [smtp.mailgun.org]:2525 {{user}}:{{password}} >> /etc/postfix/sasl_passwd
postmap /etc/postfix/sasl_passwd
rm /etc/postfix/sasl_passwd
chmod 600 /etc/postfix/sasl_passwd.db
  4. Add to postfix configuration file main.cf 
    cat >> /etc/postfix/main.cf <<EOF
relayhost = [smtp.mailgun.org]:2525
smtp_tls_security_level = encrypt
smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_sasl_security_options = noanonymous
smtp_tls_CAfile =/etc/pki/tls/certs/ca-bundle.crt
EOF
  5. service postfix restart

Amazon SES

Amazon SES is the most cost effective SMTP provider to our knowledge. However, it is also the most difficult to setup as it requires you to verify sender emails and sandbox mode (which we will not consider here).

  1. Login to your aws account
  2. Select “Simple Email Service” from the services
  3. Select Domains from the left menu
  4. Click “verify a new domain”
  5. Enter domain and click on verify
    SES will generate the TXT records you need to add to your DNS

    SES will also generate the DKIM CNAME record to be added to your DNS
  6. Ensure domain verification green status
  7. In the SES main screen, select “Email Addresses” from the left menu
  8. Enter the sender email address. Sender email address are set in Magento. Depending on your Magento configuration different emails may need to be configured. Each email needs a mailbox as a confirmation email is sent to it.
  9. Refer https://docs.aws.amazon.com/ses/latest/DeveloperGuide/smtp-credentials.html to create SMTP credentials. Note username and password.
  10. Depending on the region your SMTP endpoint (relayhost setting in the main.cf below) will be different. Refer https://docs.aws.amazon.com/ses/latest/DeveloperGuide/smtp-connect.html
    Note : The region of SES is unimportant in that there are no restrictions from where the email is sent. Your Magento instance can be anywhere – not even on AWS.
  11. Save password 
    echo [email-smtp.us-west-2.amazonaws.com]:587 USERNAME:PASSWORD >> /etc/postfix/sasl_passwd
postmap /etc/postfix/sasl_passwd
rm /etc/postfix/sasl_passwd
chmod 600 /etc/postfix/sasl_passwd.db
  12. Update main.cf 
    cat >> /etc/postfix/main.cf <<EOF
relayhost = [email-smtp.us-west-2.amazonaws.com]:587
smtp_sasl_auth_enable = yes
smtp_sasl_security_options = noanonymous
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_use_tls = yes
smtp_tls_security_level = encrypt
smtp_tls_note_starttls_offer = yes
smtp_tls_CAfile =/etc/pki/tls/certs/ca-bundle.crt
EOF

Advanced configurations

Postfix relay gateway

In a more mature environment, each php server may not be able to send on SMTP ports. Even in the presense of a NAT gateway, we would recommend a main postfix gateway to ease logging and prevent loss of emails when an instance is lost in autoscaling. With postfix we can create a relay – php servers are configured to send email to a designated email gateway server which in turn relays to the SMTP upstream server.

Sending promotional emails

Promotional emails should ideally not originate from your website – instead they should be from your marketing tools. However, Magento does support sending promotional emails. To improve deliverability of crucial transactional emails, we recommend using a different sender email domain for promotional emails. If this is configured in Magento, postfix can handle sending promotional emails from a upstream service different from the transactional service. Assuming username / passwords have been created in the SMTP service for the promotional domain, the following configuration allows selecting the relayhost to use based on the sending domain.

cat << /etc/postfix/main.cf >>EOF
smtp_sender_dependent_authentication = yes
sender_dependent_relayhost_maps = hash:/etc/postfix/relayhost_map
EOF

cat << /etc/postfix/relayhost_map >>EOF
user@promodomain  [smtp.xxx.com]:{{port}}
EOF

Add the new domains passwords as the first line in /etc/postfix/sasl_passwd

user@promodomain  {{user}}:{{password}}

luroConnect and Magento emails

The configurations described above are abridged versions of what is used in luroConnect. We use the advanced configuration with a designated server allowed to communicate on port 587 to the outside. We also use secure keys to ensure passwords are not saved in chef cookbooks / configurations. Instead they are stored in a secure key store.

Interested in knowing about the advanced architecture of luroConnect?

Fill the form below and we will contact you.


My company owns the Magento site
Yes, I am a developer
I represent a Magento Agency

Reactions to this post on linkedin

March 20, 2020 By hosting, Linux System Administration, Magento, performance, Site Speed 0 Comments Read more...

6 tips to speedup Magento websites

Introduction

Website speed is very crucial to conversions. There are many studies that indicate how website slowdown of even one second can cause drop in conversions. And websites that have taken this learning have seen conversions grow. Take Pier 1 for example in this article.

Our tips are oriented towards what you need to do on the server side to improve performance especially during high traffic times, when server response times tend to be inconsistent. These tips will speedup Magento websites.

Tip # 1 : Restrict Bots

When times are a busy, you need to be mindful of who you allow into your store. Restricting BOTs is easy to implement by checking the ‘user agent” field. With lowering of server load this will speedup Magento or help reduce resources.

Tip # 2 : Rate limit hits from the same IP

This is one more reduce tip that may work in your situation. Hits from the same IP – until your target audience is a group that is behind a firewall – will restrict simple robotic attempts to crowd your site – especially if you are running a time bound flash sale.

Bonus Tip : Use CDN to serve static content

During peak traffic, CDNs help in offloading not (just) servers but actually network bottlenecks of your servers. This causes a speedup in Magento by reducing the load on the entire system.

Rejecting unwanted traffic & BOTS is a great way to scale #Magento. http://bit.ly/2xb77Le

Click To Tweet

Tip # 3 : Use php7 instead of php5.x

php7 performance is almost 50% higher than php5. We have seen this in processing CPU intensive Magento hits. But, Magento 1.x does not officially support php7 and 3rd party patches. We like the one from inchoo. Magento 2 should be on php7.

Tip # 4 : Consider adding separate servers or pools for your checkout flow (or admin access)

While site performance is important, checkout flow performance is even more important as it is on the business end of your sales funnel. Consider adding a separate checkout servers or pools. They are akin to High Occupancy (HOV) or diamond lanes in traffic. This helps to speedup Magento’s checkout process.

The same concept can be applied to restrict number of resources you reserve for admin access.

Tip # 5 : Manage Magento Indexing

Leaving Magento Indexes to “Update on Save” can get your newly uploaded products available for shopping faster, but can slow down the site! If business permits, move indexing to slightly low volume times of the day (or night). This setting leads directly to speedup Magento front end.

Tip # 6 : Monitor the performance of your caches

A Magento site has many caches and each one helps in reducing the server load and improve response times. But you need to monitor for hit to miss ratios and out of memory conditions. This achieves a speedup in Magento as more cache hits mean faster response. The caches one should look at are php opcache, Magento block cache, FPC (Full Page Cache) and mysql cache.

Conclusion

By reducing, redirecting and monitoring server, the overall performance of a website can be improved.

We can analyze your site for free

Schedule a call

Not happy with your website performance and want an expert to look at it?

  • We will analyze your site using public information.
  • We will ask you to give us a 1 day web server log file.
  • We will try to identify what steps if any you should take to improve your sites performance goals.
Click here to schedule the call

Watch our webinar on performance and scaling in Magento

Using an analogy to vehicular traffic we explain performance and scaling in Magento.
Key takeaways

  • Know how to compare hosting options
  • Importance of good code
  • How to scale
  • Tuning Magento
Watch video now!
October 20, 2017 By hosting, Magento, page speed, performance, scale, Site Speed 0 Comments Read more...

Case Study : How bad code can hurt performance and even break

Many websites, notably eCommerce and WordPress, live with bad code. They end up spending more effort in improving performance by adding more resources. It is like adding more horsepower to an inefficient car engine – will consume more gas and in the long run fixing the engine would be a better thing.

Not all such attempts will succeed, however. Bad code takes many forms. There are times when your application is dependent on an external service. The problem comes when this service is executed inline when your visitor is waiting for your server to respond. The two popular dependencies are sending email (smtp) and curl calls for external http access. (curl is a popular library to access remote servers programmatically).

In this article we will analyse a bad code example we found in production. A customer in India (we manage the Magento site hosting) raised a ticket that their international shipping costs stopped working. Since we take care of their release process, we investigated to see if this was an inadvertent code change that made it to production. We could find no change. Infact, there was no release made from the day the problem started. So, with permission from the customer, we investigated further.

The bad code

They had overridden the default table shipping with custom logic. The primary reason was that they wanted to give shipping costs in USD for US customers. Magento has concepts of base currency and display currency. All calculations are done in base currency (in this case INR) while the charge was in display currency (USD).

 ...
    $to_Currency = urlencode($to_Currency);
    $url = "http://www.google.com/finance/converter?a=$amount&from=$from_Currency&to=$to_Currency";
    $ch = curl_init();
    $timeout = 0;
    curl_setopt ($ch, CURLOPT_URL, $url);
    curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);
 ...

I wondered why it took over 4 years since the bad code was written to break. The reason it finally broke was that google took the URL out of service. The code below would return 0 as it did not find the appropriate tag in the output. So, all shipping was free.

Analyziing the bad code

  • Instead of using Magento’s functions for currency conversion, the developer thought of
    • Accessing the google finance page
    • Scraping the HTML output
  • Might be in violation of Google’s terms of service (programmatic access)
  • The customer has a one step checkout plugin – this code gets called each step, when anything changes. This is slowing down the checkout page.
  • There is no retry or reporting of failure
  • The conversion rate used for shipping may be different for the products, as the site conversion rates are updated daily.

Conclusion

Using curl in code that is accessed as the visitor waits is a bad idea. Apart from slowing down user interaction, the probability of an error increases and then the error has to be handled appropriately.

September 20, 2017 By Magento, page speed, performance, Site Speed , 0 Comments Read more...